- Number
- 11288859
- Published
- 2022-03-29
- Filed
- 2020-06-01
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Mitchell; Kenneth J., Casas Cambra; Llogari, Li; Yue
- CPC
- G06T15/00; G06T15/205; G06T15/506; G06T19/20; G06T3/4092; G06T7/11; G06T7/50
- Verdict
- Low Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Real-time facial VFX rendering technique.
Abstract
Embodiments provide techniques for rendering augmented reality effects on an image of a user's face in real time. The method generally includes receiving an image of a face of a user. A global facial depth map and a luminance map are generated based on the captured image. The captured image is segmented into a plurality of segments. For each segment in the plurality of segments, a displacement energy of the respective segment is minimized using a least square minimization of a linear system for the respective segment. The displacement energy is generally defined by a relationship between a detailed depth map, the global facial depth map and the luminance map. The detailed depth map is generated based on the minimized displacement energy for each segment in the plurality of segments. One or more visual effects are rendered over the captured image using the generated detailed depth map.
Background
BACKGROUND Field (1) Aspects of the present disclosure relate to augmented reality systems, and more specifically to using augmented reality systems to render depth-accurate effects on an image of a user's face in real time. Description of the Related Art (2) Augmented reality systems generally are systems in which computer-rendered effects are displayed (e.g., overlaid on) in conjunction with a real-world environment. These augmented reality systems may include augmented reality systems deployed on mobile devices (e.g., smartphones, tablets, etc.), head-mounted displays, digital displays (e.g., augmented reality mirrors), and the like). In augmented reality systems, various effects can be rendered to augment a real-world environment or images captured of the real-world environment in which the augmented reality systems operate. For example, a simple augmented reality system may render additional objects and display these rendered objects in such a manner that the additional objects appear to be part of the real-world environment. In another example, lighting effects can be rendered in an augmented reality system to add artificial lighting to the real-world environment. (3) Rendering augmented reality effects is generally a task that varies in computational complexity. Simple systems in which objects are overlaid on the real-world environment may be a computationally inexpensive process. More complex effects, however, may have a degree of complexity that makes it impractical
Claims
1. A method for rendering augmented reality effects on an image of a user's face in real time, comprising: capturing an image of a face of a user; generating a global facial depth map and a luminance map based on the captured image; segmenting the captured image into a plurality of segments; for each segment in the plurality of segments, minimizing a displacement energy of the respective segment using a least square minimization of a linear system for the respective segment, the displacement energy being defined by a relationship between a detailed depth map, the global facial depth map and the luminance map; generating the detailed depth map based on the minimized displacement energy for each segment in the plurality of segments, wherein the detailed depth map captures three-dimensional facial details of the user in the captured image; and rendering one or more visual effects over the face of the user within the captured image using the generated detailed depth map, wherein a position of the face of the user and the captured three-dimensional facial details of the user in the image are preserved in the rendered one or more visual effects. ||
11. A system, comprising: a processor; and a memory having instructions stored thereon which, when executed by the processor, performs an operation for rendering augmented reality effects on an image of a user's face in real time, the operation comprising: capturing an image of a face of a user, generating a global facial depth map and a luminance map based on the captured image, segmenting the captured image into a plurality of segments, for each segment in the plurality of segments, minimizing a displacement energy of the respective segment using a least square minimization of a linear system for the respective segment, the displacement energy being defined by a relationship between a detailed depth map, the global facial depth map and the luminance map, generating the detailed depth map based on the minimized displacement energy for each segment in the plurality of segments, wherein the detailed depth map captures three-dimensional facial details of the user in the captured image; and rendering one or more visual effects over the face of the user within the captured image using the generated detailed depth map, wherein a position of the face of the user and the captured three-dimensional facial details of the user in the image are preserved in the rendered one or more visual effects. ||
20. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processor, performs an operation for rendering augmented reality effects on an image of a user's face in real time, the operation comprising: capturing an image of a face of a user; generating a global facial depth map and a luminance map based on the captured image; segmenting the captured image into a plurality of segments; for each segment in the plurality of segments, minimizing a displacement energy of the respective segment using a least square minimization of a linear system for the respective segment, the displacement energy being defined by a relationship between a detailed depth map, the global facial depth map and the luminance map; generating the detailed depth map based on the minimized displacement energy for each segment in the plurality of segments, wherein the detailed depth map captures three-dimensional facial details of the user in the captured image; and rendering one or more visual effects over the face of the user within the captured image using the generated detailed depth map, wherein a position of the face of the user and the captured three-dimensional facial details of the user in the image are preserved in the rendered one or more visual effects.