- Number
- 10321117
- Published
- 2019-06-11
- Filed
- 2014-08-25
- Assignee
- LUCASFILM ENTERTAINMENT COMPANY LTD.
- Inventors
- Koch; Hilmar, Mallet; Ronald, Libreri; Kim, Warner; Paige, Sanders; Mike, Gaeta; John
- CPC
- H04N13/111; H04N13/243; G06T15/205; G06T15/04; H04N13/282; H04N13/271; G06T13/40
- Verdict
- Medium Notable software
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Motion-controlled body capture and reconstruction (Lucasfilm mocap).
Abstract
A method of generating unrecorded camera views may include receiving a plurality of 2-D video sequences of a subject in a real 3-D space, where each 2-D video sequence may depict the subject from a different perspective. The method may also include generating a 3-D representation of the subject in a virtual 3-D space, where a geometry and texture of the 3-D representation may be generated based on the 2D video sequences, and the motion of the 3-D representation in the virtual 3-D space is based on motion of the subject in the real 3-D space. The method may additionally include generating a 2-D video sequence of the motion of the 3D representation using a virtual camera in the virtual 3-D space where the perspective of the virtual camera may be different than the perspectives of the plurality of 2-D video sequences.
Background
BACKGROUND(1) The process of recording the movement of objects or people is generally referred to as motion capture, and has many applications in entertainment, sports, medical imaging, video gaming, computer vision, and robotics. During a motion capture session, the movement of an actor can be sampled by cameras configured to track the motion of a finite number of retro-reflective markers worn by the actor. Animation data can be derived from the image samples and mapped to a 3-D model such that the 3-D model performs actions similar to the live actor by correlating motion of the reflective markers with animation controls on the 3-D model. Traditional motion capture techniques record only the movements of the reflective markers worn by the actor and purposely disregard the visual appearance of the actor.BRIEF SUMMARY(2) In some embodiments, a method for capture and reconstruction may be presented. The method may include receiving a plurality of 2-D video sequences of a subject in a real 3-D space. Each 2-D video sequence in the plurality of 2-D video sequences may depict the subject from a different perspective. The method may also include generating a 3-D representation of the subject in a virtual 3-D space. A geometry and texture of the 3-D representation may be generated based on the plurality of 2D video sequences, and motion of the 3-D representation in the virtual 3-D space may be based on motion of the subject in the real 3-D space. The method may additionally include
Claims
1. A method comprising: receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective; generating a 3-D representation of the subject in a virtual 3-D space, wherein: a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences; a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information; generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein: the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and the 3-D digital model is used to fill in the missing information of the 3-D representation by: smoothing the outer surface of the 3-D representation; filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and properly aligning or dimensioning the 3-D representation; generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences. |
8. A system comprising: one or more processors; and a non-transitory storage memory comprising instructions which, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective; generating a 3-D representation of the subject in a virtual 3-D space, wherein: a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences; a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information; generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein: the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and the 3-D digital model is used to fill in the missing information of the 3-D representation by: smoothing the outer surface of the 3-D representation; filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and properly aligning or dimensioning the 3-D representation; generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences. |
15. A non-transitory storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of 2-D video sequences of a subject in a real 3-D space, wherein each 2-D video sequence in the plurality of 2-D video sequences depicts the subject from a different perspective; generating a 3-D representation of the subject in a virtual 3-D space, wherein: a geometry and texture of the 3-D representation is generated based on the plurality of 2-D video sequences; a motion of the 3-D representation in the virtual 3-D space is based on a motion of the subject in the real 3-D space; and an outer surface of the 3-D representation is missing information, and the texture of the 3-D representation is missing texture information; generating a full 3-D representation of the subject in the virtual 3-D space by at least combining the 3-D representation of the subject with an existing 3-D digital model, wherein: the 3-D digital model is generated using a 3-D scan of a volume of the subject in the real 3-D space, and the 3-D scan is of the same subject that was captured in the 2-D video sequence; and the 3-D digital model is used to fill in the missing information of the 3-D representation by: smoothing the outer surface of the 3-D representation; filling volumetric holes in the 3-D representation that were not visible in the plurality of 2-D video sequences; and properly aligning or dimensioning the 3-D representation; generating the missing texture information using corresponding pixels in prior or subsequent frames of a motion of the of the full 3-D representation; and generating a 2-D video sequence of the motion of the full 3-D representation using a virtual camera in the virtual 3-D space, wherein a perspective of the virtual camera is different than the perspectives of the plurality of 2-D video sequences.