Photogrammetric alignment technique for virtual production.
A method of content production includes generating a survey of a performance area that includes a point cloud representing a first physical object, in a survey graph hierarchy, constraining the point cloud and a taking camera coordinate system as child nodes of an origin of a survey coordinate system, obtaining virtual content including a first virtual object that corresponds to the first physical object, applying a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object, displaying the first virtual object on one or more displays from a perspective of the taking camera, capturing, using the taking camera, one or more images of the performance area, and generating content based on the one or more images.
BACKGROUND
One method of creating a virtual reality experience can include surrounding a user with large display screens that present a virtual environment for the user. For example, an immersive content production system that can be used in production of movies and videos can include a stage or performance area that is at least partially enclosed with one or more walls and/or a ceiling each of which can be covered with display screens. One or more cameras can be placed in the performance area and live actors can interact with physical props placed on the stage, as well as with virtual elements displayed on the displays. Such immersive content production systems can present challenges for seamlessly bridging the physical props in the performance area with the virtual elements displayed on the displays so that the virtual elements appear realistic in the movies or videos captured by the cameras. SUMMARY
According to some embodiments, a method of content production includes generating a survey of a performance area. The survey includes a point cloud representing one or more physical objects including a first physical object in the performance area. The point cloud is associated with a survey coordinate system having an origin corresponding to a reference point in the performance area. The method further includes, in a survey graph hierarchy, constraining the point cloud and a taking camera coordinate system as child nodes of the origin of the survey coordinate sys
1. A method of content production, the method comprising: generating a survey of a performance area, the survey including a point cloud representing one or more physical objects including a first physical object in the performance area, the point cloud being associated with a survey coordinate system having an origin corresponding to a reference point in the performance area; in a survey graph hierarchy, constraining the point cloud and a taking camera coordinate system as child nodes of the origin of the survey coordinate system, so that the point cloud and the taking camera coordinate system inherit changes to the origin of the survey coordinate system; obtaining virtual content including one or more virtual objects associated with a virtual coordinate system, the one or more virtual objects including a first virtual object that corresponds to the first physical object; applying a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object; detecting a current position and a current orientation of the taking camera in the taking camera coordinate system; displaying the first virtual object on one or more displays surrounding the performance area from a perspective of the taking camera at the current position and the current orientation; capturing, using the taking camera, one or more images of the performance area including the first physical object and at least a portion of the one or more displays on which the first virtual object is being displayed; and generating content based on the one or more images. ||
14. An immersive content production system comprising: a performance area; one or more displays surrounding the performance area for displaying virtual content; a taking camera for capturing images of the performance area and a portion of the one or more displays displaying the virtual content; one or more sensors for detecting a current position and a current orientation of the taking camera in a taking camera coordinate system; a computer memory storing: a survey of the performance area associated with a survey coordinate system having an origin corresponding to a reference point in the performance area, the survey including a point cloud representing a first physical object in the performance area; and the virtual content associated with a virtual coordinate system, the virtual content including a first virtual object corresponding to the first physical object; and one or more computer processors configured to: in a survey graph hierarchy, constrain the point cloud and the taking camera coordinate system as child nodes of the origin of the survey coordinate system, so that the point cloud and the taking camera coordinate system inherit changes to the origin of the survey coordinate system; apply a transformation to the origin of the survey coordinate system so that at least a portion of the point cloud that represents the first physical object is substantially aligned with a portion of the virtual content that represents the first virtual object; and cause the virtual content to be displayed on the one or more displays from a perspective of the taking camera at the current position and the current orientation while the taking camera is capturing the images, the images including the first physical object and at least a portion of the one or more displays on which the first virtual object is being displayed.