Latency-reduction technique for immersive virtual-production systems.
A method of content production may include receiving tracking information for a camera with a frustum configured to capture images of a subject in an immersive environment. a first image of a virtual environment corresponding to the frustum may be rendered using a first rendering process based on the tracking information to be perspective-correct when displayed on the displays and viewed through the camera. A second image of the virtual environment may also be rendered using a second rendering process for a specific display. The first image and the second image may be rendered in parallel. The second image and a portion of the first image may be composited together to generate a composite image, where the portion of the first image may correspond to a portion of the display captured by the frustum.
FIELD
The present disclosure generally relates to generating content using one or more displays configured for operation in an immersive content production system. Specifically, this disclosure describes computer techniques for reducing latency when rendering, transmitting, compositing, and/or displaying immersive content on background screens. BACKGROUND
An immersive reality environment may be designed to provide a fully engrossing and interactive experience. At its core, this environment leverages advanced virtual reality (VR), augmented reality (AR), large screens, or mixed reality (MR) technologies to create a sensory-rich experience. In such an environment, users wear motion-tracking sensors enabling them to explore and engage with the environment. Within an immersive environment, users may interact with objects, environments, and other users in real-time. Immersive reality environments have found applications across various industries, from gaming and entertainment to healthcare, education, and training.
Immersive environments may now play a transformative role in the creation of CGI (Computer-Generated Imagery) movie sets and the way filmmakers design, visualize, and shoot scenes. These environments enable directors, production designers, and visual effects teams to immerse themselves in a digital representation of the movie's setting, characters, and special effects, enhancing the creative process and streamlining production. For example, the pre-
1. A method of content production, the method comprising: receiving tracking information for a camera associated with a frustum, wherein the camera is configured to capture images of a subject in an immersive environment comprising one or more displays; rendering, using a first rendering process, a first image of a virtual environment corresponding to the frustum, wherein the first image is rendered based on the tracking information for the camera to be perspective-correct when displayed on the one or more displays and viewed through the camera; rendering, using a second rendering process, a second image of the virtual environment for display on a display in the one or more displays, wherein the first image and the second image are rendered in parallel; compositing the second image and a portion of the first image together to generate a composite image, wherein the portion of the first image corresponds to a portion of the display captured by the frustum; and causing the composite image to be displayed on the display in the immersive environment. ||
8. An immersive content presentation system comprising: a first Graphics Processing Unit (GPU) that is configured to: receive tracking information for a camera associated with a frustum, wherein the camera is configured to capture images of a subject in an immersive environment comprising one or more displays; and render a first image of a virtual environment corresponding to the frustum, wherein the first image is rendered based on the tracking information for the camera to be perspective-correct when displayed on the one or more displays and viewed through the camera; a second GPU that is configured to: render a second image of the virtual environment for display on a display in the one or more displays, wherein the first image and the second image are rendered in parallel; and one or more processors that are configured to composite the second image and a portion of the first image together to generate a composite image, wherein the portion of the first image corresponds to a portion of the display captured by the frustum. ||
16. One or more non-transitory computer-readable media storing instructions that, when executed by one or more Graphics Processing Units (GPUs) or processors, cause the one or more GPUs or processors to perform operations comprising: receiving tracking information for a camera associated with a frustum, wherein the camera is configured to capture images of a subject in an immersive environment comprising one or more displays; rendering, using a first GPU, a first image of a virtual environment corresponding to the frustum, wherein the first image is rendered based on the tracking information for the camera to be perspective-correct when displayed on the one or more displays and viewed through the camera; rendering, using a second GPU, a second image of the virtual environment for display on a display in the one or more displays, wherein the first image and the second image are rendered in parallel; compositing the second image and a portion of the first image together to generate a composite image, wherein the portion of the first image corresponds to a portion of the display captured by the frustum; and causing the composite image to be displayed on the display in the immersive environment.