- Number
- 20260212560
- Published
- 2026-07-23
- Filed
- 2025-07-15
- Assignee
- Disney Enterprises, Inc.
- Inventors
- Zhang; Yang, Schroers; Christopher Richard, Zhang; Xiang, Mehl; Lukas Francesco, Jiao; Longxiang
- CPC
- G06N3/044; G06T15/205; G06N3/045; G06N3/0455; G06N3/0464; G06N3/047; G06N3/08; G06N3/084; G06N3/088; G06N3/0895; G06N20/00; G06T3/18; G06T7/12; G06T7/13; G06T7/20; G06T7/50; G06T9/00; G06T11/00; G06T11/60; G06T15/20; G06V10/774; G06V10/806; G06T2207/20084
- Verdict
- Medium Notable software
- First reported
- 2026-W30 (2026-07-24)
- Source
- Google Patents · FreePatentsOnline
The keeper's note
Methods and systems for performing novel view synthesis in addition to various image-to-image diffusion processes using machine learning models are disclosed.
Abstract
Methods and systems for performing novel view synthesis in addition to various image-to-image diffusion processes using machine learning models are disclosed. These include methods for performing inpainting-based novel view synthesis using depth maps and conditional models, video novel view synthesis using texture map inpainting, spatially consistent inpainting-based novel view synthesis using video diffusion models, and high-fidelity image-to-image diffusion using a texture conditional model. Further, methods for generating training data that can be used to train machine learning models to perform novel view synthesis are disclosed. These include symmetry exploiting methods of training data generation and methods using training pair alignment and splatting error simulations.
Background
BACKGROUND
The field of novel view synthesis (NVS) generally involves using input data to produce a new view of a scene, e.g., comprising a digital image or video of that scene. As an example, NVS can be used to generate a digital image showing what a scene would look like if an imaginary camera (corresponding to that scene) was moved.
Novel view synthesis can be used in various tasks, including the generation of stereo images and videos. Recent developments in virtual reality (VR) technology, including an increase in commercially available VR headsets, have led to increased interest in stereo images and stereo videos. In a stereo image (or video), two images (or videos) corresponding to slightly different camera angles are displayed to two eyes independently (e.g., from two screens located on the inside of a VR headset). The disparity between the two images or videos corresponds to the apparent displacement of objects in those images or videos when viewed from one eye or the other, which is related to the apparent distance between the objects and the observer. This is similar to the disparity of visual information that people receive when viewing objects with two eyes. As such, viewing images or videos in this manner leads to a “stereoscopic viewing effect,” allowing the brain interpret images or videos as three dimensional (3D). This can be desirable in various applications, including VR games or other media, as it can make the viewer feel like they are “immer
Claims
1. A method for generating a novel view image corresponding to an input view image using a machine learning model comprising a pre-trained diffusion model comprising one or more diffusion model layers including an input diffusion model layer, and a conditional model comprising one or more conditional model layers, the method performed by a computer system and comprising: generating, based on the input view image, a depth map; warping the input view image based on the depth map, thereby generating a warped image and a disocclusion mask; masking the warped image using the disocclusion mask, thereby generating a warped and masked image; generating a noisy embedding; encoding the warped and masked image, thereby generating a warped and masked embedding; encoding the depth map, thereby generating a depth map embedding; encoding the disocclusion mask, thereby generating a disocclusion mask embedding; combining the noisy embedding, the warped and masked embedding, the depth map embedding, and the disocclusion mask embedding, thereby generating a combined embedding; applying the combined embedding to the conditional model, thereby generating one or more conditional model layer outputs corresponding to the one or more conditional model layers; applying the noisy embedding to the input diffusion model layer and the one or more conditional model layer outputs to the one or more diffusion model layers, thereby generating an output embedding; and decoding the output embedding, thereby generating the novel view image. ||
10. A method for generating a novel view video corresponding to an input view video comprising a plurality of image frames using a mapping machine learning model and an inpainting machine learning model, the method performed by a computer system and comprising: warping each image frame of the plurality of image frames, thereby generating a plurality of warped image frames and a plurality of disocclusion masks corresponding to the plurality of warped image frames; training the mapping machine learning model to map pixels from the plurality of warped image frames to a texture map, thereby generating the texture map; generating a cumulative disocclusion mask using the plurality of disocclusion masks and the mapping machine learning model; combining the cumulative disocclusion mask and the texture map, thereby generating a masked texture map; generating an unmasked texture map by inpainting the masked texture map using the inpainting machine learning model; and generating the novel view video based on the unmasked texture map. ||
20. A computer system comprising: one or more processors; and a non-transitory computer readable medium coupled to the one or more processors, the non-transitory computer readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a method for generating a novel view image corresponding to an input view image using a machine learning model comprising a pretrained diffusion model comprising one or more diffusion model layers including an input diffusion model layer, and a conditional model comprising one or more conditional model layers, the method comprising: generating, based on the input view image, a depth map; warping the input view image based on the depth map, thereby generating a warped images and a disocclusion mask; masking the warped image using the disocclusion mask, thereby generating a warped and masked image; generating a noisy embedding; encoding the warped and masked image, thereby generating a warped and masked embedding; encoding the depth map, thereby generating a depth map embedding; encoding the disocclusion mask, thereby generating a disocclusion mask embedding; combining the noisy embedding, the warped and masked embedding, the depth map embedding, and the disocclusion mask embedding, thereby generating a combined embedding; applying the combined embedding to the conditional model, thereby generating one or more conditional model layer outputs corresponding to the one or more conditional model layers; applying the noisy embedding to the input diffusion model layer and the one or more conditional model layer outputs to the one or more diffusion model layers, thereby generating an output embedding; and decoding the output embedding, thereby generating the novel view image.