Artistically controllable 2D-to-3D stereo conversion rendering technique.
The present invention sets forth a technique for performing stereo conversion. This technique includes generating, via a first machine learning model, depth information associated with objects depicted in an input image. The technique also includes generating, via a second set of machine learning models, a disparity map for the input image based on the depth information. The technique further includes extracting, via the second set of machine learning models, feature data from the input image and warping the feature data based on the generated disparity map. The technique further includes generating, via the second set of machine learning models, an output image such that the input image and the output image, when viewed together, form a stereo pair of images. The functions and transformations used in the various machine learning models are fully differentiable, allowing for improved end-to-end training of the machine learning models.
BACKGROUND Field of the Various Embodiments
Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to techniques for performing artistically controllable stereo conversion using artificial intelligence. Description of the Related Art
Stereo conversion refers to the generation of stereoscopic video content from a traditional monocular (non-stereo) video sequence. For example, live-action feature films are often not filmed in stereo, but it may be desirable for a movie studio to produce a stereo version of a film for distribution alongside a non-stereo version of the film. This is achieved through stereo conversion implemented as a post-production process. Further, movie studios often possess large amounts of legacy content, and stereo conversion allows the studio to make this legacy content available to audiences in stereo.
Existing stereo conversion techniques may utilize computer graphics (CG) warping models to perform stereo conversion. These techniques first determine the relative differences in perceived depths among the various objects in a scene (referred to herein as “disparity values”) for a target frame. The techniques then apply backward warping or optical flows using the disparity values to generate an output image from an input image.
One drawback of the above techniques is that there is no mechanism for a user to provide additional information based on a known reference in
1. A computer-implemented method for performing stereo conversion, the computer-implemented method comprising: generating, using one or more machine learning models, a predicted disparity map for an input image based on one or more depth values associated with pixels within the input image; extracting feature data from the input image, wherein the feature data represents the pixels within the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data that are offset to a same pixel location; assigning, based on the predicted disparity map, a visibility value to each of the plurality of offset pixels; modifying, based on the visibility values, each of the plurality of offset pixels such that one of the plurality of offset pixels is visible and the others of the plurality of offset pixels are hidden; and generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereo pair of images. ||
11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating, using one or more machine learning models, a predicted disparity map for an input image based on one or more depth values associated with pixels within the input image; extracting feature data from the input image, wherein the feature data represents the pixels within the input image; applying an offset to each pixel represented by the feature data based on the predicted disparity map; identifying a plurality of offset pixels represented by the feature data that are offset to a same pixel location; assigning, based on the predicted disparity map, a visibility value to each of the plurality of offset pixels; modifying, based on the visibility values, each of the plurality of offset pixels such that one of the plurality of offset pixels is visible and the others of the plurality of offset pixels are hidden; and generating an output image based on the feature data and the modified plurality of offset pixels such that the input image and the output image form a stereo pair of images. ||
17. A computer-implemented method for training a machine learning model to perform stereo conversion, the computer-implemented method comprising: executing a first neural network that predicts pixel-wise depth and disparity values associated with objects depicted in an input image; updating parameters of the first neural network based on a first calculated training loss associated with the predicted disparity values; repeatedly executing the first neural network and updating the parameters of the first neural network; terminating the execution of the first neural network when the first calculated training loss is less than a first threshold value; sequentially executing the first neural network and a second set of neural networks while holding the parameters of the first neural network constant; updating parameters of the second set of neural networks based on second calculated training losses associated with a predicted output image; commencing a first iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the second set of neural networks; terminating the first iterative sequential execution of the first neural network and the second set of neural networks when the second calculated training losses are less than a second threshold value; sequentially executing the first neural network and the second set of neural networks; updating parameters of the first neural network and the second set of neural networks based on a third calculated training loss associated with the predicted output image; commencing a second iterative sequential execution of the first neural network and the second set of neural networks while updating the parameters of the first neural network and second set of neural networks; and terminating the second iterative sequential execution of the first neural network and the second set of neural networks when the third calculated training loss is less than a third threshold value.