EXTRACTING QUAD-MESHES WITH PIXEL-LEVEL DETAILS AND MATERIALS FROM IMAGES

Number: 20250166303
Published: 2025-05-22
Filed: 2024-11-15
Assignee: DISNEY ENTERPRISES, INC.
Inventors: RIEMENSCHNEIDER; Hayko Jochen Wilhelm et al.
CPC: G06T17/205; G06T17/20
Verdict: Low Notable software
Source: Google Patents · FreePatentsOnline

The keeper's note

Quad-mesh extraction technique for animation/VFX assets.

Abstract

One embodiment of the present invention sets forth a technique for generating a quad-dominant mesh of an object. The technique includes generating, via at least one of a first set of machine learning models, a three-dimensional (3D) triangle mesh of an object based on one or more two-dimensional input images of the object, iteratively learning, via a second set of machine learning models, an orientation field and a position field associated with a set of vertices included in the 3D triangle mesh, extracting a quad-dominant mesh associated with the object from the input triangle mesh based on the orientation field and the position field, wherein the quad-dominant mesh comprises one or more quadrilaterals, rendering an image based on the quad-dominant mesh; and optimizing the quad-dominant mesh by propagating a loss generated based on the image to the set of machine learning models.

Background

BACKGROUND Field of the Various Embodiments

Embodiments of the present disclosure relate generally to machine learning and content creation and, more specifically, to extracting quad-meshes with pixel-level details and materials from images. Description of the Related Art

In the field of computer graphics, generating high-quality 3D models from real-world images is an important task for applications in visual effects, virtual reality, and interactive media. Traditional production pipelines require numerous high-resolution meshes, often consuming extensive artist time and effort to refine raw 3D scans or model objects manually. Recent advances in neural implicit representations have shown promise in automating parts of this process, enabling more efficient extraction of object geometry and material properties from images. However, these methods typically produce dense or irregular triangle-based meshes that are difficult to manipulate and do not enable detailed control.

Existing approaches, using, for example, Neural Radiance Fields (NeRF) and Signed Distance Fields (SDFs), can generate high-fidelity views and capture object details, but fail to create explicit, editable mesh representations. When extracting meshes, these methods often yield triangle-dominant structures with excessive geometry, limiting their usability in production. Although these meshes can be converted to quad-meshes that enable more fine-grained control, the processes lack control over

Claims

1. A computer-implemented method, comprising: generating, via at least one of a first set of machine learning models, a three-dimensional (3D) triangle mesh of an object based on one or more two-dimensional input images of the object; iteratively learning, via a second set of machine learning models, an orientation field and a position field associated with a set of vertices included in the 3D triangle mesh; extracting a quad-dominant mesh associated with the object from the 3D triangle mesh based on the orientation field and the position field, wherein the quad-dominant mesh comprises one or more quadrilaterals; rendering an image based on the quad-dominant mesh; and optimizing the quad-dominant mesh by propagating a loss generated based on the image to the first set of machine learning models. || 11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating, via at least one of a first set of machine learning models, a three-dimensional (3D) triangle mesh of an object based on one or more two-dimensional input images of the object; iteratively learning, via a second set of machine learning models, an orientation field and a position field associated with a set of vertices included in the 3D triangle mesh; extracting a quad-dominant mesh associated with the object from the 3D triangle mesh based on the orientation field and the position field, wherein the quad-dominant mesh comprises one or more quadrilaterals; rendering an image based on the quad-dominant mesh; and optimizing the quad-dominant mesh by propagating a loss generated based on the image to the first set of machine learning models. || 20. A system, comprising: one or more memories that store instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: generating, via at least one of a first set of machine learning models, a three-dimensional (3D) triangle mesh of an object based on one or more two-dimensional input images of the object; iteratively learning, via a second set of machine learning models, an orientation field and a position field associated with a set of vertices included in the 3D triangle mesh; extracting a quad-dominant mesh associated with the object from the 3D triangle mesh based on the orientation field and the position field, wherein the quad-dominant mesh comprises one or more quadrilaterals; rendering an image based on the quad-dominant mesh; and optimizing the quad-dominant mesh by propagating a loss generated based on the image to the first set of machine learning models.