Facial appearance/material capture technique (VFX, granted).
Techniques are disclosed for capturing facial appearance properties. In some examples, a facial capture system includes light source(s) that produce linearly polarized light, at least one camera that is cross-polarized with respect to the polarization of light produced by the light source(s), and at least one other camera that is not cross-polarized with respect to the polarization of the light produced by the light source(s). Images captured by the cross-polarized camera(s) are used to determine facial appearance properties other than specular intensity, such as diffuse albedo, while images captured by the camera(s) that are not cross-polarized are used to determine facial appearance properties including specular intensity. In addition, a coarse-to-fine optimization procedure is disclosed for determining appearance and detailed geometry maps based on images captured by the cross-polarized camera(s) and the camera(s) that are not cross-polarized.
BACKGROUND Technical Field (1) Embodiments of the present disclosure relate generally to computer science and computer graphics and, more specifically, to techniques for facial appearance capture. Description of the Related Art (2) Realistic digital faces are required for various computer graphics and computer vision applications. For example, digital faces are oftentimes used in virtual scenes of film or television productions and in video games. (3) A digital face can be synthesized using three-dimensional (3D) geometry of the face and two-dimensional (2D) appearance maps indicating reflectance properties of the face. As used herein, “appearance” refers to reflectance properties characterizing how light interacts with materials before reaching a camera. Example reflectance properties include, without limitation, shininess (also referred to as “specular intensity”) and the shape of a specular reflection lobe (also referred to as “roughness”), which are surface properties of skin, as well shading-free color (also referred to as “diffuse albedo”), which is mostly a subsurface property but can include contributions from the surface of skin as well. The reflectance properties of human skin, and more particularly the skin on a face, can vary due to skin type, tanning, blood flow caused by muscle activation or physiological effects, scarring, stretching, oiliness, and sweating, among other things. (4) Facial capture systems have been used to capture images of individual faces, whi
1. A computer-implemented method for generating one or more maps indicating surface properties of an object, the computer-implemented method comprising: receiving a first set of images of the object captured using a first set of cameras, wherein the first set of cameras is configured to capture light that is cross-polarized with respect to a polarization of at least one light source; receiving a second set of images of the object captured using a second set of cameras, wherein the second set of cameras is configured to capture light that is not cross-polarized with respect to the polarization of the at least one light source; generating, based on initial appearance maps associated with the object, a first set of rendered images that lack highlights and a second set of rendered images that include highlights; and generating one or more appearance maps associated with the object based on (i) a first comparison of the first set of rendered images with the first set of images and (ii) a second comparison of the second set of rendered images with the second set of images. ||
11. A system, comprising: a set of light sources configured to generate light having a first polarization during a shot of an object; a first set of cameras configured to capture light that is cross-polarized with respect to the first polarization during the shot; a second set of cameras configured to capture light that is not cross-polarized with respect to the first polarization during the shot; and an appearance capture application that, during operation, is configured to: receive a first set of images of the object captured using the first set of cameras during the shot and a second set of images of the object captured using the second set of cameras during the shot; generate, based on initial appearance maps associated with the object, a first set of rendered images that lack highlights and a second set of rendered images that include highlights; and generate one or more appearance maps associated with the object based on (i) a first comparison of the first set of rendered images with the first set of images and (ii) a second comparison of the second set of rendered images with the second set of images. ||
18. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: receiving a first set of images of an object captured using a first set of cameras, wherein the first set of cameras is configured to capture light that is cross-polarized with respect to a polarization of at least one light source; receiving a second set of images of the object captured using a second set of cameras, wherein the second set of cameras is configured to capture light that is not cross-polarized with respect to the polarization of the at least one light source; generating, based on initial appearance maps associated with the object, a first set of rendered images that lack highlights and a second set of rendered images that include highlights; and generating one or more appearance maps associated with the object based on (i) a first comparison of the first set of rendered images with the first set of images and (ii) a second comparison of the second set of rendered images with the second set of images. ||
21. A computer-implemented method for generating one or more maps indicating surface properties of an object, the computer-implemented method comprising: receiving a first set of images of the object captured using a first set of cameras, wherein the first set of cameras is configured to capture light that is cross-polarized with respect to a polarization of at least one light source; receiving a second set of images of the object captured using a second set of cameras, wherein the second set of cameras is configured to capture light that is unpolarized; and generating one or more appearance maps associated with the object based on the first set of images and the second set of images, wherein generating the one or more appearance maps comprises: generating a set of inverse-rendered images based on the one or more appearance maps; and modifying the one or more appearance maps based on a rendering error between the set of inverse-rendered images and at least one of the first set of images or the second set of images, and wherein generating the one or more appearance maps further comprises: applying a set of per-camera weights to the rendering error, or applying a set of regularization constraints to the rendering error.