Capturing human performances in high-definition 3D is a complicated proposition, and among the many challenges is getting the lighting right. This impressive new project from Google researchers puts the subject in the center of what can only be described as an prismatic LED egg, but the resulting 3D models are remarkable — and more importantly, relightable.
What’s called volumetric capture uses multiple cameras in a 360-degree setup to capture what can look like a photorealistic representation of a subject, including all the little details like clothing deformation, hair movement, and so on. It has two serious weaknesses: First, it’s more like a 3D movie than a model, since you can’t pose the person or change their attributes or clothing; The second is an extension of the first, in that you can’t change the way the person is lit — whatever lighting they had when you captured them, that’s what you get.
“The Relightables” is an attempt by a team at Google AI to address this second issue, since the first is pretty much baked in. Their system not only produces a highly detailed 3D model of a person in motion, but allows that model to be lit realistically by virtual light sources, making it possible to place it in games, movies, and other situations where lighting can change.
Images from the Google AI paper that show the capture process and resulting 3D model alone and in a lighted virtual environment.
It’s all thanks to the aforementioned prismatic egg (and a couple lines of code, of course). The egg is lined with 331 LED lights that can produce any color, and as the person is being captured, those LEDs shift in a special structured pattern that produces a lighting-agnostic model.
The resulting models can be placed in any virtual environment and will reflect not the lighting they were captured in but the lighting of that little world. The examples in the video below aren’t exactly Hollywood-level quality, but you can see the general idea of what they’re going for.
The limitations of volumetric capture make it unsuitable for many uses in film, but being relightable brings these performances a lot closer to ordinary 3D models than they were before. Of course, you still have to do all your acting inside a giant egg.
“The Relightables” will be presented by the team at SIGGRAPH Asia.
