Nvidia’s new AI magic turns 2D photos into 3D graphics

Nvidia has made another attempt to add depth to shallow graphics.

After converting 2D images into 3D scenes, models, and videos, the company has turned its focus to editing.

The GPU giant today unveiled a new AI method that transforms still photos into 3D objects that creators can alter with ease.

Greetings, humanoids

Subscribe to our newsletter now for a weekly recap of our favorite AI stories in your inbox.

Dubbed 3D MoMa, the technique could give game studios a simple way to alter  images and scenes. This typically relies on time-consuming photogrammetry, which takes measurements from photos.

3D MoMa speeds up the task through inverse rendering. This process uses AI to estimate a scene’s physical attributes — from geometry to lighting — by analyzing still images. The pictures are then reconstructed in a realistic 3D form.

David Luebke, Nvidia’s VP of graphics research, describes the technique as “a holy grail unifying computer vision and computer graphics.”

“By formulating every piece of the inverse rendering problem as a GPU-accelerated differentiable component, the NVIDIA 3D MoMa rendering pipeline uses the machinery of modern AI and the raw computational horsepower of NVIDIA GPUs to quickly produce 3D objects that creators can import, edit, and extend without limitation in existing tools,” said Lubeke.

3D MoMa generates objects as triangle meshes — a format that’s straightforward to edit with widely-used tools. The models are created within an hour on a single NVIDIA Tensor Core GPU. 

Materials can then be overlaid on the meshes like skins. The lighting of the scene is also predicted, which allows creators to modify its effects on the objects.

Triangle meshes define the shapes used in 3D graphics and modeling.