NVIDIA Open-Sources Lyra 2.0, a Picture Generates a Walkable 3D World, Directly Importable to Robot Simulator

According to Perceiving Dynamics monitoring, NVIDIA has released Lyra 2.0, an open-source framework that can generate a explorable 3D world from a single image. Users provide a photo, and Lyra 2.0 first generates a roam video controlled by the camera trajectory, then reconstructs the video into 3D Gaussian Splats and mesh models, which can be imported directly into game engines and simulators for real-time rendering. The model weights and code are open-sourced under the Apache 2.0 license on Hugging Face and GitHub, allowing for commercial use.

The core technological breakthrough lies in addressing two degradation issues during long-distance roaming. The first is "spatial forgetting": when the camera moves far away and then returns, areas previously seen are beyond the model's time window, causing the model to generate from scratch, resulting in scene inconsistencies. Lyra 2.0 maintains 3D geometric information for each frame, retrieves relevant historical frames when returning, and establishes dense correspondences to help the model "remember" what it has seen before. The second is "temporal drift": autoregressive generation accumulates frame-by-frame errors, gradually distorting the scene. Lyra 2.0 uses self-enhancing training to expose the model to its degraded output during the training phase, enabling it to learn error correction rather than error propagation. Built on the WanYing 2.1-14B Diffusion Transformer, it outputs at a resolution of 832×480.

For developers and researchers, the most direct application scenario is robot simulation. NVIDIA demonstrated importing the 3D scene generated by Lyra 2.0 into its in-house physics simulator, Isaac Sim, where robots can navigate and interact. A major bottleneck in training embodied AI was the high cost and limited variety of 3D environment creation. Lyra 2.0 offers a path to batch-generate training environments from photos. Last year's September release of Lyra 1.0 only supported short-distance generation, while version 2.0 extends it to long-distance continuous exploration. Google's previous release, Genie 3, had similar capabilities but was not open source. Lyra 2.0 is currently the most comprehensive open-source solution in this direction.