header-langage
简体中文
English
Scan to Download the APP
Partnered With:

Analysis of Google's latest black technology: holographic video chat technology Project Starline

Read this article in 8 Minutes
Through Starline, people who video each other don't need to wear any glasses or helmets, it's as real as sitting across from each other and chatting
Original title: "Google reveals naked-eye 3D holographic video chat technology: 8k screen, 4 GPUs and a bunch of cameras"
Original source: Sina VR


At the Google I/O conference held in May this year, it was held outdoors in the Google campus On the opening Keynote of , the company released a series of eye-catching new products.


At the meeting, Google announced a black technology that has been secretly developed for many years: Project Starline, a holographic video chat technology. Project Starline is essentially a 3D video chat room designed to replace one-on-one 2D video conference calls, making users feel like they're sitting in front of a real person.


Through Starline, people who video each other do not need to wear any glasses or helmets. It is as real as sitting across from each other and chatting, with full details of the characters. Let's experience its effect first:



Actually, this is Real-time 3D models regenerated by high-resolution sensors, dozens of depth-scanning sensors, and a 65-inch "Light Field Display". Google says the project has been in development for several years, with a host of computer vision, machine learning, spatial audio and data compression technologies behind it. Google has also developed a groundbreaking light field system for this purpose, allowing people to experience realistic volume and depth without glasses or headphones.



We can imagine how difficult it is to implement this technology, first you need Let the brain think that there is a real person sitting not far from you; secondly, the image needs to be high-resolution and free of artifacts; in addition, there is an audio problem, because the system needs to make the sound sound like it is coming from the mouth of the person on the other side. There are minor issues such as eye contact etc.


How is the technology behind this cutting-edge black technology realized? Presumably many people want to know, recently, Google announced the technology behind the Project Starline demo in a paper.


(Paper address: https://storage.googleapis.com/pub-tools-public-publication-data/ pdf/424ee26722e5863f1ce17890d9499ba9a964d84f.pdf)


The hardware part, the Project Starline system is built around a massive 65-inch 8K panel running at 60Hz. Around it, Google engineers deployed three "capture pods" that capture color images and depth data. The system also includes four additional tracking cameras, four microphones, two speakers and an infrared projector. Overall, the system needs to capture color images from four viewpoints and three depth maps, for a total of seven video streams. The system also needs to capture 44.1 kHz audio and encode at 256 Kbps.


Obviously, all this hardware generates a lot of data that needs to be transferred, and Google says the transfer bandwidth varies from 30Mbps to 100Mbps, depending on the texture details of the user's clothing and gesture size. So it's much more than a standard Zoom call. Project Starline is equipped with four high-end Nvidia graphics cards (two Quadro RTX 6000 cards and two Titan RTX) to encode and decode all this data. End-to-end latency averaged 105.8 ms.

 

System Implementation


As shown in Figure 4 below, Google’s system mainly has two components: One is the display unit that contains the monitor, camera, speaker, microphone, illuminator, and computer, and the other is the backlight unit that contains the infrared backlight and is used as a bench seat. Both units contain white LED strips angled towards the walls and ceiling for soft reflected lighting.



The capture subsystem consists of three synchronous stereo RGBD capture pods: Two are above the monitor and one is in the "middle wall" below the monitor. The lower pod includes an additional color camera for zooming in on the subject's face. Additionally, two of the four monochrome tracking cameras sit above the display and one on either side of the remaining two, capturing high-speed wide-angle images of the eyes, ears, and mouth.


Figure 5 below shows the arrangement of the capture and display components.



More detailed system component information is as follows:



Figure 6 below is the data flow in the system, showing how the main processing components of Starline are mapped to GPUs and CPUs. All video processing in the system occurs at 60 Hz, except for face tracking at 120 Hz and IR stereo mode capture at 180 Hz.



Lighting


Google uses indirect "bounce" (bounce) light sources to create a soft lighting environment. On the sides and back of the display and backlight units, white LED strips illuminate the surrounding walls, creating a comfortable diffuse source that minimizes sharp highlights. This diffused light is also more comfortable for the user than bright LED direct lighting.


At the same time, it is also important to maintain a certain uneven distribution of light. Google found that perfectly uniform incoming light made faces and other 3D shapes look flat and unreal, hindering other 3D effects in the system. Photographers, especially cinematographers, refer to the contrast between the fully lit and shadowed sides of a subject as the "illumination ratio." To keep objects three-dimensional, Google uses stronger intensity on the side of the display unit adjacent to the wall, resulting in a roughly 2:1 lighting ratio.


The display effect is shown in Figure 7 below:


Color Calibration


Google color-calibrates the system's RGB cameras by adjusting each camera's gain, color correction (3×3) matrix, and gamma so that the standard color target [McCamy et al.


1976] Matches reference color values under D65 illuminant, offsetting the effects of indoor lighting. Images captured by a color-calibrated monitor under a D65 illuminant look like they were captured under local room lighting conditions (intensity and color). This color calibration scheme ensures that the system automatically corrects for subtle lighting differences between two user locations.

 

Object Capture


Google's goal is to render a new image for each user, this is because They should appear on the left and right eyes of other users. Obviously, if the camera can be placed precisely at these eye locations, then capture becomes trivial.


Unfortunately, this is not feasible. On the one hand, these positions are located near the center of the display and thus will be occluded; on the other hand, the user will be free to move in all 3 dimensions. Emerging see-through display technologies may be able to partially solve this problem, but transparent autostereoscopic displays do not yet exist, and in any case cannot account for viewer motion.


Google therefore placed the capture sensors on the periphery of the display. Due to the large angle of the display towards the local user, the capture perspective is far from the eye position that needs to be rendered. To account for this large parallax, Google uses a combination of visible light and near-infrared (NIR) global shutter image sensors to reconstruct a geometric approximation of the user.


As shown in Figure 5 above, the sensors are spread across three capture pods, two above the display and one on the mid-wall below it. The upper pod provides a good view of hand gestures and the sides of the head and torso, while the lower pod provides a good view of the neck, face and chin. The pod is sufficiently large at 1.4 m wide, 1.0 m high and 0.9 m deep to capture the head, torso, arms and hands of a seated user talking and gesturing naturally.

3D face tracking


Google uses four simultaneous 1280×1024 monochrome cameras, which Runs at 120Hz and has a filter to block NIR light. For each captured image, Google detects faces and localizes 34 facial landmarks [FaceDetector 2019]. Additionally, they determined the 2D positions of eyes, mouth, and ears as a weighted combination of neighboring landmarks. In order for at least two of the four tracking cameras to find these objects, Google uses triangulation to get their 3D positions.

Video compression and transmission


Four color and three depth streams from the RGBD capture pod on the GPU compressed and transmitted using WebRTC along with the tracked 3D facial points.


Using video compression techniques, this research was able to take advantage of highly optimized video encoders and decoders found in modern GPUs. Specifically, they use the NVENC/NVDEC units of four NVIDIA GPUs. This way there is enough throughput to handle four color and three depth streams at full resolution and 60Hz frame rate. Both color and depth streams are encoded using the H.265 codec with YUV420 chroma subsampling. The color stream uses 8 bits per channel, the depth stream uses 10 bits per channel, the depth data is stored in the Y luminance channel, and the UV chroma channel is set to 512 (gray). This study reduces encoding and decoding latency by omitting bidirectionally encoded (B) frames.

Rendering


After decompressing 3 depth maps and 4 color images on the receiving client, The study renders a virtual remote user's left-right perspective from the local user's eye position. It consists of three steps:

For each 4-color camera, the shadow map is computed using raycasting by finding for each ray the first intersection point of the surface fused with the input depth map;


For each of the 2 user views (left and eye), the output depth map is computed using the same raycasting algorithm;


< /p>

For each output depth map point, compute the shadow map weighted color blend from step 1.


For the left and right views, the study obtains the color of each pixel by projecting a color image onto the fused geometry (Fig. 10):



Figure 11 shows the Irregular pixelated outlines. The figure also shows that image-based fusion provides a more complete reconstruction than volumetric fusion near contours:


Sound transmission


The audio transmission technology used in this research is a combination of a series of advanced technologies: talker-tracked beamforming, reverberation reduction, WebRTC transmission, talker /listener-tracked virtual audio synthesis, binaural crosstalk cancellation crossover combination, and amplitude panning. Accurate tracking of talkers and listeners is a key factor in the authenticity of shared spaces compared to traditional video conferencing systems (Figure 12). It is understood that this is the first audio technology that does not require headphones and head tracking in a video conference.



Original link


Welcome to join the official BlockBeats community:

Telegram Subscription Group: https://t.me/theblockbeats

Telegram Discussion Group: https://t.me/BlockBeats_App

Official Twitter Account: https://twitter.com/BlockBeatsAsia

举报 Correction/Report
Choose Library
Add Library
Cancel
Finish
Add Library
Visible to myself only
Public
Save
Correction/Report
Submit