Using artificial intelligence to generate 3D holograms in real time

Despite years of exaggeration, virtual reality headsets have not yet dropped TV or computer screens as essential devices for viewing videos. One reason: VR can make users feel bad. Nausea and eye strain can occur because VR creates an illusion of 3D viewing, although the user is actually looking at a fixed-distance 2D screen. The solution for a better 3D visualization could be in a technology of 60 years remade for the digital world: holograms.

Holograms provide an exceptional representation of the 3D world around us. In addition, they are beautiful. (Go ahead – check the holographic dove on your Visa card.) Holograms offer a perspective of change based on the viewer’s position and allow the eye to adjust the focal depth to alternately focus on the foreground and background.

Researchers have long sought to make computer-generated holograms, but the process traditionally requires a supercomputer to perform physical simulations, which is time-consuming and can produce less than photorealistic results. Now, MIT researchers have developed a new way to produce holograms almost instantly – and the method based on deep learning is so efficient that it can be run on a laptop in the blink of an eye, the researchers say.

“People previously thought that with existing hardware for the consumer, it would be impossible to do 3D holography calculations in real time,” said Liang Shi, lead author of the study and Ph.D. student in the Department of Electrical Engineering and Computer Science (EECS) of MIT. “It is often said that commercially available holographic screens will be available in 10 years, but this statement has been around for decades.”

Shi believes that the new approach, which the team calls “tensor holography”, will finally bring that elusive 10-year goal within reach. The advance could boost holography in fields like VR and 3D printing.

Shi worked on the study, published in Nature, with its advisor and co-author Wojciech Matusik. Other co-authors include Beichen Li from EECS and the MIT Computer Science and Artificial Intelligence Laboratory, as well as former MIT researchers Changil Kim (now on Facebook) and Petr Kellnhofer (now at Stanford University).

The search for a better 3D

A typical lens-based photograph encodes the brightness of each light wave – a photo can faithfully reproduce the colors of a scene, but it ends up producing a flat image.

In contrast, a hologram encodes the brightness and phase of each light wave. This combination provides a more accurate representation of a scene’s parallax and depth. Thus, while a photograph of Monet’s “Water Lilies” can highlight the color palette of the paintings, a hologram can bring the work to life, reproducing the unique 3D texture of each brushstroke. But, despite their realism, holograms are a challenge to make and share.

First developed in the mid-1900s, the first holograms were recorded optically. This required splitting a laser beam, with half the beam used to illuminate the subject and the other half used as a reference for the phase of the light waves. This reference creates a unique feeling of depth in a hologram. The resulting images were static, so they could not capture movement. And they were just hard copies, which made them difficult to reproduce and share.

Computer-generated holography circumvents these challenges by simulating the optical configuration. But the process can be computational hard work. “Since each point in the scene has a different depth, you cannot apply the same operations to all of them,” says Shi. “This significantly increases the complexity.” Directing a clustered supercomputer to perform these physics-based simulations can take seconds or minutes for a single holographic image. In addition, existing algorithms do not model occlusion with photorealistic precision. Therefore, Shi’s team took a different approach: letting the computer teach itself physics.

They used deep learning to accelerate computer-generated holography, allowing the generation of holograms in real time. The team designed a convolutional neural network – a processing technique that uses a chain of trainable tensors to roughly mimic how humans process visual information. Training a neural network typically requires a large set of high-quality data, which did not previously exist for 3D holograms.

The team built a custom database of 4,000 pairs of computer-generated images. Each pair combined an image – including color and depth information for each pixel – with its corresponding hologram. To create the holograms in the new database, the researchers used scenes with complex and variable shapes and colors, with the depth of pixels evenly distributed from the background to the foreground, and with a new set of physics-based calculations to deal with the occlusion. This approach resulted in photorealistic training data. Then the algorithm started to work.

Upon learning from each pair of images, the tensor network adjusted the parameters of its own calculations, successively increasing its ability to create holograms. The fully optimized network operated orders of magnitude faster than physics-based calculations. This efficiency surprised the team itself.

“We are delighted with your performance,” says Matusik. In mere milliseconds, tensor holography can create holograms from images with depth information – which are provided by typical computer-generated images and can be calculated from a multi-camera configuration or LiDAR sensor (both are standard on some new smartphones) . This breakthrough paves the way for real-time 3D holography. Furthermore, the compact tensioner network requires less than 1 MB of memory. “It’s insignificant, considering the tens and hundreds of gigabytes available on the latest cell phone,” he says.

“A considerable leap”

Real-time 3D holography would enhance a range of systems, from VR to 3D printing. The team says the new system can help immerse VR viewers in a more realistic setting, while eliminating eye strain and other side effects of long-term use of VR. The technology could easily be deployed on monitors that modulate the phase of the light waves. Currently, the most affordable consumer monitors modulate only the brightness, although the cost of phase modulated monitors would fall if they were widely adopted.

Three-dimensional holography can also spur the development of 3D volumetric printing, say the researchers. This technology can be faster and more accurate than traditional layer-by-layer 3D printing, since volumetric 3D printing allows for the simultaneous projection of the entire 3D standard. Other applications include microscopy, visualization of medical data and the design of surfaces with unique optical properties.

“It is a considerable leap that can completely change people’s attitudes towards holography,” says Matusik. “We feel that neural networks were born for this task.”