Facebook CEO Mark Zuckerberg wants eye tracking and facial expression tracking in future iterations of Oculus Quest.
In a broad interview with The information, Zuckerberg spoke about the company’s strategy and direction in the AR / VR space.
He said Facebook is already working on its next VR headsets:
“Because of how the hardware is developed, you kind of need to know what the next three products will look like, all at the same time. It is not like the software that we change it every two weeks. We have product teams formed now, working on the next generations of virtual reality and what Quest 3 and 4 will look like.“
Zuckerberg revealed one of his most desired personal features in the next Oculus headphones; eye and face tracking to boost avatars in social experiences:
“One of the things I’m really excited about in future releases is getting eye and facial tracking on, because if you’re really excited about the social presence, you want to make sure that the device has all the sensors to really animate realistic avatars so that you can communicate well.“
Later in the interview, he talked about some of the difficulties in making this a reality:
“On the VR side, I think the most important things we’re very focused on right now are: how do you pack more sensors, to create a better social experience, on the device? To run each sensor requires more CPU power and this generates more heat and creates all these different problems.
When I think about where you are with VR today, there are some very good games and different experiences, but I would love to get to the point where you have realistic avatars of yourself, and where you can make authentic eye contact with someone and have real expressions that are reflected in your avatar.
So, what do you need for this? Well, you need to be able to track your eyes to make eye contact. You need to have some sense of face tracking or a sense of what is going on with the person’s expressions to be able to make those emotions appear naturally.“
Virtual reality headsets with eye or facial tracking are aimed at corporate buyers. Pico’s Neo 2 Eye is the first standalone headset with eye tracking, priced at $ 900 with a focus on corporate use cases. The Reverb G2 Omnicept Edition connected to the HP PC will have eye and face tracking, but it is also focused on corporate use cases with prices yet to be revealed.
Facebook Avatar Search
Facebook first showed its long-term research on photorealistic ‘Codec Avatars’ in March 2019.
With machine learning technology, avatars are generated using specialized capture equipment with 132 cameras. Once generated, they can be animated in real time by a prototype VR headset with eye tracking and face tracking cameras.
When introducing codec avatars for the first time, Facebook warned that the technology was still “years ahead” for consumer products – sending this type of photoreal avatar will require a number of innovations.
Facebook avatars today
Oculus Quest today has an integrated basic avatar system, Oculus Avatars. It is used in some applications like Poker VR Stars and XR tribe DJ, but not much more. Platforms like Bigscreen, VRChat and Rec Room use their own separate avatar systems.
Today’s headphones don’t have eye or facial tracking, but Oculus Avatars uses a neural network to simulate lip movement, and developers can set priority-looking targets to simulate eye movement.
In September, Facebook announced that the new Facebook Avatars system will replace Oculus Avatars. Apparently, it is an evolution of the current Facebook VR avatars used on Facebook Horizon and the beta version of the new Venues. Employees working on the project include former Pixar animators.
Facebook’s avatars seem to fall back on artistic realism, but it adds a full torso and simulated arms.
In today’s interview, Zuckerberg announced that Facebook Avatars will be launched this year, saying that “it will become more and more realistic over time”, suggesting that the company intends to take an iterative approach. Zuckerberg also commented on Facebook’s low-cost hardware strategy.