Microsoft recently invited me to join mixed reality pioneer Alex Kipman, the company’s technical colleague for Windows mixed reality, in an individual chat. The difference was that I wasn’t at Zoom or Teams: I was gathered around a virtual table that sprang up in my home office.
A holographic cartoon version of Alex hovered in my space and I walked around it. The only thing I needed to connect was an independent visor that I wore over my face: the Microsoft HoloLens 2. My HoloLens 2 test drive at home, for the first time, showed me where AR glasses can go. And, also, the challenges that still need to be solved. Microsoft Mesh, a technology that promises a way to get people to the same shared virtual space, shows incredible promise. But the hardware that will make the most of it hasn’t arrived yet.
See this:
Meeting with Alex Kipman of Microsoft at HoloLens …
2:18
HoloLens 2 has been around for over a year, but not for you or me. It is sold as a business device, which means it is a $ 3,500 headset intended for people in the workplace who can afford it. Unlike VR headsets, it is not designed for gaming. And Microsoft never sent HoloLens 2 analytics units before that: My demos were always in controlled spaces, for a limited period of time. When Microsoft offered to send a borrowed HoloLens 2 as part of its coverage of its mixed reality software ad, I was extremely intrigued. It is still a really new device for me.
Remember that this is an AR headset, not a VR headset: its lenses are transparent. HoloLens 2 superimposes shiny virtual objects that appear to exist in the real world. The only other headset like this is the Magic Leap One, also a business device (which I once tried in my office for about a week). It is not a matter of entering a virtual space, but of being in my own space and putting things on top of it. All those Marvel, Kingsman and Star Wars dreams about holograms that you can interact with, well, that’s Microsoft’s goal. While Qualcomm, Facebook and perhaps Apple (and others) work on AR headsets, HoloLens 2 looks like the prototype of what’s to come.
HoloLens 2 doesn’t exactly have that goal, but nobody is. Still, it can get closer now than anything else.
HoloLens 2, Oculus Quest 2: both stand-alone, both easy to start. One is AR, one is VR.
Scott Stein / CNET
It reminds me, strangely, of Oculus Quest 2
The headset is surprisingly compact and almost the same size (although lighter, in fact, than) Oculus Quest 2, Facebook standalone virtual reality headset. While Quest 2 costs $ 300 and HoloLens 2 costs more than $ 3,000, there is a spiritual similarity between the two. Both are standalone devices that do not need PCs or phones to be used. Both fit easily on my head and my glasses.
The self-contained, easy-to-use feel of both serves a similar purpose: to get people into VR (or AR) quickly and without tangles of cables or strange interfaces.
That’s where the similarities end.
Look, without controllers
Oculus Quest and HoloLens 2 allow hand tracking, but Facebook uses it as an alternative to Quest controllers. Hand tracking in Oculus Quest works surprisingly well, but HoloLens 2 has no controller: everything is done with your hands. That’s where HoloLens 2 shines … and it also has strange moments.
To touch virtual things, like buttons or keyboards, I reach out and touch them. To grab an object, pinch the edge. I open the HoloLens menu by looking at my wrist and tapping a button that appears there, glowing. To control distant things, I open my hand and cast a beam as if it were Vision. There is a sense of having supernatural powers that flow through the HoloLens interface.
On my own, I try to play a game called Roboraid on HoloLens 2, where things pop up from my walls – I tried a variation of this game many years ago in an E3 demo, but at home, I use my hands to play. Pinching, pointing and tapping my fingers together is a lot of what HoloLens 2 requires. Arm gestures can be tiring. I would like simple shortcuts. Also, a controller would be nice. I can’t get any feedback like vibration, which is where a bracelet or ring or neural input technology in the future, like what Facebook planned, comes in. Some kind of controller could help to make gestures more minimal and even let me feel what I’m doing.
Even though the display limits are lower than desired by HoloLens, I can draw in 3D in my room, scribbling lines from my bookcase and writing down real objects. I placed virtual objects next to real objects. The brilliant virtual ones remain in place, and when I come back later, HoloLens 2 on, they are still there.
The flip-down lenses of HoloLens 2. The headset can also track my eyes.
Scott Stein / CNET
Eye tracking: a technology waiting on the wings
HoloLens 2 also has eye tracking, something that today’s non-commercial virtual reality headsets do not. Eye tracking is subtle, but it allows me to look at an object – such as an open application window on the other side of the room – and say “close application” and know which one to close. When I talk to people in AR, they can see my virtual avatar’s eyes moving because eye tracking sees where I’m looking.
Facebook CEO Mark Zuckerberg wants eye tracking on future VR and AR headsets for the same purpose, to map facial expressions and eye movements into realistic avatars. Microsoft’s technology, however, being focused on business, contains the use of eye-tracking data for specific and secure instances. In conventional headphones and glasses, how will this data be used and shared? We don `t know yet.
A floating virtual desktop full of windows
I gave a demonstration of Microsoft software and also tried some other applications. I sat down and tried to open the web browsers, then played a game or two (yes, there are a few). What really impressed me was how the windows could open and float on my desk, or whatever training I wanted. I could get up and they would be stuck there. They would be there the next day.
Qualcomm’s AR smart glasses are designed to be connected displays for phones and PCs. What I’m seeing in HoloLens 2 looks like a preview of those glasses and what they’ll eventually be able to do.
In HoloLens 2, I am limited to using my hands (although I think I could pair a keyboard). I would love to see what it feels like for my laptop to suddenly create extra windows and monitors hovering in the air when I put on my future smart glasses.
The HoloLens 2 cameras, which can scan my room deeply, like the iPhone 12 Pro handle.
Scott Stein / CNET
The screens are still not perfect
The limited field of view of HoloLens 2 looks like a large floating window in front of me, where shiny 3D things appear. But the window is not big enough, which means I have to move my head to see things around the room that I don’t know are there.
The screen also has a slightly cloudy rainbow quality. It is not the perfect vivid screen that I would expect on a monitor, or even on recent virtual reality headsets. If I want to use an AR headset to watch movies or play games, I would like something more evolved. It’s not easy on a transparent lens, but maybe Micro LED technology can help make things better soon.
What will be the killer apps?
Microsoft’s HoloLens 2 uses communications and telepresence as its killer applications for business. It can also be excellent at giving instructions in the workplace. But what would be the killer apps for AR glasses sold to ordinary people? Was it fitness? Games? Virtual cinema glasses? An extra monitor that can go anywhere?
No one has discovered this yet. Companies like Niantic, creators of what is arguably the AR killer app, Pokémon Go, are exploring what it’s like play in AR glasses using a HoloLens 2. The Microsoft headset was not made to go anywhere. It is not great in daylight; it looks big and similar to a helmet; and the battery life is not long. But it’s probably the best prototype I’ve ever tried for what AR glasses will need to do next.