Mobile VR comes with its own set of challenges
July 20, 2017
Blog
Virtual reality is coming to mobile platforms, but it comes with challenges, from processing to power, and then some.
The convention floor at this year’s Mobile World Congress Shanghai had more Virtual Reality (VR) booths and demos than ever. Many visitors could be spotted wearing a head-mounted display (HMD), as in Figure 1. The hype surrounding VR has been brewing for quite a while. Sales, on the other hand, haven’t been living up to expectations. What needs to change to finally unleash the potential of VR and related technologies, like augmented reality (AR) and extended reality (XR)?
[Figure 1 | MWC Shanghai 2017: Visitors trying one of the many VR demos.]
HMDs come in a wide range of flavors, from low-cost cardboard-style designs, to high-end deeply immersive suits, like the HTC Vive, and many in between (Figure 2). On the left side of the immersion axis, all the headsets are portable and use the processing power and display of an existing smartphone which fits into the device. On the right, all the HMDs are powered by either a PC or a console, keeping the user tethered during the VR experience. This ball and chain is one thing that holds back the user experience, as my colleague pointed out over a year ago.
[Figure 2 | The graph shows the price to immersion level tradeoff in popular HMDs. (Source: CEVA)]
Sales of all the top VR headsets are forecast to grow year over year (Figure 3). That said, the portable headsets, Samsung Gear VR and Google Daydream, are expected to ship significantly more units than the PC/console-tethered counterparts. These forecasts are reinforced by other market researchers, as well. According to BI Intelligence, the smartphone-powered headsets are forecast to be fastest growing category over the next few years. The other two categories of console-powered headsets and PC-powered headsets will also grow, but not nearly as fast.
[Figure 3 | Shown are the worldwide VR headset shipments in 2016 vs. 2017.]
A new category, not included in the report, is the standalone VR headsets. There are rumors circulating about all the major players in this field, including Facebook, Samsung, Google, and Intel. The expectation is to see standalone VR headsets available by 2018. This category will face the same challenges as the smartphone-powered headsets.
In a different forecast, Mediatek shared on its blog that nearly 60 million smartphone VR headsets will ship in 2021. That represents an increase of around 240% over the 16.8 million shipped in 2016. These numbers are based on a study by Juniper Research. This means that the companies that address the challenges of mobility, i.e. power consumption, computational resources, and inside-out tracking, will be poised to sell the most. The other main challenge is content quality, which is relevant to all the categories.
Looking at two short videos captured at MWC Shanghai, we can see some of the issues that mobile VR solutions will have to solve. In the first video, we immediately see cables dangling from wearable gear, which connect it to a high-performance PC to handle processing and supply power. In the second video, you a see a little girl manipulating a basic 3D block using the Leap Motion reference platform. In the latter, the demo is made possible by external sensors, which confine the activity to a predetermined space.
Price, content, and immobility are the hold-ups
If you follow the HBO series Silicon Valley, you’ve probably noticed a new character this season, the programming whiz kid, Keenan Feldspar. In the storyline (spoiler alert!) he builds a highly immersive VR demo. After all the hype subsides, the solution is criticized because it needs a $10,000 rig to run smoothly and isn’t mobile. That’s an on-the-nose commentary on the actual VR situation. Some of the strongest factors inhibiting large scale reception of current VR products are the price tag and the immobility of the heavy equipment. Fans of the show have raised the theory that Feldspar’s character is a TV version of Oculus founder, Palmer Luckey. Luckey himself recently updated his Twitter profile picture to a picture of Feldspar.
Luckey joined Facebook when the company acquired Oculus for $2 billion in 2014. Its first HMD, the Rift, was released in 2016 and achieved disappointing sales, after which Luckey was pushed out of the company. Since then, he tweeted confirmation that he pledged $2000 to a project called Revive via the Patreon campaign. The project enables VR games designed exclusively for the Oculus Rift to run on the competing HTC Vive. Currently, quality content is a significant differentiator to attract consumers to a specific VR ecosystem. But apparently, Luckey believes that content should be cross-platform and I tend to agree with this approach.
On a related note, Khronos Group is working to create an open and royalty-free standard for VR and AR applications and devices, called the OpenXR working group. If successful, this will create a cross-platform architecture that will cut the need to port apps, games, and engines to each vendor’s API.
The challenges of mobile VR
The challenges are slightly different on mobile devices. The problem of being tethered is solved, cutting the annoying cords and opening endless opportunities for location-based content. But other challenges are introduced, starting with power consumption. Mobile means that the power source is integrated inside the HMD, and the battery’s size and weight are determined by what can be placed on the user’s head without impeding comfort. The battery location is now shifting from the front to the back to accommodate weight balance. Besides the battery’s size, which determines the run time, the amount of processing must be considered. Rendering full HD will require much less processing than 4K resolution, where the marker is heading.
As an example, the Caraok Standalone VR headset (Figure 4) boasts a 3200-mAh, 400-g lithium battery, claiming about three hours of play time. Will you be able to keep your head up for three hours with that weight perched on your head? There’s no clear-cut answer, but one thing is certain: processing must be extremely efficient to make every mAh in the batteries count.
[Figre 4 | Caraok: You might get three hours of play time, but can your neck muscles last that long? (Source: Amazon)]
Maybe the prototype that Oculus is developing for a standalone VR headset, code-named Pacific will answer some of these questions. Facebook CEO Mark Zuckerberg said that it will sit in the “sweet spot” between the Rift’s tethered, immersive experience and the Samsung Gear VR’s lightweight experience. It’s meant to do the same things as other Oculus gear—immersive gaming, video viewing, and social networking, but without a PC powering it, and without the pesky cords. Priced at $200, it just might hit that sweet spot.
Outside-in vs. inside-out tracking
Most of the stationary VR devices track the user’s movement “outside-in” by placing sensors around the room to determine the position of the HMD. The reverse approach is inside-out tracking, where sensors are placed on the device that’s worn by the user. In mobile HMDs, inside-out tracking is a necessity. This requires smarter sensors on the headset which also use the same power source as the display and processor. While inside-out tracking is more complex and challenging, its huge advantage is that it allows complete mobility. By not relying on external sensors, it can be used anywhere, not just in a preconfigured room. Inside-out tracking still isn’t as precise as outside-in, but it’s catching up. The computational challenge is to achieve six degrees of freedom tracking as well as eye tracking and hand gestures.
One thing that requires a lot of processing power in VR is generating ultra-high-resolution renderings to make the experience realistic and not pixelated. The 360-degree nature of the images makes that even more difficult. Since mobile systems typically have less processing power than gaming computers and consoles, they’ve lagged in resolution.
Foveated rendering
Foveated rendering could be a game-changer in this field. By tracking the user’s eye-movement, foveated rendering lets the device display a high-resolution image only where the user’s view is focused. Everything else is rendered at a lower resolution, until the gaze moves there. To that end, Apple has quietly acquired SensoMotoric Instruments, a company specialized in tracking people's eye movements. Similarly, Google bought Eyefluence, who specializes in the same field. These are clear signs that these companies are targeting mobile VR, and understand that foveated rendering is the way to go.
Beyond gaming: Healthcare, education, tourism, and more
VR, AR, and XR open endless possibilities to enhance our lives and experiences. Although entertainment is the most popular use of this technology, it’s only the beginning. If healthcare providers could perform procedures or train colleagues from across the globe, many lives could be saved and/or improved. Education could cross boundaries and reach many in need.
Tourism could use these technologies to allow people to experience exotic places or to enrich the adventure in familiar ones. But for all these possibilities to come to fruition, the technology must be accessible. The processing must be efficient and intelligent. The headsets need to be lightweight and comfortable while still packing enough battery power to ensure an uninterrupted activity. Once all these are achieved, all these exciting use cases are sure to follow.
An efficient vision processor must be at the core of mobile VR. Challenging computations like 4K distortion correction, foveated rendering, inside-out tracking, and sensor fusion, must be handled at low power and maximum speed for a comfortable and enjoyable experience. Find out more about how CEVA is taking imaging and computer vision mobile.
