Western's Melissa Rice using Microsoft's new HoloLens headset for her research on Mars

Curiosity, the Mars rover that Western’s Assistant Professor of Geology Melissa Rice helps to operate, sends her a 360-degree image of the Planet’s desolate landscape every morning.

As if seeing brand-new images from another planet isn’t amazing enough, the new Microsoft HoloLens headset that Rice recently received uses these photos to produce an augmented reality simulation of the Mars landscape. This allows Rice to explore, virtually, the Mars landscape around the rover.

“I can see everything on Mars to scale all around me in all directions. When I only use my computer screens, I do not get an intuitive sense for how big something is that I see in the images. I never lose that sense of scale with the HoloLens,” Rice said.

The Curiosity rover supplies Rice with both images as well as data from its on-board labs.

“Using that image, we then decide where the rover is going to drive next. We look at the rocks closest to the rover and decide which ones we want the rover to reach out and touch and take a microscopic image of, or take a chemical measurement of,” Rice said. “At the end of our day on Earth, we send those commands to the rover, which then executes those commands when it is morning on Mars. This process repeats itself every day.”

The HoloLens fits around the user’s head and can be tightened by a knob in the back.

 “The HoloLens uses augmented reality, which is different than virtual reality in that it does not block off any of your normal sensory input,” Rice said. “Instead of putting you on Mars, what it does is overlay the Mars landscape on top of your normal reality,” Rice said.

One example of augmented reality is using a smartphone to play the incredibly popular Pokémon Go! game, where “monsters” can be found near the user by looking through the phone’s camera.

Microsoft sent the Jet Propulsion Laboratory 30 HoloLens headsets for team members on the rover mission.

Rice said the goal is for the 400 other scientists to eventually use the HoloLens system for the rover mission.

“We want to all see the same thing and be able to talk to each other about what we see,” Rice said. “The intent eventually is for all 400 of us to communicate through the HoloLens.”

Right now, Rice conference-calls her team on a speakerphone while using the HoloLens to describe to them what she is seeing. She can talk to them and see what other people are pointing at in the augmented reality world as well as hear them on the phone line. 

The HoloLens also allows the team members to create markers that show up as virtual blue flags on the Martian landscape that can be seen by other team members, so they can show each other different rocks, features or places on the surface.

Rice first tested out the HoloLens in spring of 2014 while she was still working at the JPL.

“Microsoft flew me to Redmond to test an early version of the HoloLens,” Rice said. “It was closer to the size of a computer, not wireless, and they controlled the augmented reality through a desktop computer.”

Rice said she is impressed that Microsoft redesigned the HoloLens to be small and work entirely through wifi; all of its software and data is streamed from the cloud.

So while most people wake up in the morning and sip on a cup of hot coffee, Rice is scouring the Mars landscape, talking with the rover team and plotting where Curiosity will explore next - all through the technology of augmented reality.