Voice interface · Virtual reality · 2017

Alexa VR Controller

Say "take me to the gym" to an Amazon Echo and be teleported there inside a virtual town. A prototype exploring voice as a hands-free way to control a VR environment.

Role
Solo project: Alexa skill, deployment and Unity scene
Hardware
Amazon Echo and an Oculus headset
Built with
Alexa Skills Kit, AWS Lambda, Node.js, Socket.IO, Unity 5.6
Study
MSc Creative Technology, Middlesex University
The view inside the headset, shown for one eye, as spoken commands move the player between places. No audio.

The question

Voice is a natural and intuitive way to control an environment while your hands stay free. I wanted to try the Amazon Echo, "Alexa", as that voice controller for a Unity VR scene.

How it works

You say "Alexa, tell VR controller to take me to the gym". Alexa skills run in the cloud, so the command has to travel to the headset app through a server.

  1. Alexa skill

    A skill with one intent, TransportIntent, and a location slot. It runs on AWS Lambda in Node.js and understands phrases like "take me to the gym" or "move to the coffee shop".

  2. Relay server

    The skill logs into a small Socket.IO chat-style server on Azure as "alexa" and broadcasts the place name.

  3. Unity app

    The VR app logs into the same server as "VR", listens for messages from "alexa" and teleports the player to the named place.

The cinema in the virtual low-poly town, seen from the player's position.
Cinema
The gym, a red and cream building with a grey bus parked in front.
Gym
The mall and a hardware store along a street with a police car.
Mall and hardware store

The town is built from a third-party low-poly asset pack, not my own art.

Before the Socket.IO version I built a simpler prototype: a tiny Heroku server that the Unity app polled every second, so a spoken colour and shape, like a red cube, spawned in the scene.

What's mine and what's adapted

An integration project

I built this by combining existing pieces. The skill follows Amazon's Alexa Skills Kit sample, the relay is based on the Socket.IO chat example, and the overall design follows a community Alexa VR controller project on hackster.io that I studied and adapted. The town, the VR support and the Socket.IO plugin for Unity are third-party assets.

What I did: set up and deployed the Alexa skill and its Lambda function, ran the relay on Azure, connected the Unity scene and got all of it working end to end on a headset. The Heroku polling prototype is my own, as are the demo and the write-up.

What I tested

  • I tested the skill in Alexa's simulator and ran the demo on an Oculus headset.
  • I didn't run user tests, so I can't say how natural people find it, or how fast and reliable the commands are.
  • The hardest part was deployment: the WebSocket package wouldn't upload to Lambda at first because the console couldn't read the files in my zip.

Limits

  • A handful of hard-coded places, not a general vocabulary.
  • One-way only: Alexa moves you, but the VR world can't answer.
  • You have to say the skill's invocation phrase, so it isn't quite "natural".
  • It needs a cloud server between the Echo and the headset, and the Echo isn't in the headset with you.

What I'd do next

  • Test it with people: how many tries commands need, and whether talking to a speaker in the room feels right while wearing a headset.
  • Make the places data instead of code, so new locations don't need a new build.
  • Close the loop so the VR world can speak back.
Also: DiscovAR, a HoloLens experience →