Alexa VR Controller
Say "take me to the gym" to an Amazon Echo and be teleported there inside a virtual town. A prototype exploring voice as a hands-free way to control a VR environment.
- Role
- Solo project: Alexa skill, deployment and Unity scene
- Hardware
- Amazon Echo and an Oculus headset
- Built with
- Alexa Skills Kit, AWS Lambda, Node.js, Socket.IO, Unity 5.6
- Study
- MSc Creative Technology, Middlesex University
The question
Voice is a natural and intuitive way to control an environment while your hands stay free. I wanted to try the Amazon Echo, "Alexa", as that voice controller for a Unity VR scene.
How it works
You say "Alexa, tell VR controller to take me to the gym". Alexa skills run in the cloud, so the command has to travel to the headset app through a server.
-
Alexa skill
A skill with one intent, TransportIntent, and a location slot. It runs on AWS Lambda in Node.js and understands phrases like "take me to the gym" or "move to the coffee shop".
-
Relay server
The skill logs into a small Socket.IO chat-style server on Azure as "alexa" and broadcasts the place name.
-
Unity app
The VR app logs into the same server as "VR", listens for messages from "alexa" and teleports the player to the named place.
The town is built from a third-party low-poly asset pack, not my own art.
Before the Socket.IO version I built a simpler prototype: a tiny Heroku server that the Unity app polled every second, so a spoken colour and shape, like a red cube, spawned in the scene.
What's mine and what's adapted
An integration project
I built this by combining existing pieces. The skill follows Amazon's Alexa Skills Kit sample, the relay is based on the Socket.IO chat example, and the overall design follows a community Alexa VR controller project on hackster.io that I studied and adapted. The town, the VR support and the Socket.IO plugin for Unity are third-party assets.
What I did: set up and deployed the Alexa skill and its Lambda function, ran the relay on Azure, connected the Unity scene and got all of it working end to end on a headset. The Heroku polling prototype is my own, as are the demo and the write-up.
What I tested
- I tested the skill in Alexa's simulator and ran the demo on an Oculus headset.
- I didn't run user tests, so I can't say how natural people find it, or how fast and reliable the commands are.
- The hardest part was deployment: the WebSocket package wouldn't upload to Lambda at first because the console couldn't read the files in my zip.
Limits
- A handful of hard-coded places, not a general vocabulary.
- One-way only: Alexa moves you, but the VR world can't answer.
- You have to say the skill's invocation phrase, so it isn't quite "natural".
- It needs a cloud server between the Echo and the headset, and the Echo isn't in the headset with you.
What I'd do next
- Test it with people: how many tries commands need, and whether talking to a speaker in the room feels right while wearing a headset.
- Make the places data instead of code, so new locations don't need a new build.
- Close the loop so the VR world can speak back.