Technology can open up the gaming industry to people with physical disabilities like limited movement. Jason Vearing (Winx hands free gaming) is developing software that allows people to play video games with facial landmarks tracking - one of which has been developed from the GameMaker tutorial Fire Jump.

Why did you start developing for gamers with disabilities?

It's one of those things that I fell into. One of my heroes is Jason Becker, he was an amazing guitar player in the 80s and at the age of 19 he had bagged a gig playing for David Lee Roth.

Unfortunately, Becker was diagnosed with ALS, his muscles got weaker and weaker until he had extremely limited movement. It’s the same disease that Stephen Hawking had.

His dad designed a system for him to communicate by using his eyes. It works by directing your eyes to a region on a board, then selecting a letter from that region. Jason was able to use it to write lengthy articles for magazines.

I’d developed a guitar training program called ‘Riff Axelerator’ and using this platform, I did a fundraiser for Jason. I got to know the family and began to ask myself if I could make a web-based version of the communication board?

With the Families permission I made a browser based version, and combined it with predictive text and voice cloning so that by moving your eyes you can generate spoken text. Then, I began working with a team, including Dr Jordan Nguyen who does some amazing work in this space, to develop Vocal Eyes.

Following this, I started WinxHands Free Gaming with the intent to make video games accessible for people with disabilities or injuries.

Jason Becker using the region board

How do you design eye-tracking to work with video games?

The webcam measures the distance between two points on your face, and when a distance threshold is breached it triggers an action - in this case pressing a keyboard key.

Let's say a simple game like Chrome Dino Jump that has one main control (press spacebar to jump). You would have the program detect a point on the top and bottom lip and measure the Euclidean distance - lets just say it detects a distance of 50 with the mouth open and around 10 when the mouth is closed.

You would set a trigger point at say 30 and the condition is if mouth_distance >= 30 then press the space key. The webcam runs at say 30 frames a second and nothing should happen while mouth is closed, but when the mouth is open wider than 30, then the space key will press.

Chrome Dino Jump, developed by Google

What challenges have you had to overcome in your design?

You have to stay very still. If you move towards the camera closer than ‘10’, the distance of ‘mouth closed’ could become ‘40’, and therefore trigger the space key.

One trick is to add a horizontal landmark of two points on the side of your mouth and divide the above calculation (which is variable) by a relatively fixed position. This really helps to maintain stability so that if you move backward and forward in front of the webcam that vertical distance does not vary too much.

A lot of the times when someone’s finished a game, they still need their carer to move the mouse over to the restart button and click. But with some games, you can map the left mouse button to a facial movement so that when they're finished they can restart the game independently.

Calibration has been a bit of a problem, it works on my face, at a certain distance from the camera, but when I take it elsewhere it is difficult to replicate the settings, so I have been working on this and now have tools such as trackbars to make it easier to customise to new users.

What games do you use?

I chose to use the Fire Jump (demo video). It’s perfect for the software because you've basically got left and right as your main movements and one more control for firing water. Close your left eye to move left, your right eye to move right, and to shoot the water cannon you can raise your eyebrows. So you just map the keyboard to do three simple movements.

Everything already looks great about it and it’s great fun to play.

An older game that works really well but it is pretty challenging is Dragon's Lair (demo video), and it has awesome animation that still holds up.

Fire Jump, GameMaker Tutorial

Are you making any other games?

Yes. I’ve made a virtual pinball machine that works with facial tracking called ‘Winkball’, and a physical device that you attach to an actual pinball machine.

The device clamps onto the existing buttons on the side, then you control the buttons again, with your facial expressions. Allowing disabled gamers who can’t use their hands to actually play a real pinball machine in an arcade for example.

Then, I’m working on a racing car simulation. For this, we’re building a robotic device that clamps onto the centre of the steering wheel. You close your left or right eye to turn, and the device makes sure the wheel always returns to the centre. Another component that switches the acceleration and brake pedals. The trick with this was to make the accelerator ‘lock in’ when a good speed is achieved so that you can relax your face once you get going.

Winkball, by Winx Hands Free Gaming

Is there any accessibility-related tech coming into the gaming industry that you’re excited about?

There’s a new technology coming where you can control gameplay through your thoughts - think left and your character moves left. It’s called EEG (electroencephalogram).

That's obviously a whole new level and who knows where it goes from there?

Then there’s the tiktoker Waha_Taniwha who uses a quadstick to play Call Of Duty. It’s a mouth operated game controller which is essentially controlled by blowing and sucking. It’s amazing to watch him compete against (and beat!) other players who have no idea that he has paralysis from a spinal injury.

Perrikaryal gaming with EEG, video by Big Great Story

You're looking out for other people that might be able to help you grow your projects. Who are you looking for?

I want to build a kickass team of people who want to make a difference.
Someone on the social media and marketing side of it. Somebody who's really got the skills to record footage, to chop it up, and promote the videos.

A good python programmer to make the program as responsive as possible and to help improve the responsiveness on the other projects.

A robotics person, somebody who knows how to use stepper motors / robotics for the steering simulator so we can build a full racing car sim designed for somebody who uses a wheelchair and other projects.

Most of all, l want to work with game developers to see what cool games we can come up with, with these controls and user feedback as the linchpin.

How would we have anyone contact you that was interested?

At this stage, on my YouTube channel Winx910, or email: [email protected]

Written by Samuel Wain

Sam Wain is a digital marketer at GameMaker. He runs the YouTube channel, writes blogs, and leads influencer marketing. In his own time he's often found crafting stories, composing music, and implementing his sound effects into a new game or project.