Google reveals Project Gameface, mouse control using facial expressions

Marijn / ActiveB1t3 minute read

During Google I/O, the tech giant’s developer conference, they revealed Project Gameface. Project Gameface is designed to allow players to control the computer’s cursor (and keyboard) with, you guessed it, nothing but their face.

Watch Gaming with muscular dystrophy | Project GameFace featuring Lance Carr | Google on YouTube

In a video showing the technology in action, we hear from Lance Carr. Lance is a disabled gamer who lost their adaptive equipment, an expensive head-tracking mouse, in a house fire. This prompted Google to work together with Lance to design Project Gameface.

Project Gameface is a piece of software that recognizes head movements and facial gestures with a normal webcam. As such it does not rely on often expensive head tracking equipment. Google MediaPipe, an open source set of tools and libraries to easily apply Artificial Intelligence (AI) to projects powers this project.

Actions and gestures

With the current release build of Project Gameface only a few actions and gestures are available, but the source code implies a lot more is possible. You can set the following actions under mouse controls. Mouse left, right, and middle click. Pause / unpause mouse control, reset cursor to center, and switch focus between monitors.

Aside from mouse control, it is also possible to assign keyboard actions to gestures. Ironically, the Escape button can’t be assigned, as pressing it cancels the key selection. The gestures available now are: Open mouth, move mouth left / right, raise left / right eyebrow, lower right eyebrow. The source code does define over 50 possible gestures so I expect this list to grow.

Testing it out with a facial difference

Project Gameface is open source and available for download, so I decided to test out the current release. As a person with a facial difference, my experience with anything facial recognition is spotty at best. Especially when it concerns tracking of movement. Project Gameface is no different, but it is possible to adjust the detection sensitivity of gestures.

The mouse binding menu of Project Gameface showing several bindings. Each with an indicator of the detected gesture and the sensitivity slider ranging from subtle to exaggerated.

This sensitivity adjustment turned out to be a necessary feature. It shows the detected gesture size similarly to how you often see microphone volume displayed. It fills a bar depending on how much the gesture is detected. Turning from yellow to green when over the trigger threshold.

Setting the left click to moving my mouth to the left needed the setting all the way to “subtle”. Even then I needed a quite exaggerated movement to trigger it. In contrast, it constantly detected mouth movement to the right, which needed resolving with setting that to more “exaggerated”. And that still occasionally detected clicks without movement on my part. This tells me the models still need training with much more diverse faces, hardly a surprise for those who encountered these issues before.

For now, Project Gameface is definitely something that can already benefit many people. Having it work with any webcam allows for an affordable solution for accessible control, provided the user has good control over their head and facial movements.

Enjoy our work? Please consider supporting us!

Donating through DAGERSystem / AbilityPoints with PayPal may be tax deductible

ActiveB1tWebsite operationsHe/They

CIPT's resident one-person IT crew responsible for the looks, functionality, and accessibility of the site. Inclusion and accessibility troublemaker and creator of the Alt Or Not browser plugin for Twitter. Child of the 80's without an intention of growing up.

See all articles by Marijn

Follow CIPT

Latest from CIPT

(Opens in new tab) starting with