Ease of Access: Eye Tracking Tools in the Windows 10 Fall Creators Update

Last week, when Microsoft released Windows 10 Build 16257 to their global team of Insiders on the Fast Ring, they unveiled a new accessibility feature that will be an integral part of the upcoming Fall Creators Update which is expected in the September.

Eye Tracking, a technology that has been developed out of a Microsoft Hackathon two years ago, is being added to Windows 10 and it will allow users with compatible hardware to fully access their computers with just their eyes.

Right now this feature is in testing and only works with one piece of eye tracking hardware, the Tobii 4C Eye Tracker, which I had the chance to unbox and try out late last week.

As I shared then, it is amazing technology.

While any user will need to set aside some time to gain proficiency with the interface, I was able to use it without any delay after setting up the hardware on two different laptops. The Tobii 4C comes with a USB extension cable if you are setting this up on a desktop computer.

I was going to bring you a video introduction to the Eye Tracking capabilities, but in the process of trying to figure out why the Windows 10 Eye Tracking toolbar, keyboards, and mouse controls would not show up on my desktop capture software, I learned a valuable lesson about accessibility features in Windows 10.

As an able-bodied person and someone who writes about technology, I was endeavoring to show you how the Windows 10 Eye Tracking worked with the Tobii 4C hardware installed and setup. However, what I learned is these accessibility tools are no different to an able bodied person using a mouse and keyboard: They provide us the ability to use our computer but they also stay out of the way and just leave small indications they are active such as a mouse cursor or a blinking insert point for typing.

It is no different for a feature like Eye Tracking in Windows 10. These are tools that enable users who are unable to control a mouse or keyboard in the conventional method to interact with their PCs.

They should also stay out of the way when those users do things like capture desktop video of a process, etc. Thus, the Eye Tracking tools in Windows 10 do not show up in any attempt to capture the desktop for demos like I have done in the past. They are present and available to be used but they also get out of the way when these types of captures are made because they are ultimately not intended to be the focal point.

Now I did learn that you can use a hardware based capture tool to record the entire desktop interface, including accessibility tools, by grabbing the full video feed from the device and recording that video. Unfortunately, I do not have that hardware at my disposal so this walk through of the Eye Tracking interface and tools for Windows 10 will be through a series of screenshots.

Once you have compatible hardware for your system, just follow the steps provided in the Windows 10 Build 16257 Release Notes to setup the device for first use.

Next step is to activate the Eye Tracking feature which will be located in Windows Settings>Ease of Access>Other options. Just scroll to the bottom and toggle the capability on or off.

Once activated, the Eye Tracking Toolbar will be placed at the top of the screen:

The four icons on the toolbar represent (from left to right):

-- Move Eye Tracking Toolbar to bottom of screen (or move it back to the top if located at the bottom)

-- On-Screen Keyboard

-- Text to Speech (includes a full on-screen keyboard as well)

-- Activate Mouse Cursor

On-Screen Keyboard

The on-screen keyboard looks like most any other on-screen keyboard although the keys are slightly larger to make them easier to target with your gaze. This keyboard also supports shape writing when it is activated. You simply gaze at the first letter of the word you want to type and after a couple of seconds it is active and then you glance at the other letters of the word in sequence. On the final letter pause once again and the word will be provided along with suggestions for similar words in case that once is incorrect. It works just like shape writing does on Windows 10 Mobile and now tablets on Fast Ring builds of Windows 10.

If you go to the lower right corner of this keyboard and select the Fn key you will get additional keyboard options:

From this on-screen keyboard if you select the Settings icon in the lower right side, the gear icon, you will open up the Settings for Eye Tracking on Windows 10:

These are all self-explanatory but can make the Eye Tracking controls much easier to use based on your experience. I found that slowing down the dwell time helped me a lot.

Text to Speech

The third icon on the Eye Tracking Toolbar is the Text to Speech screen.

Here you can compose sentences and then have them spoken aloud. The bottom on-screen keyboard is the same as the one that pops up under the second icon on the Eye Tracking Toolbar. Across the top of this screen you will see five words. These can be customized and just by gazing on them and dwelling, they will be spoken out loud on the device. This is where a user would hold a spoken conversation with someone else in the same room/area.

Mouse Cursor

The mouse cursor is selected with the fourth icon on the Eye Tracking toolbar.

Using your gaze, move the cursor around the screen until it is near the area you want to interact further with and then just dwell at that spot with your eyes.

This is the fine mouse control interface. It includes four direction arrows for moving the centered cross hairs over the interface element you want to click on, mouse button controls for double click on top left, single click bottom left, right click (context menu) lower right, and an exit button in the top right.

Clicking any mouse control by gazing on it will close this interface automatically and take the desired action under the cross hairs such as opening a link, closing a window, or inserting your typing cursor in a text field.

As I have mentioned before, this is very impressive technology and can provide a tremendous amount of freedom to disabled users who may not be able to control a computer in any other manner.

By the way, I know that a video was not possible with my setup but Zac Bowden from Windows Central did a great video using the Elgato Capture hardware to record the desktop video of the Eye Tracking interface using a second PC. Check out his video demo to see all of this in action.

------------------------------

But, wait...there's probably more so be sure to follow me on Twitter and Google+.