[Feature] Build a set of user controls optimized for gaze input

[harishsk]

Motivation

The Microsoft.Toolkit.Uwp.Input.GazeInteraction library in the toolkit, makes handling gaze input in a UWP app incredibly easy. By adding only a few lines of XAML code it makes it possible to enable buttons and other controls to respond to gaze input. However, the default controls in UWP are optimized for mouse, pen and touch. Gaze has less accuracy, is more noisy as a signal. Therefore it is necessary to design a set of reusable controls that make it easy to use gaze input.

This feature request proposes a new library along side the GazeInteraction library under the namespace of Microsoft.Toolkit.Uwp.Input.GazeControls.

Initial controls

The sections below propose a set of initial controls that should be a part of this library.

Gaze keyboard

Eye gaze as input is used to a large extent by people with motor disabilities. One of the most common tasks they have to perform is text input. Because there is a large variability in the ability of people to use gaze input effectively, it is hard to decide on a single keyboard layout that works for all. People who are less able to use gaze reliably need larger buttons. Larger buttons take up more screen area and leave less space for the application. Larger buttons also make it difficult to fit the entire set of needed keys in a single screen (especially in languages with a larger alphabet set). This control would enable a soft keyboard that allows defining and loading custom keyboard layouts. The keyboard layout is specified in XAML with a few specific attached properties which govern the behavior of the layout.

Gaze file picker dialogs

The next most common task is to save and open files. Our experience has demonstrated that the number of elements in the Windows file dialogs and their size makes it extremely difficult to use with eye gaze reliably. This control would support a subset of the file picker dialogs and provide easy access to the user's documents folder.

Future controls

In the future, other gaze optimized controls can be added to this library. The next most important ones are combo boxes, lists and radio buttons.

Solution

I have work in progress in my fork of the toolkit in the gazecontrols branch. There is still some work to do in terms of fit and finish and interaction design review and polish. But I have a functional GazeKeyboard control and a GazeFilePicker control.
In the sample code I have written so far, having the first two controls enables a user to write a full fledged AAC (Augmentative and Assistive Communication) application that gives voice to those with speech and motor disabilities in less than 100 lines of code

Having these two controls as part of the toolkit will increase the value of the gaze interaction library in the toolkit and also accelerate the development of gaze optimized applications which would immediately benefit people with disabilities.

Request

Please review the proposal above and share your feedback.

[ghost]

Hello, 'harishsk! Thanks for submitting a new feature request. I've automatically added a vote 👍 reaction to help get things started. Other community members can vote to help us prioritize this feature in the future!

[Nirmal4G]

Do we put Gaze controls in the Input namespace or the UI.Controls namespace?

Seems the PR has it in the Input namespace.

[harishsk]

Based on discussions with @michael-hawker this work currently resides here:
https://github.com/CommunityToolkit/Labs-GazeControls