Proposal: Part 2

The Tapia Conference 2014 was accepting proposal submissions and in our case we wanted to apply for the posters. We had already discussed most of our ideas for the project but this was the opportunity to present it formally and get feedback on it. Our goal is to create an assistive technology that comprises of two open source projects: KDE's Simon voice recognition Software and GNOME's Caribou on-screen keyboard. By integrating these and expanding their features we hope to create a tool for disabled individuals that, while programming, might enhance their performance and give them access to other features that might have previously been unfeasible under their circumstances.

My part, Part 2, consisted of explaining how the programming interaction would be implemented and why would it be beneficial. This led us to discuss more about the technical limitations of the typical user interface and input mechanisms, and how we could enable a compact interface with advanced features. The integration with programming would come directly from a compatible IDE that would port through ATSPI2 it's procedures and useful data to Caribou and sub-sequentially to Simon's applications. Once the data is ported it would be managed for efficient display on the interface and voice command enabling. To achieve this, the caribou interface must be extensively modified to create an area to process the data depending on it's usefulness at any point.

As part of the proposal, a project draft was also required to be included. This draft is to be the Poster's in the final work. The following is the text submitted as requested:

The second phase of this assistive technology research project consists of integrating an on-screen keyboard (OSK) with an Integrated Development Environment (IDE) that implements a modified interface to expand functionality and facilitate interaction for users, specifically physically disabled individuals. Common assistive tools are on-screen keyboards and speech recognition software, such as Caribou, Simon and Dragon Naturally Speaking. Extensive research has been done on the efficacy of these tools, proving that they work well for controlled environments and specific tasks, but may not be useful for conditions such as motor disorders that inhibit the user’s mobility or pronunciation. Integrating their functionality to manage complex dynamic tasks is an aspect that this project explores.

Programming is an arduous task for individuals with physical disabilities that rely on independent tools to interact with their digital environment. Providing a multimodal Integrated Development Environment where programming and its complex syntax and dynamic structure is key to lessening this burden. By modifying the static structure of a on-screen keyboard to dynamically adjust to the criteria of the environment, our project provides flexibility on what type of information is displayed to the users at a given moment. While facilitating and reducing keystrokes, the application’s speech recognition mode can perform any task available to it through the interface thereby reducing stress on extremities by relying on voice commands and taking advantage of convenient features as word completion, word prediction, embedded application commands (open, close, save), or grammar specific commands (comment, collapse, expand) that are available through both methods.