My First Week with the Kavita Project

First of all, I thank my predecessors and contemporaries in the project for all their work to date; and of course, Dr. Ordoñez for inviting me to contribute. Beyond a worthwhile endeavor in developing accessibility tools for programming, working with spoken interface software presents many opportunities for insight into how much more we can integrate computing into daily life.

Building on the progress of the past few years, I, along with Valerie and Andrea, have been configuring Simon to write code on behalf of us, instructed (mostly) by our voice. As we implement the recommendations of the previously administered survey, we're taking note of limitations as well as how to work around them in our effort to create something truly useful.

As well as using the previously set up Simon, I've been trying to get the program running in my computer - technical post to follow. I have run into some hurdles, but as these are worked around I'll learn more about Simon and a little about my computer as well.

This has been my first post, stay tuned for more.

Final Proposed Vocabulary and Grammar

       A detailed survey was developed and administered with the purpose of gathering the feedback from the programming community regarding alternative options for the proposed vocabulary and grammar.  The questions selected for the survey represent an assortment of programming commands that fall under each of the programming control flow structures: Sequential Structures; Decision or Conditional Structures, and Looping or Repetitive Structures.  The participants were students of Computer Science and Computer Engineering  from the University of Puerto Rico and the Polytechnic University of Puerto Rico. 

       After performing a qualitative and quantitative analysis of the results obtained from the survey, which contains 10 questions (each one is a multiple choice featuring two or more possible verbalizations of the code and one write-in answer slot for alternative suggestions), a final version of the grammar and vocabulary was proposed, and is presented in Table 1. It is expected to be embraced by the programming community.

       The proposed voice command is written in bold, arial, and red, the corresponding C++ code, as a consequence of the voice command, in italics, bold, and black (this is the output produced by the voice-recognition software in an editor), and variables and literals presented in italics, and green.  

Table 1: Final Proposed Vocabulary and Grammar      

Programming structure	Voice commands
Variable definition	defining integer variable             int ; total             int total;
Standard output: literals	printing literal             cout << “ ”; hello world            cout << “ hello world”;
Standard output: variables	printing variable             cout << ; total             cout << total;
Mathematical operations	printing             cout << ; grouping             cout << () ; total adding 2             cout << ( total + 2 ) ; moving right, symbol frontslash 4             cout << ( total + 2 ) / 4 ;
Standard input	getting input into variable total cin >> total;
Modulus operations	total modulus 3             total % 3
Relational operations	total symbol greater than seven total > 7
Logical operations	(statementA) operating and             (statement) && negating (statementB)             (statement) && !(statement)
Loops	inserting for loop             for ( ; ; )             {             } inserting initiation (initializeExpression) inserting condition (conditionalExpression inserting modification (modifyExpression)             for ( initializeExpression; conditionalExpression; modificationExpression)             {             } entering loop (statement(s))             for ( initializeExpression; conditionalExpression; modificationExpression)             {                         statement(s);             }
Conditionals	inserting conditional if() { } inserting condition total symbol greater than seven if( total > 7) { } printing literal eight if( total > 7) { cout << “8”; } else printing literal six if( total > 7) { cout << “8”; }else { cout << “6”; }

Prototype using for loop

Today, Jean K. and I finished implementing the vocabulary showed in an earlier post. We also made changes to the vocabulary by adding and eliminating some words, commands, sentences in Simon's Grammar module. During the implementation we create the commands necessary to create a program that would print the numbers from 0 to 9. Here is a video that we made showing the creation of this program.

Note: This video does not has sound because there was some technical difficulties when it was recorded.

Categorized vocabulary

In the last few weeks, Jean Karlo and I were working with the vocabulary that was developed last semester. We made a few tables to categorized the vocabulary. The categories were also established last semester in the proposal of Andrea and Natalia.

Here is the first table (draft) that I made 

action	object	operator	text	command	launch	key
printing	variable	adding	literal	symbol	entering	else
grouping		frontslash		operating		for
getting		modulus		initiation
defining		greater than		condition
inserting		and		modification
		negating

Later, Jean Karlo made a few changes expanding the vocabulary and adding new categories

Action	Object	Operator	String	Command
inserting	variable	adding	literal
entering	loop	frontslash		operating
defining		modulus		initiation
		greater than		condition
		and		modification
		negating
		less than
		equal
		grouping ()

Launch	Key	Text	Type	Number
opening	else	accum	character (char)	1 to 9
closing	for	total	integer (int)
starting	printing	a - z	float
ending	while		double
	conditional (if)		string
	function
	class

Our First Prototype of the Semester.

Last week in our weekly meeting Gustavo Gratacos, an undergraduate student who worked in this project in previous years, came by to help us with the problem that we were having with Simon. The problem was that Simon recognized the commands that we created but didn't executed them. After showing Gustavo the problems that we were having and walking him through the steps that we were following, he managed to solve the problem. When we were adding the plugins in the ‘Commands’ Module we needed to click on the plugin that was added and delete what was in the Trigger space, which in this case was ‘Computer’ as shown in the picture below. After doing this, all of the commands that we created worked.

In this meeting we also created a Simple "Hello World" program in a text editor called ‘Kate’, which we made a video of .

Implementation of the Vocabulary

   During the last few weeks, Valerie and I have been working on implementing the vocabulary and grammar that was created last semester into Simon. I have a windows computer and it hasn’t been responding to the voice commands from the scenario that I downloaded which was “EN/H4W] Mouse”. We created a scenario of my own and my goal was to open an executable file, which in this case was a Python command line. We were able to create a command using the "Program" type and activate it manually by clicking on the Trigger button that is in the Grammar module, but we were not able to activate it by voice. Valerie and I tried to do the same thing in her Mac computer but had the same results as before.

   Another thing that happened these past few weeks is that Simon has been giving me the following error every time I activate it:

The recognition reported the following error:

Failed to setup recognition: Julius did not initialize correctly

After asking the KDE community what might be causing this error and how can I solve it through an email, they were able to point out that I was mixing a speech model ([EN/VF/JHTK] Voxforge) and a scenario (EN/H4W] Mouse) that had different phonemes. According to Peter, the person who respond my question, “the EN/H4W mouse scenario uses the H4W phoneme set, while the EN/VF base model uses the VF phoneme set”. Now I need to correct this mistake and use a scenario that matches the phoneme of the speech model. Valerie and I need to figure out how to activate the commands that we create and I need to do the necessary adjustments to correct the mistake I made by selecting a scenario with a matching  EN/VF phoneme. 

Hello Kavita!

Hello! I am the newest member of this research.  I’m so excited to join this project and I’m willing to keep up with all of the work that all of you have done already. It is amazing what technology can do through developing different tools to help and improve assistive technologies. Still there are countless problems that we can assist and make it possible for people with disabilities. It is important to mention that having a disability does not mean that you can’t achieve a task.

            Last week, Professor Ordóñez presented me Kavita through facebook. It was a pleasure to know about the inspiration of the project and it will be an honor to have the opportunity of meeting her in person some day. She will now be like a role model to me because of all of the achievements that she done in her life. I will be happy to work in this research with all of you and Kavita.