Prototype using for loop

Today, Jean K. and I finished implementing the vocabulary showed in an earlier post. We also made changes to the vocabulary by adding and eliminating some words, commands, sentences in Simon's Grammar module. During the implementation we create the commands necessary to create a program that would print the numbers from 0 to 9. Here is a video that we made showing the creation of this program.

Note: This video does not has sound because there was some technical difficulties when it was recorded.

Categorized vocabulary

In the last few weeks, Jean Karlo and I were working with the vocabulary that was developed last semester. We made a few tables to categorized the vocabulary. The categories were also established last semester in the proposal of Andrea and Natalia.

Here is the first table (draft) that I made 

action	object	operator	text	command	launch	key
printing	variable	adding	literal	symbol	entering	else
grouping		frontslash		operating		for
getting		modulus		initiation
defining		greater than		condition
inserting		and		modification
		negating

Later, Jean Karlo made a few changes expanding the vocabulary and adding new categories

Action	Object	Operator	String	Command
inserting	variable	adding	literal
entering	loop	frontslash		operating
defining		modulus		initiation
		greater than		condition
		and		modification
		negating
		less than
		equal
		grouping ()

Launch	Key	Text	Type	Number
opening	else	accum	character (char)	1 to 9
closing	for	total	integer (int)
starting	printing	a - z	float
ending	while		double
	conditional (if)		string
	function
	class

Our First Prototype of the Semester.

Last week in our weekly meeting Gustavo Gratacos, an undergraduate student who worked in this project in previous years, came by to help us with the problem that we were having with Simon. The problem was that Simon recognized the commands that we created but didn't executed them. After showing Gustavo the problems that we were having and walking him through the steps that we were following, he managed to solve the problem. When we were adding the plugins in the ‘Commands’ Module we needed to click on the plugin that was added and delete what was in the Trigger space, which in this case was ‘Computer’ as shown in the picture below. After doing this, all of the commands that we created worked.

In this meeting we also created a Simple "Hello World" program in a text editor called ‘Kate’, which we made a video of .

Implementation of the Vocabulary

   During the last few weeks, Valerie and I have been working on implementing the vocabulary and grammar that was created last semester into Simon. I have a windows computer and it hasn’t been responding to the voice commands from the scenario that I downloaded which was “EN/H4W] Mouse”. We created a scenario of my own and my goal was to open an executable file, which in this case was a Python command line. We were able to create a command using the "Program" type and activate it manually by clicking on the Trigger button that is in the Grammar module, but we were not able to activate it by voice. Valerie and I tried to do the same thing in her Mac computer but had the same results as before.

   Another thing that happened these past few weeks is that Simon has been giving me the following error every time I activate it:

The recognition reported the following error:

Failed to setup recognition: Julius did not initialize correctly

After asking the KDE community what might be causing this error and how can I solve it through an email, they were able to point out that I was mixing a speech model ([EN/VF/JHTK] Voxforge) and a scenario (EN/H4W] Mouse) that had different phonemes. According to Peter, the person who respond my question, “the EN/H4W mouse scenario uses the H4W phoneme set, while the EN/VF base model uses the VF phoneme set”. Now I need to correct this mistake and use a scenario that matches the phoneme of the speech model. Valerie and I need to figure out how to activate the commands that we create and I need to do the necessary adjustments to correct the mistake I made by selecting a scenario with a matching  EN/VF phoneme. 

Hello Kavita!

Hello! I am the newest member of this research.  I’m so excited to join this project and I’m willing to keep up with all of the work that all of you have done already. It is amazing what technology can do through developing different tools to help and improve assistive technologies. Still there are countless problems that we can assist and make it possible for people with disabilities. It is important to mention that having a disability does not mean that you can’t achieve a task.

            Last week, Professor Ordóñez presented me Kavita through facebook. It was a pleasure to know about the inspiration of the project and it will be an honor to have the opportunity of meeting her in person some day. She will now be like a role model to me because of all of the achievements that she done in her life. I will be happy to work in this research with all of you and Kavita.

Development of a Vocabulary for Speech Recognition Software

  Last semester Andrea, Natalia, Xiomara and I developed a vocabulary to use while programming  using speech recognition software such as Simon. The vocabulary uses gerunds to identify the key words that are commonly used in many programming languages.  We then created a survey that consisted of 10 statements with three options on how the person would verbalize it. The statements used basic programing structures such as loops, conditional, creating variables, among others to determine the best way to verbalize them.  

   A total of 33 students from the University of Puerto Rico, Rio Piedras Campus and Polytechnic University of Puerto Rico responded to the survey.  Based on the results of the survey, we developed a final version of the vocabulary that we expect would be accepted by the programming community.

Our vocabulary in the works

Xiomara, Andrea, Natalia and I have been working on developing a vocabulary that can be used to program by voice. Our goal is that this vocabulary can be used in any programming language such as Python and C++. First Xiomara selected a piece of code from the internet that was written in Python. We used this piece of code to write down how we would say it if we were to speak the code out loud. Having an idea of how we would command by voice, we each wrote down a list of words and phrases that we thought were essential to program in any programming language. For example we each wrote a phrase that would be used to create a Function. Now that we each created our own list, we are in the process of selecting the best words from our lists to create a vocabulary that will be used to program by voice.