Our approach tries to involve general scenarios for the end-user, who is not an expert programming person. Therefore the task is not fixed a priori, but chosen by the end-user, her/himself. The challenge we want to face up to is to make all robot skills accessible to the end-user so he/she could have fun using and combining them in
new skills increasing robot capabilities in an easier way by natural interaction with the robot.

The robot can guide the user in that construction showing the possibilities in each interaction context and by precise queries about necessary information, maintaining
the coherence of the interaction process. Therefore, the implemented system doesn’t just allow the natural programming of a social robot, but also helps to improve the naturalness in Human-Robot Interaction.

A Sequence Verbal Edition System (SVE) bridge between external utterances of the user and internal skills of the robot, making these skills accessible by voice. This system allows the user to build new skills (new sequences) from primitive skills, and reuse the created
skills for subsequent sequences.

Interactive Sequence Edition

Semantics and Verbal Commands for the Edition of
a Sequence

A sequence is essentially a graphical structure. The main goal of this work implies to make the sequence verbally accessible. Therefore we have studied how to design such a system that translates form verbal utterances to graphical information.

Referential Commands

1. Reference to a Node, that is in some part of the already created sequence. E. g. “After raising the left arm…”

2. Reference to a Branch, that allows to distinguish between different branches inside the sequence. E.g. “In the first branch…”.

3. Reference to a Structure of Multiples Branches This type of reference is used to conclude a part of the sequence made by some branches in parallel, that could be a conditional selection or a parallel execution. E. g. “You finish all the branches, and then, wait until I touch your left shoulder”.

If the user does not specify explicitly a focus of attention, the Sequence Generation System takes one by default. For instance, if the user adds an action, it is
supposed that the next structural command is going to be in reference to that action, then the SGS establish by default the focus of attention in that new added action.

Structural Commands

1. Addition of a node. The node could be a step (action) or a transition (condition). E. g.: The utterance “Raise the left arm” will add an action. The utterance “then…” or “Wait until I touch your head” will add a condition.

2. Removing of a node. All nodes (steps or transitions) in the edition focus will be removed. E. g.: “Remove the last action.”

3. Opening or closing a structure of multiple branches that can be a sequence selection or simultaneous sequences, explained above. E. g.: “You consider four possibilities” will open a selection between 4 branches; “You do three things,
   at once” will open 3 simultaneous branches.

4. Loops. That is, the union of one part of the sequence with a previous one. E. g.: “go back to the beginning”.

The Natural Programming System presented here allows
multiple functionalities that can be easily implemented.
One open question of a programming system is
how to add new functions to the initial set of actions and
conditions without changing the design of the system,
that is, at execution time. In our approach, as semantic
grammar can be changed at run-time, it is possible to
add new terms, e.g. new action names, to the vocabulary
of the robot. A detection capability of the spelling
can also help in these sense. In the other side, as actions
and conditions are implemented as a xml-sequence with
Python functions, the new created function could be expressed
in these terms.

New skills are created from primitive ones. But each
new created skill can be reused for subsequent sequence
editions. However, the innate primitive set of actions/-
conditions plays an important role: this set has to cover
all the possibilities of the robot. Here we have used just
a small subset of the possibilities of Maggie: the low level
movements of the different DOF and touch sensors.
But also Maggie has many other skills that can be
also used in a new sequence. For instance, verbal skills
such as ettsSkill and asrSkill could be used as follows:
“when I say you ”stop“ you stop.”, or “If I touch you in
the side, you begin to laugh.” Many conditions can be
also included in the set of primitives. For instance, “wait
5 seconds”, “when the battery level is below 50%…”, “If
you detect an obstacle in the Laser…”, etc.

In other natural programming systems, the action
primitives are one by one designed from a set of cue
user utterances, therefore the primitives are well
adapted to the user speech. This method has the advantage
that easily covers a great range of the user utterances,
if the application domain is well constrained.
Nevertheless, this design method of the “innate set”
does not assure that the primitives are going to cover
all the action possibilities of the robot. In our approach
it was very important to cover all such possibilities,
and Maggie has many of them. Therefore, the innate
primitive set is closer to the low-level robot action-perception
space.

In the speech or symbolic level, the
semantic-CFG rules perform the necessary transformation
from the user utterance, that is natural, into the
action-perception robot space, that follows a formal
model. It is the DMS that is in charge of guiding the
user and translate his/her natural language to an internal
representation, in terms of these primitives.

Experimental tests with end-users show that they enjoy
a lot playing with Maggie to the game of “teaching
how to do things”. The robot still is not able to adapt
itself to the complexity of user’s communication rhythms, and is the user who has to
learn how and when articulate the utterances. Nevertheless,
we have found that end-users perceive this adaptation
intuitively and easily, and also that they enjoy doing
so. End-users see Maggie as a baby-agent that still is in
development, what produces a kind of sense of superiority.

The system easily strikes end-users in two ways.
In the interaction side, by the naturalness of the interaction.
In the execution side, as the robot performs what
the user has told.