which aims to help children with autism
to converse to engage better in the
conversation and to use better their
language in a social way so we I
conducted research with my colleague
professor sure from computer science at
Monmouth University we thought that we
would try if children with autism must
speak better with a robot than with a
peer why do we do this research because
we know that children with autism try to
tend to isolate themselves not answer
questions if they are asked don’t
respond to their name and take as much
and so we also know that they have a
fascination for objects so whereas with
peers and with children like them they
have a different time in getting engaged
and playing with them and talk to them
they can be captivated by objects
especially technological tools and
computers for hours so I thought that
maybe proposing the child a conversation
with the robot speaking in a human voice
that would help them start the
conversation and start talking and
answering the questions and then they
could actually generalize this skill
once obtained to other contexts like the
school or the family so basically what
are the problems of the children with
autism in language and using language in
a in a social way they are they have
problems in also nonverbal communication
so looking at people in the eyes smiling
while they’re talking taking turns so
waiting for death time to
answer and answering with an appropriate
topic so we call this skills pragmatic
language skills so the first goal of the
study is to actually rehabilitate
pragmatic language skills in children
with autism the second week ability in
children without this means engaging in
emotional exchanges so when they talk to
people they seem to be not they seem not
to engage in an emotional exchange and
they don’t resonate with the emotions of
the other person they maybe do not
understand the feelings of the other
person or they are not able to analyze
their own feelings or to answer to
express their emotion through with
speech so this is the other goal that I
wanted to achieve to let them actually
express their emotions better with the
robot so what did we do we recorded the
speech of children who are native
speakers of Italian because I conducted
this part of the research in Italy at
the Klean at speech pathology clinic
directed by a friend of mine dr. Madani
Renzo the clinic is the Institute of the
ortho phonology and Rome so they gave us
kindly access to their patients and we
could record nine children with autism
and and six normal children children in
normal development so what what did I
use in this experiment to help the
children express themselves some
sentences that are designed to elicit
emotions in this in this in the listener
in the person who has to respond
sentences like do you want to be my
friend or how do you feel when you see
your mom do you like football
do you are you happy when you go out
with your dad
so children are actually asked to answer
and hopefully they will express how they
feel and start being gaining contact
with these emotions so what children
children between the age of five and
fifteen and children who don’t have
other cognitive impairments but just
autism these are children who are more
verbal so more functional
we know that autism is a spectrum so
there is a variety of symptoms children
are very different but we chose the ones
who can engage in a conversation so what
do we analyze we analyze how much they
engage in the conversation and whether
the intonation the porosity of the
conversation is actually appropriate in
their speech why because through the
intonation the melody of speech we
express actually our emotions so it’s
not enough that we say the appropriate
words but also the intonation is
important so how did we run this
experiment we chose a robot would look
like a toy so it’s a mind mind storm
Lego robot very easy to build it was
built by the other students in the
computer science department in at
Monmouth University and in this case it
has a humanoid shape but we tried
different shapes and what does it do so
it speaks with the voice of a child what
I did is I recorded some sentences
spoken by a real child asking these
questions and we uploaded the sentences
in the memory of the robot so basically
there is not a synthesizer it’s very
simple there is not a speech recognizer
it’s just
playing the robot is just plain
sentences that are uploaded in its
memory at a few seconds interval to lead
let the children to lead to the children
some time to answer and so what it does
the robot walks towards the child has
some vision sensors identifies the child
and stops at approximately 15
centimeters and then starts talking so
how did we organize the experiment in a
first phase the child who is tested
speaks to the robot so the robot says
asks question you want to be my friend
do you like candy and the child
interacts so we will see how then we
remove the robot and we actually take in
the room
a real boy who is asking the same
questions what do we want – what do we
want to test first whether there are
patterns in the conversational
interaction with the robot this is
because the autism is a puzzle and we
don’t know how to treat it appropriately
because every child has different
symptoms so it would be very useful if
we could find some patterns especially
in verbal exchanges so we could target
our therapies to those patterns so this
is the first time I was looking for
second thing was – I was testing whether
the children with autism would respond
differently from children in normal
development – the interaction with the
robot and the boy and third I wanted to
see also if the children of course if
the children with autism speak better
more easily to the robot than to the
child so in that case we can use the
robot as a therapy so what did we find
actually we found that
I found some patterns of interaction so
this is a good result so as I said this
is a preliminary study on children who
speak Italian nine children so I’ll tell
you the patterns so the first the first
pattern is used by two children they’re
all boys so I will use masculine
currents as you know there is a ratio
four to one boys to girls for children
with autism so it’s easier to find boys
participants so first pattern is a goal
area so echolalia is a type of response
that she’s done with autism have which
means they imitate the person what the
speaker said so the robot asks do you
want to be my friend and they repeat do
you want to be my friend so they do it
however in two ways in terms of porosity
of intonation the first is just
repeating the intonation that this the
robot said and the second is to speak in
an iphone ‘voice which is like this so
like a whisper how is it produced it
means that you are not vibrating your
vocal folds so since the melody of
speech is generated by moving the vocal
folds it means that they eliminate cross
V so this children in some way do not
really engage in a conversation because
they repeat what the robot just said and
they do it without a prosody so you see
that actually it’s the for my intention
to communicate in some way that is
damaged in this case so they don’t have
the concept of exchanging speech in a
conversation but at least this children
express something verbally with the
robot because with the real child they
would not talk to him okay at all so
it’s something the second pattern was
actually
to use some stereotyped sentences and
intonation so they when the child a
robot asks do you want to be my friend
they would repeat hello hello
as if they were on the phone and they
were trying to understand here the other
person so they would they would keep
saying hello hello hello hello all was
the same many times two of them and I
have to say I also found this in
different languages I tested this do the
same they say hello hello in Italian
pronto pronto
and they repeat the same intonation over
and over again again this is not an
engagement in conversation but it’s some
kind of acknowledgment that somebody has
talked to them and they are trying to
exchange some speech again this children
with the real boy do not do that they
they don’t engage at all so one of these
children also after this the first
interaction is stepped away removed
himself from the robot and went to a
doll house that was in the room and
starting started playing a game
narrating the game to himself and
talking to himself action this is
another pattern that we find in children
with others they talk to themselves so
he was telling himself moved the chair
there is somebody coming in you have to
move you have to move you know open the
door
so he was talking to himself always with
a high very high pitch and stereotyped
intonation so this is again is not a way
to interact in a conversation but it’s
speaking at least so the other way the
other pattern that I found was the other
children were more verbal so other two
children answer the questions but with
mono syllables very short or two words
sentences too short to find an
intonation and it’s not really
conversation but at least they give an
answer so
acknowledged they have an interlocutor
somebody to talk to so that’s a step
ahead let’s say and in this case they
these children answer differently to the
questions with the robot and with the
boy
so they answer all the questions with
the robot but only some of them with the
boy so it’s still an improvement the the
final pattern is for the rest of the
boys they were more fluent in speech so
they would answer with more words and
with the an intonation that seemed okay
and however in the end of the sentence
the intonation became a lullaby like a
little singsong so they’ve changed the
rhythm the rhythm is also connected to
the melody of speech so all of them did
that so in some way there is a control
of porosity which is a different aspect
of speech than words that is affected by
autism and needs to be rehabilitated
this is also something that wasn’t
studied before so it’s a new finding and
prosody we think it’s processed in the
right half of the brain whereas speech
in the left hemisphere so we have some
new information from this experiment and
so I want to just say in the end that
the children with autism responded
differently to the robot and to the boy
children know in normal development they
answered exactly the same sentences to
the robot and the boy the two tasks were
one after the other so they remember the
questions of course but I noticed that
they are used exactly the same answers
so it I think as a tentative
interpretation is that we can say that
when a robot speaks with the voice of a
boy the children in normal development
think it’s a boy
so they would answer as if it was a
person exactly like they would answer to
the real boy but children with others
who think that actually is a different
entity and so they answer differently
and in some cases the less verbal so the
functional children actually had some
kind of start in the conversation so
anything is good you know as a
improvement in therapy so we will start
from there to try to improve the
prototype and try to make it better so
it can elicit more interactions and
hopefully there is some improvement in
the future for these skills in the
season [Applause]