Press "Enter" to skip to content

Building an Intelligent Robot | Sanja Fidler | TEDxUofT


[Music]
[Applause]
all right so I’m Sonia I’m part of the
artificial intelligent group and
actually AI is really blooming these
days right so everywhere I can basically
answer any questions you’ve tried it and
then there is robot I don’t ask any
question they just do basically whatever
you tell them and I know you know there
is algorithms that did world champions
in really complicated games but perhaps
the most life-changing domain is sign
yourself for autonomous driving class
right just imagine a car that can you
know drive you to school or work on its
own you know how cool would that be
right and maybe not so far out actually
the movies from the eighties my actually
true pretty soon all right so all this
is amazed how much of this is actually
powered by this really powerful concept
neural networks which is a machine
learning water and basically you give it
some data and then it goes through
layers and layers of computation until
it basically does whatever you want it
to do
all right and this concept has been
around for a decade but it has been 2012
that actually a group at U of T really
made it a significant breakthrough this
was a group of Geoff Hinton here and
this really started to work it is much
better than anything that has been
around earth before actually why are
just recently posted this top hundred
people in the world have the most
influence on the full digital digital
world and actually Jeff made it on this
list
probably one of the few professors that
made it in this list and what’s more
impressive is actually only a few spots
below Taylors
I’d call that a success all right so
let’s start with the beginning of the
story actually my story so the kid on
the left the cute kid on the left is me
I’m on the right is my first cognitive
agent so my sister actually
so you’re basically looking at the
moment I kind of decided on my life
right it was AI so that’s what I wanted
to do and you kind of see the
expectation right so there I wanted me
to perceive the world needs to talk to
the person definitely do stuff for the
person right and entertain the owner so
you know there was the first day of my
page dear you know how how to actually
get there all right so now we have a
robot in this scene or an apartment and
I wanted to actually you know bring your
coffee or something like that so at
least it just understands to go to the
kitchen it’s actually you know recognize
a coffee pot right not to mention sugar
a meal right so what I started with was
the problem of recognition right which
basically means that you have a photo
and you want to recognize the object in
from the photo but what makes this
problem rich challenging is there’s so
many different classes so many different
types of objects in the world and each
object can be basically in any location
and any size and viewpoint and so many
different poses or number or even
abstraction so how can you make a model
that can actually deal with this kind of
you know complexity so but people
believed in that those days including me
is that there’s some compositionality to
objects right subjects are composed of
parts like a car is composed of wheels
and wheels may be composed of other
elements right so how can we translate
that idea to vision okay so let’s take a
very simple example right you were a kid
you were probably playing with Lego
right see it took really simple pieces
and you compose them a little bit more
complicated
or shapes doing that until in the end
you had maybe a full sculpture a full
object right so how can we translate
that to vision okay so does something
like this it can start a really small
tiny patches like little pieces of
curves and that keep combining them
until in the end the top of this
hierarchy you have something that looks
like an object right and this model
which kind of preceded this big success
of neural networks actually had a
structure a special structure of a
neural net all right but by the time it
was actually one of the deepest model
existed it worked pretty well so I was
super proud of myself but I’m Jeff came
in kind of ruined the dream basically in
the bottom you see the original image
and then you know this color show you
what the model which kind of sits in the
top there is looking at when it’s
recognizing objects so it was doing
pretty well I’m just going to show you a
little video can a robot write a
symphony can a robot turn a canvas into
a beautiful masterpiece can you all
right so this basically brings us to the
other issue right so now we have a robot
in this thing that cannot recognize the
world but the scene is actually not
empty right there’s people in this world
and this people are gonna want to talk
to the robot right so in this case is
grandma saying something and the robot
needs to understand language needs to
understand what the grandma is actually
referring to this this little bottle in
this case right I know it’s very sad the
robot needs to also be able to talk talk
back right it’s so engaging the
conversation suddens to be able to
describe the world and I’m gonna tell
you a little bit about what we did on
this description party so this is
basically what the current systems do so
they take a picture and then they’re
able to write a little sentence that
describes this picture my robo take the
picture and basically generates word by
word describing what’s on the picture
right so this would be one example of
I’m holding a metal ball in the table
all right but if you look at this
picture it’s so much richer than while
it caption is actually saying right
there’s a party maybe the gondola
graduated the gondola ride is worried
that there is no more food writers why
stop so much more alive and this reason
caption over here right so maybe I can
we actually want a robot you know to do
more maybe they want to understand what
these people are actually feeling and
you know why they’re feeling you
particularly why they’re gonna be
thinking and basically what they will do
next or want to do next so maybe we want
our robots to actually write a much more
descriptive text about this picture
right maybe write a little story and
it’s gonna describe it right so if we
actually had a date I forgot the
pictures in this story then we would be
able to train on your lines gonna be
able to produce this kind of data and
this kind of stories right so the
question is we have pictures but how can
we get such really rich descriptions
okay books all right so books are really
rich in kind of bringing the characters
to life for you right you really imagine
how they look like what they’re thinking
and feeling basically any like
throughout the story right I’m gonna
show you as I walk towards the bar
across the concrete parking lot I looked
straight down the road and saw the river
moving a pace with the river was a long
single line of men eyes aimed at their
feet walking steadfastly nowhere I felt
an immediate need to get inside so
basically there’s this three sentences
from from this book are kind of really
telling you a lot right so basically the
author is in the parking lot and it’s
this man walking across the street it
makes me feel very uneasy okay but and
this book on burner maybe some of your
read ready
actually it was converted into a movie
right and then you can see the same
scene basically seen from the eyes of a
director so our idea was can we take
move in a book basically a movie that
was reported after a book and try to
align them so basically take a clip so
each clip from the book from the movie
and find where it actually happens in
the book okay and we can find that text
you can just copy it over to our clip
and then we have the data that we were
looking for all right
but this far is actually super
challenging alright so movie contains
two hundred thousand pictures okay and
the book has almost a say one hundred
and fifty thousand words it’s really
challenging and furthermore there’s so
much diversity in books right so here is
the longest sentence this is from
American Psycho one sentence four
hundred words all right and this is the
shortest one there’s so much variety
right and actually the student or the
student told me it’s not possible so
spend Friday night annotating one movie
in a book and this is what you can come
up with so I just see on the bottom is
basically a movie in minutes from left
to right laughs is the beginning the
right is then and on the top there are
red bubbles are sentences in the book
again from left in the beginning right
is the end and the edges that you see is
basically correspondences that means
that the particular chunk in the movie
is corresponding to the sentence in the
book so you can actually do it it took
me ten hours for a two-hour movie that’s
not too bad and then we actually hired
annotator basically we’re just watching
movies and reading books for one month
and she she even got paid for it so I
think wasn’t too bad
alright so now you can take this data
and what you can do is take the movie
clip it in
short clips and then I wanna do some
matches I’m gonna compute a similarity
between each clip and a particular
sentence in the book and you can use a
neural net to do this for you okay so
hopefully if sound works you’re gonna
see a clip from Harry Potter and what
you’re gonna hear is not the movie
you’re gonna hear the narration of the
part of the book this clip was matched
to Harry please relax or Madame Pomfrey
will have me thrown out
Harry swallowed and looked around him he
realized he must be in the hospital wing
he was lying in a bed with white linen
sheets and next to him was a table piled
high with what looked like half the
candy shop then you can take a movie
that doesn’t have a book and just match
it to you know like all sorts of books
and try to find some Paraguayans when I
kind of best match to this clip right so
with this for Batman if you got matched
to some facts and fiction cube I believe
you will enjoy your time here I am NOT a
harsh master but I am strict when we are
with others I expect you to present
yourself properly what we do here in
your room and in the dungeon is between
you and I it is a testament to the trust
and respect we have for each other and
no one else needs to know about our
arrangement
I’m sure the past few days have been
overwhelming thus far but I have tried
to give you as much information as
possible do you have any questions I
need an umbrella and I shake Spock this
robot actually be able to answer these
questions right so we also this is
something like this so basically this is
where the current methods are doing so
they are taking a picture and they’re
putting a little quiz to the robot so
the robot needs to look at the picture
and the question and then be able to
answer questions to it
if you are able to ask to answer any
questions that someone can ask you know
about the picture this really means that
the robust understanding was there all
right but there is just so many things
you can ask about a single photo right
what is there maybe with the color so
it’s kind of boring right but the
picture right there’s like the world is
not started there’s actually a story
with her in every everything
all right so this is like 20 second flip
actually you know there’s so many more
rich questions that you can ask right so
what’s the relationship between
character is what it is character lie to
the other what’s gonna happen is what’s
there so there’s so much more you can
ask so idea was not just take a single
photo in a quiz take two hundred
thousand photos so full movie and give
them the robotic quiz and now the robot
needs to read this quiz and learn
basically what’s going on so understand
the story
alright so this is basically our robots
so they’re just chilling out and
watching TV watching videos and
basically learning how the world and
people work right basically what how and
why things are happening right don’t
worry it’s actually not working very
well yet it’s a really challenging
problem twenty percent is random the
best we can do is forty percent but I
think is gonna work you know soon enough
all right so what that is every need
right so now they can talk to each other
I can ask questions but this brings us
to the basic thing why did the grandma
actually got the robot right to
basically do stuff for her right maybe
she wants to have the robot make tea or
coffee and so on right so basically the
robot needs to take action not just
pursue the world right now imagine the
scenario so how does grandma and the
answers make me coffee right now there
is all this robot or any other that have
been pre-programmed to know how to do
coffee right all the stuff that are
there right but maybe say the grandma
actually bought the robot things have
changed right the world is changing
maybe she wants to do other stuff right
more interesting stuff
and they’re like crazy things happen
every day
so how can we actually have this robot
or that and actually be able to follow
anything that we were gonna be asking it
alright so here was our idea so let’s
say we actually as the robot to do
something that it has never heard how to
do before right so how would you do it
as a human you would describe the steps
how to do it you will describe I know
the recipe you know do this and this and
this in natural language so the idea was
in the robot read this instruction just
a natural language and self program
itself basically write code that is
going to be able to execute and actually
learn how to do this task right so what
you see on the left is just kind of a
set of really simple instructions a
sequence of instructions that the robot
knows how to execute but actually if
it’s executing it then it basically does
what the grandma wants right so then in
the end you can have an environment and
then it knows how to do that stuff right
luckily no sound so here is a
description the robot read this
description basically does its own code
in a material gonna see is this agent
performing this action this is actually
done in a game engine called unity and
it was done in collaboration but the
main driving force was actually a view
of the undergrad so she came to my
office he said I know computer vision
when I have been programming games and
was like all right you’re hired
we’re gonna be doing this cool project
it’s a simulation is really the thing to
do these days because if you have a
robot do stuff in a real environment you
can just you know go really wrong so
first needs to the stuff in the
simulated environment alright so now we
can kind of close you know bit knows how
to perceive understand language even do
stuff right but what else can we do
right and since this grammar is actually
kind of tech savvy right maybe we can
throw it I
a little phone and do all these nice
things for her as well right so maybe
she’s interested in fashion or other
possible applications so our idea was
can you take a photo and maybe the user
is going to tell you the settings right
go into the beach or something like that
and you want to predict how fashionable
this this user actually looks for the
occasion okay so what we did was take
this website for chief topia we’re
basically a lot of lifestyle enthusiast
suppose their latest outfit and then
other people can vote on them so
basically tell you how well you actually
look and that was a proxy so basically
we took the number of votes and this was
our proxy as fashionability this is the
stuff that you want to predict when
someone shows you the photo so this is
basically our model thinking that these
people are fashionable so this is a
prediction and the nice thing is that it
as a side effect also discover styles
right so imagine you give it a photo and
then basically it’s able to show you the
pictures of all the other people who
have the same style maybe that gives you
some idea of what you want to wear next
right it can also do recommendations so
let’s say that you know this photo
doesn’t maybe look so good and it’s
gonna tell you you know maybe one of our
something else to increase how well you
look for that occasion maybe that gives
you some idea and maybe you just want to
describe in a little bit more detail of
what you want to kind of see on yourself
and then you can actually generate that
kind of clothing on the user maybe the
user wants to do say something else a
blue shirt and then you can generate
that kind of clothing right so in a
couple of minutes if I’m actually see
like so many outfits on yourself I kind
of see what goes well on you and maybe
just maybe that’s the only way to get a
nerd like me in cosmopolitan
it was the first moment and my mom was
actually proud all right so this project
was actually thanks to Edgar should I
say Edgar before the project
all right so I hope in this 15 maybe a
little bit more minutes I have convinced
you that AI is the thing to do and AI is
a new cool so you should all be working
on it
[Applause]
[Music]
Please follow and like us: