
so as as you all know robotics
artificial and tell
learning this all buzzwords
can not only find in research and
science anymore you can find them more
and more in public media so when there
is a phase where you have this this big
hype about a domain it is always good to
start with the definition so what is
artificial intelligence machine learning
means software software algorithms that
learn from data robotics means we have a
connection between a computer system a
digital system that can physically
interact with the with the environment
and we’re going to talk about that much
much more in detail today but taking a
step back so there are self-driving cars
I’m sure that every one of us has
thought about that looking at Apple Siri
Google’s assistant Amazon’s Alexa so
we’re seeing more and more of these
technologies entering our everyday lives
and what makes this different so when it
comes artificial intelligence what
computer software used to be imagine you
had three colors white grey and black
and you would ask a computer program
what color is this white grey or black
there’s a unique answer to this if you
show a photograph to an algorithm and
you would ask who is on the photograph
then the answer might be this could be
Linda but maybe it’s some someone else
or this could be Chris but it could be
someone else but the same is true for us
humans right if I would show you a
photograph in many cases assuming you
know that person you can tell who is on
the photograph but in many cases also
you you’re not really sure it could be
that person but could be someone else so
we’re seeing more and more of these
breakthroughs in semantic understanding
speech recognition speech translation
now looking at robotics let’s take a
step back to 1997 when IBM’s deep blue
bet Garry Kasparov in chess this is what
we we call it artificial intelligence at
that point and I’m sure that you’re for
of alphago are much more recently
however I have not yet seen a computer
system or robot that can take a
box of chess figures take each
individual figure put it on the board
and physically play chess with me so
let’s let’s talk about robotics what is
what is robotics robotics is the
intelligent connection between
perception and action so what that means
is on the right side you can see a robot
and the robot has is installed in some
environment works in some environment
but it’s also about physical contact
having the system making physical
physical contact with the environment
possibly with the human and the robot
consists of actuators in many cases
electrical motors but there are many
other actuators and then it consists of
sensors so it can sense its own position
it might have a camera for visual
perception it might have a forced walk
sensor or skin for tactile perception
and then we have a a control system and
what I want to focus on today is this
control system so let’s let me let me
start with something that I’ve done
quite a while ago and this is one of the
first projects that I did and in
robotics what you can see here is a
robot inserting a battery into a cell
phone the robot is equipped with a four
stroke sensor so it can can feel but it
cannot see the phone and it doesn’t
exactly know where the phone is so you
can see that it is it is displaced right
so that you know we don’t know the
position and orientation of the phone so
we’re just feeling like blindly trying
to insert the battery into the phone and
what you can see on the right hand side
is a graph so the graph is static and
each node in this graph is a command for
the robot so depending on the sensor
signals from the robot we take a a path
through this graph that really depends
on what is happening in the robot work
cell now when we switch from one node to
another we don’t know in what state we
are in so that means we need to be able
to switch to react instantaneously in
this case within like one millisecond we
need to change from one behavior to
another so
this is comparable to the human reflex
you can see me soldering and I’m burning
my fingers and these these reflexes are
very often triggered in our spinal cord
or in our surah bellum and we react to
pain or something unforeseen right away
and very often actually not very often
we we really first react and then
afterwards we consciously notice oh I
just brought my fingers so we use this
technology and we implemented a Jenga
playing robot so Jang has a parlor game
so many of you know that it consists of
a tower of wooden bricks and the goal is
to pick a brick take it out and put it
back on the top of the tower and the
robot that we used to play this game was
equipped with position sensors in the
joints so the robot knew where it was
it had a forced walk sensor so that it
could feel it had a distance sensor and
it had two cameras and what you know can
can see we’re now about to to grasp this
one brick of wood and while we’re now
trying to to pull out this this break
we’re eliminating all transfers of
forces and torques and we do that to
prevent the tower from toppling and we
really have to do this in a very very
careful careful manner and this again is
implemented using the same primitives
there’s the same graph principle that
you that you’ve seen before however now
this graph is much more complex um you
can see a view from from the cameras and
we were actually cheating a little bit
so we we colored all of the blocks with
black and white color so that it’s not a
computer vision problem so we simplified
our task that way you can see the the
force torque sensor to really get the
the tactile perception you can see it in
on the robot and the distance sensor as
well now taking a step forward
programming here was was really a
challenge
and looking at the control system what
could we do to kind of make the control
system more intelligent the the green
block here on the left so we could
imagine to put a aun in there right so
the human is Telly operating the robot
and if the human also gets a sense of
touch then this is what we call haptics
so the human can haptically telle
operate the robot so the human can feel
what the robot is feeling and here’s one
one example this is something that we
did at Stanford University you can see
this wealth on the right and the robot
as well and this is a haptic device a
haptic device is a imagine a joystick
and you have force feedback but with
this joystick you you get force feedback
and up to six degrees of freedom so you
can feel exactly what the robot is
feeling so in this case the human
operator can actually see the valves but
of course there could also be a camera
so that the operator would only see the
the camera image No
obviously that doesn’t scale right we
would need one human per robot so would
be easy to program because it’s done by
the human but we can scale that so we
need something else a much more
interesting aspect is actually putting
the human not at the channel operation
seat but have the human and the robot
interact maybe physically interact so
here’s another project where you can see
the robot and the human physically
interacting this actually was a student
project at Stanford University I’m using
this this framework that I mentioned
earlier where we are able to generate
notions within less than one millisecond
comparing this is what what is
comparable to the human reflex and here
the students were able to implement this
in in less than three weeks and it’s
actually it looks complex but it’s it’s
very very simple so the robot moves
towards the human sort and the moment we
detect the contact within one
millisecond the robot starts recoiling
exactly what a human would do as well
and here’s the defense mode also very
simple all the the only thing that the
human and sorry that the robot tries to
do is to put it sort perpendicular to
the human sort you can see that again so
you can see me fighting with the robot
and all the robot is trying to do is to
put it it’s sort perpendicular to mine
and you can see the 3d camera there and
also the camera image in in the
background
now looking at this this actually looks
dangerous right and imagine you would
have humans and and robots interacting
sharing the same Rock space so safety is
something physical safety for the human
is something that really needs to have
the highest priority whatsoever so we
don’t want to harm to injure or even to
kill a human right so this has to be um
the the foremost priority for for
roboticists when they work on
applications where robots and humans
share the same workspace so we then
worked on on an algorithm that allows
the robot to prevent from collisions so
you can see on the right-hand side here
a a depth image and based on the depth
image we can detect the human we can
detect obstacles and we then compute a
velocity vector so we compute a point in
space and the velocity vector is
pointing into the opposite direction of
the human and to the opposite direction
of of the obstacle and this way we can
compute a motion in real time that
allows the robot to move away from the
obstacle in the best possible way as
fast as physically possible so the robot
tries everything it’s possibly Candace
physically can to prevent from that
collision so that is one trend but even
here programming again this is very very
expensive
there is another development in the
domain of Robotics research and that’s
based on on unmoral it’s a lot of
engineering we put a lot of
and thought into the controllers into
the motion planners and when I worked at
Google I had the honor and the pleasure
to also collaborate with the team at
Boston Dynamics and I would like to give
great credits to to them for the for the
videos we’re seeing here this robot is
called Atlas it’s a humanoid robot and
this robot is fully force and torque
controlled and what they were able to do
here is to have the robot being able to
walk alongside with humans on very
uneven terrain – to manipulate objects
like here and there’s another robot
called called spot so this spot is the
UM
splat mini to be precise it’s a
quadrotor robot within with an arm on
top of it and what you can see here in
this demonstration is actually whole
body controlled so we keep the position
of the end effector where it is and then
we move move around the latest version
of Atlas can even jump and this is
really an amazing achievement they’ve
just was implemented very recently and
it’s not only about jumping and looking
at some of you might have seen this
already
so this is really an amazing engineering
achievement and and credit to the team
at Boston Dynamics at this point however
in all three cases the graph that you’ve
seen human robot collaboration and the
the latest videos it requires a lot of
skills expertise and experience to
program these robots so how can we
overcome this this problem you’re right
we cannot have like people who are
highly educated in robotics to program
all of these robots it will not work it
will not scale so this is where machine
learning might come to help and this is
ongoing research in many research labs
in in the world including the ones here
in in Karlsruhe and machine learning as
I mentioned means software algorithms
that learn from data now data is
something that is sometimes hard to find
in robotics so we can find a lot of text
and images in the internet to train
algorithms this is why they have become
very very trained in speech recognition
image recognition however they’re not
too many websites that show motor
currents motor torques joint torques
joint positions right we just don’t have
enough enough of their data so the
problem we have here is we actually need
a lot of data it’s really about data and
and we come to data driven robotics and
one experiment that we did it at Google
awhile ago this was last year we had 14
robots trying to grasp objects and we
did there for two months there’s a
method that is called reinforcement
learning and that means the robot is is
taking an action and if it does a a good
action it gets a reward for it so in
this case the reward was the success of
grasping an object and we did that as I
mentioned for about two months that was
a little more than 800,000 grasps and
there was one thing that happened it was
truly amazing and surprising to us so
look at this scene you can see the
stapler
the yellow DUPLO break here and now in
the next scene here the robot is trying
to grasp the yellow brick and what it
does is it pushes away the stapler to
then grasp the object and there was no
engineer that wrote a single line of
code to do this this was truly learned
by the system and when we’ve seen that
this was a big surprise and a big
success to thee to the entire team there
now this is truly amazing however even
that doesn’t scale because this works
for this robot with this camera with
this grouper if you change the setup the
neural network that was trained will not
work anymore so machine learning is no
magic and we have to be aware of it it’s
a tool and it just means it’s a set of
algorithms that learn from data now
taking us stop here if if you are
interested in in this domain and in
robotics there’s a fantastic a book and
also a website where you can find
hundreds of such videos and technologies
with descriptions it’s all openly
available and if you take a look at this
this book and website I’m sure you will
find more interesting information about
these technologies and now concluding
looking a little bit into the future
of what’s what’s gonna happen next so
we’re seeing almost weekly right now
technical breakthroughs in visual
perception and and speech recognition
however the challenges that we are still
facing and where we do not yet have good
solutions is safe actuation having
robots and humans working together in a
safe way having robots physically
manipulating the environment and having
robots and humans working together
sharing the same workspace and doing
that in a in a safe way so to conclude
this is what we’ve seen before so the
intelligent connection between
perception and action safety is one very
important very fundamental aspect and
data so this is where we are going to
see in the next decade more and more
breakthroughs as well and I hope that I
will be able to contribute that further
but I would also like to say that we
should not be afraid of this future
robots are not gonna become self-aware
they will help us support us assist us
at work and in everyday life thank you
so much
[Applause]
artificial and tell
learning this all buzzwords
can not only find in research and
science anymore you can find them more
and more in public media so when there
is a phase where you have this this big
hype about a domain it is always good to
start with the definition so what is
artificial intelligence machine learning
means software software algorithms that
learn from data robotics means we have a
connection between a computer system a
digital system that can physically
interact with the with the environment
and we’re going to talk about that much
much more in detail today but taking a
step back so there are self-driving cars
I’m sure that every one of us has
thought about that looking at Apple Siri
Google’s assistant Amazon’s Alexa so
we’re seeing more and more of these
technologies entering our everyday lives
and what makes this different so when it
comes artificial intelligence what
computer software used to be imagine you
had three colors white grey and black
and you would ask a computer program
what color is this white grey or black
there’s a unique answer to this if you
show a photograph to an algorithm and
you would ask who is on the photograph
then the answer might be this could be
Linda but maybe it’s some someone else
or this could be Chris but it could be
someone else but the same is true for us
humans right if I would show you a
photograph in many cases assuming you
know that person you can tell who is on
the photograph but in many cases also
you you’re not really sure it could be
that person but could be someone else so
we’re seeing more and more of these
breakthroughs in semantic understanding
speech recognition speech translation
now looking at robotics let’s take a
step back to 1997 when IBM’s deep blue
bet Garry Kasparov in chess this is what
we we call it artificial intelligence at
that point and I’m sure that you’re for
of alphago are much more recently
however I have not yet seen a computer
system or robot that can take a
box of chess figures take each
individual figure put it on the board
and physically play chess with me so
let’s let’s talk about robotics what is
what is robotics robotics is the
intelligent connection between
perception and action so what that means
is on the right side you can see a robot
and the robot has is installed in some
environment works in some environment
but it’s also about physical contact
having the system making physical
physical contact with the environment
possibly with the human and the robot
consists of actuators in many cases
electrical motors but there are many
other actuators and then it consists of
sensors so it can sense its own position
it might have a camera for visual
perception it might have a forced walk
sensor or skin for tactile perception
and then we have a a control system and
what I want to focus on today is this
control system so let’s let me let me
start with something that I’ve done
quite a while ago and this is one of the
first projects that I did and in
robotics what you can see here is a
robot inserting a battery into a cell
phone the robot is equipped with a four
stroke sensor so it can can feel but it
cannot see the phone and it doesn’t
exactly know where the phone is so you
can see that it is it is displaced right
so that you know we don’t know the
position and orientation of the phone so
we’re just feeling like blindly trying
to insert the battery into the phone and
what you can see on the right hand side
is a graph so the graph is static and
each node in this graph is a command for
the robot so depending on the sensor
signals from the robot we take a a path
through this graph that really depends
on what is happening in the robot work
cell now when we switch from one node to
another we don’t know in what state we
are in so that means we need to be able
to switch to react instantaneously in
this case within like one millisecond we
need to change from one behavior to
another so
this is comparable to the human reflex
you can see me soldering and I’m burning
my fingers and these these reflexes are
very often triggered in our spinal cord
or in our surah bellum and we react to
pain or something unforeseen right away
and very often actually not very often
we we really first react and then
afterwards we consciously notice oh I
just brought my fingers so we use this
technology and we implemented a Jenga
playing robot so Jang has a parlor game
so many of you know that it consists of
a tower of wooden bricks and the goal is
to pick a brick take it out and put it
back on the top of the tower and the
robot that we used to play this game was
equipped with position sensors in the
joints so the robot knew where it was
it had a forced walk sensor so that it
could feel it had a distance sensor and
it had two cameras and what you know can
can see we’re now about to to grasp this
one brick of wood and while we’re now
trying to to pull out this this break
we’re eliminating all transfers of
forces and torques and we do that to
prevent the tower from toppling and we
really have to do this in a very very
careful careful manner and this again is
implemented using the same primitives
there’s the same graph principle that
you that you’ve seen before however now
this graph is much more complex um you
can see a view from from the cameras and
we were actually cheating a little bit
so we we colored all of the blocks with
black and white color so that it’s not a
computer vision problem so we simplified
our task that way you can see the the
force torque sensor to really get the
the tactile perception you can see it in
on the robot and the distance sensor as
well now taking a step forward
programming here was was really a
challenge
and looking at the control system what
could we do to kind of make the control
system more intelligent the the green
block here on the left so we could
imagine to put a aun in there right so
the human is Telly operating the robot
and if the human also gets a sense of
touch then this is what we call haptics
so the human can haptically telle
operate the robot so the human can feel
what the robot is feeling and here’s one
one example this is something that we
did at Stanford University you can see
this wealth on the right and the robot
as well and this is a haptic device a
haptic device is a imagine a joystick
and you have force feedback but with
this joystick you you get force feedback
and up to six degrees of freedom so you
can feel exactly what the robot is
feeling so in this case the human
operator can actually see the valves but
of course there could also be a camera
so that the operator would only see the
the camera image No
obviously that doesn’t scale right we
would need one human per robot so would
be easy to program because it’s done by
the human but we can scale that so we
need something else a much more
interesting aspect is actually putting
the human not at the channel operation
seat but have the human and the robot
interact maybe physically interact so
here’s another project where you can see
the robot and the human physically
interacting this actually was a student
project at Stanford University I’m using
this this framework that I mentioned
earlier where we are able to generate
notions within less than one millisecond
comparing this is what what is
comparable to the human reflex and here
the students were able to implement this
in in less than three weeks and it’s
actually it looks complex but it’s it’s
very very simple so the robot moves
towards the human sort and the moment we
detect the contact within one
millisecond the robot starts recoiling
exactly what a human would do as well
and here’s the defense mode also very
simple all the the only thing that the
human and sorry that the robot tries to
do is to put it sort perpendicular to
the human sort you can see that again so
you can see me fighting with the robot
and all the robot is trying to do is to
put it it’s sort perpendicular to mine
and you can see the 3d camera there and
also the camera image in in the
background
now looking at this this actually looks
dangerous right and imagine you would
have humans and and robots interacting
sharing the same Rock space so safety is
something physical safety for the human
is something that really needs to have
the highest priority whatsoever so we
don’t want to harm to injure or even to
kill a human right so this has to be um
the the foremost priority for for
roboticists when they work on
applications where robots and humans
share the same workspace so we then
worked on on an algorithm that allows
the robot to prevent from collisions so
you can see on the right-hand side here
a a depth image and based on the depth
image we can detect the human we can
detect obstacles and we then compute a
velocity vector so we compute a point in
space and the velocity vector is
pointing into the opposite direction of
the human and to the opposite direction
of of the obstacle and this way we can
compute a motion in real time that
allows the robot to move away from the
obstacle in the best possible way as
fast as physically possible so the robot
tries everything it’s possibly Candace
physically can to prevent from that
collision so that is one trend but even
here programming again this is very very
expensive
there is another development in the
domain of Robotics research and that’s
based on on unmoral it’s a lot of
engineering we put a lot of
and thought into the controllers into
the motion planners and when I worked at
Google I had the honor and the pleasure
to also collaborate with the team at
Boston Dynamics and I would like to give
great credits to to them for the for the
videos we’re seeing here this robot is
called Atlas it’s a humanoid robot and
this robot is fully force and torque
controlled and what they were able to do
here is to have the robot being able to
walk alongside with humans on very
uneven terrain – to manipulate objects
like here and there’s another robot
called called spot so this spot is the
UM
splat mini to be precise it’s a
quadrotor robot within with an arm on
top of it and what you can see here in
this demonstration is actually whole
body controlled so we keep the position
of the end effector where it is and then
we move move around the latest version
of Atlas can even jump and this is
really an amazing achievement they’ve
just was implemented very recently and
it’s not only about jumping and looking
at some of you might have seen this
already
so this is really an amazing engineering
achievement and and credit to the team
at Boston Dynamics at this point however
in all three cases the graph that you’ve
seen human robot collaboration and the
the latest videos it requires a lot of
skills expertise and experience to
program these robots so how can we
overcome this this problem you’re right
we cannot have like people who are
highly educated in robotics to program
all of these robots it will not work it
will not scale so this is where machine
learning might come to help and this is
ongoing research in many research labs
in in the world including the ones here
in in Karlsruhe and machine learning as
I mentioned means software algorithms
that learn from data now data is
something that is sometimes hard to find
in robotics so we can find a lot of text
and images in the internet to train
algorithms this is why they have become
very very trained in speech recognition
image recognition however they’re not
too many websites that show motor
currents motor torques joint torques
joint positions right we just don’t have
enough enough of their data so the
problem we have here is we actually need
a lot of data it’s really about data and
and we come to data driven robotics and
one experiment that we did it at Google
awhile ago this was last year we had 14
robots trying to grasp objects and we
did there for two months there’s a
method that is called reinforcement
learning and that means the robot is is
taking an action and if it does a a good
action it gets a reward for it so in
this case the reward was the success of
grasping an object and we did that as I
mentioned for about two months that was
a little more than 800,000 grasps and
there was one thing that happened it was
truly amazing and surprising to us so
look at this scene you can see the
stapler
the yellow DUPLO break here and now in
the next scene here the robot is trying
to grasp the yellow brick and what it
does is it pushes away the stapler to
then grasp the object and there was no
engineer that wrote a single line of
code to do this this was truly learned
by the system and when we’ve seen that
this was a big surprise and a big
success to thee to the entire team there
now this is truly amazing however even
that doesn’t scale because this works
for this robot with this camera with
this grouper if you change the setup the
neural network that was trained will not
work anymore so machine learning is no
magic and we have to be aware of it it’s
a tool and it just means it’s a set of
algorithms that learn from data now
taking us stop here if if you are
interested in in this domain and in
robotics there’s a fantastic a book and
also a website where you can find
hundreds of such videos and technologies
with descriptions it’s all openly
available and if you take a look at this
this book and website I’m sure you will
find more interesting information about
these technologies and now concluding
looking a little bit into the future
of what’s what’s gonna happen next so
we’re seeing almost weekly right now
technical breakthroughs in visual
perception and and speech recognition
however the challenges that we are still
facing and where we do not yet have good
solutions is safe actuation having
robots and humans working together in a
safe way having robots physically
manipulating the environment and having
robots and humans working together
sharing the same workspace and doing
that in a in a safe way so to conclude
this is what we’ve seen before so the
intelligent connection between
perception and action safety is one very
important very fundamental aspect and
data so this is where we are going to
see in the next decade more and more
breakthroughs as well and I hope that I
will be able to contribute that further
but I would also like to say that we
should not be afraid of this future
robots are not gonna become self-aware
they will help us support us assist us
at work and in everyday life thank you
so much
[Applause]
Please follow and like us: