great fantastic yeah it’s really a
pleasure to be here and I have to say
that we had the coolest Prime Minister
ever and he’s is even cooler than you
can’t think of it
great so this is my pleasure to be here
I’m gonna try to throw some light into
the life of a self-driving car so first
let me start by saying you know why we
potentially my wanna have surviving cars
in the first place
unfortunately this more than 1 million
people a year actually died due to
traffic accidents so imagine the
potential impact that we can have if you
can reduce that number significantly the
population get older and medicine gets
better you know we are having more and
more issues of mobility so surviving
cars can definitely help us out a lot we
shall decrease our pollution right our
planet is now getting healthier and what
I’m most excited also driving cars is
really try to think about what is new
way what are new ways to do public
transportation that you know will allow
us to commute faster and in a much
better way and if you think about it you
know having a car is really a waste
right so we should be thinking of
sharing resources both with materials as
well as well as a space so for those of
you that own a car so your car is bored
to death it’s basically sitting around
95 percent of the time just maybe
talking to the car next body right and
so imagine you know the space that the
car is taking right we could use a short
parts you know that we could use that to
make our cities much better so you
probably all have seen some way you know
one of these systems for example the
Google car the uber cars etc somewhere
in the media right and you know these
systems are really good that even things
like localization but planning avoiding
obstacles
however there is two problems that this
systems have that prevent them from
scaling up to you know a case where each
one of us can actually use as a driving
car mainly they use this thing on the
top the typical is a spinning is a
leader and the thing cost this day is
our
100,000 Canadian dollars right so that’s
really expensive and they use
pre-recorded manually annotated maps and
just for example to map the cold us it’s
gonna take around one to ten billion
dollars so in Mayan you know having maps
up to date now so the goal of my
research has been to try to create
solutions that animals live in which
rely on cheap sensors okay as well as as
little prior knowledge as possible and
this is actually possible revived
because us as humans can actually solve
this task and what do we use we use our
eyes right so we can the same way we can
actually use cameras in order to try to
solve this at this problem now from the
point of view of computer vision right
which is trying to use computers to
understand the world via cameras we have
to solve many many different tasks to do
so driving cars we need to recognize the
environment we need to be able to
estimate three-dimensional
representation of the world we need to
be able to know how our car is moving on
the other cars we need to know what is
happening in front of us right
understand motion and intention of those
cars and we need to do this in three
dimensions and we need to again
understand everything that is in the
scene that we see now if we want to do
this what do we need to do so first of
all we need data and this might sound
you know something that this is C but
it’s not so easy because having access
to one of these platforms costs already
half a million dollars right so it’s not
something that you can just buy in the
supermarket also it’s very important to
think of the role of machine learning we
cannot fund craft brewers that will
represent all possible scenarios for a
car we need to be able to learn by
driving by example we also need to think
about models that are able to solve many
many many tasks that we need to solve
for doing self-driving cars and it has
to be efficient right these systems have
to react this scenario they have to be
real-time and they seem implies working
on you know so well I’m a computer
scientist but also in hardware solutions
for this so in the remaining of this
talking on I a study
what we have done in the spectrum of
making self-driving cars affordable for
each one of us
so I mentioned data at the beginning
right so seven years ago we started you
know trying to think of okay when I work
on Vista this Tasi of self-driving cars
can we actually do that and it turns out
that it was not possible because we
couldn’t have access to any datasets
right so this was you know we spend the
first two and a half years actually
building the benchmarks for people to
compete in order to build the next
generation of algorithms that would
solve other Nomos driving so here you
see the car that we use for this we
basically mounted with four cameras this
later is spinning on the top as well as
GPS and IMU and we went around and drive
and in this case the city of cars room
okay so let me play a video so here is
the car in the parking lot of ki T so
this is in Assisi culture this is in
collaboration with one of the labs over
there and you know and as I said we went
around I’m driving around you see
scenarios not so easy scenarios you know
where we have illegal maneuvers to you
know really crowded the scenes and then
use these fancy sensors to create ground
truth in order to evaluate how well we
can do with yes cameras okay so here you
see examples of ground truth for Delta
current with for how the vehicle moves
as well as to the detection and tracking
okay so this benchmark was actually a
huge success and it is still the only
benchmark out there for self-driving
cars despite that it’s been online for
now almost five years we have more than
20,000 folks actually working on this
and more than 500 submission to the
benchmarks were basically every car
company and every you know group working
on this is actually working with our
data so let me tell you a little bit of
what are the tasks that one needs to
solve for self-driving cars so the first
and canonical one is actually trying to
yes reconstruct the environment without
necessarily understanding what is in
front of us right so this is a Tacitus
as humans we solve all the time we have
right so this you know imagine you have
two cameras and we want to reconstruct
right for every pixel that we see in our
eyes right how far away it is from us
right so this is basically you know our
one of their latest algorithms that is
able to after 40 meters we almost as
accurate as this really fancy laser
scanner while costing you know a camera
these days cost $30 right okay so then
once we reconcile the environment we
need to perceive understand what it
means what is in front of us right so
one of the tasks that one is to solve is
pixel labeling right so for every pixel
in one of those images on the Left you
want to be able to say okay is there a
card there it’s the road there is there
a person as well as the the thing that
you also need to understand is being
able to discriminate between you know
every one of the individuals every one
of the cars that is out there so systems
right now using convolutional nets using
dip lorina Sonya was talking about
before are able to generalize really
well to unseen in this case and since
teens from different cities and create
you know incredible results in terms of
performance this is you know only using
a single images import once we are able
to understand pixels we need to
understand three dimensions right we
need to be able to predict trajectories
of you know the traffic participants and
this is what you see over here so this
is our latest results that were actually
done by a third year you of the
undergrad Y which is basically showing a
set of their results better than you
know any of these car companies so when
you see here is to see trajectories in
2d and 3d that the system is able to
estimate again in places that the car
has never driven before so it doesn’t
record to use Maps or anything it’s just
given the images it’s able to estimate
this next we can you can close that so
next we can actually estimate you know
everything that is happening in the
scene so what you see here is a single
camera mounted on top of the car and
base
develop the algorithms that are able to
translate that image that video into a
representation of the scene where you
can actually do control right so you see
here in black the Eco car the car that
is driving autonomously
did you see the race or the field of
view of what the car sees and you see
that is able to compute maps on the fly
and it’s able to estimate what every
single car participant is in the same
now it’s color-coded by the intention of
those cars and on the bottom left you
can see the traffic scenario without
looking at traffic lights
so basically saying what is the last set
of maneuvers that our car can actually
do safely another thurston we need to
solve is actually the task of set
localization but if you win this vehicle
to be able to drive us around you need
to know at every point in time where it
is in the scene needed to click on the
slide thank you and so the way that
typically this is done in industry is
that so a car manually drives around the
world and everything that it sees it’s a
story in a dataset okay and then one is
driving autonomously you look in your
data set will have a scene a place that
looks exactly the same as what I see in
front of me okay so so it’s a bit like
web search right so this you know works
fairly well but it’s not a scalable
right imagine that you need to capture
how the world looks like at every point
in time in every season every time of
the day you know there is construction
all the time right the world changes so
this is why doing self-driving cars it’s
also very expensive right so what we
look at is can we actually create
solutions to localization where we don’t
need to know the appearance of the world
and as humans right we can do this we
can navigate and then we never know how
a particular Street like Laura Street
looks like so it turns out that yes you
can actually do this and it’s very
simple you can download freely from the
web opening street maps which is these
maps that you see in blue in the screen
and what you can basically do is
estimated the trajectory of the car of
the eco car biasing yes the cameras and
the motion of the car is enough to tell
you where you can potentially be in the
same so at the beginning we’re going to
have a region of interest where we know
we are more or less there somewhere and
it’s gonna be a uniform distribution
right because we don’t know where we are
and as we start driving around then
we’re going to shrink it the probability
distribution of possible locations where
the car could potentially be okay and
this is where you’re gonna see on the
bottom right in a second yes for
comparisons in black you’re gonna see
what the ground truth is okay so the
beginning is uniform you see a few modes
but very quickly is able to say we’re
exactly in this location okay so you see
if here yes assuming of the same idea
right a few modes and very quickly
there’s only two months now and we turn
alive therefore this you know this is
the only location where we could
potentially be and this you can narrow
the maps from the web is free right you
don’t need to pay anything
yes mount one or two cameras in your car
so here is a mama Hatton like a scenario
so you see that in this case there is
two moles of the distribution right it’s
very symmetric but again at some point
with one manoeuvre there is only one
possibility let me just zoom into
different regions sorry with that so
what you’re gonna see now is the same
thing but in a map that is the whole
city of cultural there is 2,000
kilometers or potential roads okay which
means that you know there is a lot of
uncertainty of where you can potentially
be in the scene and with only 30 seconds
of driving it’s able to localize very
accurately where you are in the scene
okay so hopefully you can see that in
the screen right again very quickly it
says we are in this highway here this is
where we are the mod is relatively big
and at some point we take an exit and
then it strings to you know very precise
location of well this is where we are
Thanks so I mentioned before the mapping
was very expensive right and the reason
why is just the way that people are
actually tackling this problem however
if you could potentially have a further
worse evolutions to map Ian then we
should be able to use those maps that
are up-to-date so why is this so
expensive so this is the what companies
are doing these days and you probably
have seen one of these in you know in
the city so the idea is that you have a
car you mounted with a whole bunch of
fancy sensors and then you drive around
the world and then you capture how the
world looks like again right so it has
very small coverage and it’s really
expensive this is what the car sees
right it is a miniscule part of the
world right however you know there is so
many other ways to see the world that we
could use in order to improve the
coverage of our maps one example right
is you know Big Brother right
it sees us all the time of what we are
doing right so we could use drones UAVs
planes satellites go around the world
twice a day every single day right so
it’s very very up-to-date the imagery
now it’s not so trivial how to estimate
maps from yes aerial views okay so
hopefully thanks so what you’re gonna
see here is a system that is able to as
it drives and using aerial views
localize where the car is in terms of
the location in geo coordinates and it’s
able to also map clear HT maps of where
the roads where are the lanes where the
parking spots are in yellow here as well
as in blue where the sidewalks okay so
this is really efficient it takes three
seconds per kilometer of road so you
know in a simple of the computer cluster
you can actually map the whole world and
so again for mapping the problem is the
same as it was for cell driving cars
that again only companies like Nvidia
for this have access to data so one of
the last things that the most recent
things that we have done in the in the
lava saga is actually to create the next
big benchmark for people to compete in
to creating maps in an affordable manner
and the cool thing is a lecture
we selected you know Toronto area to be
the main area right so the way that we
created this but was by going to the
government and going to the libraries
and you seen the HD maps that have
already been created and these are
really really rich they have all sort of
very interesting metadata so we have 3d
buildings of the whole recreated Toronto
as well as the location of every single
three the location of every single pole
the shape of every traffic light every
every tree every pole in the city by an
the idea is that now we want
participants to be able to estimate all
these different things automatically
and we also generated using again deep
learning neural networks solutions that
are able to estimate this map this Maps
so here you see related results you’ve
seen only I read news right so there are
almost as good as the ground truth we
can estimate you can estimate the
topology of average streets and I
mentioned before that you know for the
localization we use these OpenStreetMaps
these are generated by people clicking
in aerial images the location of streets
and we are only 1% below what people do
for this for this task and it’s fully
automatic so you can imagine and we can
do this at a larger scale of the whole
world so you can imagine the impact of
doing things like this other things that
we can do is create centimeter accurate
maps for Nia’s Google Street View
panoramas which are available out there
for the whole world as well okay so just
to conclude I show you a couple of
things that one is to do in order to
create self-driving cars for each one of
us so things like sensing perception
localization mapping I show you our
latest coolest priority in to you know
bring in Toronto to the community at
large and there is you know many many
many things to do in research in order
to really solve all the different task
forces driving cars and you know it’s
one acknowledge the fantastic team that
is behind what I show you today and I
just want to finish also with a promise
that we will have self-driving cars in
Toronto really really sir thank you
[Music]
[Applause]
pleasure to be here and I have to say
that we had the coolest Prime Minister
ever and he’s is even cooler than you
can’t think of it
great so this is my pleasure to be here
I’m gonna try to throw some light into
the life of a self-driving car so first
let me start by saying you know why we
potentially my wanna have surviving cars
in the first place
unfortunately this more than 1 million
people a year actually died due to
traffic accidents so imagine the
potential impact that we can have if you
can reduce that number significantly the
population get older and medicine gets
better you know we are having more and
more issues of mobility so surviving
cars can definitely help us out a lot we
shall decrease our pollution right our
planet is now getting healthier and what
I’m most excited also driving cars is
really try to think about what is new
way what are new ways to do public
transportation that you know will allow
us to commute faster and in a much
better way and if you think about it you
know having a car is really a waste
right so we should be thinking of
sharing resources both with materials as
well as well as a space so for those of
you that own a car so your car is bored
to death it’s basically sitting around
95 percent of the time just maybe
talking to the car next body right and
so imagine you know the space that the
car is taking right we could use a short
parts you know that we could use that to
make our cities much better so you
probably all have seen some way you know
one of these systems for example the
Google car the uber cars etc somewhere
in the media right and you know these
systems are really good that even things
like localization but planning avoiding
obstacles
however there is two problems that this
systems have that prevent them from
scaling up to you know a case where each
one of us can actually use as a driving
car mainly they use this thing on the
top the typical is a spinning is a
leader and the thing cost this day is
our
100,000 Canadian dollars right so that’s
really expensive and they use
pre-recorded manually annotated maps and
just for example to map the cold us it’s
gonna take around one to ten billion
dollars so in Mayan you know having maps
up to date now so the goal of my
research has been to try to create
solutions that animals live in which
rely on cheap sensors okay as well as as
little prior knowledge as possible and
this is actually possible revived
because us as humans can actually solve
this task and what do we use we use our
eyes right so we can the same way we can
actually use cameras in order to try to
solve this at this problem now from the
point of view of computer vision right
which is trying to use computers to
understand the world via cameras we have
to solve many many different tasks to do
so driving cars we need to recognize the
environment we need to be able to
estimate three-dimensional
representation of the world we need to
be able to know how our car is moving on
the other cars we need to know what is
happening in front of us right
understand motion and intention of those
cars and we need to do this in three
dimensions and we need to again
understand everything that is in the
scene that we see now if we want to do
this what do we need to do so first of
all we need data and this might sound
you know something that this is C but
it’s not so easy because having access
to one of these platforms costs already
half a million dollars right so it’s not
something that you can just buy in the
supermarket also it’s very important to
think of the role of machine learning we
cannot fund craft brewers that will
represent all possible scenarios for a
car we need to be able to learn by
driving by example we also need to think
about models that are able to solve many
many many tasks that we need to solve
for doing self-driving cars and it has
to be efficient right these systems have
to react this scenario they have to be
real-time and they seem implies working
on you know so well I’m a computer
scientist but also in hardware solutions
for this so in the remaining of this
talking on I a study
what we have done in the spectrum of
making self-driving cars affordable for
each one of us
so I mentioned data at the beginning
right so seven years ago we started you
know trying to think of okay when I work
on Vista this Tasi of self-driving cars
can we actually do that and it turns out
that it was not possible because we
couldn’t have access to any datasets
right so this was you know we spend the
first two and a half years actually
building the benchmarks for people to
compete in order to build the next
generation of algorithms that would
solve other Nomos driving so here you
see the car that we use for this we
basically mounted with four cameras this
later is spinning on the top as well as
GPS and IMU and we went around and drive
and in this case the city of cars room
okay so let me play a video so here is
the car in the parking lot of ki T so
this is in Assisi culture this is in
collaboration with one of the labs over
there and you know and as I said we went
around I’m driving around you see
scenarios not so easy scenarios you know
where we have illegal maneuvers to you
know really crowded the scenes and then
use these fancy sensors to create ground
truth in order to evaluate how well we
can do with yes cameras okay so here you
see examples of ground truth for Delta
current with for how the vehicle moves
as well as to the detection and tracking
okay so this benchmark was actually a
huge success and it is still the only
benchmark out there for self-driving
cars despite that it’s been online for
now almost five years we have more than
20,000 folks actually working on this
and more than 500 submission to the
benchmarks were basically every car
company and every you know group working
on this is actually working with our
data so let me tell you a little bit of
what are the tasks that one needs to
solve for self-driving cars so the first
and canonical one is actually trying to
yes reconstruct the environment without
necessarily understanding what is in
front of us right so this is a Tacitus
as humans we solve all the time we have
right so this you know imagine you have
two cameras and we want to reconstruct
right for every pixel that we see in our
eyes right how far away it is from us
right so this is basically you know our
one of their latest algorithms that is
able to after 40 meters we almost as
accurate as this really fancy laser
scanner while costing you know a camera
these days cost $30 right okay so then
once we reconcile the environment we
need to perceive understand what it
means what is in front of us right so
one of the tasks that one is to solve is
pixel labeling right so for every pixel
in one of those images on the Left you
want to be able to say okay is there a
card there it’s the road there is there
a person as well as the the thing that
you also need to understand is being
able to discriminate between you know
every one of the individuals every one
of the cars that is out there so systems
right now using convolutional nets using
dip lorina Sonya was talking about
before are able to generalize really
well to unseen in this case and since
teens from different cities and create
you know incredible results in terms of
performance this is you know only using
a single images import once we are able
to understand pixels we need to
understand three dimensions right we
need to be able to predict trajectories
of you know the traffic participants and
this is what you see over here so this
is our latest results that were actually
done by a third year you of the
undergrad Y which is basically showing a
set of their results better than you
know any of these car companies so when
you see here is to see trajectories in
2d and 3d that the system is able to
estimate again in places that the car
has never driven before so it doesn’t
record to use Maps or anything it’s just
given the images it’s able to estimate
this next we can you can close that so
next we can actually estimate you know
everything that is happening in the
scene so what you see here is a single
camera mounted on top of the car and
base
develop the algorithms that are able to
translate that image that video into a
representation of the scene where you
can actually do control right so you see
here in black the Eco car the car that
is driving autonomously
did you see the race or the field of
view of what the car sees and you see
that is able to compute maps on the fly
and it’s able to estimate what every
single car participant is in the same
now it’s color-coded by the intention of
those cars and on the bottom left you
can see the traffic scenario without
looking at traffic lights
so basically saying what is the last set
of maneuvers that our car can actually
do safely another thurston we need to
solve is actually the task of set
localization but if you win this vehicle
to be able to drive us around you need
to know at every point in time where it
is in the scene needed to click on the
slide thank you and so the way that
typically this is done in industry is
that so a car manually drives around the
world and everything that it sees it’s a
story in a dataset okay and then one is
driving autonomously you look in your
data set will have a scene a place that
looks exactly the same as what I see in
front of me okay so so it’s a bit like
web search right so this you know works
fairly well but it’s not a scalable
right imagine that you need to capture
how the world looks like at every point
in time in every season every time of
the day you know there is construction
all the time right the world changes so
this is why doing self-driving cars it’s
also very expensive right so what we
look at is can we actually create
solutions to localization where we don’t
need to know the appearance of the world
and as humans right we can do this we
can navigate and then we never know how
a particular Street like Laura Street
looks like so it turns out that yes you
can actually do this and it’s very
simple you can download freely from the
web opening street maps which is these
maps that you see in blue in the screen
and what you can basically do is
estimated the trajectory of the car of
the eco car biasing yes the cameras and
the motion of the car is enough to tell
you where you can potentially be in the
same so at the beginning we’re going to
have a region of interest where we know
we are more or less there somewhere and
it’s gonna be a uniform distribution
right because we don’t know where we are
and as we start driving around then
we’re going to shrink it the probability
distribution of possible locations where
the car could potentially be okay and
this is where you’re gonna see on the
bottom right in a second yes for
comparisons in black you’re gonna see
what the ground truth is okay so the
beginning is uniform you see a few modes
but very quickly is able to say we’re
exactly in this location okay so you see
if here yes assuming of the same idea
right a few modes and very quickly
there’s only two months now and we turn
alive therefore this you know this is
the only location where we could
potentially be and this you can narrow
the maps from the web is free right you
don’t need to pay anything
yes mount one or two cameras in your car
so here is a mama Hatton like a scenario
so you see that in this case there is
two moles of the distribution right it’s
very symmetric but again at some point
with one manoeuvre there is only one
possibility let me just zoom into
different regions sorry with that so
what you’re gonna see now is the same
thing but in a map that is the whole
city of cultural there is 2,000
kilometers or potential roads okay which
means that you know there is a lot of
uncertainty of where you can potentially
be in the scene and with only 30 seconds
of driving it’s able to localize very
accurately where you are in the scene
okay so hopefully you can see that in
the screen right again very quickly it
says we are in this highway here this is
where we are the mod is relatively big
and at some point we take an exit and
then it strings to you know very precise
location of well this is where we are
Thanks so I mentioned before the mapping
was very expensive right and the reason
why is just the way that people are
actually tackling this problem however
if you could potentially have a further
worse evolutions to map Ian then we
should be able to use those maps that
are up-to-date so why is this so
expensive so this is the what companies
are doing these days and you probably
have seen one of these in you know in
the city so the idea is that you have a
car you mounted with a whole bunch of
fancy sensors and then you drive around
the world and then you capture how the
world looks like again right so it has
very small coverage and it’s really
expensive this is what the car sees
right it is a miniscule part of the
world right however you know there is so
many other ways to see the world that we
could use in order to improve the
coverage of our maps one example right
is you know Big Brother right
it sees us all the time of what we are
doing right so we could use drones UAVs
planes satellites go around the world
twice a day every single day right so
it’s very very up-to-date the imagery
now it’s not so trivial how to estimate
maps from yes aerial views okay so
hopefully thanks so what you’re gonna
see here is a system that is able to as
it drives and using aerial views
localize where the car is in terms of
the location in geo coordinates and it’s
able to also map clear HT maps of where
the roads where are the lanes where the
parking spots are in yellow here as well
as in blue where the sidewalks okay so
this is really efficient it takes three
seconds per kilometer of road so you
know in a simple of the computer cluster
you can actually map the whole world and
so again for mapping the problem is the
same as it was for cell driving cars
that again only companies like Nvidia
for this have access to data so one of
the last things that the most recent
things that we have done in the in the
lava saga is actually to create the next
big benchmark for people to compete in
to creating maps in an affordable manner
and the cool thing is a lecture
we selected you know Toronto area to be
the main area right so the way that we
created this but was by going to the
government and going to the libraries
and you seen the HD maps that have
already been created and these are
really really rich they have all sort of
very interesting metadata so we have 3d
buildings of the whole recreated Toronto
as well as the location of every single
three the location of every single pole
the shape of every traffic light every
every tree every pole in the city by an
the idea is that now we want
participants to be able to estimate all
these different things automatically
and we also generated using again deep
learning neural networks solutions that
are able to estimate this map this Maps
so here you see related results you’ve
seen only I read news right so there are
almost as good as the ground truth we
can estimate you can estimate the
topology of average streets and I
mentioned before that you know for the
localization we use these OpenStreetMaps
these are generated by people clicking
in aerial images the location of streets
and we are only 1% below what people do
for this for this task and it’s fully
automatic so you can imagine and we can
do this at a larger scale of the whole
world so you can imagine the impact of
doing things like this other things that
we can do is create centimeter accurate
maps for Nia’s Google Street View
panoramas which are available out there
for the whole world as well okay so just
to conclude I show you a couple of
things that one is to do in order to
create self-driving cars for each one of
us so things like sensing perception
localization mapping I show you our
latest coolest priority in to you know
bring in Toronto to the community at
large and there is you know many many
many things to do in research in order
to really solve all the different task
forces driving cars and you know it’s
one acknowledge the fantastic team that
is behind what I show you today and I
just want to finish also with a promise
that we will have self-driving cars in
Toronto really really sir thank you
[Music]
[Applause]
Please follow and like us: