Press "Enter" to skip to content

Unlocking the creative power of drones | Mike Roberts | TEDxBerkeley


drone cameras are everywhere in the last
few years
Hollywood cinematographers and hobbyists
have started to use this new technology
to capture shots that would be almost
impossible to get otherwise that’s an
example here’s some pretty dramatic
drone footage of the Hoover tower on the
Stanford campus you tough crowd now what
might be less obvious from watching this
clip is that it was captured fully
autonomously with no human pilot to get
this shot we used an app that I built
with my colleagues in the Stanford
computer graphics lab the app lets you
set up shots ahead of time in a virtual
environment and then capture them later
fully automatically on a real drone so
you don’t need to know how to fly a
drone at all and you can still get the
shots you want I want to share our
design process with you of how we built
the app and also how we can apply the
underlying ideas more broadly to unlock
the enormous creative power of drones
but first let’s take a step back to
understand the thinking that went into
our app we need to understand this photo
the man in the photo is the actor James
Dean and it was taken just before his
first movie came out eventually this
photo went on to become one of the most
iconic images of the 20th century but
part of what makes this photo so special
is the technical craftsmanship that went
into it here we’re seeing the original
annotations by Pablo and Ariel the
darkroom artist whose job it was to
develop the photo and in these
annotations we see just exquisite
attention to detail where he would use a
different mixture of chemicals and light
in each region of the image to highlight
the interesting stuff I mean can you
imagine putting this much effort into
your most recent brunch selfie
some people are nodding like uh yeah
definitely my point here is that
photography used to be a highly
technical endeavor as well as a creative
one fast forward to today we all have
access to essentially Studio quality
photography with a single button click
on our phones for many of us photography
has become non technical photography has
been democratized but drone photography
has not not yet here we’re seeing a
couple of expert drone pilots as though
they’re on set capturing footage for a
music video the guy on the left is
flying the drone the guy on the right is
controlling the camera and sometimes
this team even works with a third expert
whose entire job is just to control the
cameras focus and the whole time that
they’re doing this they have to use
these remote control helicopter style
joysticks that can be counterintuitive
to use so in some sense this is the
modern equivalent of spending all day in
the darkroom painstakingly developing an
analogue photo so we saw how hard it was
for people to use drones in these
demanding creative scenarios we thought
what if we built a flight planning app
specifically for drone cinematography
can we keep the creativity but remove
some of the technical barriers
associated with working with drones it
took us a while to wrap our heads around
what this meant but we eventually
figured something out so let me show you
what we came up with the most important
thing in our app is the 3d preview
window the idea here is that you set up
the shots you want by navigating around
in 3d and specifying the still framing
that you want at specific times then
behind the scenes our app synthesizes a
smooth drone trajectory that connects
the still frames and obeys the laws of
drone physics we also have this 2d map
where you can get an overview of your
shot and we have these curves that allow
you to control the timing of your shot
now so far this interface is very
similar to how an animator at Pixar we
set up a camera path in a virtual
environment but we have to do a bit of
extra stuff to make sure that our user
interface works with drone cameras and
that’s because drone cameras are
physical objects if I want to get a
drone to do anything I need to apply
just the right thrusts on each of the
propellers at just the right times it’s
like a whole symphony of gas pedals then
I need to play just right to get the
drone to do what I want and there’s
physical limits to the kinds of
maneuvers the drones can actually do so
to account for these limitations we
added these feasibility plots to the
side of our app the idea here is that as
you’re setting up a shot we compute all
the required thrusts and velocities and
that kind of thing we display them on
these plots we also display the drones
physical limits with these horizontal
red lines so if you’re setting up a shot
that exceeds your drones limits if
you’re setting up a shot that is
infeasible then you get visual feedback
right away and you can adjust your shot
accordingly then at any time during the
design process you can preview what your
shots gonna look like and because we’re
reasoning you know we’re going to some
lengths to reason about the physics in
this app these previews tend to be quite
accurate
one of the most inspirational takeaways
from this project was seeing the shots
the total beginners could create this
shot was created by one of our beginner
users that had never flown a drone
before we also brought in some super
expert users to try out our app
including some guys that get paid to
travel around the world and shoot drone
videos but this shot from the total
beginner ended up being my favorite it
was almost like the beginner users had
never had their brains polluted by how
hard it was to fly a drone manually and
their creativity could just flow however
there’s a problem I kind of swept it
under the rug so far so far in what I
showed you our our app just notifies you
if your shot is infeasible but it
doesn’t do anything to help you make it
feasible
this behavior is kind of like a
spellchecker that can tell you your
spelling a word wrong but doesn’t offer
any suggestions for how to spell it
correctly now of course in our research
paper we were like hey that’s not a bug
that’s a feature come on it’s more
expressive that way but honestly I never
thought that this was a feature I always
thought that this was a bug and to see
why it’s a bug let’s have a look at a
typical shot that you might set up in
our app and in this example I’m color
coding the shot according to whether or
not it’s feasible where blue is feasible
and red is infeasible so notice that
this shot is not even close to feasible
in the middle of the shot it requires
almost three times as much thrust and
three times as much speed as our drone
can deliver if you tried to execute this
shot on a real drone its behavior would
be unpredictable it would have to cut
corners to try to keep up with where
it’s supposed to be and it could easily
crash into this building or something
else in the environment in general
adhering to these physical limits is
important for safety so I really wanted
our app to deal with these physical
limits automatically the solution we
came up with is that the app is going to
autocorrect this shot for us but in a
way that respects the drone physics and
under the hood the app is just looking
for opportunities to just slightly bend
the timing of the shot to make it less
aggressive so here’s where we slightly
bend the timing and notice that the auto
corrected shot is bumping right up
against the physical limits but not
going past them so we’re keeping as much
of the excitement and pacing of the
original shot as possible this
autocorrect feature expands the creative
possibilities for beginner users because
it’s easy and safe now for beginners to
set up shots that are at the physical
limits of what their drone can do this
is that same shot I showed on the
previous slide after it’s been auto
corrected by our app and there’s me and
a colleague running after the drone
and this is probably the most aggressive
drone footage of the Stanford campus
that’s ever been recorded and it’s worth
mentioning that I’m not speeding up the
video at all so I’m just playing it back
at 1x speed so we’ve seen how drones can
be used to create compelling
cinematography but they can also be used
to create compelling virtual worlds for
our video games and our VR experiences
and that’s because drones can be used to
scan our physical world in 3d to
demonstrate this idea let’s try to scan
this industrial site that’s actually
just down the street from here in dare I
say a slightly less picturesque part of
Berkeley the standard way that you would
scan this site with a drone is to fly
overhead at a safe height above the
scene in kind of a lawnmower shape and
maybe you’d do an extra loop around the
scene if you had enough battery this is
what existing flight planning software
for 3d scanning would do for you today
it’s also kind of a microcosm for how
Google Earth acquires its 3d data but
our approach is gonna be a little bit
different at first we’re just gonna do
that quick loop around the scene like
this and based on this quick loop we’re
gonna reconstruct approximate scene
geometry and for this step the geometry
we reconstruct does not have to be
perfect as you can see here this
geometry is really nothing to write home
about but it turns out that it’s going
to be really useful in deciding where to
fly next so based on that approximate
geometry we generate this scanning
trajectory that’s customized and
tailor-made for this specific scene and
the idea here is that we’re gonna try to
get into all the nooks and crannies of
the scene and try to observe it from as
many different views as possible this
will make it easier for our computer
vision algorithms to do a more detailed
3d reconstruction for example this is
the best 3d reconstruction that you can
get from doing that standard overhead
lawnmower pattern
and this is the 3d reconstruction we get
from our method I’ll toggle that one
more time
standard overhead lawnmower ours so
we’re starting to get to the point where
our automatically scanned 3d models are
good enough to use in a video game or a
virtual reality experience we also
repeated this experiment one more time
at this barn and the quality
improvements that we observed or even
more pronounced so here’s the
reconstruction we got from our
trajectory and here’s the one we got
from that overhead trajectory I’ll
toggle that again one more time
ours overhead so in this talk we’ve seen
a couple of exciting examples of how we
can use drones to help us in our
creative endeavors but these are just
baby steps I’m excited about the next
generation of drone apps that don’t even
exist yet so I’m excited for a
journalist to throw a drone in the air
and immediately start covering a
breaking news story or for biologists to
start observing wildlife in totally new
ways or for archaeologists to launch a
team of drones to scan some big ancient
monument in other words I don’t just
want to unlock the creative power of
drones
I want to unlock the creative power of
you thank you
you
Please follow and like us: