Google brain on the magenta project with
magenta project we’re seeking to answer
the questions can machine learning be
used to create compelling pieces of
music and art if so how and if not why
not now these are easy questions to ask
but surprisingly difficult to answer a
truly compelling piece of music or art
transcends a generated data point music
and art connects us to one another
engages us surprises us and provides a
means of expression for some of our most
personal and powerful stories can a
machine learn to do that well it turns
out we aren’t the first people to ask
these questions people have been trying
to compose music using algorithms for a
long time back in the 1700s musical dice
games were very popular
they allowed any player to compose a
piece of music just by rolling some dice
now of course the compositions that
these games produced were limited to the
snippets of music written by the creator
of the game so in the end most of the
composition sounded fairly similar to
one another one way to solve this
problem of limited compositions is to
make the rules for producing the
compositions much more complex as far
back as the 1800s Ada Lovelace was
writing down ideas about how computers
could one day be used to create
elaborate and scientific pieces of music
in the 1950s researchers at the
University of Illinois I finally got a
chance to try out some of those ideas
they use there iliacs supercomputer to
write music for a string quartet and the
music that they produced was much more
interesting than complex than dice game
music and researchers and musicians have
been exploring those ideas ever since
however this approach still has a
fundamental limitation the computer can
only write music that follows the rules
of the algorithm programmed into it so
if you’ve carefully written an algorithm
for producing string quartet music that
same algorithm isn’t going to wake up
one day and decide to write some dubstep
what if there was a way to have
computers write music the same way that
humans write music so instead of using
handcrafted algorithms and the way that
humans write music is first we listen to
music thousands and thousands of hours
of music from the earliest stages of our
lives
subconsciously picking up on patterns
and learning about different genres then
when we want to write something new
we can draw on that experience and our
own inspiration to write a new
composition this idea of having
computers learn not from handcrafted
algorithms but from data is called
machine learning
but how can a machine learn from data
there are many ways that this can be
done but let me give you one quick
example let’s say that you wanted to
write a system that could tell you
whether a given photo was of a cat or of
an elephant and let’s say that you
already had systems in place that could
recognize features such as fertilize if
you’ve had enough photos of elephants
into this system only telling the system
that these are photos of elephants over
time the system would learn that
features like tusks and eyes are highly
correlated with an elephant being in the
photo similarly if you fit in enough
photos of cats over time the system
would learn that fur and eyes are highly
correlated with a cat being in the photo
now in this system the the important
point is that we didn’t write an
algorithm that listed out the specific
features for each animal instead the
system learned on its own which features
correspond to which animal and it even
learned that some features in this case
eyes correspond to both animals now in
this example we have just a few of these
kinds of nodes but if you take many of
these kinds of nodes sometimes many
thousands of these nodes and arrange
them in multiple layers you create
what’s sometimes called an artificial
neural network now it’s called this
because it’s loosely based on the way
that biological neurons work together in
the human brain to solve problems now
the system that we have here has has
many flaws there are many situations
where it won’t work for example not all
cats have fur and I’m sure you could
come up with a long list of other
examples that’s why for real image
recognition systems we take those
neurons that I was talking about we
arrange them in multiple layers and then
we let the system learn on its own which
features are important for recognizing
objects and photos usually the first few
layers learn to recognize things like
lines and basic shapes then the
intermediate layers are able to piece
together those lines in basic shapes and
recognize more complex shapes and
patterns and by the time that we get to
the final few layers it’s able to detect
things like spirals there’s something
there that looks kind of like a bird
maybe some honeycomb in the lower left
all right so now we have a way for
machines to learn based on data but how
can we apply that machine learning to
creation of art here’s an example of art
created by another team at Google they
call their system deep dream it works by
taking a neural network that’s trained
to recognize objects and faces and it
runs that network on an image in such a
way that it dreams up objects and faces
that aren’t actually there and in fact
there was a recent charity art show in
San Francisco where people paid
thousands of dollars to take home deep
dream generated artwork so clearly we’re
at a point where machine generated art
is able to make some kind of human
artistic connection all right now how do
we apply machine learning and neural
networks to music creation one way to
approach this problem is as a prediction
problem so you can say given a sequence
of notes predict what the most likely
next note is in order to do this we’ll
need to use a slightly different kind of
neural network than the one I’ve been
talking about so far we’ll need to use
what’s called a recurrent neural network
and this type of network has access to
not only the information that it’s
currently being presented with so in
this case the most recent note of the
melody but it also has access to a
representation of information in the
past so the previous notes of the melody
with this type of network you can ask
the question given a sequence of notes
what’s the next note and then if you
repeat that process and ask it to keep
predicting the next note you can
generate a whole new melody all right so
now we’ve got a way to train a machine
to produce a melody we just need some
training data so we collected a library
of songs tens of thousands of songs
everything from Bach to Radiohead and we
took the melody lines from those songs
and we fed them into the network one
note at a time and then we as a as a
starting point we gave the network the
first four notes of twinkle twinkle
little star dun dun dun dun and asked it
to continue the melody and this is what
[Music]
all right go to our website you can
listen to the rest of it there all right
so that’s pretty good but we wanted to
explore ways that we could make melody
generation even better focusing on just
next step note prediction has a few
important limitations one is long term
structure if you’re only asking the
model to predict the next note it has
difficulty coming up with things that we
like to listen for in music like
repeated motifs and longer melodic arcs
another problem is repetition if your
training data has a lot of repeated
notes and you can hear this for example
even in twinkle twinkle little star dun
dun dun dun then over time the model
might learn that given a particular note
if you ask it to predict the next note a
reasonable prediction might be well just
repeat that same note similarly if the
last thing that happened was arrest or
silence a reasonable prediction might be
well just add another rest extend the
silence and this works well in the
short-term but as you can imagine it
breaks down in the long-term so one of
our interns this last summer Natasha
Jake’s came up with the idea of using a
technique called reinforcement learning
to improve our models now reinforcement
learning works a little bit different
than the training method I’ve been
talking about so far instead of showing
the model a list of examples and saying
these examples are correct instead we
ask the model to produce some output and
then we give it a reward mathematically
speaking based on how well it did this
is conceptually similar to the idea the
composition student and their teacher
the student writes composition and then
the teacher gives them feedback on how
good or bad the composition is so using
this system over time the model will
learn to favor producing outputs that
receive a positive reward now of course
the important part of this is we have to
define what sounds good is we certainly
didn’t have time for a human to sit
around and listen to all of the
thousands upon thousands of melodies
that the model produces and rate them so
what we did is we found a composition
textbook and we just took the list of it
listed some rules for what makes a good
melody and we took that list and
programmed it in and of course the first
rule that it lists is avoid excessive
repeated notes just one of the problems
we were trying to solve ok so let’s see
how well we did first I’ll play you an
example of what the model produced
before reinforcement learning
you’ll notice that the melody isn’t very
end
all right and here’s what it sounds like
after reinforcement learning all right a
lot more interesting after we posted
that result online MIT Technology Review
said that that particular melody sounded
a bit like a jingle from the latest
toothpaste campaign which we take as a
high compliment all right so that’s far
I’ve been talking about music but
magenta is interested in visual art as
well
a recent breakthrough in machine
learning has been the ability to
transfer the style of one image to
another so for example you can see what
any given photo looks like painted in
the style of starry night we wanted to
take this idea a little bit further and
so in collaboration with another team at
Google brain we developed a network that
could learn multiple styles and mix them
together this is a video of peekaboo
my teammates dog transformed to look
like several different painting styles
so you’ll see the the slider for starry
night just got turned up to 100% it’s
being mixed with a few other painting
styles and then we’ll remove the starry
night style from the transformed image
all right I can go to the next slide
okay and we’re working on a bunch of
other projects too but I don’t have time
to talk about all of them today so what
we’ve got here is pretty good but
remember what I talked about at the
beginning about how music and art
connects us to one another engages us
surprises us has what we’ve done here
accomplished that perhaps not on its own
but we believe the key to magentas
success will be the community we
recognize that music and art is too big
of a problem to tackle on our own so
we’re building a community of musicians
coders machine learning researchers to
work on this together we’ve posted the
code for our models up on github and we
encourage anyone to try it out tweak it
use it improve it in ways that maybe we
wouldn’t have thought of here’s a great
example of how community can take a
relatively simple beginning and
transform it into something really
interesting
you remember that tune from earlier done
dunh dunh well when we posted that
example online Jason Wynn who studies
Vietnamese traditional music wanted to
play along with it so he and another
artist collaborated and created this
video
all right that’s great we love how the
tune the magenta created inspired these
musicians to come together and create
something new next slide please
and we wanted to expand on this idea of
musicians interacting with magenta so
for the neural information processing
systems conference this past December we
put together a demo that allows a
musician to play with magenta in
real-time so in this video and the
electric piano sound that you hear will
be the human performer plain and melody
and then magenta will respond in
alright next like this we think that
these kinds of interactions are where
things get really interesting with
magenta we think that machine learning
applied to music and art is going to be
similar to win Rickenbacker and Gibson
figured out the electric guitar they
created this new musical instrument but
really all they were trying to do was
come up with a way to make the acoustic
guitar sound a little bit louder they
could have never imagined what artists
like Jimi Hendrix or Saint Vincent would
end up doing with it
similarly we think that the tools that
machine learning researchers will come
up with for working with music and art
will be used by artists in ways that
those researchers could have never
expected and I hope that some of those
artists are right here today everything
that I’ve talked about is up on our
website the code is all up on github you
can download all of these experiments
and run them on your own computer see
for yourself how they work try them out
maybe you’ve got a great idea for how
much into could be used to create new
music and art go ahead try it out I’m
looking forward to seeing what you create thank you