humans make sense of the world in a very
sophisticated way we make use of our
five senses of seeing hearing touching
smelling and tasting and we combine all
of that information in such a way that
we can understand the world now
computers don’t have access to these
perceptual modalities computers do
things differently the stuff that they
do still amazes us if you think of what
Google does and how effortlessly it
retrieves the things that you’re looking
for or how Facebook organizes your
newsfeed or how you can order
practically and anything on Amazon it’s
amazing and we’re so used to interacting
with computers and they’re so useful to
us that we tend to kind of take them for
granted and what happens then is that we
overestimate their capabilities let me
give you an example of what I mean so if
we have these two sentences John and
Mary’s pets like playing in the yard
they constructed a fence to keep them
safe
if I give these two sentences to a human
and then I say who does they refer to
and who does them refer to we can do
this without any trouble
so we understand that John and Mary are
people and that they have pets and that
they like their pets and want to keep
them safe and protected so if you have
your pets playing around in your yard
they might run out onto the street and
maybe get hit by a car and this is
something that they don’t want to happen
so for that reason they construct a
fence to keep their pet safe so they is
John and Mary and then it’s their pets
if I give this to a machine it’s much
harder and the reason for that is that
if I grammatically analyze this sentence
there’s actually no way of knowing what
they are what them refers to the way as
humans do this is by looking at the
meaning of these words so looking at the
meaning of fence for example another way
of saying this is that what humans are
very good at is having common-sense
reasoning and this is something that
machines really struggle with so the
core to getting to common-sense is
having meaning in computer
and the famous linguist ray jackendoff
has said that meaning is the holy grail
of a variety of disciplines not only
linguistics but also philosophy
psychology and the neurosciences and I
would like to argue here that artificial
intelligence the research discipline is
very much a part of that list of
disciplines where meaning is the Holy
Grail so the end goal and the reason for
that is that we absolutely need meaning
if we want to add common-sense reasoning
so my research has been concerned with
trying to improve the way that computers
understand meaning by making them learn
to understand meaning in a more human
way one of the problems that we
absolutely need to solve if we want to
have meaning is the so called symbol
crowning problem and this is a problem
that has a very long history in
philosophy arguably going back all the
way to Plato and Aristotle but it’s most
famous recent incarnation with an
application to AI is a thought
experiment by the philosopher John
Searle and Searle says suppose you have
what he calls the Chinese room and
there’s a person inside this room with a
big book of rules that tell him what to
do when Chinese characters get input
into the room so we can have someone
standing outside of the room scribbling
Chinese characters on pieces of paper
putting them in and if the person inside
the room does a good job following the
rules you might output sequences of
Chinese characters that are completely
fluent so it might look like he’s
actually fluent in Chinese and then
Searle asked you do you think that this
person is fluent in Chinese and phrase
this way of course we’re inclined to say
no it doesn’t he’s just following the
rules in a book that’s exactly the
problem here what computers do is
something that’s very similar to this so
you might phrase this as saying can you
learn Chinese from a Chinese dictionary
that’s written completely in Chinese
because that is what computers do so the
way computers learn meaning these days
is through something called the
distributional hypothesis and the
distributional hypothesis says you shall
know a word by the company that it keeps
so let me give you an example of what
that
means we take a large corpus of text
let’s say Wikipedia and if we want to
know the meaning of cat and dog we look
at all the occurrences of these words in
Wikipedia and then we look at the
company that they keep so we can see
that cat Co occurs very often with fur
and with birth and dog Coker’s with Bart
so this allows us to see something about
what these things mean and if we add up
all of these different counts then we
can construct something that’s called a
vector space model and a vector space
model allows us to see how similar two
things are to each other so every word
is a vector and we can calculate how
similar these two things are so that
allows us to see that dog and cat are
closer to each other in meaning than
they are to moon because they occur in
more similar contexts now nice thing
about these vector spaces is that we can
visualize them in a two-dimensional plot
so every one of these points here is a
word and its location is determined by
its meaning so for example here we would
have captain politician and soldier
because they’re all humans and down
there we have animals snakes and birds
and hawks so looking at this space we
can investigate how close to things are
in meaning when we look at something
from a particular angle and the way we
learn where to put these points these
days is by using neural networks you
might have heard of neural networks or
deep learning it’s very much the hype
these days as a machine learning
technique for learning representations
and the way we do that is by trying to
predict the context which means that we
suffer from the grounding problem
because we’re defining these meanings in
terms of other words now you might say
the grounding problem is not really a
problem at all if I want to understand
the meaning of democracy I can go to
Wikipedia and I can read that it’s a
political system etc etc and in this
case for abstract words like democracy
you’re probably right
so the textual modality or the type of
information text is enough for learning
the meaning of democracy and indeed if I
switch to a different modality pictures
for example this is not really telling
me much about what democracy actually
means sure it’s an
creation so if you type democracy in
Google Images and you look at the
pictures that you get you get American
flags and things like that but that’s
not really what democracy means so so
far so good
we don’t really suffer from the
grounding problem however if I want to
understand the meaning of cat and I
again go to Wikipedia I can learn that
it’s a small carnivorous furry mammal
that people like to use as pets and sure
that’s part of what cat is to us but to
us humans the cat is much more than that
because the cat is much closer to this
and the reason for that is because of
the way we interact with the world so we
understand the meaning of cat through
the perceptual experiences that we have
so there’s a thing that we look at and
that we can hear and that we can touch
and that together makes the meaning of
cat so my research is about trying to
make computers do this so trying to make
computers use perception in a similar
way to how humans do that so how would
we go about doing that we can take
inspiration from the human brain so if I
show the cat picture to the brain what
happens is that the photons fall on your
retina and you process this in your
brain trying to make sense of it and as
you go deeper inside your brain you
become more and more abstract so you get
a concept representation of cat at some
point deep in your brain and you can
reason with that concept representation
and for example think about what would
happen if your cat is not fenced in in
the yard and gets hit by a car so we can
do a similar thing again using neural
networks which are inspired by the brain
and we show pictures of cats – neural
networks and we show lots of these
pictures and we let the neural network
learn what the meaning is of cat and
then we look at the far end at the
things that it’s learned and this allows
us to see the visual meaning of a
concept so instead of the textual
meaning that we had earlier we can now
have the visual meaning so what I’d like
to show you is some examples of what
happens when we make this transition
from textual information to visual
information so that you can really see
the difference so here we have some
concepts highlighted again in the
spacer remember every point is a worth
and his location is determined by its
meaning give and take are so close
linguistically that they actually
overlap so you only see take here that’s
how close they are but of course we know
that generous should be much closer to
give than it should be to take we also
know visually that love and valentine
should probably be closer to each other
and the same goes for white in panorama
what I’d like to show you is what
happens when we move to visual space and
then you can see the transition
happening life so wide and panorama
moves together love and Valentine give
moves towards generous and away from
take so what happens there is that you
have a different perspective on the
meaning of these concepts and in this
case their perspective is probably
closer to what you really want them what
you got from the textual modality but we
can also do this the other way around
remember I said that for democracy you
probably want to learn it from Wikipedia
and you don’t want to learn it from
Google Images so you want text rather
than vision so if we start from vision
we can see for these abstract concepts
chaos Anarchy destruction they’re kind
of evenly distributed in this space and
this is not what we want we want these
three things at the bottom to be close
and far away from order because they are
the opposites but because these concepts
are abstract it’s very hard to learn
that from pictures but if we make the
transition we can see that these
concepts move together as we turn
towards the textual perspective so this
shows that for different concepts there
are different ways of looking at things
and sometimes one way is better and
sometimes the other and so far I’ve
talked about vision and text but it’s
not restricted to that we can do the
same thing but then with audio so now
we’re thinking about what things sound
like so if I have a rocket and an
explosion demolition and fireworks they
should all be very close because they
sound like explosions
and the same goes for sounds produced by
water so splash and river and waterfall
and the same goes again for sounds
produced by humans so protests or sing
or the sound of children so we can make
that transition from text
to the audio space and we can see the
water sounds moving to the top left the
explosion sounds coming together and the
people sounds coming together so if
we’re thinking about the auditory
modality and taking an auditory
perspective on the meaning of these
words
then this is a much better way of
thinking about the meaning of those
concepts I’m not trying to say that
there’s a best way to look at the
meaning of these concepts what I’m
trying to say is I pass with so many
things the truth is probably somewhere
in the middle so what you want ideally
is to take information from all of these
different perceptual modalities and
combine it into one multimodal meeting
representation that covers all of these
different aspects of these concepts so
we can have waves and water and ocean
and surf and think about them in all of
these different modalities and then as
we make the transition towards
multimodal space these things come
together and we more accurately model
their meaning in the sense of what they
mean for humans experiments have been
done with this and and it’s shown that
if you go multimodal then you become
much closer to how humans actually think
about meaning so ultimately this is
something worthwhile pursuing and my
general point is that if we want
computers to get common sense reasoning
and if we want machines to develop human
level artificial intelligence then we
need to think about the world in a
similar way to how humans perceive the
world that is we need to make use of
these perceptual modalities in order to
learn the meaning of the things that
happen in the world around us
so I’ve talked about vision and an audio
but you can also think of making
machines smell things or taste things or
touch things and ultimately this will
get us much closer to human level
artificial intelligence so let’s return
to the example that we started with the
core here is in understanding the
meaning and one of the key words there
that we need to understand is the word
fence because the fence is something
that you use to keep things in in such a
way that they don’t run out onto the
road and get hit by a car so how would
we go about understanding the word fence
well one way to do this is we
can construct a physical robot and have
him tried to climb the fence that we
built maybe you can try to move around
it every once in a while it might
succeed in climbing over this fence and
experience what it’s like to be hit by a
car and maybe discover that that’s not
something that you want to have happen
to you but this would be very expensive
not only because we keep destroying our
robot but because it would take a very
very long time for us to learn the
things that humans take for granted if
we have to do all of these different
things individually it would take ages
but the alternative is to do this not
with the physical fence but with a
virtual fence so now we have a virtual
robot in a virtual world trying to climb
the virtual fence and maybe every once
in a while getting hit by a virtual car
but that’s not much of a problem because
we can do this an infinite amount of
times and that means that we can learn
things much more efficiently especially
with neural networks that require a lot
of data and for that reason there’s a
lot of interest in the AI community now
to be able to solve these complex
problems that you see in video games
because if AI is ever going to emerge
it’s much more likely to do so in this
virtual world than it is in our physical
world and who knows maybe one day we’ll
have machines that have common sense
thank you
[Applause]
sophisticated way we make use of our
five senses of seeing hearing touching
smelling and tasting and we combine all
of that information in such a way that
we can understand the world now
computers don’t have access to these
perceptual modalities computers do
things differently the stuff that they
do still amazes us if you think of what
Google does and how effortlessly it
retrieves the things that you’re looking
for or how Facebook organizes your
newsfeed or how you can order
practically and anything on Amazon it’s
amazing and we’re so used to interacting
with computers and they’re so useful to
us that we tend to kind of take them for
granted and what happens then is that we
overestimate their capabilities let me
give you an example of what I mean so if
we have these two sentences John and
Mary’s pets like playing in the yard
they constructed a fence to keep them
safe
if I give these two sentences to a human
and then I say who does they refer to
and who does them refer to we can do
this without any trouble
so we understand that John and Mary are
people and that they have pets and that
they like their pets and want to keep
them safe and protected so if you have
your pets playing around in your yard
they might run out onto the street and
maybe get hit by a car and this is
something that they don’t want to happen
so for that reason they construct a
fence to keep their pet safe so they is
John and Mary and then it’s their pets
if I give this to a machine it’s much
harder and the reason for that is that
if I grammatically analyze this sentence
there’s actually no way of knowing what
they are what them refers to the way as
humans do this is by looking at the
meaning of these words so looking at the
meaning of fence for example another way
of saying this is that what humans are
very good at is having common-sense
reasoning and this is something that
machines really struggle with so the
core to getting to common-sense is
having meaning in computer
and the famous linguist ray jackendoff
has said that meaning is the holy grail
of a variety of disciplines not only
linguistics but also philosophy
psychology and the neurosciences and I
would like to argue here that artificial
intelligence the research discipline is
very much a part of that list of
disciplines where meaning is the Holy
Grail so the end goal and the reason for
that is that we absolutely need meaning
if we want to add common-sense reasoning
so my research has been concerned with
trying to improve the way that computers
understand meaning by making them learn
to understand meaning in a more human
way one of the problems that we
absolutely need to solve if we want to
have meaning is the so called symbol
crowning problem and this is a problem
that has a very long history in
philosophy arguably going back all the
way to Plato and Aristotle but it’s most
famous recent incarnation with an
application to AI is a thought
experiment by the philosopher John
Searle and Searle says suppose you have
what he calls the Chinese room and
there’s a person inside this room with a
big book of rules that tell him what to
do when Chinese characters get input
into the room so we can have someone
standing outside of the room scribbling
Chinese characters on pieces of paper
putting them in and if the person inside
the room does a good job following the
rules you might output sequences of
Chinese characters that are completely
fluent so it might look like he’s
actually fluent in Chinese and then
Searle asked you do you think that this
person is fluent in Chinese and phrase
this way of course we’re inclined to say
no it doesn’t he’s just following the
rules in a book that’s exactly the
problem here what computers do is
something that’s very similar to this so
you might phrase this as saying can you
learn Chinese from a Chinese dictionary
that’s written completely in Chinese
because that is what computers do so the
way computers learn meaning these days
is through something called the
distributional hypothesis and the
distributional hypothesis says you shall
know a word by the company that it keeps
so let me give you an example of what
that
means we take a large corpus of text
let’s say Wikipedia and if we want to
know the meaning of cat and dog we look
at all the occurrences of these words in
Wikipedia and then we look at the
company that they keep so we can see
that cat Co occurs very often with fur
and with birth and dog Coker’s with Bart
so this allows us to see something about
what these things mean and if we add up
all of these different counts then we
can construct something that’s called a
vector space model and a vector space
model allows us to see how similar two
things are to each other so every word
is a vector and we can calculate how
similar these two things are so that
allows us to see that dog and cat are
closer to each other in meaning than
they are to moon because they occur in
more similar contexts now nice thing
about these vector spaces is that we can
visualize them in a two-dimensional plot
so every one of these points here is a
word and its location is determined by
its meaning so for example here we would
have captain politician and soldier
because they’re all humans and down
there we have animals snakes and birds
and hawks so looking at this space we
can investigate how close to things are
in meaning when we look at something
from a particular angle and the way we
learn where to put these points these
days is by using neural networks you
might have heard of neural networks or
deep learning it’s very much the hype
these days as a machine learning
technique for learning representations
and the way we do that is by trying to
predict the context which means that we
suffer from the grounding problem
because we’re defining these meanings in
terms of other words now you might say
the grounding problem is not really a
problem at all if I want to understand
the meaning of democracy I can go to
Wikipedia and I can read that it’s a
political system etc etc and in this
case for abstract words like democracy
you’re probably right
so the textual modality or the type of
information text is enough for learning
the meaning of democracy and indeed if I
switch to a different modality pictures
for example this is not really telling
me much about what democracy actually
means sure it’s an
creation so if you type democracy in
Google Images and you look at the
pictures that you get you get American
flags and things like that but that’s
not really what democracy means so so
far so good
we don’t really suffer from the
grounding problem however if I want to
understand the meaning of cat and I
again go to Wikipedia I can learn that
it’s a small carnivorous furry mammal
that people like to use as pets and sure
that’s part of what cat is to us but to
us humans the cat is much more than that
because the cat is much closer to this
and the reason for that is because of
the way we interact with the world so we
understand the meaning of cat through
the perceptual experiences that we have
so there’s a thing that we look at and
that we can hear and that we can touch
and that together makes the meaning of
cat so my research is about trying to
make computers do this so trying to make
computers use perception in a similar
way to how humans do that so how would
we go about doing that we can take
inspiration from the human brain so if I
show the cat picture to the brain what
happens is that the photons fall on your
retina and you process this in your
brain trying to make sense of it and as
you go deeper inside your brain you
become more and more abstract so you get
a concept representation of cat at some
point deep in your brain and you can
reason with that concept representation
and for example think about what would
happen if your cat is not fenced in in
the yard and gets hit by a car so we can
do a similar thing again using neural
networks which are inspired by the brain
and we show pictures of cats – neural
networks and we show lots of these
pictures and we let the neural network
learn what the meaning is of cat and
then we look at the far end at the
things that it’s learned and this allows
us to see the visual meaning of a
concept so instead of the textual
meaning that we had earlier we can now
have the visual meaning so what I’d like
to show you is some examples of what
happens when we make this transition
from textual information to visual
information so that you can really see
the difference so here we have some
concepts highlighted again in the
spacer remember every point is a worth
and his location is determined by its
meaning give and take are so close
linguistically that they actually
overlap so you only see take here that’s
how close they are but of course we know
that generous should be much closer to
give than it should be to take we also
know visually that love and valentine
should probably be closer to each other
and the same goes for white in panorama
what I’d like to show you is what
happens when we move to visual space and
then you can see the transition
happening life so wide and panorama
moves together love and Valentine give
moves towards generous and away from
take so what happens there is that you
have a different perspective on the
meaning of these concepts and in this
case their perspective is probably
closer to what you really want them what
you got from the textual modality but we
can also do this the other way around
remember I said that for democracy you
probably want to learn it from Wikipedia
and you don’t want to learn it from
Google Images so you want text rather
than vision so if we start from vision
we can see for these abstract concepts
chaos Anarchy destruction they’re kind
of evenly distributed in this space and
this is not what we want we want these
three things at the bottom to be close
and far away from order because they are
the opposites but because these concepts
are abstract it’s very hard to learn
that from pictures but if we make the
transition we can see that these
concepts move together as we turn
towards the textual perspective so this
shows that for different concepts there
are different ways of looking at things
and sometimes one way is better and
sometimes the other and so far I’ve
talked about vision and text but it’s
not restricted to that we can do the
same thing but then with audio so now
we’re thinking about what things sound
like so if I have a rocket and an
explosion demolition and fireworks they
should all be very close because they
sound like explosions
and the same goes for sounds produced by
water so splash and river and waterfall
and the same goes again for sounds
produced by humans so protests or sing
or the sound of children so we can make
that transition from text
to the audio space and we can see the
water sounds moving to the top left the
explosion sounds coming together and the
people sounds coming together so if
we’re thinking about the auditory
modality and taking an auditory
perspective on the meaning of these
words
then this is a much better way of
thinking about the meaning of those
concepts I’m not trying to say that
there’s a best way to look at the
meaning of these concepts what I’m
trying to say is I pass with so many
things the truth is probably somewhere
in the middle so what you want ideally
is to take information from all of these
different perceptual modalities and
combine it into one multimodal meeting
representation that covers all of these
different aspects of these concepts so
we can have waves and water and ocean
and surf and think about them in all of
these different modalities and then as
we make the transition towards
multimodal space these things come
together and we more accurately model
their meaning in the sense of what they
mean for humans experiments have been
done with this and and it’s shown that
if you go multimodal then you become
much closer to how humans actually think
about meaning so ultimately this is
something worthwhile pursuing and my
general point is that if we want
computers to get common sense reasoning
and if we want machines to develop human
level artificial intelligence then we
need to think about the world in a
similar way to how humans perceive the
world that is we need to make use of
these perceptual modalities in order to
learn the meaning of the things that
happen in the world around us
so I’ve talked about vision and an audio
but you can also think of making
machines smell things or taste things or
touch things and ultimately this will
get us much closer to human level
artificial intelligence so let’s return
to the example that we started with the
core here is in understanding the
meaning and one of the key words there
that we need to understand is the word
fence because the fence is something
that you use to keep things in in such a
way that they don’t run out onto the
road and get hit by a car so how would
we go about understanding the word fence
well one way to do this is we
can construct a physical robot and have
him tried to climb the fence that we
built maybe you can try to move around
it every once in a while it might
succeed in climbing over this fence and
experience what it’s like to be hit by a
car and maybe discover that that’s not
something that you want to have happen
to you but this would be very expensive
not only because we keep destroying our
robot but because it would take a very
very long time for us to learn the
things that humans take for granted if
we have to do all of these different
things individually it would take ages
but the alternative is to do this not
with the physical fence but with a
virtual fence so now we have a virtual
robot in a virtual world trying to climb
the virtual fence and maybe every once
in a while getting hit by a virtual car
but that’s not much of a problem because
we can do this an infinite amount of
times and that means that we can learn
things much more efficiently especially
with neural networks that require a lot
of data and for that reason there’s a
lot of interest in the AI community now
to be able to solve these complex
problems that you see in video games
because if AI is ever going to emerge
it’s much more likely to do so in this
virtual world than it is in our physical
world and who knows maybe one day we’ll
have machines that have common sense
thank you
[Applause]
Please follow and like us: