[Music]
about 12 years ago I moved to New York
City with the hope that I would become a
photographer what I realized was that I
did not want to do photography
photography to me became a way of
enforcing power a way of capturing
something in a way of exerting control
over other people so a few years later I
started down a different path and I
began exploring ways to defeat
photography when you’re looking at this
slide here you see a computer vision
algorithm analyzing an image looking for
faces and in 2010 I developed this
proof-of-concept project thinking about
a future in a more machine-readable
world where our faces are being analyzed
more often and computer vision now
exerts control over those images
captured through photography and to me
photography which is done with a camera
became inseparable two surveillance
cameras so my interest shifted away from
thinking about how to capture the world
into an experiment like this we’re all
thinking about how to disappear into the
world and what do you see happening here
is on the left the face is detected and
you can see that by the number of
confidence scores overlapping it each
green box represents one possible face
and on the right you see not many
confidence boxes as this very slow down
version of face detection plays through
what you see is that on the right there
is no face detected and this project
comes out of kind of reverse engineering
the way that a face detection algorithm
works and then undermining it exploiting
that vulnerability through something low
costs that could be simple such as hair
makeup or fashion styling
so now let’s look through the algorithm
and see what the face detection
algorithm is looking at here we can see
a heat map overlay showing on the left
very very face like image and on the
right an on face like image an important
concept to realize with computer vision
is that there’s no such thing as an
absolute face you only have more face
like and less face like and what do you
see here is how that plays out through
the algorithms reading of an image and
that’s an important concept to
understand if you’re interested in
creating ways of disappearing or types
of computer vision camouflage that
camouflage in this case is about
understanding the threshold of detection
at what point a face becomes not a face
and then figuring out how to appear just
one step below that I took this way of
thinking to another kind of imaging in
2013 here looking at thermal imaging
thermal imaging looks at the heat
radiated by a body which emits about 100
watts it’s a very effective way to look
at someone over a long distance and
especially from a drone a military drone
the idea with this project is to explore
is there a way to disappear to exist
under that threshold of detection of
military observation and the solution an
experiment and a provocation is called
stealth wear and it looks at using
silver-plated fabrics to shield the
thermal energy emitted from a body and
here you see the title of this one is
called the anti drone burqa deliberately
provocative but the idea here is to
reimagine a religious garment in the age
of mass surveillance and military
surveillance and think about it instead
of providing a separation between man
and God providing a separation between
and drone and there’s also a hijab as
part of that collection now you can look
at this image from thermal test footage
which shows a few people standing around
a van and one of the people is learning
the anti drone burka if you look in the
center you can see a slight difference
in the tonality and now when I play it
you’ll see the motion and it’ll be very
clear and the legs were deliberately
left not concealed but of course now
that you have the idea and the
technology that thermal imaging can be
blocked you have the solution after that
project I began thinking what’s
happening next and what’s the big
picture where are we headed in terms of
a machine-readable world with computer
vision analyzing more of our appearances
on social media but also more in the
physical world and it began looking at
what does it mean to be captured as a
pixel how much data can you represent in
a pixel and so to understand what it
means to be captured and represented as
a pixel I want to talk about how much
data can be represented in something
that normally we would think of as very
boring one pixel is not impressive
compared to the capabilities of your 8
megapixel camera phone but one pixel
contains 256 different values in
grayscale if we were to look at that and
try to understand the difference between
two colors that to a computer vision
algorithm are completely different we
can’t interpret those so this space of
256 variables not all of them are
legible to a person while they are to a
computer until we put text on it showing
each different color now we can do an
experiment with just one pixel of
information we can encode nearly every
letter in the times in Roman font said
with about 97% uniqueness we find out
that the most unique character is M
meaning it has the highest amount of
separation from all the other characters
probably thinking this is not that
interesting except a lot of people post
texts that’s been pixelated online maybe
you have I find it and I use it for my
own research looking at how dangerous is
it to post pixelated text because that’s
not a redaction it’s a reduction in
information and in this experiment you
can see up on the left hand side the
pixelated text which is only 12 pixels
when you run that against a genetic
algorithm you discover that those 12
pixels said Ernest now if we go up to 2
by 2 pixels that’s still not very
impressive but if you were to look at
every combination of 2 by 2 pixels so
here we have about 4.3 million
combinations which would take you about
136 years to look at every single
combination that you can represent a 2×2
I can speed up that animation and would
only take 13 years to look at every
combination of it now if we go up to 4
by 4 the amount of variables that can be
represented in this space increases
exponentially and we’re already at a
point that exceeds the capacity of the
entire human races perceptual
capabilities meaning if you tasked every
person on the planet with looking at
that image for their entire life we
would never succeed in looking at every
combination which is to say that’s a lot
of information now if we go to 8×8
that’s even more information and you can
represent quite a lot of different
things at a very low resolution
if we go to color we have to the third
more possibilities so often in computer
vision that’s actually too much
information and you’ve reduced that back
to grayscale because you have so many
representations just in grayscale that
8×8 what can we do with six by seven
pixels we can do face recognition on on
a sample size of about 50 people with
95% accuracy we can build a visual
search engine
reducing every image to about 8 by 8
pixels this is called the perceptual
hash and that’s how you search for
copyright infringement when we go to 14
by 14 we only have about a 18 percent
reduction in performance and a facial
recognition system compared to the
original face when we go up to 12 by 16
we can begin to encode detect human
activity which brings up the question if
you are building a system to detect
motion to look at whether people are
standing or walking or riding a bike if
you can do that at 12 by 16 do you
really need to store HD footage at 1920
by 1080 and at 24 by 24 we have the
definition of the boundaries and the
parameters of the human face when we go
up to 40 by 40 this is all the amount of
information needed to do facial analysis
many smartphone apps that will show you
things like age estimation emotion that
will find your face and an image forty
pixels and as we increase that space up
to 100 100 by 100 pixels 10,000 pixels
in total this is everything that you
need to do face recognition at scale
meaning government agencies law
enforcement agencies when they look at
you
it’s been 100 pixels days now of course
we go to color and this is more similar
to where we’re capturing images and
posting them if we take that 100 by 100
pixel image and compare it to the amount
of information posted on Instagram
that’s only 2.5 percent of one photo a
very small of information leads to a
very large amount of information and
some of it can be quite uncomfortable
here’s a research study that came out
last year looking at inferring
criminality basically an updated version
of phrenology but with AI so here in
this example it’s looking at the lip
curvature and it’s basing the way that
you look
against a database of criminals and if
you look like a criminal you have a
higher criminal score not fair is it now
here’s another one that’s looking at
psychological attributes and again it’s
an inference if you look like someone
who’s intelligent if you look like
someone who’s kind chances are you may
also be kind according to this algorithm
and some of the labels they have are
literally weird one of the labels is
weird intelligent kind so being able to
infer this information from a 100 by 100
pixel space remotely meaning I could
capture one image and do that to every
single person in the room right now
other things that you could do predict
decision-making capabilities with about
a 20% higher accuracy than people and
here’s a company that eventually sold
their product to Google in 2014 called
jetpack and what’s really interesting
here is what they’re doing is analyzing
images on Instagram and you can see the
guy with a mustache but you probably
can’t guess why that’s valuable well
they use that image to detect where
there might be a hipster cafe
and then they build a visual guidebook
so if you’re visiting that city you
could go to the hipster cafe the image
on the bottom is even better this one is
looking at girls wearing lipstick and
the location of the photo posted to tell
guys were to find bars with single women
now this one I think is great looking at
with computer vision whether someone is
going to be a high-performing CEO based
on the width of their mouth but it’s
okay if you don’t have a wide mouth as
long as they’re interested in running an
NGO because the opposite is true for
nonprofit agencies a narrower mouth is
good these are just a few of many
research papers that have come out in
the last few years that are mining
attributes inferring attributes from
images appearing mostly on social media
and here’s a glance at what some of
those datasets look like one of them is
quite unbelievable which is Facebook
does not have only four thousand images
but in the publicly available ones the
number is always going up and the one in
particular that the research paper is
from Microsoft recently published a data
set of about 10 million photos with a
hundred thousand identities where do
these people come from celebrities
photos posted online that’s what’s being
used to train algorithms that’s what’s
being used to train new products for
emotion recognition and that’s also
what’s being used to train government
surveillance programs here’s a slide
from 2014 and this one if you look at
the pictures on the bottom they’re
babies these are images being used to
hone government surveillance
programs unfortunately for people in
Lithuania
it does include you as a US citizen for
some reason I’m excluded from this
search uncomfortable things happening
where do these images come from Twitter
Instagram Facebook of course Flickr so
this is what happens when photos are
posted online
this is who’s looking at them and when
you have enough photos you can create
more photos you can create resolution
through those photos what’s happening
now is a field of study called
generative adversarial networks and
super-resolution were instead of taking
a 100 by 100 pixel image you take an 11
by 11 pixel image and then you
synthesize the larger image so starting
with 11 by 11 you can generate a 3d
profile of someone then you can change
lighting and the pose and then you can
build a more robust facial recognition
profile from 11 by 11 pixels is this
still photography at this point no this
is a visual database and these are
visual databases that hold the unique
story of your biometrics your friends
biometrics your kids biometrics your
location your emotion and your movements
many of your life’s moments can be
recorded and reduced into a very small
pixel space but I think that if we look
critically at this pixel space as images
as a visual database and if we look
deeper into these pixels we can begin to
understand how we’re appearing to them
and we can also begin to imagine new
technologies for enhancing privacy and
information
security and possibly also imagine new
ways of appearing into the world and
disappearing into the world thank you
[Music]
about 12 years ago I moved to New York
City with the hope that I would become a
photographer what I realized was that I
did not want to do photography
photography to me became a way of
enforcing power a way of capturing
something in a way of exerting control
over other people so a few years later I
started down a different path and I
began exploring ways to defeat
photography when you’re looking at this
slide here you see a computer vision
algorithm analyzing an image looking for
faces and in 2010 I developed this
proof-of-concept project thinking about
a future in a more machine-readable
world where our faces are being analyzed
more often and computer vision now
exerts control over those images
captured through photography and to me
photography which is done with a camera
became inseparable two surveillance
cameras so my interest shifted away from
thinking about how to capture the world
into an experiment like this we’re all
thinking about how to disappear into the
world and what do you see happening here
is on the left the face is detected and
you can see that by the number of
confidence scores overlapping it each
green box represents one possible face
and on the right you see not many
confidence boxes as this very slow down
version of face detection plays through
what you see is that on the right there
is no face detected and this project
comes out of kind of reverse engineering
the way that a face detection algorithm
works and then undermining it exploiting
that vulnerability through something low
costs that could be simple such as hair
makeup or fashion styling
so now let’s look through the algorithm
and see what the face detection
algorithm is looking at here we can see
a heat map overlay showing on the left
very very face like image and on the
right an on face like image an important
concept to realize with computer vision
is that there’s no such thing as an
absolute face you only have more face
like and less face like and what do you
see here is how that plays out through
the algorithms reading of an image and
that’s an important concept to
understand if you’re interested in
creating ways of disappearing or types
of computer vision camouflage that
camouflage in this case is about
understanding the threshold of detection
at what point a face becomes not a face
and then figuring out how to appear just
one step below that I took this way of
thinking to another kind of imaging in
2013 here looking at thermal imaging
thermal imaging looks at the heat
radiated by a body which emits about 100
watts it’s a very effective way to look
at someone over a long distance and
especially from a drone a military drone
the idea with this project is to explore
is there a way to disappear to exist
under that threshold of detection of
military observation and the solution an
experiment and a provocation is called
stealth wear and it looks at using
silver-plated fabrics to shield the
thermal energy emitted from a body and
here you see the title of this one is
called the anti drone burqa deliberately
provocative but the idea here is to
reimagine a religious garment in the age
of mass surveillance and military
surveillance and think about it instead
of providing a separation between man
and God providing a separation between
and drone and there’s also a hijab as
part of that collection now you can look
at this image from thermal test footage
which shows a few people standing around
a van and one of the people is learning
the anti drone burka if you look in the
center you can see a slight difference
in the tonality and now when I play it
you’ll see the motion and it’ll be very
clear and the legs were deliberately
left not concealed but of course now
that you have the idea and the
technology that thermal imaging can be
blocked you have the solution after that
project I began thinking what’s
happening next and what’s the big
picture where are we headed in terms of
a machine-readable world with computer
vision analyzing more of our appearances
on social media but also more in the
physical world and it began looking at
what does it mean to be captured as a
pixel how much data can you represent in
a pixel and so to understand what it
means to be captured and represented as
a pixel I want to talk about how much
data can be represented in something
that normally we would think of as very
boring one pixel is not impressive
compared to the capabilities of your 8
megapixel camera phone but one pixel
contains 256 different values in
grayscale if we were to look at that and
try to understand the difference between
two colors that to a computer vision
algorithm are completely different we
can’t interpret those so this space of
256 variables not all of them are
legible to a person while they are to a
computer until we put text on it showing
each different color now we can do an
experiment with just one pixel of
information we can encode nearly every
letter in the times in Roman font said
with about 97% uniqueness we find out
that the most unique character is M
meaning it has the highest amount of
separation from all the other characters
probably thinking this is not that
interesting except a lot of people post
texts that’s been pixelated online maybe
you have I find it and I use it for my
own research looking at how dangerous is
it to post pixelated text because that’s
not a redaction it’s a reduction in
information and in this experiment you
can see up on the left hand side the
pixelated text which is only 12 pixels
when you run that against a genetic
algorithm you discover that those 12
pixels said Ernest now if we go up to 2
by 2 pixels that’s still not very
impressive but if you were to look at
every combination of 2 by 2 pixels so
here we have about 4.3 million
combinations which would take you about
136 years to look at every single
combination that you can represent a 2×2
I can speed up that animation and would
only take 13 years to look at every
combination of it now if we go up to 4
by 4 the amount of variables that can be
represented in this space increases
exponentially and we’re already at a
point that exceeds the capacity of the
entire human races perceptual
capabilities meaning if you tasked every
person on the planet with looking at
that image for their entire life we
would never succeed in looking at every
combination which is to say that’s a lot
of information now if we go to 8×8
that’s even more information and you can
represent quite a lot of different
things at a very low resolution
if we go to color we have to the third
more possibilities so often in computer
vision that’s actually too much
information and you’ve reduced that back
to grayscale because you have so many
representations just in grayscale that
8×8 what can we do with six by seven
pixels we can do face recognition on on
a sample size of about 50 people with
95% accuracy we can build a visual
search engine
reducing every image to about 8 by 8
pixels this is called the perceptual
hash and that’s how you search for
copyright infringement when we go to 14
by 14 we only have about a 18 percent
reduction in performance and a facial
recognition system compared to the
original face when we go up to 12 by 16
we can begin to encode detect human
activity which brings up the question if
you are building a system to detect
motion to look at whether people are
standing or walking or riding a bike if
you can do that at 12 by 16 do you
really need to store HD footage at 1920
by 1080 and at 24 by 24 we have the
definition of the boundaries and the
parameters of the human face when we go
up to 40 by 40 this is all the amount of
information needed to do facial analysis
many smartphone apps that will show you
things like age estimation emotion that
will find your face and an image forty
pixels and as we increase that space up
to 100 100 by 100 pixels 10,000 pixels
in total this is everything that you
need to do face recognition at scale
meaning government agencies law
enforcement agencies when they look at
you
it’s been 100 pixels days now of course
we go to color and this is more similar
to where we’re capturing images and
posting them if we take that 100 by 100
pixel image and compare it to the amount
of information posted on Instagram
that’s only 2.5 percent of one photo a
very small of information leads to a
very large amount of information and
some of it can be quite uncomfortable
here’s a research study that came out
last year looking at inferring
criminality basically an updated version
of phrenology but with AI so here in
this example it’s looking at the lip
curvature and it’s basing the way that
you look
against a database of criminals and if
you look like a criminal you have a
higher criminal score not fair is it now
here’s another one that’s looking at
psychological attributes and again it’s
an inference if you look like someone
who’s intelligent if you look like
someone who’s kind chances are you may
also be kind according to this algorithm
and some of the labels they have are
literally weird one of the labels is
weird intelligent kind so being able to
infer this information from a 100 by 100
pixel space remotely meaning I could
capture one image and do that to every
single person in the room right now
other things that you could do predict
decision-making capabilities with about
a 20% higher accuracy than people and
here’s a company that eventually sold
their product to Google in 2014 called
jetpack and what’s really interesting
here is what they’re doing is analyzing
images on Instagram and you can see the
guy with a mustache but you probably
can’t guess why that’s valuable well
they use that image to detect where
there might be a hipster cafe
and then they build a visual guidebook
so if you’re visiting that city you
could go to the hipster cafe the image
on the bottom is even better this one is
looking at girls wearing lipstick and
the location of the photo posted to tell
guys were to find bars with single women
now this one I think is great looking at
with computer vision whether someone is
going to be a high-performing CEO based
on the width of their mouth but it’s
okay if you don’t have a wide mouth as
long as they’re interested in running an
NGO because the opposite is true for
nonprofit agencies a narrower mouth is
good these are just a few of many
research papers that have come out in
the last few years that are mining
attributes inferring attributes from
images appearing mostly on social media
and here’s a glance at what some of
those datasets look like one of them is
quite unbelievable which is Facebook
does not have only four thousand images
but in the publicly available ones the
number is always going up and the one in
particular that the research paper is
from Microsoft recently published a data
set of about 10 million photos with a
hundred thousand identities where do
these people come from celebrities
photos posted online that’s what’s being
used to train algorithms that’s what’s
being used to train new products for
emotion recognition and that’s also
what’s being used to train government
surveillance programs here’s a slide
from 2014 and this one if you look at
the pictures on the bottom they’re
babies these are images being used to
hone government surveillance
programs unfortunately for people in
Lithuania
it does include you as a US citizen for
some reason I’m excluded from this
search uncomfortable things happening
where do these images come from Twitter
Instagram Facebook of course Flickr so
this is what happens when photos are
posted online
this is who’s looking at them and when
you have enough photos you can create
more photos you can create resolution
through those photos what’s happening
now is a field of study called
generative adversarial networks and
super-resolution were instead of taking
a 100 by 100 pixel image you take an 11
by 11 pixel image and then you
synthesize the larger image so starting
with 11 by 11 you can generate a 3d
profile of someone then you can change
lighting and the pose and then you can
build a more robust facial recognition
profile from 11 by 11 pixels is this
still photography at this point no this
is a visual database and these are
visual databases that hold the unique
story of your biometrics your friends
biometrics your kids biometrics your
location your emotion and your movements
many of your life’s moments can be
recorded and reduced into a very small
pixel space but I think that if we look
critically at this pixel space as images
as a visual database and if we look
deeper into these pixels we can begin to
understand how we’re appearing to them
and we can also begin to imagine new
technologies for enhancing privacy and
information
security and possibly also imagine new
ways of appearing into the world and
disappearing into the world thank you
[Music]
Please follow and like us: