[Applause]
you have almost all been subject to a
computerized predictive algorithm of the
following form if you like X you may
like Y music movie streaming services
online shopping services routinely look
at your pattern of behavior compares
that to previous customers and tries to
make predictions for you sometimes it
works and sometimes it doesn’t work now
you may be less aware that you’ve also
may have been subjected to a predictive
algorithm of the form if you are like X
we may not give you a loan banks
financial institutions routinely use
your personal attributes compares them
to previous customers to assess the risk
to the institution in terms of giving
you a loan or mortgage you may have also
recently been subjected to a predictive
algorithm of the form if you are like ax
we may not give you a job employers and
in some cases university admissions are
using predictive algorithms to make
hiring and admission decisions and if
you’ve recently had a run-in with the
criminal justice system
well I’m sorry first of all and second
of all you may have been subjected to a
predictive algorithm of the following
form if you are like X then you may go
to jail courts now are routinely using
predictive algorithms to make decisions
in the courtroom one particularly
popular and now widespread algorithm is
used at the point of arrest to make bail
decisions and conceptually it’s a pretty
simple idea a defendant who’s been
charged with a crime is information is
extracted from them and is fed into a
computer algorithm and that computer
algorithm outputs a risk factor and that
risk factor is meant to quantify the
likelihood that person will commit a
crime in the future think of a really
simple version of the Minority Report
without the creepy people in the pool
and if you are assessed to be high-risk
well then the judge may deny bail and
you will be held in prison awaiting
trial and if you are assessed to be
low-risk then you may be released
pending trial now in 2016 investigative
journalists from ProPublica published a
troubling report on one particularly
popular and widespread predictive
algorithm that is used in the courts
around this country and what they found
is that this particular algorithm was
significantly disadvantageous to black
defendants what they found is that if
you were a black defendant you are
nearly twice as likely to be predicted
as reoffending when you did not that
happened 44 percent of the time if you
were black and only 23 percent of the
time if you are white that is a
staggering difference and at the same
time you were almost twice as likely to
be predicted as not offending when in
fact you did if you were white as
compared to black that happened 47
percent of the time versus 28 percent of
time so in other words the algorithm is
biased to black defendants against black
defendants in other words if you are
black and you are a subject to this
algorithm you are more likely to be
assessed as high-risk and incarcerated
than if you are white so shortly after
this article was published a student of
mine Julia Dressel came to me and she
said how is this possible
we’re computer scientists how are we
allowing this to happen how are we
allowing software to make life-altering
decisions on people when they have these
types of inequalities and it was the
right question to ask and it got us
spending about a year and a half journey
trying to understand what was happening
here and how and if we can do better and
I want to tell you a little bit about
that journey and a little bit about what
we learned along the way
so before I dive into the algorithmic
details and sort of understand where
this bias is coming from we started with
a really simple question
we started with the question of are
these algorithms actually better than
you than the average person because
after all presumably we use these
algorithms because they’re more accurate
more objective and more fair and they
remove any biases that may exist in the
human mind and that’s a good idea but we
don’t know if it actually is true or not
so we set out to determine that so
here’s what we did we recruited four
hundred participants from an online
crowdsourcing platform and we asked him
a very simple question
we gave them a short paragraph about an
actual defendant this was a with 8,000
people for whom we had demographic
information we had information about
their past criminal history if any um
and then we knew the risk score
according to the software that is being
used in the court and then they were
tracked for two years so we actually
know if they reoffended or not and so
our participants read a short paragraph
was seven pieces of information the age
and the sex of the person and then five
pieces of data that summarized their
juvenile and adult criminal history they
did not know the race of the person
importantly okay and then we simply
asked them please read this short
paragraph it’s an online survey and just
tell us if you think this person will
reoffended
in a two-year window following their
last arrest yes no yes no yes no here’s
what we learned the computerized
software had an accuracy of 65% our
humans had an accuracy of 67% these are
random people on the internet responding
to a survey for a buck and they spent
about 10 seconds reading that little
short paragraph and making a
determination they are as good as the
commercial software being used in the
courts today oddly they made the same
errors as a software they were more
likely to say that a black defendant at
a rate of 37 percent would reoffended
–nt compared to 27 percent for white
defendants and they’re more likely to
say that a white defendant would not
reoffended at a rate of 40 versus 29
percent how is this possible they don’t
know the race of the person and by the
way neither does the software so both
the humans and the software have a
racial bias and they
no the race that’s weird okay so first
of all it’s a little troubling that
random people on the internet are as
accurate as a software that’s been
deployed in our courts and we’re gonna
talk some more about that in a minute
but there’s two questions that came out
of this one is how is it that humans and
the software have a racial bias when
they don’t know the race of the person
and number two is how is it that random
people on the internet being paid a buck
to answer our survey question are as
good as commercial software that is
being deployed in our course and we
needed to answer those two questions now
at this stage what we should have done
is reached into this commercial software
this predictive algorithm and asked how
does it work what’s going on in here
what is it what information is it using
how is it doing that mathematically and
really understand how the algorithm is
making the mistakes and why it’s not
more accurate
unfortunately the software is
proprietary unfortunately the software
is a very tightly held corporate secret
that has been protected by the company
and we do not have access to it and
neither do the courts by the way so
here’s what we did we reversed
engineered it we said okay fine you
don’t want to tell us the answer we’ll
figure it out ourselves and so let me
tell you a little bit about how we did
that so here’s what we’re gonna do I’m
going to assume that I’m going to only
try to make a prediction of a person
reoffending from two factors these could
be any factors you want it to be how old
you are how many juvenile convictions
you have how many years you were
unemployed whatever whatever numbers you
can quantify and imagine I just have two
of those factors for now just to keep
things simple and when we do is I’m
gonna reach into that database uh that I
use for my human participants where I
know the demographic information I know
whether they reoffended or not and I’m
gonna plot in this space factor a factor
B all the people who did reoffending
data point here every point here
corresponds to an individual and their
corresponding value of factor a and
factor B and now I reach into my
database and I do the same thing for the
people who did not reoffending factor B
and so now I populate this space of
these two factors and I know who did and
who did not refine and my job is to
build a classifier and the
simplest possible thing you can do in
this space the kind of thing we teach
our undergraduates at Dartmouth College
is to draw a line through the space
that’s it that’s all it takes and all we
now say is well anybody that is to the
left of that line is at risk of
reoffending because that’s where the
majority of the offenders are and
anybody who’s to the right of that space
is not at risk now I have a classifier
and notice that I’ve trained the
classifier on data that I know the
answer to but now any individual that
comes in anybody in the audience I can
compute your factor a compute your
factor B plot in the space are you to
the left or to the right and I have a
prediction for you okay so now the
question is what are the factors what
are the factors that allow me to predict
whether you will commit a crime in the
future if any now remember I said that
our human participants had seven factors
age sex and then five factors that
quantified their previous criminal
history and importantly not race so
here’s what we did we did hundreds and
hundreds of classifiers on all possible
subsets of those seven factors we didn’t
know which ones were the most important
ones but we wanted to start there and we
build hundreds and hundreds of
classifiers with all possible subsets
whether it was five or two or one or all
seven and then we just asked which
classifier is the most accurate which
one gives us the best predictive
accuracy and we learned something really
surprising from just two factors how old
you are and the total number of crimes
whether that was a juvenile a dolt is
enough to get the same accuracy as the
commercial software so commercial
software is an inaccuracy of sixty-five
percent our simplest possible classifier
with two factors and only two factors is
it 66 percent and our classifier by the
way still had the same racial bias
misclassifying blacks at a higher rates
than whites so how does this help us
answer this question as to what’s going
on here right how does this help us
determine why is there racial bias and
how is it that this commercial software
is as good as random people on the
internet getting paid a buck to do a
survey one of the two most important
things
the classifier is number of prior crimes
right I think we can all agree that age
is probably race blind but let’s talk
about number of prior crimes here’s what
we know we know that in this country if
you are black you are much more likely
to be arrested charged and convicted of
a crime the New York Times just last
week for example had a report that said
in New York City black people are 15
times more likely to be charged on
low-level marijuana charges than their
white counterparts number of crimes is a
proxy for race because of this social
inequity because of this inequity in the
criminal justice system and so race was
slipping into the classifier unknown to
us and had we not reverse engineered the
classifier had we not figured out what
are the two most important factors it
would not have all been clear how race
was slipping into here and so the race
bias is coming in because we are feeding
data into this thing that tells you the
race of the person inadvertently number
two how is it the commercial software
isn’t that is making life-altering
decisions on people is the same as
random people on the internet with
little to no criminal justice experience
well as it turns out what we learned is
it’s actually a really simple classifier
here’s what you do you take two pieces
of information how old somebody is in
the total number of crimes they’ve
committed and here’s basically what it
comes down to what the classifier is
doing and what we think humans are
unconsciously doing is the following if
you are young and you’ve committed a lot
of crimes you’re at high risk and if
you’re old and you’ve committed very few
crimes you’re at low risk that’s what it
comes down to right it’s pretty simple
now you could argue that simplicity’s
not bad it’s not bad to be have a simple
classifier but I argue that hiding
behind big data data analytics
artificial intelligence machine learning
you give the appearance that your
predictive algorithms can make more
sophisticated inferences than they
actually can when the company says we
use the latest and data analytics we use
the late
and artificial intelligence it appears
as if they can make very sophisticated
inferences about the future for criminal
defendants when in reality they are
actually doing something incredibly
simple and why does that matter
it matters for the following reason
imagine the following scenario you were
a judge in a courtroom and there is a
defendant before you and you are told
that a proprietary software built on the
latest advances in data analytics and
artificial intelligence and machine
learning predicts that this person is at
high risk of offending you would imagine
that that judge would take that
recommendation relatively seriously it
sounds compelling now imagine that the
judge is told that a random person on
the internet tweeted that they think
that person is at high risk
you’d imagine the judge wouldn’t give a
damn and rightfully so but here’s the
problem that seemingly sophisticated
software and that random person tweeting
on the Internet had exactly the same
accuracy and so it does matter it does
matter how simple and it does matter
that it is no better than humans because
it tells us how much credibility it
tells us how much weight to put into
these predictions and I argue that
weights should be very very little given
what we have learned so in this study we
learned two important things and they go
beyond these predictions in the criminal
justice system one big data data
analytics AI ml are not inherently more
accurate more fair less biased than
humans simply hiding behind technology
does not make you better and this is
particularly true when the data that is
being fed into these algorithms mirror
social existing social inequalities and
the danger here is that you hide behind
that right you forget that you’re
feeding data into the system that is
what is happening in the world around us
these predictive algorithms are meant to
overcome the limitations and they are
simply mirroring them and that is very
dangerous
number two
it is dangerous and it is reckless to
unleash predictive algorithms like this
without thoroughly understanding how
they work without thoroughly
understanding their accuracy and without
thoroughly understanding their fairness
and that is true in the criminal justice
system it is true for employers and
universities making admissions and
hiring decisions it is true in financial
institutions making loan and mortgage
decisions those three things are some of
the biggest things that we do in our
lives our civil liberties our financial
well-being and our employment and we are
turning over decisions to computerized
algorithms and that is very worrisome to
people like me as technologists I will
mention by the way that the
much-talked-about GDP our that went into
effect in Europe recently has given
European citizens the right to audit the
data and the algorithms that are used
against them or on them or to them if
they believe they are incorrect that is
a very thoughtful and very good idea and
it’s something that we in this country
should start thinking about there was a
larger lesson here in the 1980s we were
doing with highly complex issues having
to do with medicine and biology we had
amazing advances and we were really
struggling with how to reconcile that
with our social norms with ethics with
morality and with religion and we were
really struggling in society to come to
grips of that in the u.s. propped up a
bioethics panel and that bioethics panel
was populated with incredibly smart
people from all walks of life from the
religious part from the scientific from
the ethics from the legal and from the
policy and from the economic side and
they were they were their task was to
help us manage these highly complex
issues that we were dealing with and we
didn’t have a roadmap for how to go
forward I advocate for the creation of a
national or an international cyber
ethics panel a panel to help us contend
with what is turning into an incredibly
complex cyber landscape whether it is
the types of predictive algorithms that
I’m talking about here that are being on
sign-u is the public whether is how the
internet has been weaponized by hate
groups and extremists and terrorists
whether it is the privacy issues that
we’ve been struggling with over the last
few years
whether it is security issues whether it
is how the internet is being used to
traffic children in the sex trade all
the problems that we are seeing online
we have to start getting a grip on
technology can be a force for tremendous
progress and tremendous good but as we
have seen over the last few years left
unchecked it can just as well plunge us
into a digital dystopia thank you [Applause]
you have almost all been subject to a
computerized predictive algorithm of the
following form if you like X you may
like Y music movie streaming services
online shopping services routinely look
at your pattern of behavior compares
that to previous customers and tries to
make predictions for you sometimes it
works and sometimes it doesn’t work now
you may be less aware that you’ve also
may have been subjected to a predictive
algorithm of the form if you are like X
we may not give you a loan banks
financial institutions routinely use
your personal attributes compares them
to previous customers to assess the risk
to the institution in terms of giving
you a loan or mortgage you may have also
recently been subjected to a predictive
algorithm of the form if you are like ax
we may not give you a job employers and
in some cases university admissions are
using predictive algorithms to make
hiring and admission decisions and if
you’ve recently had a run-in with the
criminal justice system
well I’m sorry first of all and second
of all you may have been subjected to a
predictive algorithm of the following
form if you are like X then you may go
to jail courts now are routinely using
predictive algorithms to make decisions
in the courtroom one particularly
popular and now widespread algorithm is
used at the point of arrest to make bail
decisions and conceptually it’s a pretty
simple idea a defendant who’s been
charged with a crime is information is
extracted from them and is fed into a
computer algorithm and that computer
algorithm outputs a risk factor and that
risk factor is meant to quantify the
likelihood that person will commit a
crime in the future think of a really
simple version of the Minority Report
without the creepy people in the pool
and if you are assessed to be high-risk
well then the judge may deny bail and
you will be held in prison awaiting
trial and if you are assessed to be
low-risk then you may be released
pending trial now in 2016 investigative
journalists from ProPublica published a
troubling report on one particularly
popular and widespread predictive
algorithm that is used in the courts
around this country and what they found
is that this particular algorithm was
significantly disadvantageous to black
defendants what they found is that if
you were a black defendant you are
nearly twice as likely to be predicted
as reoffending when you did not that
happened 44 percent of the time if you
were black and only 23 percent of the
time if you are white that is a
staggering difference and at the same
time you were almost twice as likely to
be predicted as not offending when in
fact you did if you were white as
compared to black that happened 47
percent of the time versus 28 percent of
time so in other words the algorithm is
biased to black defendants against black
defendants in other words if you are
black and you are a subject to this
algorithm you are more likely to be
assessed as high-risk and incarcerated
than if you are white so shortly after
this article was published a student of
mine Julia Dressel came to me and she
said how is this possible
we’re computer scientists how are we
allowing this to happen how are we
allowing software to make life-altering
decisions on people when they have these
types of inequalities and it was the
right question to ask and it got us
spending about a year and a half journey
trying to understand what was happening
here and how and if we can do better and
I want to tell you a little bit about
that journey and a little bit about what
we learned along the way
so before I dive into the algorithmic
details and sort of understand where
this bias is coming from we started with
a really simple question
we started with the question of are
these algorithms actually better than
you than the average person because
after all presumably we use these
algorithms because they’re more accurate
more objective and more fair and they
remove any biases that may exist in the
human mind and that’s a good idea but we
don’t know if it actually is true or not
so we set out to determine that so
here’s what we did we recruited four
hundred participants from an online
crowdsourcing platform and we asked him
a very simple question
we gave them a short paragraph about an
actual defendant this was a with 8,000
people for whom we had demographic
information we had information about
their past criminal history if any um
and then we knew the risk score
according to the software that is being
used in the court and then they were
tracked for two years so we actually
know if they reoffended or not and so
our participants read a short paragraph
was seven pieces of information the age
and the sex of the person and then five
pieces of data that summarized their
juvenile and adult criminal history they
did not know the race of the person
importantly okay and then we simply
asked them please read this short
paragraph it’s an online survey and just
tell us if you think this person will
reoffended
in a two-year window following their
last arrest yes no yes no yes no here’s
what we learned the computerized
software had an accuracy of 65% our
humans had an accuracy of 67% these are
random people on the internet responding
to a survey for a buck and they spent
about 10 seconds reading that little
short paragraph and making a
determination they are as good as the
commercial software being used in the
courts today oddly they made the same
errors as a software they were more
likely to say that a black defendant at
a rate of 37 percent would reoffended
–nt compared to 27 percent for white
defendants and they’re more likely to
say that a white defendant would not
reoffended at a rate of 40 versus 29
percent how is this possible they don’t
know the race of the person and by the
way neither does the software so both
the humans and the software have a
racial bias and they
no the race that’s weird okay so first
of all it’s a little troubling that
random people on the internet are as
accurate as a software that’s been
deployed in our courts and we’re gonna
talk some more about that in a minute
but there’s two questions that came out
of this one is how is it that humans and
the software have a racial bias when
they don’t know the race of the person
and number two is how is it that random
people on the internet being paid a buck
to answer our survey question are as
good as commercial software that is
being deployed in our course and we
needed to answer those two questions now
at this stage what we should have done
is reached into this commercial software
this predictive algorithm and asked how
does it work what’s going on in here
what is it what information is it using
how is it doing that mathematically and
really understand how the algorithm is
making the mistakes and why it’s not
more accurate
unfortunately the software is
proprietary unfortunately the software
is a very tightly held corporate secret
that has been protected by the company
and we do not have access to it and
neither do the courts by the way so
here’s what we did we reversed
engineered it we said okay fine you
don’t want to tell us the answer we’ll
figure it out ourselves and so let me
tell you a little bit about how we did
that so here’s what we’re gonna do I’m
going to assume that I’m going to only
try to make a prediction of a person
reoffending from two factors these could
be any factors you want it to be how old
you are how many juvenile convictions
you have how many years you were
unemployed whatever whatever numbers you
can quantify and imagine I just have two
of those factors for now just to keep
things simple and when we do is I’m
gonna reach into that database uh that I
use for my human participants where I
know the demographic information I know
whether they reoffended or not and I’m
gonna plot in this space factor a factor
B all the people who did reoffending
data point here every point here
corresponds to an individual and their
corresponding value of factor a and
factor B and now I reach into my
database and I do the same thing for the
people who did not reoffending factor B
and so now I populate this space of
these two factors and I know who did and
who did not refine and my job is to
build a classifier and the
simplest possible thing you can do in
this space the kind of thing we teach
our undergraduates at Dartmouth College
is to draw a line through the space
that’s it that’s all it takes and all we
now say is well anybody that is to the
left of that line is at risk of
reoffending because that’s where the
majority of the offenders are and
anybody who’s to the right of that space
is not at risk now I have a classifier
and notice that I’ve trained the
classifier on data that I know the
answer to but now any individual that
comes in anybody in the audience I can
compute your factor a compute your
factor B plot in the space are you to
the left or to the right and I have a
prediction for you okay so now the
question is what are the factors what
are the factors that allow me to predict
whether you will commit a crime in the
future if any now remember I said that
our human participants had seven factors
age sex and then five factors that
quantified their previous criminal
history and importantly not race so
here’s what we did we did hundreds and
hundreds of classifiers on all possible
subsets of those seven factors we didn’t
know which ones were the most important
ones but we wanted to start there and we
build hundreds and hundreds of
classifiers with all possible subsets
whether it was five or two or one or all
seven and then we just asked which
classifier is the most accurate which
one gives us the best predictive
accuracy and we learned something really
surprising from just two factors how old
you are and the total number of crimes
whether that was a juvenile a dolt is
enough to get the same accuracy as the
commercial software so commercial
software is an inaccuracy of sixty-five
percent our simplest possible classifier
with two factors and only two factors is
it 66 percent and our classifier by the
way still had the same racial bias
misclassifying blacks at a higher rates
than whites so how does this help us
answer this question as to what’s going
on here right how does this help us
determine why is there racial bias and
how is it that this commercial software
is as good as random people on the
internet getting paid a buck to do a
survey one of the two most important
things
the classifier is number of prior crimes
right I think we can all agree that age
is probably race blind but let’s talk
about number of prior crimes here’s what
we know we know that in this country if
you are black you are much more likely
to be arrested charged and convicted of
a crime the New York Times just last
week for example had a report that said
in New York City black people are 15
times more likely to be charged on
low-level marijuana charges than their
white counterparts number of crimes is a
proxy for race because of this social
inequity because of this inequity in the
criminal justice system and so race was
slipping into the classifier unknown to
us and had we not reverse engineered the
classifier had we not figured out what
are the two most important factors it
would not have all been clear how race
was slipping into here and so the race
bias is coming in because we are feeding
data into this thing that tells you the
race of the person inadvertently number
two how is it the commercial software
isn’t that is making life-altering
decisions on people is the same as
random people on the internet with
little to no criminal justice experience
well as it turns out what we learned is
it’s actually a really simple classifier
here’s what you do you take two pieces
of information how old somebody is in
the total number of crimes they’ve
committed and here’s basically what it
comes down to what the classifier is
doing and what we think humans are
unconsciously doing is the following if
you are young and you’ve committed a lot
of crimes you’re at high risk and if
you’re old and you’ve committed very few
crimes you’re at low risk that’s what it
comes down to right it’s pretty simple
now you could argue that simplicity’s
not bad it’s not bad to be have a simple
classifier but I argue that hiding
behind big data data analytics
artificial intelligence machine learning
you give the appearance that your
predictive algorithms can make more
sophisticated inferences than they
actually can when the company says we
use the latest and data analytics we use
the late
and artificial intelligence it appears
as if they can make very sophisticated
inferences about the future for criminal
defendants when in reality they are
actually doing something incredibly
simple and why does that matter
it matters for the following reason
imagine the following scenario you were
a judge in a courtroom and there is a
defendant before you and you are told
that a proprietary software built on the
latest advances in data analytics and
artificial intelligence and machine
learning predicts that this person is at
high risk of offending you would imagine
that that judge would take that
recommendation relatively seriously it
sounds compelling now imagine that the
judge is told that a random person on
the internet tweeted that they think
that person is at high risk
you’d imagine the judge wouldn’t give a
damn and rightfully so but here’s the
problem that seemingly sophisticated
software and that random person tweeting
on the Internet had exactly the same
accuracy and so it does matter it does
matter how simple and it does matter
that it is no better than humans because
it tells us how much credibility it
tells us how much weight to put into
these predictions and I argue that
weights should be very very little given
what we have learned so in this study we
learned two important things and they go
beyond these predictions in the criminal
justice system one big data data
analytics AI ml are not inherently more
accurate more fair less biased than
humans simply hiding behind technology
does not make you better and this is
particularly true when the data that is
being fed into these algorithms mirror
social existing social inequalities and
the danger here is that you hide behind
that right you forget that you’re
feeding data into the system that is
what is happening in the world around us
these predictive algorithms are meant to
overcome the limitations and they are
simply mirroring them and that is very
dangerous
number two
it is dangerous and it is reckless to
unleash predictive algorithms like this
without thoroughly understanding how
they work without thoroughly
understanding their accuracy and without
thoroughly understanding their fairness
and that is true in the criminal justice
system it is true for employers and
universities making admissions and
hiring decisions it is true in financial
institutions making loan and mortgage
decisions those three things are some of
the biggest things that we do in our
lives our civil liberties our financial
well-being and our employment and we are
turning over decisions to computerized
algorithms and that is very worrisome to
people like me as technologists I will
mention by the way that the
much-talked-about GDP our that went into
effect in Europe recently has given
European citizens the right to audit the
data and the algorithms that are used
against them or on them or to them if
they believe they are incorrect that is
a very thoughtful and very good idea and
it’s something that we in this country
should start thinking about there was a
larger lesson here in the 1980s we were
doing with highly complex issues having
to do with medicine and biology we had
amazing advances and we were really
struggling with how to reconcile that
with our social norms with ethics with
morality and with religion and we were
really struggling in society to come to
grips of that in the u.s. propped up a
bioethics panel and that bioethics panel
was populated with incredibly smart
people from all walks of life from the
religious part from the scientific from
the ethics from the legal and from the
policy and from the economic side and
they were they were their task was to
help us manage these highly complex
issues that we were dealing with and we
didn’t have a roadmap for how to go
forward I advocate for the creation of a
national or an international cyber
ethics panel a panel to help us contend
with what is turning into an incredibly
complex cyber landscape whether it is
the types of predictive algorithms that
I’m talking about here that are being on
sign-u is the public whether is how the
internet has been weaponized by hate
groups and extremists and terrorists
whether it is the privacy issues that
we’ve been struggling with over the last
few years
whether it is security issues whether it
is how the internet is being used to
traffic children in the sex trade all
the problems that we are seeing online
we have to start getting a grip on
technology can be a force for tremendous
progress and tremendous good but as we
have seen over the last few years left
unchecked it can just as well plunge us
into a digital dystopia thank you [Applause]
Please follow and like us: