all right so I’m gonna talk about do
numbers lie I’ll come back to that but
for now let me start with a quote from
president Trump’s State of the Union
address just one month ago African
American unemployment rate stands at
lowest rate ever recorded impressive
right the suggestion here if you were to
look back and look at his tweets you
realize that hey we all have his
policies to thank for here’s another
example now this time let’s rewind our
clock back to the last election in 2016
so mr. Evan Bayh Democrat from Indiana
he was running for the US Senate okay so
he had served in the Senate before so
this ad attacked him for voting with
then President Obama 96% of the time 96%
Wow the suggestion here is quite obvious
this guy is just so partisan do not vote
for him okay so these are powerful
statements with powerful suggestions but
let’s for now just ignore the suggestive
parts and focus on the factual claims
like you know lowest rate ever or 96%
what do they have in common
first of all they’re both based on
numbers a numbers are great we never use
numbers they automatically make your
claim look more believable second not
only numbers you can also get the data
to verify these numbers for example from
the Bureau of Labor Statistics you know
lowest rate ever or 96 percent what do
they have in common first of all they’re
both based
on numbers a numbers are great we never
use numbers they automatically make your
claim look more believable second not
only numbers you can also get the data
to verify these numbers for example from
the Bureau of Labor Statistics were from
congressional records so great another
stamp of approval right now this is
number backed by data and indeed if you
go ahead and take the effort and check
these claims what do you find
they’re both factually correct end of
discussion right except that
unfortunately in spite of being
factually correct claims can also
mislead for example yes yes indeed black
unemployment rate has being at the
lowest level ever but if you were to put
this claim in the larger historical
context you will realize that this rate
has been steady decline for the past
seven years long before our current
president took office so there now I
don’t want to sound like I’m just always
picking our president the sad truth is
that these sort of claims are everywhere
around us across ideologies across
cultures and these are nothing new they
have been with us throughout history in
fact I’m gonna quote from this very
interesting book titled how to lie with
statistics written probably long before
many of you were born and here’s the
quote if you torture the data along
enough it will confess to anything I’m
gonna tell you how to torture the data
okay so you’ve got some data right okay
say congressional voting records do you
want to do some analysis on it say I
want to calculate how often people voted
with Obama now you can think of this
analysis as a black box
okay so it takes data as input crunching
some numbers and outputs a number say
96% would you care to stick into your
claim now what’s interesting about this
black box is that it’s got these knobs
okay so for example one knob may control
who we’re comparing
another knob may control the time period
of comparison like over the last month
last year last 200 votes whatever yet
another knob may control what goes into
the calculation do you include votes
that are procedural do you include
nominations so on and so forth all those
technical details now even for a simple
analysis there could be a lot of knobs
you may not realize all of them but you
know they still exist this is where the
torture comes in because you can tune
these knobs you tune the results changes
your keep tuning until you get what you
want
60% no no no 70% still boring 80% we’re
talking 96% bingo right that’s also
called cherry-picking data dredging
there’s lots of other names for it now
if this has been going on for ever and
ever in our history what’s the big idea
now well thanks to technology we’ve got
more data than ever deform and
increasingly important decisions are
driven by and justified by data and
there’s more opportunities than ever and
also temptation to use the data to
generate these quote-unquote half-truths
can you see them everywhere they can be
used to bias public opinions they can be
used to control this course narratives
they can be used to justify actions
sometimes with serious consequences so
you see data in numbers really don’t
automatically make us any safer actually
what I would argue we need more
fact-checking instead of less
and as we have discussed this is really
really hard work right because you have
to check beyond just correctness so as a
computer scientist I asked
it’s very any way that I can help is it
possible for algorithms to actually help
us with this we figured out one way to
do it the idea is really remarkably
simple it’s embarrassing you torture the
data even more okay now I realize I used
the term torture quite a lot in this
talk but I assure you no bits or pixels
were ever harmed in the production of
this talk so so there we have a claim
which is based on some analysis over the
data yeah and this analysis comes with a
particular setting of its knobs which in
essence captures what particular view of
the data that the claim wants you to see
but there is no reason why we should
blindly trust that particular view so
instead what we’re gonna do we’re gonna
go in and tweak these knobs more a lot
more and we’re gonna see how they impact
the conclusion of the claim by exploring
a lot of possible knob settings we
actually expose this larger context in
which we can evaluate to the original
claim you can ask how robust it is you
can ask how unique it is and so on now
this is hard work and this is why this
doesn’t get done often enough but hey
that’s the type of the work that
computer is really great at so we can
develop algorithms that will help us
tirelessly and meticulously tweak these
knobs and synthesize this much bigger
context for us to fact-check ooh indeed
this is the approach we have taken so
together with a group of students and
colleagues into computer science and
public policy we set out and build a
system for the last election that allows
you to actually fact check statements
based on congressional voting records
and here are some examples of the
insights we were able to find yes mr.
Evan Bayh indeed voted will
president obama 96% of the time but that
was in 2010 now you tweak this knob just
a little bit to 2009 what do you see
that percentage drops to 77% yeah
combined two years together that gives
you 85 percent still not as high as 96
percent right so in a sense the original
claim is not very robust because even
with a small tweak you can weaken its
conclusion very drastically
now you may still wonder hey 77% maybe
that sounds still a little high so for
that insight you have to tweak a
different knob and this is knob that
controls who you are comparing so by
tweaking this knob we see how the other
senators voted with Obama and this
creates the bigger context in which you
can evaluate mr. Evan theis performance
okay so in this visualization each bar
is basically a senator okay so the
Democrats are blue
Republicans are red okay and these bars
are sorted according to their height so
the height of the bar indicates how
often somebody voted with Obama now in
2009 where do you think mr. Evan Bayh
was well he’s all the way there in the
red territory as a matter of fact he was
a Democrat who agreed least with mr.
Obama and in fact less than even three
of the Republicans so they’re 77% is now
that height sprite though even ninety
six percent is not as high as what they
would want you to believe so our
algorithms were able to find these
insights and visualizations
automatically for you so to conclude my
talk let me ask this question again do
numbers lie no I would say no numbers
never lie but you can lie with numbers
second point algorithms can help us
fight flies with numbers great but
there’s just one catch
I did not talk about this but some of
you might have noticed the same
algorithms that tirelessly and
meticulously tweak these knobs to help
us fact check can also be used to
tirelessly and meticulously cherry-pick
that knob setting that will advance your
point of view however bias that might be
so in other words algorithms can also
help us lie with numbers this irony
leads to my final point really it’s not
about numbers or data or algorithms it’s
about you because it is you who must
choose wisely
how you want technology to shape our
future thank you for listening [Applause]
numbers lie I’ll come back to that but
for now let me start with a quote from
president Trump’s State of the Union
address just one month ago African
American unemployment rate stands at
lowest rate ever recorded impressive
right the suggestion here if you were to
look back and look at his tweets you
realize that hey we all have his
policies to thank for here’s another
example now this time let’s rewind our
clock back to the last election in 2016
so mr. Evan Bayh Democrat from Indiana
he was running for the US Senate okay so
he had served in the Senate before so
this ad attacked him for voting with
then President Obama 96% of the time 96%
Wow the suggestion here is quite obvious
this guy is just so partisan do not vote
for him okay so these are powerful
statements with powerful suggestions but
let’s for now just ignore the suggestive
parts and focus on the factual claims
like you know lowest rate ever or 96%
what do they have in common
first of all they’re both based on
numbers a numbers are great we never use
numbers they automatically make your
claim look more believable second not
only numbers you can also get the data
to verify these numbers for example from
the Bureau of Labor Statistics you know
lowest rate ever or 96 percent what do
they have in common first of all they’re
both based
on numbers a numbers are great we never
use numbers they automatically make your
claim look more believable second not
only numbers you can also get the data
to verify these numbers for example from
the Bureau of Labor Statistics were from
congressional records so great another
stamp of approval right now this is
number backed by data and indeed if you
go ahead and take the effort and check
these claims what do you find
they’re both factually correct end of
discussion right except that
unfortunately in spite of being
factually correct claims can also
mislead for example yes yes indeed black
unemployment rate has being at the
lowest level ever but if you were to put
this claim in the larger historical
context you will realize that this rate
has been steady decline for the past
seven years long before our current
president took office so there now I
don’t want to sound like I’m just always
picking our president the sad truth is
that these sort of claims are everywhere
around us across ideologies across
cultures and these are nothing new they
have been with us throughout history in
fact I’m gonna quote from this very
interesting book titled how to lie with
statistics written probably long before
many of you were born and here’s the
quote if you torture the data along
enough it will confess to anything I’m
gonna tell you how to torture the data
okay so you’ve got some data right okay
say congressional voting records do you
want to do some analysis on it say I
want to calculate how often people voted
with Obama now you can think of this
analysis as a black box
okay so it takes data as input crunching
some numbers and outputs a number say
96% would you care to stick into your
claim now what’s interesting about this
black box is that it’s got these knobs
okay so for example one knob may control
who we’re comparing
another knob may control the time period
of comparison like over the last month
last year last 200 votes whatever yet
another knob may control what goes into
the calculation do you include votes
that are procedural do you include
nominations so on and so forth all those
technical details now even for a simple
analysis there could be a lot of knobs
you may not realize all of them but you
know they still exist this is where the
torture comes in because you can tune
these knobs you tune the results changes
your keep tuning until you get what you
want
60% no no no 70% still boring 80% we’re
talking 96% bingo right that’s also
called cherry-picking data dredging
there’s lots of other names for it now
if this has been going on for ever and
ever in our history what’s the big idea
now well thanks to technology we’ve got
more data than ever deform and
increasingly important decisions are
driven by and justified by data and
there’s more opportunities than ever and
also temptation to use the data to
generate these quote-unquote half-truths
can you see them everywhere they can be
used to bias public opinions they can be
used to control this course narratives
they can be used to justify actions
sometimes with serious consequences so
you see data in numbers really don’t
automatically make us any safer actually
what I would argue we need more
fact-checking instead of less
and as we have discussed this is really
really hard work right because you have
to check beyond just correctness so as a
computer scientist I asked
it’s very any way that I can help is it
possible for algorithms to actually help
us with this we figured out one way to
do it the idea is really remarkably
simple it’s embarrassing you torture the
data even more okay now I realize I used
the term torture quite a lot in this
talk but I assure you no bits or pixels
were ever harmed in the production of
this talk so so there we have a claim
which is based on some analysis over the
data yeah and this analysis comes with a
particular setting of its knobs which in
essence captures what particular view of
the data that the claim wants you to see
but there is no reason why we should
blindly trust that particular view so
instead what we’re gonna do we’re gonna
go in and tweak these knobs more a lot
more and we’re gonna see how they impact
the conclusion of the claim by exploring
a lot of possible knob settings we
actually expose this larger context in
which we can evaluate to the original
claim you can ask how robust it is you
can ask how unique it is and so on now
this is hard work and this is why this
doesn’t get done often enough but hey
that’s the type of the work that
computer is really great at so we can
develop algorithms that will help us
tirelessly and meticulously tweak these
knobs and synthesize this much bigger
context for us to fact-check ooh indeed
this is the approach we have taken so
together with a group of students and
colleagues into computer science and
public policy we set out and build a
system for the last election that allows
you to actually fact check statements
based on congressional voting records
and here are some examples of the
insights we were able to find yes mr.
Evan Bayh indeed voted will
president obama 96% of the time but that
was in 2010 now you tweak this knob just
a little bit to 2009 what do you see
that percentage drops to 77% yeah
combined two years together that gives
you 85 percent still not as high as 96
percent right so in a sense the original
claim is not very robust because even
with a small tweak you can weaken its
conclusion very drastically
now you may still wonder hey 77% maybe
that sounds still a little high so for
that insight you have to tweak a
different knob and this is knob that
controls who you are comparing so by
tweaking this knob we see how the other
senators voted with Obama and this
creates the bigger context in which you
can evaluate mr. Evan theis performance
okay so in this visualization each bar
is basically a senator okay so the
Democrats are blue
Republicans are red okay and these bars
are sorted according to their height so
the height of the bar indicates how
often somebody voted with Obama now in
2009 where do you think mr. Evan Bayh
was well he’s all the way there in the
red territory as a matter of fact he was
a Democrat who agreed least with mr.
Obama and in fact less than even three
of the Republicans so they’re 77% is now
that height sprite though even ninety
six percent is not as high as what they
would want you to believe so our
algorithms were able to find these
insights and visualizations
automatically for you so to conclude my
talk let me ask this question again do
numbers lie no I would say no numbers
never lie but you can lie with numbers
second point algorithms can help us
fight flies with numbers great but
there’s just one catch
I did not talk about this but some of
you might have noticed the same
algorithms that tirelessly and
meticulously tweak these knobs to help
us fact check can also be used to
tirelessly and meticulously cherry-pick
that knob setting that will advance your
point of view however bias that might be
so in other words algorithms can also
help us lie with numbers this irony
leads to my final point really it’s not
about numbers or data or algorithms it’s
about you because it is you who must
choose wisely
how you want technology to shape our
future thank you for listening [Applause]
Please follow and like us: