that word means different things to
different people to us it means
aggressive investments in a number of
issue areas from evidence-based policy
criminal justice education healthcare
pension reform public accountability and
many more I get asked a lot of questions
about our work why did you choose the
shoe areas that you did and how do you
choose new ones I always feel that when
responding to that question people are
expecting me to tell a story about some
transformative event that happened in my
life for John’s life that led us to
embark on this path maybe they want to
hear that we invest in health care
because we’ve had family members who
unfortunately have had cancer maybe they
want to hear story about my life growing
up in Puerto Rico and how that fueled my
interest in and passion for fighting
social justice issues and addressing
poverty it said we’ve come to expect
that every talk will start with a story
like that something intensely moving and
personal anything that starts with when
I was a kid but I’m not going to tell a
story like that although I could I think
personal stories are so important they
are what fuels our passion for many of
us they’re the reason why we do the work
we do but I believe that they’re
precisely the wrong place to look for
insight in fact I believe that we as
policymakers government officials
philanthropist even as individuals spend
entirely too much time on anecdote and
not enough time when evidence I’m here
today to argue that we need to change
that that we’re routinely making all
kinds of decisions based on incomplete
inconsistent flawed or even non-existent
data based on anecdote that this is
harming all of us in ways that we’re not
appreciating and that we can and should
do better now I’ve just made a bunch of
provocative statements so let me get to
work at convincing it I’ll start with
one example from the Ted global stage in
2012 the second most viewed TED talk
of all time Amy Cuddy psychologist
Harvard Business School professor famous
in large part for her TED talk for her
work on power posing so she basically
argues that standing in a power post
position like Wonder Woman like this or
like this you know whatever it makes you
feel super powerful changes the hormone
levels in your body and might even be
conducive to greater success it’s
amazing right I mean that’s so cool I
should just stand like this the entire
time the Ted community eighths is up
over 38 million views over the last four
and a half years the talk is been
subtitled into 48 languages and Counting
here’s how the Ted website today
describes professor Cuddy’s work it says
that professor Cuddy’s work reveals the
benefit of power posing and it makes a
reference to this hormone level issue
that I just talked about national news
outlets gushed about the work and many
of us rushed to buy her best-selling
book now I need to be a downer but I
think for LG powerful thing doesn’t work
at least not as advertised look at the
myriad studies that have been conducted
on power posing ever since professor
Cuddy released their findings they point
the glaring errors in her methodology
they questioned the entire theory of
power posing some researchers tried to
reproduce her work and couldn’t others
tried to reproduce her work and got
exactly the opposite result that power
posing reduces feelings of power and it
gets worse even professor Cuddy’s
co-author who herself as a prominent
academic has completely disavow the
study and she did so in no uncertain
terms here’s where she said I do not
believe the powerful effects are real
and she goes farther than that she
confesses to glaring errors in the way
that they conducted their study tiny
sample sizes fluency data selectively
reported findings even professor Cuddy
herself who continues to stand by the
power post Theory now has sort of
amended her story and now she says that
she’s agnostic about power poses effects
on hormone levels
Gnostic that was the whole reason she
went on the Ted stage into this and did
this and made us all do this and feel
powerful you might think this is a trite
example and it’s by no means meant to be
a personal attack who cares if I stand
in front of you like this if that makes
me feel good or not and what does it
have to do with solving the country’s
problems and the world’s problems which
is why we’re all here well a lot because
this example is endemic or something
that we see throughout academic research
and throughout research in general
virtually everywhere you look you’ll
find researchers many of them prominent
and most of them well-intentioned
actively misleading us into believing
that bad research is proven fact we’ve
seen this time and again in our work and
when John and I started the foundation
we set out to solve the country’s
problems by attacking root causes so we
were pretty new at this so we figured
that the best place to start was to try
to get our head around what we did and
didn’t know as a community collectively
as a country what do we know about what
doesn’t doesn’t work after all
philanthropist has been trying to save
the country and make the country better
for many many years and have spent
billions of dollars doing it governments
state governments local governments have
spent exponentially more on health care
programs social programs anti-poverty
programs job training you name it so we
assumed that there had to be a massive
body of evidence that could help guide
our investment decisions what we found
was really alarming we saw bad research
everywhere it didn’t matter where we
looked for example we turned to
nutrition as potentially an avenue to
address our health epidemics we wanted
to get smart about what factors and
foods might be conducive to chronic
diseases obesity heart disease diabetes
and we found some very good research but
we also found an abundance of studies
like these chocolate makes you skinny Oh
beer helps you work out in case you
didn’t know cause knee prevents
Alzheimer’s super catchy
loses a researcher notices the people
under study who are eating chocolate are
also skinny and summarily comes to the
conclusion or at least tells us that
they’re skinny because eat chocolate
same thing with a Alzheimer study shoddy
research bad methodologies small sample
sizes correlations powdered as
causations selectively reported findings
just like the Amy Cuddy study my way
there’s a lot of these one might say a
lot of alternative facts in 2012 a group
of researchers randomly chose 50
ingredients from a cookbook normal stuff
like milk eggs and they took a look at
the body of research relating to these
ingredients to see what they could learn
about whether these ingredients did or
did not cause cancer seems like a
worthwhile exercise here’s what they
found for every ingredient that you see
listed in the slide lying Tomatoes milk
eggs corn coffee butter there was at
least one research study that argued
that the ingredient causes cancer and at
least one research study that argued
that the ingredient prevented cancer
what are we supposed to do with this
information we turn to health care same
thing we saw that the authors of the
vast majority of clinical trials
reported and top medical journals
silently changed the outcomes that they
were reporting so they said they were
going to study one thing but they
reported on another now why would they
do that now that I don’t claim to be an
expert scientific research is enormously
complicated certainly clinical
researchers as well but one theory might
be that the original studies didn’t pan
out the way they wanted so they cherry
picked positive findings from those same
studies on secondary outcomes and
reported on those instead so they could
get published now you don’t have to have
a PhD to start wondering whether maybe
there could be something fishy going on
here and these shenanigans happen
everywhere take a recent project that we
did a reproducibility project so we
asked researchers to reproduce one
hundred psychology experiments that had
been published in top psychology
journals in 2008 so go do them again if
you do them again will you find the same
result that’s what we wanted to know you
know often they could find the same
results one third to one half of the
time now I’m not claiming that
scientists and researchers are actively
and intentionally committing fraud I’m
saying there’s something broken in our
system where scientists are feeling the
need to report only positive findings at
the expense of the whole story how would
they do that well both ways so let’s say
that a researcher wants to prove that
there’s a relationship between eating
blue M&Ms and prostate cancer because
that would be newsworthy so he designs
the conducts an experiment but he
doesn’t find a relationship okay so he
does it again the same experiment does
it again no relationship
he doesn’t 17 more times in our
relationship but on the 20th time he
does find a relationship boom he can
publish blue M&Ms cause prostate cancer
new study shows he doesn’t tell us about
the 19 other times that he conducted
that same study and failed he just tells
us about the one time that he succeeded
that’s called the Fowler effect the
researcher could also tweak the
statistical analysis until he gets the
results that he wants that’s healthy
hacking or he could report is to a very
narrowly defined group when in fact the
original research study was meant to
address a much much larger group why
does it happen well there could be lots
of reasons but I would argue that a
reason is it the incentive system in
science and in research is broken in an
ideal world scientists and researchers
will be motivated by one thing the
pursuit of truth but in the real world
scientists and researchers are equally
motivated by the desire to publish
because that’s the vehicle for achieving
tenure that’s the vehicle for getting
funding and for achieving notoriety
scientific journals are clamoring for
articles that report flashy results so
that those articles can get cited they
would have a term for this is called the
impact factor then of course there’s the
media and we as consumers we all want to
hear those flashy results as a
researchers deliver even if it’s at the
expense of full transparency and rigor
and while the worse off for it our
foundation we try to break this cycle
and reform the system by funding
organizations that are promoting
transparency in good practices and
collaboration and data sharing the
Center for open science the Center for
evidence-based medicine at Oxford the
metric Center at Stanford health news
organizations that are holding the
media’s feet to the fire as to what they
report on research studies and they’re
all doing terrific work but we need to
do so much more we didn’t become
philanthropists to hang out with
academics and role in the hallowed halls
of universities we became
philanthropists to change the world but
how can we even think about what to
change if we don’t know what works and
how are we supposed to know what works
if we can’t trust research so this is
our problem
this is our issue this goes to the core
of who we are and what we do we can’t
function if we don’t have a healthy
research ecosystem this problem of bad
research and bad science isn’t limited
track edenia this is just as bad in
public policy and remember the Scared
Straight program you know at-risk youth
would go to maximum-security prisons or
meet prisoners and the prisoners would
yell at them and tell them about their
bad choices and how they have to live
their life on the straight and narrow we
spent millions of dollars on these
programs and they made intuitive sense
except that they didn’t work research
showed that these programs actually
increase the likelihood that these kids
would commit criminal acts we should
have known this much earlier we would
have saved millions of dollars and more
importantly maybe we would have saved
some of these
kids three federal departments the
Department of Labor HHS and the
Department of Education have funded
randomized control trials on social
programs that have either been
administered by the government or by the
private sector
now randomized control trials are the
gold standard in research here’s what
they found
70% of the employment and job training
programs that the Department of Labor
looked at had weak or no positive
effects of the 28 teen pregnancy
prevention programs that HHS looked at
only three were worthwhile and eighty to
90 percent of the education programs
that the Department of Education looked
at had weak or no positive effects
relative to what squares were doing
already so we know so little about what
works in education and in job training
and in policy in general even when we
think we do take the federal
government’s what works Clearinghouse
which was established as a resource for
practitioners to determine what works in
education it purports to rely on
rigorous research let’s look at this
research there is a study that had only
a few dozen participants there’s another
study that was conducted over the course
of around 12 weeks then there’s a bunch
of positive labels on secondary outcomes
that ultimately don’t matter
for example the website labels is
positive a reading preparation program
because research showed that after going
through the program kids were able to
recognize letters of the alphabet well
you might say that’s a stepping stone to
reading so that sounds pretty reasonable
to me and it does except that the
website doesn’t tell us that those same
researchers studied that same program
and found that the program had no
effects on kids ultimate ability to read
well isn’t that what we care about so
we’re spending millions and millions of
dollars on programs that at worst don’t
work and at best we don’t know well we
don’t have sufficient data or we don’t
have reliable data so this needs to
change we need to stop as a
philanthropic community as a
policymaking community we need to stop
funding these programs and relying on
this ecosystem that isn’t giving us the
results that we need
and I’ve got some ideas on how to do
that how to reform the system first more
randomized control trials those of us
who work with governments for
governments collaborate with governments
need to demand more randomized control
trials so that we can get evidence and
understand what doesn’t doesn’t work
second we all need to follow the
evidence this is on all of us
governments and philanthropist need to
stop funding what doesn’t work and start
funding what does work we need to hold
our own selves accountable for these
results and for this problem this isn’t
something that was imposed upon us –
something that we’ve created and that’s
on us to fix and third we need better
data we can’t do research if we don’t
have healthy data systems that are
speaking to each other that are
harmonized that researchers can access
to give us the answers that we need now
here – there are excellent organizations
both within and outside the government
that are working on these issues within
the government the Commission on
evidence-based policymaking the Social
and Behavioral Sciences team are
conducting randomized control trials
pushing an evidence-based policymaking
agenda within government and they’re
doing terrific work in collaboration
with stayed in local governments in the
nonprofit sector the rhode island policy
innovation labs j-pal results for
America are all contributing to
reforming this ecosystem by either
conducting randomized control trials or
working at promoting evidence and thanks
to the work of these organizations and
many others we now have some answers we
now know which programs do move the
needle on things that we care about like
child welfare and education and job
training and recidivism armed with that
information aren’t we in a better spot
as people who care as concerned citizens
as philanthropists as policymakers to
make good decisions we need to follow
data we need to focus more on research
and we need to demand that of
governments and of ourselves because
policy ideas that truly are we’re
spreading
or the ones at work thank you
[Applause] [Music]