disparities and who is impacted by the
American criminal justice system perhaps
you’ve heard some of these black men are
six times more likely to be incarcerated
than white men on the road minorities
are more likely to be stopped and
searched than whites but are these
disparities evidence of racial bias we
often turn to data for a semblance of
truth for rigor objectivity I love data
it can reveal patterns tell stories and
at their very core data can confirm or
challenger theories but when it comes to
racial bias definitive answers have been
hard to come by
although data initiatives have been
sweeping the country drawbacks of
current approaches have limited our
ability to have an informed discussion
sometimes contributing more confusion
rather than clarity here’s one example
one might argue that the higher search
rates for minorities are due to racial
bias the reality of policing is that
officers tend to be deployed in high
crime areas areas that tend to be
predominantly black and Hispanic so a
common counter-argument is that
the elevated search rates we see are
simply a result of effective policing
how can we move forward and propose
solutions if we can’t accurately
identify or measure the very thing we’re
trying to address racial bias our team
at Stanford has been working on methods
to overcome some of the current
limitations but I first want to show you
why it’s so difficult to measure racial
bias as think of traffic stops now in
order to identify bias we would need to
know whether an office
would given to drivers in otherwise
identical contexts react differently
when one is white and the other is not
now obviously this is impossible so what
can we do well the most commonly used
tests to measure racial bias is known as
the outcome test it compares the search
success rate or hit rate among race
groups what’s a hit well a hit is when
an officer searches a car and finds
contraband that is something illegal so
it’s kind of like a homerun for the
officer the idea is if searches of
minorities turn up contraband 10% of the
time where are the searches of whites
turn up contraband 90% of the time then
this adjusts officers are searching
minorities on the basis of less evidence
than whites now this test sounds very
reasonable right but it turns out it has
a well-known limitation and to see this
let’s think about what happens during a
traffic stop so here’s your typical
driver during the course of the stop the
officer has discretion to search if they
have discretion to search if they have
sufficient reason to believe the drivers
can contraband
so they observe the demographics of the
driver the time of day whether it’s a
dangerous neighborhood and they look for
clues whether the driver is acting
nervous or evasive there’s any odors in
the car more factors than we can ever
hope to measure and in order to make
their decision we imagine they distill
all this information down into a single
number which represents their subjective
estimate that the driver is going to
possess contraband now if their estimate
is sufficiently high they’re going to
conduct a search that’s a pretty
suspicious-looking driver right so now
imagine we had this information for all
drivers then we could order the drivers
an increasing order of suspicion we’d
expect that above a certain threshold of
suspicion all drivers would be searched
now say two out of the four searches
were successful the hit rate would be
one half okay so everyone with me so far
now let’s see how the outcome task can
fail so here we have two race groups
green and blue the hit rate are equal
indicating that the outcome test does
not find evidence of bias but notice
that the least suspicious green driver
that was searched had an estimated 25
percent chance of having contraband
whereas the lis suspicious blue driver
that was searched had an estimated 51%
chance so these green drivers would not
have been searched had they been blue
the lower threshold for green suggests
that police require less evidence when
searching green drivers so what we
actually want to measure are the search
thresholds themselves the only problem
is we can’t observe them we can only
observe the hit rates so we’re left with
a flawed metric to measure one of the
most invasive actions police can take to
search someone it’s like we’re trying to
diagnose a patient with unreliable tests
using miscalibrated equipment so in
order to address this problem our team
at Stanford developed a mathematical
model to determine whether searches are
consistently effective given the number
of stops searches and hits we can infer
the search thresholds being used by a
department for each race group we apply
the test to 4.5 million stops in North
Carolina each point here is a department
the x-axis is the white threshold and
the y-axis is the minority threshold so
points below the line indicates that
thresholds for minorities are
consistently lower across the largest
100 departments in the state now we’ve
tested and ruled out legitimate reasons
why thresholds might vary such as time
of day year or policy changes and
crucially we cannot determine whether
the lower thresholds are due to
discrimination or implicit bias
or some other unknown factor but the
results do suggest there are some
striking irregularities in how search
decisions are made we must also be
cognizant that the data we choose to
collect and the models we build are
laden with values the way we choose to
discretize and record the world around
us
will determine the stories our data will
be able to tell and models are built on
assumptions ones that we must
continuously test and perfect but the
very fact that we lack perfect tools
today is all the more reason to strive
for rigor if we are to ever reach a
common ground we are we must have the
courage to ask some uncomfortable
questions perhaps find some
uncomfortable answers and start all over
again when our solutions don’t work
police undoubtedly have a tough job
there’s a high cost to search in someone
innocence the lack of trust and there’s
a high cost to not search in someone
who’s potentially dangerous and Safe
Communities and higher crime Sir Robert
Peel who’s commonly regarded as the
father of modern policing envisioned a
society where the bond between police
and the communities they served was so
strong that it would be like they were
one in the same and until the day our
data and our narratives will speak of
Kiehl’s vision we mustn’t stop working [Applause]