Press "Enter" to skip to content

What superhuman poker bots can teach us about decision making | Adam Kucharski | TEDxLiverpool


twenty-one years ago IBM’s deep blue
chess computer defeated Garry Kasparov
is the first time that an artificial
chess player had beaten the world’s best
human it was an incredible technical
achievement but it wasn’t entirely
satisfying deep-blue had essentially
beaten Kasparov by brute force it just
outs competed him and it meant that
although the game taught us a lot about
computers
Kasparov felt we didn’t really learn
that much about human intelligence and
decision-making
why could a computer get away with
playing the game in this way well one
reason is the complexity if we think of
complexity in terms of the number of
potential positions you could find
yourself in during a game in chess
there’s about a hundred billion billion
billion billion billion such
combinations obviously a lot and it’s
certainly more than checkers which was
mastered by computers in the early 90s
but it’s nowhere near as complex as
something like go which was only really
mastered by computers in the past couple
of years but it can’t just be about
complexity because to play poker with
limited stakes actually has fewer
possible game positions than checkers
and yet this is a game that computers
have struggled with for much longer so
why is this well as well as complexity
we need to think about information
that’s available checkers chess and go
or what’s known as perfect information
games all the information you need to
make a decision is right there in front
of you
poka of course is a game of hidden
information you don’t know what your
opponents have they don’t know what you
have and if you think about it this
makes the game much closer to the sort
of situations we face in real-life
negotiations auctions bargaining these
all involve hidden information and it
means that arguably if a computer were
to ever achieve true human-like
intelligence in effect it would have
mastered a highly complex hidden
information game and for this reason
researchers have had a long-standing
interest in poker as a way of
understanding human decisions in fact
the whole modern theory of games
originated with poker originated in the
1920s with a mathematician called John
von Neumann now
Philemon was a brilliant researcher but
he wasn’t so good at games like poker
and maybe at first glance pokers the
perfect game for mathematician it’s a
game of probabilities it’s the chance
you get a decent hand the chance your
opponent gets a better one but von
Lohmann realize there’s more to it than
that
because you have this hidden information
there’s an element of deception you just
want to guess what your opponent has
they’re trying to guess what you have
you just want to guess what they’re
guessing and he wanted to understand
this back and forth process that was
going on and to do that he looked at
really really simplified forms of the
game and one of these was a game where
there’s just two players they each get a
single card and there’s one round of
betting and in this situation for know
him and realized that there was a
tug-of-war going on because in poker
everything you win comes out of your
opponent’s pocket so both players are
simultaneously trying to maximize what
they win will minimize what they handed
their opponent and studying this game
von neumann showed that in his
tug-of-war there’s an equilibrium point
there’s a point where the force is
balanced and each player has an optimal
strategy so that they wouldn’t expect a
better outcome if they did something
different
when he worked out this optimal strategy
for this very simple game he found that
for the player who goes first
if they get dealt a high card they
should bet now intuitive that makes
sense if you got a decent card you put
money on it he found that if that first
player gets dealt a kind of middling
value card they shouldn’t bet they
should check and see what happens again
intuitively that makes sense we found
that the first player if they get dealt
a low value card should bet in other
words they should Bluff and of course
for centuries players have been bluffing
in games like this but often it was
thought of as something of a quirk of
psychology and here was von neumann
showing that that wasn’t the case at all
bluffing was a mathematical necessity in
these games if you wanted the optimal
strategy now it was a powerful insight
but it’s still a really simple form of
poker and it meant that essential
von Lohmann could write down fixed rules
for each player so the first player for
example if they get doubt a high or low
card they bet if they get dealt a
middling value card they check they
follow those rules in the long run they
get the best expected outcome of course
in real life isn’t like that
take for example the penalty kick all
things being equal a right footed player
will have a better chance of scoring if
they put the ball on the left side of
the goal their strong side
so perhaps players should always shoot
on this side well of course if the
keeper works out what they’re doing
they’re always going to go that way and
reduce the chance of scoring so perhaps
instead players should always shoot on
their weak side this should be the rule
that they follow but again if the keeper
guesses what they’re doing they’re going
to go that way and reduce their chance
and we’ve got this back and forth
second-guessing that von Oy moon was
interested in in this situation the
optimal strategy is to play with an
element of randomness don’t always put
it in the same side mix it up a bit and
analyzing the penalties the
mathematically optimal thing to do is to
shoot on your strong side about 58% the
time and your weak side the remainder of
the time and what’s amazing is if you
look at real-life penalty data this is
almost the exact ratio that players
shooting now I’m not claiming that they
sit in the changing rooms working
through the mathematics but by trial and
error
they’ve converged to the optimal
strategy in this game but again here
we’re looking at a relatively simple
situation we’ve got two options and in
that poker game I mentioned there were
only two players and two cards but of
course real life and real poker is far
more complex than this it’s not
something where we can just neatly write
down the combinations and work out the
mathematics but fortunately we can turn
to another method it’s a technique that
was developed by a colleague of John von
Neumann it was a mathematician called
stanislav olan olan was once playing
solitaire he was this card game laying
out cards and wondered what’s the chance
I’ll lay out the cards in a way that
allow me to finish the game he started
writing down some different combinations
and possibilities and it just got far
too complicated too quickly and just
realized he wasn’t going to get an
answer in this way they thought well why
not instead just lay out the cards
deal the cards a few times and see what
happens in other words why not simulate
the game to understand how it works and
he’d end up calling this method the
Monte Carlo method on this idea that you
can simulate a random process that
understand how it works and the Monte
Carlo method is now hugely important in
science and technology and it’s really
common in AI as well and in fact teams
are now using this method to develop
poker programs or poker BOTS
that can play the game when poker BOTS
first became popular about 10 or 15
years ago a common strategy was just to
tell it what to do if you’re building a
bot you have your algorithm and you say
in this situation you do this in this
situation you do that but the problem
with that approach is your bot will only
ever be as good as you are if you’re
telling it exactly what to do
your bot will be limited by your own
abilities as a player it won’t be able
to exceed your ability but there’s a
better approach we can use and it was
one mentioned by Alan Turing who is a
pioneer of computing back in the 1950s
and he noted that if you’re trying to
build an intelligent machine it doesn’t
make sense to build the adult mind with
all its fixed rules and strategies what
you want to do is build the child mind
and let it work out for itself how it
should improve and learn in these games
and this is exactly what researchers are
now doing their building poker BOTS that
can play against each selves themselves
and each other and work out how the game
should be played and what’s perhaps
surprising is the way in which they
learn to play the game as they’re
playing at each point in the game rather
than looking forward and saying what
strategy would give me the best possible
outcome instead they make decisions and
look back and ask what have happened if
I done something differently as methods
known as regrets minimization in fact
the bots at each point is working out
how much it regrets its decision and
there’s evidence that this is important
for human learning as well people have
damage to the part of the brain that’s
associated with the ability to
experience regret can struggle in games
that involve uncertainty and chance in
studies where they’ve been given games
of logic for example putting cards in
order they
absolutely fine but as soon as there’s
hidden information and risk they really
struggle to identify the optimal
strategy so it suggests this ability to
experience regrets may be very important
for how we learn to make decisions in an
uncertain world and these techniques
have certainly been very powerful for
these bots in fact these poker BOTS have
been playing against each other billions
and billions of times the point that
they’ve got so good that in 2015
researchers in Canada announced that
poker is solved now let’s be specific
here what they’re shown is that for the
two platform of the game with limited
stakes they’re bought had found a
strategy that in the long run would not
lose even against the perfect opponent
in the long run this bot was essentially
unbeatable it was an amazing achievement
not least because a lot of these
researchers by their own admission
aren’t really poker players but all they
are good at is getting machines that can
learn and this bot has learnt some of
the strategies I’ve already mentioned it
Bluffs its plays with an element of
randomness and when you play this BOTS
it can feel quite uncomfortable
it feels somehow like you’re being
deceived and manipulated and it’s not
just these world-class sports where this
can happen when I was writing about this
topic I built some very very simple BOTS
to try and understand more about them
and two things struck me first it didn’t
take much to build something as better
than I was and that’s a weird feeling
losing to something that you’ve built a
few hours earlier and the second thing I
found is it didn’t take much for the
bots to frustrate me and frustrate me in
quite human ways it sometimes felt like
it was almost toying with me and again I
knew that wasn’t – I’d built it after
all
but I still have that sensation I’ve
noticed that and interviews that people
will often talk about these BOTS in
vitamin terms they’ll slip into using
human pronouns and perhaps it’s not that
surprising given that these machines are
evolving many of the techniques and
behaviors that we often associate with
human decision-making in some cases
these machines are challenging our ways
of making decisions this bot that solves
poker would sometimes do something a bit
odd it would get out a pretty bad
and and bet on it would get dealt the
best possible hand it wouldn’t bet the
maximum that’s not just this but that
would do it the following year a book
called Claude eco played some of the
world’s top humans at No Limit poker so
a more complex form of the game again it
would do some weird things is sometimes
bet loads of money to win a small amount
or it would seemingly hesitate and be
indecisive and in poker there’s a tactic
known as limping which is where the
start of the round you bet the minimum
possible amount you limp into the game
as it were and it’s often seen as a
nuisance tactic it’s how many beginners
do poker professionals look down on it
but portico limps surprisingly often
about 10% of time in fact in Latin
Quarter ko means to limp now why do
these BOTS come out with these seemingly
odds behaviors well when we play games
we have to make mental simplifications
to deal with the complexity involved we
say this is a good situation this is a
bad situation we do exactly the same in
daily life we all make approximations to
deal with the complexity of the world
around us but these BOTS are showing
that those simplifications we use in
games are too narrow that sometimes
these extreme seemingly weird behaviors
can actually be beneficial and these
BOTS because they’re not constrained by
what is reportedly a good strategy
sometimes picked these behaviors and do
well out of it and these BOTS are also
challenging the role of emotion in these
games when researchers first shows from
the early version these BOTS to poker
professionals the PERT is noted that
these bots were quite passive and in
human games aggression can be a very
effective tactic it puts people under
pressure it forces them to make mistakes
but BOTS don’t really react to emotion
in the same way and as these BOTS
improve is likely that we’re going to
see changes in how humans play these
games and the role of emotion in those
kind of situations and these BOTS are
improving last year four of the world’s
top players played a BOTS at two-player
no-limits poker so again a more complex
form of the game and a form of the game
that for a long time has been thought
safe from computers many have seen it as
a game of humans
psychology and creativity is more of an
art than a science
unfortunately no be told the bot it
turned up and it won and these bots are
going to continue to improve
historically you need a supercomputer to
build a superhuman poker BOTS but last
month researchers showed that you could
now get these BOTS to master poker on a
desktop computer soon these things will
be running on smartphones perhaps all
these developments sound alarming but I
believe there’s an upside to this
progress because games like poker so
similar to many of the situation’s we
face in real life we could potentially
learn a lot about the sorts of decisions
and risks we face every day by losing to
machines we have the chance to gain a
better understanding of ourselves thank
you
[Applause]
[Music] [Applause]
Please follow and like us: