Press "Enter" to skip to content

GOTO 2012 • Frankenbuilds; if Agile is so good, why are our Products so bad? • Gabrielle Benefield


and now we’re going to talk about how
agile esters might be killing the planet
who was here for Jazz’s talked earlier
on value hands up most of you and who
was here for Josh’s talk okay there’s
going to be some overlap hopefully I can
get into a little more depth because I
came along to their sessions to make
sure that we weren’t going to have too
much overlap so similar theme let’s go
through it agile esters are killing the
planet so agile lean scrum Kanban they
have all the answers do they are they
really solving our problems or are they
just helping us to build the wrong thing
writer there’s been a really big focus
on let’s make our development work
better the issue of that is we’re
delivering all these milestones we’re
delivering working software so the great
promise of agile was that hey you know
what for so terrible because they look
at phases we say that we’ve done
planning we’ve done delivery we’re
making all this progress and we sort of
trashed that in the agile community we
said well hey that’s that’s useless or
we could have as a pile of documents we
might not have anything useful to show
for it so instead we’re going to quickly
build working software show it to our
clients let’s see how that’s turned out
for us in fact all they did was push
more through the system faster so now
what we’ve done is we’ve gone from a
model where we tracked activity right we
track that we’d actually finished some
work activity we’ve gone to tracking
outputs we have built more stuff but is
that actually a good thing
so I hear so many people saying we get
our products out on time and on budget
does it matter if it’s the wrong thing
for the wrong people
a dog takes a machine gun to product
development so what we’re effectively
doing is taking lots of shots and hoping
that some of them hit our target the
issue of that is we take out a lot of
innocent bystanders on the way and
effectively we could be creating a whole
lot of waste so output is not
necessarily a good thing
right what we’ve done is effectively by
building more increased the odds of
success by using the machine gun whereas
what we really want is a very targeted
sniper gun
I was at a meeting once and the VP’s
stood up and said hey team it’s so great
we delivered 35 features on-time and
on-budget and everyone’s clapping away
now at the back saying you have it did
we make money did we deliver results and
in fact would to have done the same job
capers Jones had an interesting
statistic in one of his books on
estimation he said that 30 development
accounted for about 30% of product
development costs which means there’s
70% surrounding it over the years I’ve
been doing agile for a very long time
and various forms I found that you know
it’s not their jobs being a bad thing
it’s cleaned up a lot of the mess in the
middle zone that development zone and
the great promise of frameworks like
scrum is to make the dysfunctions
visible so what we’ve now found is that
we clean that middle mess up we’re
making stuff get through the pipe
quicker and instead what we found is
that what’s coming in the requests and
what’s getting out are the wrong things
so instead of building the wrong thing
writer so instead of saying you know the
big questions mean which is better
Kanban or scrum completely the wrong
question it does not matter if you’re
building the wrong thing who are doing
is gold-plating the development
mechanisms we’re not actually improving
the product itself today we want to
build the right thing so I’m going to
talk to you a lot about building
outcomes over outputs now people who now
have Tom Gill
back in the 60s he’s been saying this
he’s been saying hey we need to measure
value and it’s taken a long time for the
rest of the world to start catching up
Tom’s a brilliant guy almost too smart I
think it took this long for everyone to
finish competitive engineering book but
what Tom’s been saying for a long time
is you need to measure value nothing
else matters his whole models based on
Evo which is little one-week iterative
steps to get feedback and to look at
your progress measures so I’ve been
working Tom a long time to figure out
you know how do we make this more used
and more acceptable to the world so
that’s been a bit of a lynchpin in what
I’ve been doing to give a bit of
background in the late 90s we were doing
some X
he just very technically focused 2003 I
was in a San Francisco startup I came in
to run the product team and then they
ended up giving me everything they gave
me product development operations etc
and they said in six months we want to
take the company public so I want to do
a big IPO and at this point this is a
company that has like six warring VPS
this is like Afghanistan and practice we
have everybody disagreeing we have
separate lists think of those as six
different back logs for all the agile
people and we don’t have that bigger
development team so we have everyone
building lots of things lots of work and
progress nothing actually getting done
so when I and I said ok well let’s start
cleaning up this mess and we started to
use a bit of lightweight sort of scrum I
started bringing in lean thinking lean
portfolio thinking and the number one
standout item was that we were on this
massive feature farming Drive what we
were trying to do is put so many
features out at Lightspeed and we were
going to make lots of money
who seen those proposals which come
through all of these features each
features worth X amount of money I’m
having a big argument with my co-authors
of the scrum primer there’s a little
column in the backgroud backlog which
I’d sort of missed the first time which
assigns value to each product backlog
item I’m like how do you know you know
what’s the value of something like a
logon that’s way too detailed and so
these people thought that each feature
was worth money I looked at I thought
that that’s not our problem at all so I
went up to the board of directors and I
looked at them I said ok so who goes to
Amazon and eBay and I’ll ask you all
this who goes and uses Amazon and eBay
regularly like at least one for once a
month right sure we do all the shopping
there because we don’t like to get up
from programming so I said okay do you
who was one of the people who uses it a
lot who would say they like a hardcore
user of Amazon or Ebay hands up okay ask
you do you go to Amazon or Ebay every
week because you’re wanting to see
what’s the new feature that they’re
going to launch for you to use no what
do you go there for for the books and
what else
eBay
No so you go there for the books you
want to buy something it’s pretty simple
if your Amazon simple value statement I
want to buy something if you’re someone
like Skype simple value statement I want
to make a call and yet most of these
products need to go out of their way to
put as many obstacles and features in
your way as they possibly can
so the more features you send out the
worse it actually gets so this was that
problem we’re on this feature farming
exercise and in fact that wasn’t a
problem at all so I said we’re not going
to build lots of new features we’re
going to perform improve the performance
and usability all the qualities of our
website our revenue curve went from
being small hardly going up until so
steeply vertical than in six months time
when we were ready to go public the
bankers called us in and said stop
you’re not allowed to go public I like
what we’ve done a great job we’re almost
down to zero defects we’d actually
halfway through being actually replacing
legacy code where the best XP is on the
planet everything was going so well and
they said I said look at the revenue
curve and they said that’s the problem I
don’t know if anyone’s ever tried to
take a company public why would our
revenue curve be a problem because it
goes up any ideas okay be careful of
this if you get really good at what what
you’re doing the problem is that it’s
not sustainable they’re like no one goes
into a vertical cliff jump looks like
you’re actually fiddling the books so I
had to literally go back to my team and
say okay guys stop what you’re doing
let’s actually tell that off get a
sustainable curve so we could take the
company public so that’s the dramatic
difference that stuff we’re hearing
today about lean startups etc that’s
what matters we were lucky enough to
have the chief scientist from Amazon at
the time Amazon it’s a company that
drives themselves by data everything
they do they test it’s amazing Amazon in
the early days if anyone used it back
then but it used to take about three
weeks to get a book two to three weeks
right you’d order it get a book some
went up and said well how can we get a
book and maybe you know let’s get it
down to a week three days one day then
who had the brilliant idea you know some
wants to get a book would you like to
get a book in under one
it cause right it’s the worst worst
product in the world for me I’m sitting
at something like a conference I’m there
and I’m like one click by it’s
downloading to my Kindle fantastic so
they get the whole idea that’s all about
that user experience so one thing that
I’ve been working a lot with lately is
guy called Range Rover in the US and
I’ve been creating something called the
target outcomes framework so we really
want to drive home you’ve got to go for
outcomes not outputs okay something that
we need to re-educate our clients in and
as you heard joshing around talking
about earlier velocity all these
estimator measures they are just getting
in the way of what we need to do which
is to deliver value faster so think of
outcomes outcomes are the results we
expect the the destination we want to
get to options are once you create
outcomes these are the results there’s
many ways to get there so I’m going to
talk through how to get started I wanna
give you a little bit of practical
information we’ve only got a little bit
less than an hour this is something
we’re going to have a book coming out on
this and some more detailed guidance
what I found
I love the Lean Startup book but when
you go through it it sounds fantastic
until you try to do it and if you’ve got
a little website that’s okay but when
you’re running this at scale I run it
with company who has literally millions
of users and very complicated product
sets if you’re doing this at enterprise
levels I find that it starts falling
down they can be little there’s not a
lot of information out there on how to
get this going big scale one thing you
like to do there’s a lot of lean people
in the room who knows about a3 thinking
coming out of Toyota ok not a lot of
lean people lean gets kind of
bastardized in our community it tends to
be about reducing waste it’s all about
it in a way trying to make things more
efficient great ideas but one of the key
tenants is about this idea of learning
now a3 thinking there’s some good books
on it there’s a Sobek book on a3
thinking there’s another managing to
learn book and it’s a very deep
discipline it makes you think about
systemic reasons and why we have
problems or causes for things so this is
actually something we’ve blended into
model so that we understand the context
the users the environment extremely
deeply before we start deciding where to
go I think sometimes we sort of leapfrog
that we jump straight into we want to
get here but actually we don’t know the
problems who don’t know the reasons why
so one thing you can do we’re hearing
about user experience I’m going to tell
you to be the method actor of user
experience you know robert de niro these
famous actors go out they actually have
a movie where they’re going to be
working behind the counter in
supermarket they get a job in the
counter of the supermarket so never ask
you users what they want and in fact the
worst thing you’d ever do is ask your
developers what they think the users
want or even your on-site customer right
they might be million away at miles away
from your actual users I always love
that quote which you probably seen if
I’d asked my customers what they wanted
we would have built faster horses so one
thing you never do is go and get that
product requirements backlog list for
users you’ll build the wrong thing
almost guaranteed get out of your chair
and go find out what they need I’ll talk
about more about this later but this can
be done very quickly and easily there’s
a great story I do one of these Silicon
Valley think tanks they build great
products they were investigating they
got a job to create children’s
toothbrush so everybody at that time
thought children are little people right
little people have little hands we’re
going to build a little toothbrush so
everybody had built these little
toothbrushes right thinking little hands
little people little toothbrush they
went out and actually watched little
people doing their toothbrush stuff and
guess what they found that everyone’s
got kids little kids are little
toothbrushes they brush their teeth like
that right they have no fine motor
skills so actually completely wrong
solution they came up with those big
toothbrushes they were the guys who came
up that invention so the kids could
grasp them easily
they went from something like under 20
percent market share to 75 percent
market share surely through going out
and observing the users Yahoo we used to
do a lot of the stuff we’d beg borrow
steal go and get friends whatever we
could do to get a sin
and to understand them so remember I
said earlier you want to go to Amazon to
get a book you are on skype to make a
call those are some of the things that
when we go in we both look at those
goals if you’re in existing products
which a lot of us are could even be your
business processes doesn’t matter figure
out what it is they want to do we use
some of your lean techniques like cycle
times to measure those what we’re going
to do is keep iterating to make those
better and better and better so figure
out what the core jobs are and always be
very specific about making that as easy
and quick for our users to do create
strategies for getting to market at the
right time I saw a Jess had a little
target with the one person in the middle
too etc great idea
be aware that product development is a
bit of a black card it’s far more
sophisticated if you’re at a startup you
have different users at different life
cycles for example if I’m a startup I
want to get viral thought leaders so I
want someone like one of our personas we
created I was in the innovation group at
Yahoo we went out we’re looking at a
youth product and we targeted the
scholar called Susan because we saw in
the breaks
she had a tech piece of technology she’d
be looking at it there’d be like eight
kids all around here watching what Susan
was doing so she was the thought leader
we thought you know what if we want to
get this product out to many kids we
want her to be our target that was one
of our personas after a while we then
want to go for the kids who are going to
be the repeat users so again a lot of
the skits Center a lot of sophisticated
thinking about our users I mean at the
moment with some teams I pretty much
start with drawing a little stick figure
bubble head it’s better to have a user
than no user so don’t want you to think
you have to get super sophisticated but
again create empathy and get people to
drive products our software is used by
real people um once we do this analysis
again and going through fairly quickly
but we’ve decided what’s really going on
with our products um I do a lot of get
data gathering that’s one of the first
things I do at companies one of our
clients
who uses builds a product you’ve
probably all have used millions people
are using it around the world we went in
and said okay what are the usage
patterns what are people really doing
with this product it’s really
interesting three days later one of the
product owners came back and said you
know how you told me to investigate all
the stuff and get some of this data back
he said we found this one feature that’s
been on the backlog for a while if we
launch it now it will actually work if I
said we’re about to launch it tomorrow
it’s pretty simple that’s going to save
us 12 million pounds a year so
immediately by looking at the data they
could figure out what’s going on that
was great that was our first week of
consulting at a client so that pretty
much paid for us so once we’ve done this
data gathering we’ve understood our
users because what we’re trying to do is
do deep deep thinking we can use
techniques like five why’s value stream
mapping what’s really going on once we
understand it now we can do the impact
mapping if you know the severity of the
problem or the opportunity that helps
you immediately prioritize I love those
agile classes just a quick aside where
they say you can do bubble sorting you
get two stories you go let’s do a or B
which one goes higher and it’s like
seriously is that a way to build a
product you know how do we that makes no
sense to me instead where we’re going to
get the most impact
put on your 8020 thinking caps we’ve all
seen those Standish Group reports but
are the ones that show that about 20% of
the features used 80% of the time this
is what we’re doing as the agile
community we have a great method for
getting products out quickly optimized
for that 20% and we’ll talk about ways
you know which is the right 20% but it
starts before you waste all your time
building user stories understand where
is the greatest upside the problems that
we experienced we set the target
outcomes so target outcomes I’m going to
call them the pocoyo key of product
development anyone heard that term
Pocoyo key before yeah see if you throw
Japanese words and he was like how cool
is that so Pocoyo key make it easy to do
the right thing and hard to do the wrong
thing
haysoms exam because it makes me sound
like I’m about 200 but floppy disks when
you opened up the little fluffy disk
container
there was one way to put it in right and
not the way to put it and wrong right so
they designed in idiot proofing right
they made it very easy to choose and
this is what we need to do is
idiot-proof our products so that we can
tell which features should go in and
better still which features not to build
remember we’re dealing with the world
that we’re getting so good at building
stuff fast that’s making us lazy and
it’s making easy to build too many
things my best product donors I work
with are very good at saying no well not
now I’m not going to do it now
target outcomes need to be clear
measurable we’ve kind of used a
simplified version of Gil’s work so we
have a very clear baseline again you
need data to set a good baseline we have
a very good target this is the point we
want to get to for example at Amazon it
takes two weeks to buy a book we want to
get that down to three days let’s keep
building things and features and
enhancements to get that cycle down okay
that’s a simple version of setting
target outcomes have fun and if you
don’t measure how will you know when you
get there it is a discipline I get
people it’s hard to get the Cystic at
first so I get people saying well we
can’t quantify things you know we can’t
quantify the user experience right who
thinks that who thinks it’s hard to
quantify everything any hands yeah ok so
I had this recently so one of the other
things I’m doing at the moment is
writing a book on legal contracts that
can be used for agile development it’s
actually why I spent a lot of time
building a target outcomes framework
because fundamentally legal contracts
are based on a goods and service
contract it’s based on coming up with
description of the goods you want like
you go to a shop hand over I want
buttermilk eggs they give you buttermilk
eggs you get paid for it fundamentally
flawed for the complex development we do
in software so now we need a way to say
let’s be results-driven let’s come up
with outcomes that you measure and based
on that we have to make everything
measurable so I was working with a big
charity I went
do a quick contract review just to see
what they were looking at they had some
words in there we want our system they
are building an internal CRM so they get
a lot of people contributing and
donating to their cause they want to
track what they’re doing at the moment
everybody is using things like Excel 10
different products etc they want to
centralize it be able to do better data
reporting etc and so they had written
the contract for the suppliers which
said we want our system to be easy to
use and to be very consistent ok so what
is consistent mean any ideas throw some
ideas out consistent first thing that
pops to mind same thing not go down
anyone else
predictable what a predictor will mean
see I like that version predictable that
should be fragile team so you can guess
what they might do ok right ok so I’m
going through it and I’m mentally
assuming it consistent while it means
that it should work like if you click on
a right button here it should be the
same right button and you should go back
and it should act that way and I said is
that what you mean let’s really define
it and she said no no I mean that once
you run your report the data is correct
so consistent mean there was actually
data that was validated and it was
actually correct and more precise data
whoa who knew right so you can imagine
the kind of systems we were about to
build with these people so when people
say you can’t measure everything yeah
there are some difficulties last week I
was down with a very big video gaming
company helping them to build their
video games
very cool industry and I was kind of
thinking how do I get this working what
a target outcomes for gamers right so
first I said them who’s your persona who
you’re building it for a gamer like ok
you’ve got at least three I can think of
you’ve got a hardcore gamer a core gamer
and a casual gamer very different news
cases one has to get into the game as a
casual gamer has to be easy to use
hardcore gamers need skill difficulty I
said ok and what are the goals for these
gamers well we’re going to build a game
where you win at the end measure Nets
whatever a racing game you hit the
I’m like wow that’s gonna be a great
game I can make that now I can build a
little racing track where they hit a
finish line we’re good to go
well no it’s about more than that what
is it about no it’s about fun it’s about
what is fun okay so we started breaking
it down it’s really interesting we
started building a little I was calling
them an emotion map so we had things
like attractors how do you attract
people into the game we had frustrated
if something’s not annoying enough then
you get a bit bored with the game right
we had to get to a point that someone
gets to massive elation and winning and
then there’s tail off what do they do
next how do you make it so that they can
re-experience that feeling get addicted
back to your game so there are ways to
kind of figure out what are the pieces
that matter how do we measure them right
a measure can be quantified but there’s
other qualitative measures as well but
what we’re trying to do and this is the
thing that I think most people miss
we’re not measuring to get the most
precise number just like we don’t do
things like poker planning to actually
get the best estimates why do we do
something like poker planning wow you’re
really proud you’re there to share
knowledge right yeah so what we’re
trying to do when we quantify these
things is get down to a level of
understanding so that we’ve created
poker your key easy to do the right
thing hard to do the right roll thing
okay so that you don’t want your product
on or your customer who’s got customers
that they can’t even get a hold of
anyone work with those people right I
get people where they’re dealing with
generals in the army who are like okay
product development team or war in
Afghanistan yes he’s going to win right
when you got those people you’ve got one
opportunity to really try and understand
their greater outcomes that make this
work because you can’t sit there going
should it be blue or green what do you
think how many guys like you know those
aren’t so things I can spend time on so
what we’re trying to do is we set a
direction once we have a very very clear
direction if you do nothing else then
this will give you a lot of power over
choosing what to do next in your
products then we want to go into
something called options thinking so
back to agile or whatever methodology
a lot of them based on outputs what
you’re doing as soon as you start
writing up outputs but your product
backlog your roadmap you’re committing
to a certain future you’re saying this
is what we think we’re going to build
this is the danger of current legal
contracts they force you into defining
upfront and committing even if you can
change it round it’s harder because once
you give a feature a name or put it in a
document has a life of its own and
people feel like you’re killing it at
that point of view take their life away
so we’re going to stop pretending that
you can see the future and again you’ve
had this theme pushed at you today about
Lean Startup create options so instead
of having one thing and to have multiple
ideas right we’ve got an outcome to get
to but there’s many ways to reach that
outcome so keep your options open what
we want to do is again this is back to
responsible moment thinking you want to
make decisions as reversible as possible
and irreversible decisions as late as
possible options thinking is all about
setting up potential ideas and then
being able to execute on them rapidly I
was talking the other day that’s
interesting but had a different slide up
there we were talking about we were
talking about yeah we were talking about
Darwin’s theory survival of the fittest
it’s not about survival survival there’s
a movie called dudes like a teen movie I
think they talk about survivals the
slovers film was suicide that’s a way of
trading order arrival as the of the
fittest is where it’s actually at so
it’s the people who can get to market
with the right product the quickest not
the ones who can just survive in that
market we want to be better than
surviving the outcomes options idea is
fractal so what you can do is have high
level outcomes for the company things
like reach engagement with our customers
monetization they should link through
and align to every area of development
right down to the qualities of your
system right so every big outcome for a
company breaks down into multiple
options think of that as your portfolio
your roadmap all of those break down and
two different options and streams and so
forth okay so it’s a very flexible more
of a model that looks like a very
complex system as opposed to the linear
stuff that we’ve got ourselves into an
agile so if you think about your product
backlogs they’re very linear they lead
to very tactical thinking that’s why
originally the talk was called Frank and
builds we keep hacking things together
and hoping for a good result so all this
does that sounds simple you can go out
and do that right how do you know what
the right thing is we don’t we have no
inclu right we never know I like the
idea we keep saying there’s no such
thing as product requirements their
product guesses they are options and
experiments we should run never think
you know in advance because we don’t so
stop guessing start testing we’ve talked
about the idea running multiple options
fast the thing is we’re all about Nigel
making the unknowns known that’s why we
do vertical slices of our product etc
we’re trying to highlight risks we’re
trying to get ideas and information back
quickly we talked about set based design
with Josh already so I talked about that
that was a slide I thought I’d have so
get an affray of your users early and
often another big tip I give you I
noticed a dramatic change we had very
advanced researchers at Yahoo user
experienced people we had people from
Apple’s innovation lands we had the
whole gamut what I found is they would
say hey this doesn’t work the people
don’t use it
the developers never quite got it so we
started taking our development teams out
on field trips with us
seeing as believing you need to
experience it even in our usability labs
I’ll give you an example Ajax had just
come out so you could drag and drop and
so the team had designed this great
photo album feature we could drag and
drop your photos all the execs were like
oh that’s the best thing ever you know
it’s going to be the breaking ground and
we got the lab going so we had someone
coming in behind that two-way glass and
they get in and they’re looking at it
and someone says okay so how would you
create an album and move your photos
into it and that clicking around the
light
no I give up behind the screen as the
developer he said anything I know what
you do you move you stupid person you
know because of course that user was an
idiot because the developers you know
obviously smart design this very cool
system so uh anyway they walk out and
Scott just said give me a laptop runs
over do we push it through to the next
room before the next person got into for
the next test problem fixed we’ve been
telling him this for two months Scott
they don’t get it people don’t get this
drag-and-drop thing it’s not it’s not
intuitive so by taking them out we
change the whole model I’m ignore your
mother be fast dirty and cheap hopefully
my mother’s not watching this everyone
comes to me I got someone the other day
we were I’m like they were building
something and I’m like why are you
building that way you don’t even know if
it works and they said well in q4 we’re
going to get a user testing group to
come in and do all this research on it
what do you mean don’t you have like
friends or family now like go and get
cutlet computers will set up this test
so immediately we get feedback to give
an example I was in New Zealand giving a
class there and I had someone from the
police department in the government they
were building all the software all the
tracking of the police records and she
used to be a policewoman and she said
you know what
our software is so unusable it makes me
sick but I tell the guys and they don’t
get it I said you’re a policeman right
yeah we’re local yep still got friends
there yep okay I said how much will it
cost for a little taxi to take everyone
in and no 20 bucks I said what about the
police people like doughnuts is that a
methyl true and she went actually they
really do and I said okay get a box of
doughnuts get your team go and spend the
afternoon with the police people so they
did so for 20 bucks in the cost of a box
of doughnuts they went and sat beside
the police people immediately they saw
Joe who was 56 with two kids exhausted
after being all down the beat coming and
trying to add his time and it was just
miserable they felt so bad
they created a little picture of Joe put
it on the wall and every time they went
to build something they went yeah
but would you really get that probably
not right so this is like fast dirty
chief a team in Bangalore India every
lunchtime they go out on a Friday so hey
we’ll give you a free lunch come and try
it a yahoo product that’s all it costs
to get rapid feedback so don’t think it
has to be expensive to do this remember
that features are not the experience
again poka-yoke k creates this is the
goal this is the job our customers need
to get done more features are actually a
problem what we need to do is remove the
in essential so the essential can speak
right the more features on the page of
the higher it is to use when the iPod
came out they had about 40% of the
feature set of mostly other mp3 players
because they got what the job people
wanted to get done was to actually get
music quickly and listen to it right
which meant you had to be able to
connect someone to get somewhere to get
the music and get it and listen to it so
again the game is to hit the target with
as few bullets as possible that’s why
I’m very concerned by current legal
contracts they incentivize building more
I’m concerned about things like velocity
cycle time measures which measure how
much throughput we can get to me that’s
telling your developers to build as much
as possible right I want you to do the
opposite so the game we have very smart
people we work with we set a game target
which is we need to get here as quickly
and with as few features as possible if
you do that maybe you can go home early
right I don’t want more code so this is
about that idea of Lean Startup we’re
building minimal viable not maximum
possible your job who are the product
owners in the room any yeah your job is
to try and filter that to get as little
through that pipe as possible measure as
you go now everyone’s listened to
metrics going yeah I get that we say
that we need 20 new signups or make X
amount of revenue what they do is they
put in a pitch has anyone seen those
bids for work we want these two hundred
resources and a year and we’re going to
make fifty million dollars the greatest
product ever
right we will see move don’t believe it
what we do is we constantly measure that
way we get instant feedback are we in
the right direction we set up automated
metrics frameworks so that we can get
this real-time feedback back at that
beginning step where we talked about go
and gather the data this is when you’ll
start to see the holes in your
organization’s do you have the usage
statistics do you know how people are
using your products if you don’t start
building those in as quickly as possible
this is what will drive your company
this is why Amazon so good they
constantly look at the numbers they set
up very sophisticated frameworks so if
your metrics are at lag your product
will lag you need to get those quickly I
talk too much about continuous
deployment this is where the agile group
can come in again that we say our job is
to make it as fast as possible to get
that product to market user testing we
talked about that I although I said I
used some lab testing I really like
sitting up multivariate testing the idea
of split testing with multiple outcomes
if you do that why test with five people
in a lab can actually give you
substandard results we used to be able
to very quickly at a start-up in San
Francisco flip the switch and
immediately we could siphon off a couple
of percent and be testing real-time with
thousands of users that gave us much
more contextual data one thing I haven’t
heard a lot about is we talked about
metrics and gathering them and a lot of
people think that’s for the managers or
the product owners I always treat my
developers I think essentially they have
a gamer mind so they like playing games
maybe but also they like things like the
thrill of the kill finding defects you
know they have a game and mindset so
when I set targets transparency is key
give all those numbers back to your
developers at the startup every month
we’d have a pizza meeting we will give
them all the spreadsheets we would show
which stuff we’d launched what feedback
we got what changed in the company so
again this is not for management this is
for everybody to drive by
try to have very visual progress
indicators so ways that you can say you
know if we want a certain outcome are we
making incremental progress keep those
very front of mind you can drop your
burn down charts I don’t need any of
that stuff I want to see are we actually
getting valid business results back as
we go so how do we know we’ve succeeded
we get to where we’re going and so we
hit the target outcomes and if we don’t
we very quickly adapt so we pivot at
this point to use Lean Startup language
so I get people saying well that’s all
well and good but does it really work
so which sounds better our velocity is
thirty six point forty six points and
we’ve delivered a hundred new features
or we just implemented one feature in
three days which will save us 12 million
pounds over the next year what would you
choose – here we are we did a pilot with
a company recently and trained up one of
our favorite product donors and they had
an All Hands meeting they had big
off-site with all the product managers
all of them put up their powerpoints and
we’d a little bit of these features on
time and on budget of course and so went
and Jamie just got up and went yeah well
we made X million and this is what we’re
doing and this is how we did it and all
the managers just went oh that’s the guy
and he’s you know going off in the
company pretty rapidly what industry can
he do this with we’re currently building
we’ve been building these huge automated
metrics frameworks we not only measure
products but also transformation areas
so I’ve gone to companies where they
have like an Aikido test are you doing
daily stand-ups like a maturity model
like oh my god why do you do daily
stand-ups to communicate so let’s
measure communication saturation if
we’re looking at Cost Reduction let’s
measure total cost of ownership for
systems so we’re putting all these
measures in and not only at a big energy
trading company we’re doing metrics
video gaming and we’re doing it for a
big voice over IP telco so this stuff
I’m again huge agile person have been
doing it for a long time to me this is
way way evolved
right this is so much better than what I
was doing before I mean yeah we’re kind
of doing a startup and then I got
distracted trying to make agile better
but actually this is the game we should
be playing so I want to talk too much
about how this changes everything I
mentioned legal contracts but let’s go
back so I know we always go to a time
where we started why do you think I said
add your Lister’s are killing the planet
that’s a pretty strong statement
especially for a CST by the way come to
my class on Thursday Friday night I when
you use the machine gun approach to
develop your products you’re basically
building more features faster the more
features you build the more resources
you use so if you think about it every
feature you build takes a certain amount
of machinery power etc to use each time
you have to maintain it the more servers
that build all your crap and store it up
the more beautiful feature farms that
you develop that’s using a lot of power
right so you’re actually helping to kill
the planet faster good on you so so the
features save the planet um any
questions yes ah great question
question was you know I talked about
this 12 million we can make how do you
measure the money uh well again back to
sort of the thinking of testings we can
get down to quite sophisticated
financial models because at the end of
the day I look at it that we’re asking
for investments and we’re showing that
we can create value back on that
investment right each bit of money give
to development’s investment so let’s say
usually I do it to give us some plastic
example I can’t use that one because I
get killed apparently in my India but
Amazon let’s say that I go through
Amazon it’s taking two weeks to get
books or maybe actually a very easy one
is if you look at the customer
conversion funnel so from when they land
on something like Amazon to when they
actually get through and get the book
where do we lose people so often you
know in the early days people found that
they’d get
percent drop off at certain pages you
can look at them and say oh weird they
get to the shipping page and then they
disappear
turns out all they wanted was the
shipping information maybe they get
through into the payment scheme and find
that they can’t actually use a certain
payment type this is we would start
doing split testing to try the things so
now we know that actually there’s a
lifetime revenue amount that many
companies can work out that for each
person who converts there’s an immediate
money but also you turn them into a
longer lifetime customer so you might
have a revenue model where 50 bucks a
year for each person that we have is a
certain conversion statistic so now I
know based on that for each one that
completes that conversion that if they
were worth $50 to me then say if I have
a million people we’re off at 10% okay
then you can go through and say for each
percentage that’s worth X amount of
money to the company right so now if I
know that hey when I’ve got to choose so
I’ll give you an example let’s say we
found that there was a big drop-off in
conversions when people bounced if the
page loaded slowly right so that causes
a huge drop-off I could look at three
options to fix that one could be by
looking at I say well our hosting is
based in England and a lot of the
drop-off happens for a lot of our users
in Australia because the data center is
so far away that causes a slowdown it
could be Network latency maybe we could
just catch some page elements there’s a
few options right thus we’re getting
into options now I can start costing it
out I know that if we improve that we
could potentially make X amount of money
which one of these options do I would
play out so I probably start with
caching the page elements I look at
maybe some network improvements what is
it but there’s a point where I know that
maybe the data center is too expensive
for the amount of value I potentially
get back this is where you get into fine
testing they have time to go through all
that is that okay yeah Wow you look
puzzled still good come to one of my
other classes about stuff you
and actually it’s a good point so the
question was how much is it do you
measure too much and again think back to
investment what’s the value we actually
are very clear about trying to measure
very few things to start with so we
might only take three simple measures if
you measure too much it’s confusing also
this is like you know if I launch 100
features all at once this is the problem
with big batch product releases how do I
know which of the killer features within
it right by doing that with taking away
our ability to understand what’s going
on
same with metrics you’ve got to be very
careful because it’s a system each thing
you change can change five other things
so actually you want to be very specific
about what you measure and when for
example if I’m a start-up I’m going to
be looking at virality how quickly does
my product get taken out their
stickiness to people come back to it I’m
probably not going to be looking at
revenue so much to begin with because if
no one uses it who cares right then I’ll
bring in other measures as we go so you
do have to know what you’re doing it’s
not a simple thing to get going but
figure out if this is our outcome what
are the things to measure at this point
all right
so you say about that maybe HB
split-testing and stuff like that can
help when your release products etc so
what I’m thinking is what about the
marketing side they are doing stuff as
well so how do I make sure that they are
not affecting the same stuff I’m
creating do you understand what I mean
you might have two answers for that
about marketing one is involved them I
get them involved in all of the product
discovery thinking because they can work
at odds for example if the game is to
get someone to buy a book marketing goes
and puts 200 packages up that might get
the marketing and come up but doesn’t
actually help the user experience in
fact it can detract from it that’s a
problem and then the other part of the
question was rummy I got back two
o’clock the morning he can’t remember
either
he can’t remember what was it what was
it about marketing those are really
important thing that I totally forgot oh
I’ll causation this is a big thing
so one of the nerve owners is that we’ll
have an outcome-based contract we’ve
actually got three one-page contracts
that I’ve developed with the UK lawyer
called minimal viable contracts how
original and one of them is a pure
outcome based model that you only get
paid if you deliver value and if you
deliver it early you can get huge upside
all the consultants can you imagine a
world I save that company twelve million
in three days if I got 10% of that how
cool would that be right so now you get
into causation ah but we just launched
200 other things how do we know that
what you’ve done has impacted the
results this is where you need to
measure at different levels so remember
I talked about the model being fractals
so we have target outcomes and options
underneath it for example you could say
we want to increase revenue that’s a
huge outcome for the company like Amazon
I might be at the level where I know if
I’m in the back-end server area if I can
keep the system make it more available
we go from 96% availability to 99 then
we actually keep more people on the site
I can show that that has an alignment to
the high level outcome right so what
you’ve got to be able to do is agree
that this is a level that you’re
comfortable with you want to keep
linking it you should see some variation
in the revenue but just as we could do
something that improves that bottom line
someone else could do something that
tracks by it so we have to agree that
our product level this is the right
level to measure it there’s a little bit
of an act of faith there but if you do
enough testing and can get feedback
again this gets into how you do split
testing etc we can show that there is a
link this is the hardest part about
writing the contracts is to get that
linkage correctly done so that we get
the suppliers saying okay we’re going to
do this that actually helps the
grade-level outcome the other questions
other questions okay thank you everybody
you
you
Please follow and like us:

Be First to Comment

Leave a Reply