to explore standard deviation standard
deviation is a statistical concept we
are going to talk about what it is why
it exists and what it’s good for and how
to implement it and we are going to do
that in JavaScript that is what we’re
going to talk about today I am mpj and
you are watching fun fun function I know
that some of my longtime subscribers
might be thinking why statistics why a
statistical concept this is a
programming channel what we are learning
some statistics because statistics is
the basis of of machine learning this
episode is kind of like an alpha beta
thingy I want to do more episodes about
data science because I think that we as
programmers are moving into a future
where we cannot get away with not
knowing that data science machine
learning and and analysis and all that
kind of stuff is becoming more and more
used in in business and pretty much all
all venues of software development is
just growing more and more and more and
let me talk a little bit about why I
think that is it’s 45 it’s actually an
interesting example in this context
whence what if I got started and some 10
years ago they did some user testing and
they they asked people what they wanted
from a cloud based music player and they
people were at this point they had these
mp3 libraries with a couple of thousand
thousand songs so that is there was
their frame of mind they wanted they
said that they just wanted everything in
a list the problem is that if everything
in in a list you would you would start
scrolling from the top like living on a
where our bar is and stuff and then you
will scroll scroll and scroll and scroll
and scroll and scroll and scroll and
scroll but the the the little cursor
scrolling thing wouldn’t even have moved
because there was so much music on the
service when
you have a million bajillion songs like
the old conventions of just displaying
data straight up it does not make sense
when I started using computers I don’t
know exactly like 20 years ago perhaps
the everything was sort of very database
II like everything all the interfaces
were tables it’s kind of like everything
breeds from Excel and databases because
the in computer science and programming
the tuple or the database Row is a very
useful useful form to think about data
and to store data it’s basically how
data is often stored I mean in no sequel
databases it’s kind of like these data
objects but they’re they’re still kind
of sort of rows so the interface is
twenty years ago they were basically
just a very thin layer on top of the
data I just showed you where your files
your mp3 files in a music player like
winamp
and it gave you some search ability
things you could put them in other lists
and other tables like playlists but in
the end it was pretty much like showing
you the data in a very unobstructed form
and this is well meaning now it just
it’s it’s meant to give the user as much
power as possible and let the user
figure out for their o on their own what
they want to do with all this this data
and it gives you some data management
tools as well however over time we kind
of learn that this thinking might be may
be problematic the user doesn’t really
like spending time with your software
the user does not want to use software
they when they are using a music player
they want to they want to play a song
that fits their mood for instance or in
the case of a a check out a shopping
cart they want to just whenever they
this as soon as they have made their
purchasing decision they want to move as
fast as possible from that point to
you the actual checkout and payment and
the the more time the user has spent in
that process the the worse the worst of
they are and the the the more you have
failed as a software development team in
a utopian world the the software just
magically knows what the user wants and
gives it to them and the user really
doesn’t have to use the software feel
wrong with it at all
so in recent years we have more and more
software that kind of learns and figures
out and looks at data for the user and
figures out what the user wants Amazon
started doing this with the shopping
experience it’s kind of started figuring
out that if if a user that bought this
thing also bought this thing and there
is a big pattern there you you you can
do you can kind of when a user picked
put something in their shopping cart you
can based on that data recommend them
that they also put this thing in their
shopping cart Spotify has an extremely
successful product called discover
weekly which is again just a list but it
is an automatically generated list that
is based on your previous listening
history in the last months or year I
don’t know it it generates a list of
recommendations for you that you will
probably with a some more tied degree of
probability that you will like and that
you haven’t listened to yet there’s no
interface for discover weekly other than
the user just doing what they they do
and based on that data
Spotify generates generates a this this
magical playlist for you every Monday so
there is this kind of movement in the
software industry from this this notion
that instead of giving the users
software that is kind of like a very
powerful powerful database where the
user can do these slew lookups we have
software that does the lookup for the
user
20 years ago the user was expected to do
the data science and analysis themselves
and 20 years from now I firmly believe
that software is expected to do that job
for the user and that means that we as
software developers need to learn data
science we need to figure out we need to
learn how to do data analysis in our
software because that is what software
will primarily be about don’t get me
wrong there’s going to be interfaces and
apps and click it click II things
forever but I think that a growing and
maybe even majority part of of software
is going to be software like discover
weekly or Amazon’s recommendations or
Netflix recommendations this kind of
thing that just based on a big data set
figures out what it is that the user
wants and that is why you as a developer
needs to start looking at learning data
science okay so what is standard
deviation why why do we want to use it
and I know that some people are getting
antsy right now like oh just show me how
standard deviation works but it is
extremely important that you understand
why you are learning something because
if you don’t your brain will just start
you okay be critical and like why is he
learning this night no no what stop you
this is I don’t understand how this is
useful and your brain will start
identifying this thing it’s boring and
when it’s boring you just lose all kind
of motivation and you will not be able
to learn it or if you are you will
eventually forget it this is it’s
extremely important to learn why okay so
what is standard deviation let’s just
standard deviation is given a data set
of points like to do like the example
that we’re going to talk about today is
average know the order
per day so on Monday you had 10 orders
on Tuesday you had 13 orders on the next
day you had 20 orders and and so on
imagine a data set like that and so they
differ from day to day right different
on Tuesday different on Wednesday
different on Thursday standard deviation
is the deviation from the from the norm
from the average in that in that data
set it’s a number that describes how
much the the order number
normally fluctuates during a time period
okay so let’s let’s draw this on this
scale here we we have orders and on this
scale here we have time time so in our
imaginary ecommerce store we have like
five orders on on Monday we have say
seven orders on Tuesday we have we have
nine orders on Wednesday and then it
drops a little and we have have seven
orders again then here it goes up to 16
orders and then it drops again to say
six orders let’s say that we are we want
to in our e-commerce system we want to
create some kind of alerting system when
we see spikes like this partially
because if you get a lot of orders we
might need to the person planning
staffing might need to be alerted of
that or maybe the drop is huge and there
might be some problem with the system or
some internet connection thing or you
know some bad news or whatever we want
we want to detect anomalies and how do
we write software for a thing like that
I mean we cuneus hard-code the values so
that we kind of figure out what that the
app we we just do some back of the
napkin calculation and we figure
that the Ambridge is kind of I don’t
know perhaps perhaps four or something
the average is here and then we just do
like some send an email when the the and
the order amount per day goes would like
far enough from the average that it’s
its range we was just write some numbers
but it doesn’t really scale because as
the as the store grows and the varies
from from week to week like absolute
numbers well just it’s just gonna break
down and this is where standard
deviation comes in it is the average
distance to the average so like we
figure out that the average distance
from the average is somewhere here now
let’s say that say that it’s here
actually this one this regardless like
so all of these are within the standard
deviation because it’s this distance
from this line and this distance from
from this line all of these points here
they are normal but this one is
considered to be outside of the norm and
thus we can trigger the alert after
everyday using the standard deviation we
can just check we can just calculate the
average for the current time period like
say that to the last sixty days and we
can see like how far the the point that
the current of the current day that we
were checking how far that is from the
from the the mean or the average and we
can then calculate the standard
deviation and see if that distance a low
or over the standard deviation here so
this one is okay because it’s not too
far away from them but this one is we
know that oh it’s all over the standard
deviation this this is this is
suspicious we should throw an alert oh
right let’s see if we can implement this
and you might wonder
suddenly wearing glasses and it is
because I shot this two segment before
but i forgot to screen recording the
entire thing so all right so let’s first
invent our dataset let’s call them blue
blue blue orders I guess or like yeah
let’s call them orders this is gonna be
our orders per day the first day on
Monday three orders tie Tuesday’s five
orders seven orders on Wednesday 8:00 on
some other day perhaps five again and
then it’s Christmas so it’s like lot of
orders and then eight and then four okay
so the first objective if you remember
my crappy drawing is to figure out the
the average the average number of orders
per day because without that we can’t do
anything so let’s first figure that out
so how do we figure out the average
orders I guess we need the total orders
divided by orders the number of data
points we have orders dot length I want
a little bit of inline evaluation here
so I’m going to start
guaca guaca is a cool little plugin for
visual studio code that allows you to do
inline evaluation which is really good
if you’re doing screencasts like I do so
we see here that we have an error on
this line a total orders it’s not
defined because well we don’t have that
and total orders is a bit wrong we want
the sum of the orders and some orders
how do we do that go to juice reduce
orders dot reduce if you are not
familiar with reduce I have a video on
that topic in
that corner or that corner or in the
episode description if you’re being
confused reduce takes two arguments the
second argument is the sum that we are
starting with which is zero and the
first argument is a function which has a
the first argument is the sum which on
the first first iteration is going to be
zero and then the second argument is the
order that we are iterating and I’m
gonna call it X because I feel like I
might be being an asshole today that’s
why I’m using X and it’s could be X plus
some this is the most basic reduce
operation so many things oh I
accidentally opened up a output terminal
hi let’s close that and all right so if
I just remove this no yes no okay so in
order to get quokka to in line evaluate
you have to do this then it in line
evaluates so that means that this is now
65 that’s nice but see if average orders
works as well I mean this look quokka
highlighting thing it has to touch it
won’t work
splendid all right so now we have this
line here the next step is to figure out
these lines here which is the average
distance from the mean the average like
I’m like what is the average distance
from this line between all these dots in
order to calculate this average we first
need all the differences so let’s see if
we can do that I’m just gonna call it
differences and it’s going to be orders
go to use math if you don’t use math I
have a video on that too you can check
that out here and in the episode
description map takes
one argument a another function which is
a it’s sometimes called a mapper
sometimes called a callback sometimes
called a predicate it’s the same thing
they it is a function that takes the
each order each item in the array and
feeds it into it and it returns
something new and then that becomes
another array with the same with those
new values I’m just going to make a
function here that doesn’t transform
anything so I can illustrate this by
saying like I if I text plus one you see
that all the numbers increase by one if
I do a to all numbers increase by two
and so on
however that is just an example what we
want to do is subtract the average
orders per day which gives us these
differences here remember we are looking
for the average distance from the mean
I’m just gonna call it this average if
that and you know what we now that we we
have this array here that we want to
calculate the average of and we have
that exact logic because we do that on
the orders up here so we’re already
doing this here so I’m going to break
this out into a function so I’m gonna
call this left array average and it’s
going to be take it be a function that
takes an array that array we’re gonna
call this on steel that oak and I’m
going to steal this go there that should
do it and then I can just if we remember
that this is eight point one two five
just going
place that with Ray have read of orders
and it’s the same cool I can just remove
this that oh this one this is a mistake
though it’s gonna be had to be are there
okay now we can use do this ever in
order to get the average difference he
has to average ray no no array average
of the differences that will give us 0
that’s wrong
well this is a problem because all of
these these are differences and the
thing is the nature of differences is
that if we sum them all together they
are per definition going to be zero
because they they negate each other so
what we want to do yes that we want to
basically remove the minus a zero
because we are interested in in the
distance of these things from the mean
so now look at have a look at this array
how it changes when I do another map and
I pass in math dot absolute so you see
here that it just takes the and makes
the makes a number into its absolute
counterpart which is basically removing
the the minus sign let me show you when
I remove this here see BAM – oh yeah so
now it works we have these differences
here they are correct and the array
average is also correctly calculates
four point two so now we have these
but now we are getting into the annoying
part of this presentation because what
we’re looking at here is almost but not
quite standard deviation to do the
strict definition of standard deviation
we need to square all these numbers
instead of absolute in them and we then
should we will calculate the average
just like we did do here but after we
calculate the average we have to do take
the square root of that average so we
square the differences and then average
them and then we square root them to
unsquare them the reason we need to do
that is sort of because of math
compatibility reasons with algebra and I
will go into that a little bit more
later but for now just just go with me
here and we’re just gonna do the the
squaring thing so that we knew what we
are doing before we we talked about
about why we need to do it so let’s
first square this and you square things
by multiplying them with themselves and
we let the standard deviation be equal
to the average difference or the square
root of the average difference BAM and
now we have the standard deviation and
now we again have this thing these lines
here but we have it in the
mathematically strictly correct way
which is called the standard deviation
okay I think that we can actually use
this to check if something is an outlier
now like remember that we are we are
writing this code in order to identify
these outliers and
new alerts for them so let’s figure out
they have something as an outlier let’s
go through the orders array create a map
of that and for each order we’re going
to see how far it is from the
t-t-t-t-t-t-t-t-t-t
the average order let me just show that
is displayed in line and and then I
think that we can just subtract the
standard standard deviation – and check
if if that turned out to be more than 0
yes you see that 3 5 7 8 5 25 false
false false false false true false false
so types correctly identified 25 as the
as the outlier so you see that you can
easily use standard deviation to detect
anomalies and in datasets like this and
you can use standard deviations for for
lots of things but this is one of the
many use cases standard deviation is one
of the most common phrases or concept
that you will you will hear in data
science being thrown around a lot and
it’s very useful but I also promised
that we would talk a little bit why we
have to do this like why do we have to
square this and why do we have to then
do the square root of this thing it was
so much cleaner to just do math dot
absolute like from a programming
standpoint this seems a bit nonsensical
this number also doesn’t seem quite as
clean I feel what I mean it absolutely
works but why why do we do this well
mathematicians are strange well it’s
strange but they like relationships
between numbers I’m pretty new at this
thing so I
don’t exactly know what what it will
entail but I can tell you that this
thing here the one that we call the
average difference actually has a a name
in in statistics and it’s called the
variance variance is a very useful
concept in in statistics it’s it has all
these interesting and rubric
characteristics or so I’m told I’m new
at this
and it has that partially because it is
it is squared its squaring things has
introduc it’s very good when you’re
doing algebra I’m making absolute
numbers it’s very tricky to to use that
in in algebra at all like it just misses
algebra up and even though I don’t know
exactly what this will be used for down
the line I can make it parallel to
something that you might know and that
is the Pythagorean Pythagorean theorem
which is the relationship between the
sides of a triangle mathematicians has
figured out that there is a relationship
that between the 3d lines of a triangle
so the I don’t remember which I think
it’s a with the side a squared plus the
sine b squared is always equal to side c
squared when you’re first exposed to the
Pythagorean theorem it seems like an
interesting novelty but not more but
it’s actually ludicrously useful because
it can be used to calculate the distance
between points and all kinds of
interesting things and then you can
shuffle this around to figure out the
distance even though you only know two
points and it’s just it’s it’s pretty
cool from an algebraic standpoint and
very very useful and I and I trust the
mathematicians that variance will
eventually become very useful and we
will be able to use it for many things
like that but basically it’s because of
the nice algebraic properties of squared
numbers
do this squaring and then on the square
in using the square root kind of thing
and that is all I had we have explored
standard deviation today and then I had
a long rant about why we need to learn
statistics as programmers and I it’s
because I think that we currently
delegate a lot of sifting through data
and figuring things out to the user but
in the future we will be expected to do
that for the user to a bigger extent and
to do that we need to know data science
then I made a spectacularly crappy
drawing where I showed a bunch of points
on an hour in an e-commerce system show
that we were supposed to calculate
average and that the standard deviation
are these these lines that indicate how
far on average data points in our data
set tends to stray and then we can use
those averages to figure out when some
bloody outlier comes along and we can
create an alerting system for that for
instance this is just one of the many
uses for standard deviation and then we
actually did an implementation where we
calculated the standard deviation of
this of a given data set we used map and
reduce to do it and initially we did a
simple implementation where we just used
made the differences in two absolute
numbers before creating an average of
them then I explained that that was
almost but not quite the strict
definition of standard deviation and
then we did the strict definition where
you do a squared version of each
difference and then do an average of
that and then pick the square root of
that number and that you get out and
that is the standard deviation you have
just watched an episode of fun fun
function I release these every Monday
morning Oh 800 GMT but if you don’t want
to wait until next Monday morning you
can check out this episode
which Google’s machine learning worm
that they have in their basement has
determined is the correct episode for
you to view right now I am mpj until next Monday morning thank you