oh good so hi good Monday morning it’s
not morning as you can see it’s evening
in New York and this is this is what you
get if you procrastinate and show again
intro all day anyway
so while I am while I’m in New York for
vacation then I thought that it would be
nice to shoot a video with a couple of
developer friends of mine were like
prominent fascinating and brilliant
people that I’m super proud you know and
just have a chat yes start with talking
about who they are and then jump up from
there and what we ended up with was
talking about data visualization data
processing in JavaScript Python d3 like
a bunch of that stuff and it turned out
to be super super super interesting
discussion so if we’re interested in
looking a little bit into data science
and whatnot this episode should be
really nice for you I think before we
get on with that though I would like to
read today’s sponsor which is brilliant
thank you brilliant brilliant is a
problem-solving website that teaches you
how to think like a computer scientist
if you’re interested in machine learning
and linear algebra and all that other
map that goes goes on behind it
brilliant is your friend they have
interactive challenges with introducing
increasing difficulty it can like a game
where the sense of progression it’s
really satisfying use brilliant dot org
slash FFF weren’t signing up to let them
know that you came from here also if you
are one of the first 200 to use that
link you will get 20% off an annual
subscription if you get that check them
out after the show the link is also in
the episode description something like
that ah that’s fine this is my good side
I can’t tell like they’re like do you
guys know is is it is it a thing with
good sites or not I’d perhaps we can
drop it a little bit as well right I
don’t know I always hear people say that
so I’m just saying that’s but I
pretty sure both I don’t have a good
side both no I don’t know which one it
is
I would like to know please let me know
which is my good side yeah that is very
valuable information we can start like
okay so we’re gonna do this do some
introduction for people that don’t know
you so that this is gonna be put up on
my channel but yeah you like people
don’t know who you are necessarily
Daniel tell us a little bit about who
you are
my name is Dan Schiffman that’s my name
got that
I guess there’s probably like three
things that are most relevant to mention
where that I do and work with and and
participate in one is I also have a
youtube channel called coding train
where I do a lot of coding tutorials
could say more about that I also teach
at a program here in New York City
called ITP it’s called the Arts New York
University and then I also work on this
thing called the processing Foundation
which is a nonprofit entity that manages
well it does a lot of different
community and education initiatives but
also maintains creative coding
educational software toolkits things
that are open source like the JavaScript
library p5.js processing original
processing software which is Java based
and then there’s also a Python
processing which I don’t know how to use
I do know how to use it I just I’m yeah
you probably shouldn’t sell that to you
because you’re probably like anti
allergic to Python actually maybe yeah
tell us a little bit about yourself yeah
sue hi I’m Shirley my name is Shirley
Liu I’m in I think the title I finally
settled on for myself is independent
creator of data visualization yeah yeah
it took me two and a half years to do
this but I’m basically a freelance
software engineer and I concentrate on
data visualization within that and they
do the whole data design code process
for all of my clients and a lot of my
clients are either kind of journalism so
like The Guardian Google News labs
actually um or they’re more for like
internal data analytics kind of things
so that’s basically what I do so what
but what does that entail technically
like what what what do you build the
visualizations and yeah so I started out
as a front-end engineer and I’ve kind of
been like in that domain and haven’t
branched out like I was planning to like
five years ago I’m and so everything I
do is in JavaScript there was a time in
which I was like I would not be
pigeonholed into this language which I
was like I’m gonna do Python for all of
my data processing and quite cleaning
and then it I realized that I was like
2x lower in Python then I was in
JavaScript and after just because like I
had at that point been like using
javascript for years and like nothing
else and I swear I had a like semi
proper in computer science education
where you know I was taught like Python
and Java and C I don’t know how to use
any of that anymore so yeah i’m i also
don’t and yes after a month of python i
was like this is really inefficient and
I went back to new Jess forget it
Marco yeah hi I’m Marco I what developer
yes I work for Google Chrome’s web
developer ecosystem team which means we
are a group of web developers building
webby things and talking about
everything within browser engineering
team so half of our work is talking
about what’s coming through the browser
to people like you all to talk about
like you know hey these features are
coming and then another half of the job
is to talk to browser engine you’re
saying hey here’s how web developer
building you know these api’s maybe
should be there so like we want these
features in browsers or natively support
it so those communication work I’ve been
trying to cover up my title called jaws
planar like because of that job I do a
lot of like blog posts and like talks
and like slides and like things so
within that I try to put my touch by
hand draw
diagrams of things I do I use a lot of
metaphors and drawings to communicate
technical concepts so I’ve been trying
to sneak in like whenever they say oh
we’re having conference and we need your
bio
I just acne Kinross planar and then see
if they accept me they did yeah what was
it you that had like some kind of
knitting project as well so the way this
job happened was so so I was a front-end
engineer well I was JavaScript engine
you I was suing server-side and then
also from inside I was doing solar
engineering like excited or startup
thing and while there I started doing
knitting with JavaScript side project
that would involves a lot of things with
what like what is that so is there a
machine that doesn’t it so there is a
knitting thing called knitting machine
from 80s that has microchip that talks
binary somebody reverse engineered how
to send the data to them so there was
there was a Python code to do that and I
was like oh I can write a node occurred
to do that same thing because it’s just
sending a buffer so that was one project
of me saying like hey so I have this
image I like do image filters and I’m
changing the color and things because
like yarn doesn’t come in twenty two
hundred and five fifty two colors right
yeah like you only have like limited
choice of color so if you have like you
know a cat picture that you want the
sweater the sweater
not exactly the sweater but I did figure
out how to draw this precisely by using
my software so like you know you can do
like Miss processing that was part of my
kneading project putting data into the
knitting machine that was part of my
knitting project and then I like there’s
like to two schools of knitting
I’m very I go to I I’m very experienced
with rabbit holes a lot of time so a lot
of people go into the deep into hardware
and figure out they’re like DIY hardware
and stuff but I went into a softer size
so I have project costs verify which is
just kind of like started as a joke like
Oh make your own ugly sweater so it’s a
combat up saying I pulled your selfie
and then I will put it into a like
sweater a template image and then you
can download to tweet it but also like
you you have a holiday mode and I put
like cheeky like reindeer and things
we’re gonna get one of these why wanted
that’s because I’m planning a holiday
special yeah I actually my my my
brother’s girlfriend has this thing with
ugly holiday sweaters if she has one you
I’ll never get every year like I would
love this so so for my personal
production I can I have made a
production for Paul Lewis who you did oh
yeah so he has a scarf of like his show
logo thing that I have made him when he
left our team I was like a gift but yeah
it takes time takes time so a lot of
people ask me oh you should do it see
shop and I’m like no no you don’t
understand
making those sweater takes like a week
yeah pricing point software engineering
job third party that would take care of
all of that for you you just send them
like the design and they will like knit
ship etc or like it’s just a software
engineering prize it’s like yeah made by
real software like a bicep tendon
there’s like grass fed by yeah either
so yeah that’s not like to Wall Street
people who look down on software
engineering so so so that that was like
for a while my side project and that’s
how I kind of gone into speaking at a
conference or writing blog about it
because I was just doing just like
projects and people gave me opportunity
talk about thanks and when the timing
was way
team was hiring and I was like I talk
about JavaScript in the form of knitting
but I can talk about JavaScript like I’m
interested in like browsers like
performant canvas and like you know how
to write performing JavaScript code with
a lot of data so yeah that’s how this
job happened brilliant so how long have
you been here almost two years two years
in April what it like utes
you also work with Jake Archibald yes
sometimes so that’s a medium yeah and
you do like I see you doing a bunch of
devrel videos and shooting shooting
videos do you do that in this building
or like ah so we have one studio in this
building that we’ll shoot but London
where Jake is based they also have a
studio there so all of their HTTP 203
and super choice video they’re not
shooting at wlky even though it looks
like it they have a fake set in a tiny
corner yeah but that’s how it works
it’s very unfortunate that like there is
a support to do those things yeah cool
so I guess we’ll link to whatever your
channel gone I keep forgetting like it’s
there that that what what the card where
are you oh the car Tube cards I will
link in one of those corners no no it’s
it’s definitely this corner right yeah
is that mirrored that’s mirrors yeah
maybe it’s actually this exactly that’s
like man it’s harder than you think to
point one thing that I like while you
were talking that I wanted to get back
to is like Python versus JavaScript as
data languages okay like because right
now it feels to me like Python is like
it’s the it’s a language that data
scientists tend to use and like but
whenever I abuse it I’ve kind of like
this would be so easy to make in
JavaScript I keep coming back to that I
mean it seems it seems to me like the
the what javascript would need in order
to like start grabbing that share is
some kind of equivalent
pi because it feels like numpy is what’s
really keeps yeah I was just going to
say like so actually in some sense that
Google’s JavaScript version of
tensorflow has that built into it oh yes
actually what it what it most but it but
but it essentially is underneath the
hood is a highly optimized you know
WebGL compatible numpy of course you can
then do all sorts of machine learning
stuff with that and that’s really what
people are using it for but the core the
core library is really just math with
the really cool sue I was going to say
that proper data engineers I think like
actually proper data people I think use
Python I’m not an actual proper data
engineer so who’s a proper engineer like
heard about Netflix has actual data
engineers
it’s like Netflix like uber I’m gonna be
here in this building somewhere properly
what do they look like what do they
either like oh you’re not a real
developer like oh yeah I’ve been told
I’m not an engineer my boyfriend my
fiance is a mechanical engineering oh
yeah sooo I don’t actually use Python
and for my own purposes of like you know
fetching data so like when I program
like it like and grab from api’s or when
I’m like cleaning data I find like
JavaScript to be perfectly fine and I
actually think a lot of my friends out
do data science they actually they
actually use our oh yeah yeah so but I’m
I’m feeling like if you just give
JavaScript another year or two it feel
like but I think it also depends like if
what you’re trying to do is more it’s
sort of like pure data science where the
end result is kind of like I need to get
this like number out versus trying to
communicate an idea or make something
interactive or easily shareable like the
like be able to do this stuff in the
browser with the javascript that gets
you that extra piece so I don’t view one
is this a better or worse but it’s just
a different okay well I can’t wait to
read the comments of this video
but um but I do i you know for me it’s
like I’m excited about JavaScript
because with students that I work with
if I can just send a URL and start
recording at 30 minutes maybe does maybe
does that would be an interesting like
know but it’s only been 90 minutes so
it’s probably like it didn’t stop maybe
it was it was it not recording from the
beginning no it probably no no God like
that is that it’s the nasty thing about
like not using here that you are not
completely used to all the time it’s
like this is this is camera you sort of
it’s kind of like I still haven’t
learned all the nooks and crannies of it
but it’s fine I will you just replace it
with us with like picture you still have
the sound yeah but I perhaps I can
usually animator yeah animated versions
of ourselves
oh no no yeah I just do the thing where
draw splaining know the multipotent
thing like where you just do the floppy
heads no I love that one was not better
than the other one time went into like
long Twitter conversation of like if you
should use Python or JavaScript for data
like analysis or whatever because uh so
I I started as a like internal tools
engineer was my first kind of job as a
programmer
before that I was product product
manager so like I learned to code and
then kind of switched the team and
things so like it was like you know
alright you have a lot of data business
people let me make data visualization
for you and then that’s around the time
I met early and then I switched job that
was Python shop and then they gave me
opportunity to live person on the job so
it’s like a great and then I was just
like on Python like how Python is
so good like pison hood like you know
data great so I was just like had a big
JSON files a CSV of someplace
data source and I was trying to get to a
minimize data that I just needed for
this visualization and I’m basically at
one time
operation of like taking the data out of
it I was just like writing Python code I
was like tweeting like a Python is so
nice like I read that they’re like you
terrible a psycho it’s so nice and then
somebody’s hit that like all jobs good
can do the same thing and I was it feels
like slow I had no idea
like I had no like was so new to me I
had no like you know but information two
pockets and I went into like some people
just sending me just stuff like you can
do this way it should be as fast thank
you but it came down at the end it is
overrated
like I only have that Python job show
for like six months yeah and then after
that like I kept doing the other
visualization project but it came down
to me that it’s just like what’s
familiar with you right is like most of
the time what you only doing is like
iterating through the line items and
then finding the information you want
the key value yes and then put it into
different JSON yeah and then for that
like Python or JavaScript like doesn’t
matter like whichever were comfortable
and I was crew way more comfortable to
write for loop in Java Script and
figuring out like Python and loopy
things I just like one box you’re like
I’m just gonna use Java
I feel like unless I’m doing like really
really intense data analysis I really
don’t think I need Python and which is
probably also chicken and egg thing
where I haven’t done the really really
intense data analysis because I don’t
want to go into but it’s sometimes also
depends on like there are certain
packages and libraries for Python that
may not have like an obvious easy
equivalent like NLT kay not that I’m
some expert nail 2k by the natural
language toolkit that’s in Python is
just very like you know sort of like the
the ubiquitous ubiquitous like
historical like standard for doing
natural language process and there are
like I’ve used because I teach like my
classes which I was like look there’s
like no natural and there’s this other
text analysis thing and now maybe you
could use attention flow Jas and the
node version others but there isn’t that
like historical standard of years and
years and years and years of material
not like sense that kind of tool but I
agree like if you’re big
for me like if I’m just iterating over
the array like I’m gonna do that where I
don’t know how to do that one thing I
did when I was sort of learning to use
JavaScript was like so I learned jobs
good first I was like a coding language
but then there was it was just like oh I
have as a knowledge of like how code
works and then I was still doing like
business job and I thought like Oh it’ll
be cool like I already deal with data
and I’m comfortable with code it’ll be
cool my next step would be like data
science right like business intelligence
so I did like one of like our Casella
data science class and I loaned our and
you know or has a lot of good features
for like having like you know that one
quantile data or like plotting and then
all of that and then I went into data
visualization so I was dealing with a
lot of JavaScript and I were using
underscore do you just luck be done if
yeah which is really nice like when I do
like when I need to do any kind of data
president I just fall back to like
underscore or our way around our or any
of those languages which is like makes
it so easy to process things so one of
like practice project that I did was
taking ARDS nice functions like
quantiles
or median and underscore didn’t support
but underscore support a mix-ins so you
can write eight basically pure like loop
functions oh yeah I have my own time
yeah yeah you I have my contour not seen
but there is also I’m so I use a little
– now but there’s also like I’m simple
statistics is a JavaScript like MA or
library that does all of it like
everything for statistics and then
deeply actually also has I’m a lot of
like data like array methods that will
help for like what you need to visualize
the date it’s there is a liova script
library called simple statistics yeah
what there’s basically so I also give
some talks and I give talks two types of
talks one is about like general things
I’ve learned doing data visualization
and then
and then the other type of talk is using
d3 would like react or I’m actually
gonna I’m actually gonna stop talking
about react because I stopped using yet
yeah Daniel was like the idea was that
Daniel teaches me d3 but then it’s very
simple it’s a letter and a number
yeah next week will do and you have an
m5 now ml5 yeah I mean I would say I
have an ml like you’re very good at
naming things why these things are not
mine along for the ride it’s an engineer
so get it but either way like we ended
up like oh but how do we like get
automatic reloading and obvious like yes
yes you create react up and then we just
like the episode turn into like an
exploration of how you would integrate
p5 and yeah and how they would function
well together so we taught each other’s
things back and forth it turned out real
nice you can find the episode at this
point to all the director I actually
tried to do that once I’m so I used I
used to do a lot of d3 with react and
now I do d3 with view I’m and I can talk
about how much I like view I’m we can
you can do that in a little bit I try I
do try to do p5 within react and I found
that p5 I couldn’t figure out how to
make PFI modular to import it into RIA
and and then I stopped using it right
looks like I have an answer to this
question but I’m like not sure like we
but yep though I think just the in
summary to summarize that what was sort
of interesting about that exploration is
that p5 and its core is really designed
to be a beginner friendly educational
environment it scales to doing many
exciting and amazing like animation you
can build real projects with that I
don’t mean it’s like some sort of toy
library but the way it does some certain
things are non-standard in JavaScript
yeah so in order to be extra beginner
friendly so mixing that with things that
are I guess like really strictly
standard
can get get blood funny business because
I was trying to use I’m the p5 noise
function for um for that SFMOMA project
right and then I think and then I use
and then I also use like random Gaussian
and like these kind of things that like
you know isn’t in low – isn’t and III
isn’t in simple statistics and I was
trying to like load that in and I think
I ended up having to be like import p5
and then I have to access with like p5
dot prototype dot like random Gaussian
is that what the uh yeah it’s like this
is probably not yeah well we can we can
have a side but I would not feel like
actually have a separate like technical
discussion about this because I’m very
curious about these kinds of
integrations but so so what about d3 and
view and react and how they how they
integrate talk a little bit about that
because like I really haven’t gotten
into view because I just looked at it it
seems like yeah it seems like it
approves a little bit on react but I’m
really good at react I figure that yeah
I’m gonna next wait for the next cool
thing after view because before I like
make the effort just because I don’t
know anything view react came before
view yeah and then view is view react I
know is maintained by Facebook is view
an independent open-source project or is
there an entity that maintains it as
well independent yeah yeah so view is
originally I think so okay historically
okay so I think d3 was 2011 and I’m just
putting d3 in there cuz that’s the thing
but that’s a completely different thing
than like reactive view which other what
do you call those things content
frameworks
yes yes there like sue okay so I think
there’s two to answer your question but
did we react view
I’m d3 was 2011 it’s not one of those I
think people call it like MV star
frameworks but but react is have we
talked about what d3 is for the people
that don’t know my house was shut yeah
so d3 is stands for data-driven
documents that’s why it’s it’s three DS
and that’s why it’s d3 um and what it
does is it’s this really really large
library to help you translate your raw
data into what you need to draw on to
the screen because the web is kind of
it’s it’s kind of weird for drawing
shapes on the web UI you either use SVG
or canvas and with SVG you have like
elements like rectangles and circles but
you also have to specify you know the XY
positions where everything should be
drawn or the the radii or something and
raw data wouldn’t have XY positions or
radii so d3 is the library that helps
you kind of like translate like let’s
say you have you know temperature data
so you might want to translate like
temperature into the Y positions or so
it really it’s really helpful for doing
that it’s really helpful for calculating
like you know layouts like trees or like
network diagrams so that’s what d3 is
and just just it’s also the important to
note that d3 is SVG rendering and p5 is
entirely canvas based so there’s like
some similarities between these library
sources or huge amount of differences
obviously but that’s one kind of
important distinction yeah well actually
d3 since version 4 has actually also
supported count interesting ok so what
do I know
oh yeah I think version 4 was 2016 and
that was when it became modular and that
was when I Mike Bostock who’s kind of
like the core person that does it I
guess he was like canvas is really
important for performance because when
you do data visualization you might have
thousands of data points or like tens of
thousands and um you can’t do SVG is not
performant after about a thousand yeah
about three thousand and then if you’re
animating and you’ve got SVG it’s not
performant after like a thousand I’m so
so it has really great canvas support so
interesting yeah so side note when I was
building that ugly sweater generator
my uploads was so basically it just your
your binary like to color pictures
translate it into a v-shaped knit stitch
and then I just lay them so like each
stitch is one pixel in the image my
first iteration was each of them or SVG
object and then I just like lay them out
of course it’s kind of going like even
like 100 500 pixel image gonna go into
so many elements and it was just like
unusable oh I need to use canvas for
this and then change it to canvas senses
yes Wow didn’t know that SVG had such
performance problems
yeah because SVG is essentially like
everything you want to draw is a single
Dom element so if we can imagine like if
you have a thousand things you’re
drawing there’s a thousand dominants and
you even with modern browsers especially
if you’re going to be transforming that
like I’m manipulating that in any way
like most browsers would be like what
are you doing to me I have like this
grass is greater thing of like I always
do everything in canvas and I keep
thinking Oh SVG that must be the most
wonderful place to be because everything
is like the data itself and it’s not
just render to pixels in there I can’t
so I think I aspire to be an SVG is so
like elegant form a react standpoint
because everything becomes like these
domino meds and like the diffing thing
like mesh is really well it’s like super
alluring you use it and then you like
build your entire application and then
you hit that limit then you’re like no
in some ways I think that I’m a canvas
is actually easier to use I’m at least
from a data visualization problem like
perspective so a lot of times what we
might do with data vis is like you might
want to translate from like one state to
the next like if you have a data set
about I don’t knew I’m the state of the
economy in one year and then you want to
animate that to the next year or
something traditionally when it was just
d3 d3 has a way where you have to like
you know give it the first set of data
and then give it the second of set of
data and d3 will tell you okay so um
these are the new things that you have
to create you know to do this these are
the things that you want to like you
know take out of the Dom these are the
things that you want to update to the
Dom and that’s what it’s called the
enter update exit pattern um and that’s
like a really should turn himself to
keep track of as soon as you have like
you know thousands of things you need to
like keep track of so canvas is really
nice because all you have to do is like
and at this millisecond draw this at
this next millisecond draw this I am et
cetera et cetera one thing I always
think about canvas though is that it’s I
always think of canvas design-wise was
like oh it’s this big block that sits on
the page whereas I think about like when
I think of d3 and SVG’s I think about it
it’s like little drawings and animations
and shapes all integrated within the
content itself but in sort of like in
you know what
things I love to look at are these like
explorable explanations Nikki cases work
for example and so for some reason I
imagine that it would be that SVG and
dative is having like an essay and
interactive essay about data that doing
that sort of stuff would be harder or
somehow to do with canvas in terms of
but I guess that’s not actually true I
may have just lost a little canvasses
everywhere yes so actually I think the
hardest thing about canvas is
interactivity because like that’s
probably also what you’re alluding to
right yes it’s just one big thing so if
you can’t there’s no built-in support
for like clicking on last circle or oh
yeah this is what I was thinking about
but the really nice thing is I’m d3 also
has a functionality like I’m quadtree
the it like has this yeah this module
called country where you feed it all of
the positions of your data and it will
kind of quadtrees are like then there
was subdivide and subdivide and it’s
just like tree that it’s really fast
like searches like n log n right um and
so it’s a it’s a fast way of searching
so all you have to do all I do is I have
a canvas limit
I’ll draw everything and because when
I’m drawing I’ve already calculated the
data and the XY positions of the data
I’ll then take that to your data passing
into d3 quadtree and then any time a
person like clicks I’m and d3 also has
like really like helpful is call like d3
Mouse and then it will gives you exactly
the XY position relative to the
container so you know what to do and
you’ve got like the top of the window
document open my like thing
yes okay yeah like this a lot of Mormons
in the JavaScript community they
complain about it but they know but
they’ve already left my channel binary
gives you is like nice waiting truck
waits comp yeah but yeah tell my voice
PG it’s like you know in mine okay I
think you know my knitting things I have
exact same thing of like Oh somebody
like click event well somebody hit this
coordinate
figure out where that was figure out
what should be underneath and figure out
what I need to be drawing your one thing
I’ve always wanted about d3 is its looks
when I see people use it it’s so
powerful and beautiful and all the
different examples and project I’m
amazed by but it doesn’t strike me as
necessary a library that you would start
with if you’re like learning to code or
if you’re like first doing database or
is that not actually true there are ways
to kind of like learn use is it beginner
Frank that it’s actually like d3 like
it’s almost harder to learn if you know
a lot of program because rate paradigm
it’s guys only thing completely so I had
a basic idea of what JavaScript is I was
able to like you know buttons count up
quick thing and then I went into oh I
know JavaScript now and I can do like
you know let’s do data visualization and
I go there’s like this week and what’s
great about d3 is that they have a lot
of examples yeah so – at the beginning
of my career I went into these were you
thinking like oh they have a lot of code
I can copy it’ll be fine easy and it was
easy the first one was like oh I can
just have like circles and bar chart
next up was really hard you realize that
you need to do something creative that
you you want to do something outside of
the box and then you have to figure out
this d3 language yep that is very
particular flavor of JavaScript hi I
have two parts two parts to the answer
to that question and the first part is I
like so I uh I went in and d3 was the
first JavaScript library I learned other
than jQuery um and I just thought that
all JavaScript libraries worth as hard
and I just kind of like barreled through
it would like purely no idea that this
was like kind of weird and not like a
standard JavaScript library and but like
I was I had to do it for work so I was
like this must be how it’s always like
and so I kind of just barreled through
it I’m but the second part of that
answer is actually I’ve been thinking a
lot about that problem that like you
know d3 does have a little bit of a
learning curve when it starts cuz
because the way you think about it is
different um so I actually now have a
workshop on front-end masters
that’s called I’m d3 for yak developers
so if you already know
a lot of fiat and then this actually
makes getting into t3 a lot easier and
the reason fray is that I think the main
learning curve for d3 is kind of the
enter update exit pattern uh-huh I’m but
that’s actually basically how react
works under the hood and so if you
realize that then you only need d3 to do
the data calculations of like you know
the XY coordinates or like that d3 mouse
or like you know that the quadtree kind
of things but you don’t have to use d3
for like the actual Dom you just told
reacts like these are my like SVG
elements I’m this is the data like this
is the state I want you to render in
that state this is the stay I want you
to render I’m and you don’t have to do
the actual like hard part of Gabriel so
that’s another thing I’ve kind of
wondered about libraries like this is
like d3 is so it’s one thing if what you
let’s say you get hired for a job like
please here’s our data I would like to
see any bar chart please yeah but III
has that put in the code you find the
example you adapt it you have the bar
chart of your data but for something
like what you worked on with like SFMOMA
for example if you’re trying to kind of
innovate we’re talking about that maybe
not but so so now all right finishing
thing but then we’re gonna talk about
yeah I guess that was so cool
so I didn’t mean that so I but it lured
more broadly if you’re trying to invent
your own language for data visualization
and come up with new ideas with your own
design that’s unique or creative do just
the library sometimes limit that those
or do does something like one thing
we’ve been always conscientious about
with processing in p5 is we try to build
in such a way that you don’t end up with
people making things that look like oh
that was made with this tool like it has
the there’s not like an automatic effect
that like really it’s really
cool-looking that everybody’s using and
every project kind of things so do you
do you struggle with that in using
libraries or no because it’s really yeah
go ahead
actually you probably also have an input
I’m I really dislike libraries built on
top of t3 that are like charting
libraries or and because I’m to me I
think d3
the right amount of flexible and the
right amount of open-ended kind of like
what you were saying I’m where I can
literally just pick and choose the
things that I want out of that library
and I don’t feel any obligation to use
other parts of the library um whereas
with a lot of charting libraries they’re
like you can build a bar chart but
that’s all you can do for me I do a lot
of like very non-standard visualizations
I don’t strictly just do a bar chart I
don’t strictly just do like a line chart
those I’m and it’s it’s all very fluid
so I only ever ever use like d3 itself
and I’ll use I use view these days and
how you get green sock for animation and
that’s all I use
so like talk a little bit about your
other V like you’re weird visualizations
I wanna hear about the project because
that was that the one with trees yeah
yeah flowers and trees yeah so tell me
about that how did that came to be and
what was it and what was I a lot of my
stuff are more in that direction than
like what you might think of as like
data visualization I’m I don’t know if
like I should be showing you or like how
do you want to do it wait what is that
project oh do you want to see this yeah
oh screen record so that people know
what we look at I need to replace this
get rid of all your secret passwords for
CD SFMOMA one and this one was I’m
basically what they asked me to do I
noticed this bugs me this is a bug right
here let’s ignore it and after I pointed
out and so what they asked me to do is
last year summer time they release this
program called semi SFMOMA
and you can text this number and you can
say like send me art sent me love send
me this emoji like send me I’m and they
have they got like 5 million texts and
what they would do is then they would
send back a artwork that they think
matched that request and then they came
to me and another creative agency and
asked us to like do something with that
data
I’m and so what I did was I’m five
million is a lot so I decided to
constant
on just five individuals and kind of
their interaction with the service for a
week and the way that you read it is I’m
each one of these leaves or flowers are
an individual text that they sent so a
flower means that the requests that they
sent was something positive like send me
love some me happiness a leaf meant it
was something neutral like semi-abstract
or send me re m and then when it’s one
of these kind of like empty leaves and
means that they asked for something
negative so it’s like something like
semi deaths that mean I don’t know I’ve
seen the there isn’t that they don’t
actually ask for that many negative
things which is really cool and it’s
colored by basically the artwork that
they received back so I extracted I
think the top few colors and then I’ll
like kind of watercolor effect it in I’m
and then you can also interact with it
to kind of see the order in which they
asked for things so it’s really cool to
see like for example this person asked
for something bakery and then they got
this thing that was the bakery which
somehow inspired them to ask for you for
like semi-smile
and they got this kind of like sorry to
the artist like slightly creepy looking
thing that looks like it’s crying and
then this person asked for semi tears
and and so like this kind of like the
way that this the text interactions
would send me SFMOMA
I’m kind of affected their mood it’s
kind of what I was going for so you can
so each of them are drawn in d3 no so
this one was an interesting exploration
in tech I’m so this is essentially I’m
what did I do
I have actually a really long write-up
for data sketches which is this project
I do with my friend Noddy and Doddy
Bremer and I and we have I’m basically
this is so cool so we you know have a
basically a topic and then we’ll
basically go out and find a data set for
that topic and then
do the whole visualization from start to
finish and then we do write-ups about
the whole process so you can actually
read the whole technical like how we
went and got the data so this is kind of
like um all the data I had on hand um
how I decided to go about like
prototyping so then and this is kind of
like you know I saw Tyler Hobbs like
watercolor effect and I really wanted to
recreate that and this is like
expectation versus reality and then so
then you know then he helped me various
people helped me and I got it right and
this is how it looks like with actual
data I’m and then I was in Japan at the
time so then I’m I started being like I
want to make flowers and so I did so
this is actually when I knew when I was
looking through Perlin noise because if
you notice I’m at each of these strokes
are varied and like kind of like
thickness so it looks like it’s hand
drawn yeah I’m and so what I did was
yeah I did SVG paths and then went
through every path with like carlin
noise and did this in canvas I’m so
there’s a lot of things overlaid I think
I only use d3 for this one to do like
basic like scaling um these actual
shapes are basically hand calculated and
then I have links to code like every
single step of the way I’m but this is a
combination of um it’s just pure like
you know a canvas Bezier curve commands
I’m used green sock to do the actual
kind of timing a when they’ll show up
when they’ll draw and I used view to
kind of just manage like like very basic
interactions I’m yeah I we are a bound
at fifty minutes and that’s not probably
problem is that the the battery is
running out on that camera but it’s
like I think it’s kind of like a good
place to wrap up and I think we should
finish up where people can find you
online oh yeah um so my twitter handle
is sxyw you I’m yeah yeah you do still
do twitch streaming no yes yeah yes well
I’m trying to get back into it like my
personal life has like exploded in the
last year yeah so people should follow
you social pressure like 600 followers
on Twitch in the light three months like
since in the three months I was not
doing any streams and I’m like what what
happened here yeah I’m yeah so I’m
hoping to get back into it like January
February or so I’m but yeah so I’m on
I’m not handle everywhere twitched
Twitter github on my own website is calm
and alright brain to to just look for
the train stuff so coding train on
YouTube and I’m at Schiffman which is no
seat Shi FF MN on Twitter all right and
I’m sorry ko and say ma RI on Twitter I
do not have YouTube although yeah I
think we had like a long three-hour
conversation while you were telling me
to start you’re always welcome to come
if you all both you will always want to
do like a guest spot if you’re in New
York again or anytime you want to we had
a lot of fun yeah whenever I do a like
girl splaining I usually end up I call
it tweeting like one time I call it
tweet Dean with like four pictures and
then like it had like starting to end
like story of like explaining a
technology and then somebody said like
can you do your drawers play anything on
this topic and I’m like I like that I
was doing yeah every time I do draw
spinning it always a nothing Twitter so
Oh Oh brilliant thank you so much
[Laughter]
this is still recording it is and here’s
me again with a battery in the camera
fantastic
I’m here to remind you that today’s
sponsor was brilliant if you are feeling
like you should probably stop
procrastinating on learning machine
learning brilliant is your friend
brilliant org slash
FFF also the same link is in the episode
description if you’re new to the show
welcome this was an episode of fun fun
function I release these every Monday
morning oh wait hundred GMT but you will
forget that so you can subscribe by
clicking here or watch another episode
right now by clicking here I am mpj until next Monday morning thank you
not morning as you can see it’s evening
in New York and this is this is what you
get if you procrastinate and show again
intro all day anyway
so while I am while I’m in New York for
vacation then I thought that it would be
nice to shoot a video with a couple of
developer friends of mine were like
prominent fascinating and brilliant
people that I’m super proud you know and
just have a chat yes start with talking
about who they are and then jump up from
there and what we ended up with was
talking about data visualization data
processing in JavaScript Python d3 like
a bunch of that stuff and it turned out
to be super super super interesting
discussion so if we’re interested in
looking a little bit into data science
and whatnot this episode should be
really nice for you I think before we
get on with that though I would like to
read today’s sponsor which is brilliant
thank you brilliant brilliant is a
problem-solving website that teaches you
how to think like a computer scientist
if you’re interested in machine learning
and linear algebra and all that other
map that goes goes on behind it
brilliant is your friend they have
interactive challenges with introducing
increasing difficulty it can like a game
where the sense of progression it’s
really satisfying use brilliant dot org
slash FFF weren’t signing up to let them
know that you came from here also if you
are one of the first 200 to use that
link you will get 20% off an annual
subscription if you get that check them
out after the show the link is also in
the episode description something like
that ah that’s fine this is my good side
I can’t tell like they’re like do you
guys know is is it is it a thing with
good sites or not I’d perhaps we can
drop it a little bit as well right I
don’t know I always hear people say that
so I’m just saying that’s but I
pretty sure both I don’t have a good
side both no I don’t know which one it
is
I would like to know please let me know
which is my good side yeah that is very
valuable information we can start like
okay so we’re gonna do this do some
introduction for people that don’t know
you so that this is gonna be put up on
my channel but yeah you like people
don’t know who you are necessarily
Daniel tell us a little bit about who
you are
my name is Dan Schiffman that’s my name
got that
I guess there’s probably like three
things that are most relevant to mention
where that I do and work with and and
participate in one is I also have a
youtube channel called coding train
where I do a lot of coding tutorials
could say more about that I also teach
at a program here in New York City
called ITP it’s called the Arts New York
University and then I also work on this
thing called the processing Foundation
which is a nonprofit entity that manages
well it does a lot of different
community and education initiatives but
also maintains creative coding
educational software toolkits things
that are open source like the JavaScript
library p5.js processing original
processing software which is Java based
and then there’s also a Python
processing which I don’t know how to use
I do know how to use it I just I’m yeah
you probably shouldn’t sell that to you
because you’re probably like anti
allergic to Python actually maybe yeah
tell us a little bit about yourself yeah
sue hi I’m Shirley my name is Shirley
Liu I’m in I think the title I finally
settled on for myself is independent
creator of data visualization yeah yeah
it took me two and a half years to do
this but I’m basically a freelance
software engineer and I concentrate on
data visualization within that and they
do the whole data design code process
for all of my clients and a lot of my
clients are either kind of journalism so
like The Guardian Google News labs
actually um or they’re more for like
internal data analytics kind of things
so that’s basically what I do so what
but what does that entail technically
like what what what do you build the
visualizations and yeah so I started out
as a front-end engineer and I’ve kind of
been like in that domain and haven’t
branched out like I was planning to like
five years ago I’m and so everything I
do is in JavaScript there was a time in
which I was like I would not be
pigeonholed into this language which I
was like I’m gonna do Python for all of
my data processing and quite cleaning
and then it I realized that I was like
2x lower in Python then I was in
JavaScript and after just because like I
had at that point been like using
javascript for years and like nothing
else and I swear I had a like semi
proper in computer science education
where you know I was taught like Python
and Java and C I don’t know how to use
any of that anymore so yeah i’m i also
don’t and yes after a month of python i
was like this is really inefficient and
I went back to new Jess forget it
Marco yeah hi I’m Marco I what developer
yes I work for Google Chrome’s web
developer ecosystem team which means we
are a group of web developers building
webby things and talking about
everything within browser engineering
team so half of our work is talking
about what’s coming through the browser
to people like you all to talk about
like you know hey these features are
coming and then another half of the job
is to talk to browser engine you’re
saying hey here’s how web developer
building you know these api’s maybe
should be there so like we want these
features in browsers or natively support
it so those communication work I’ve been
trying to cover up my title called jaws
planar like because of that job I do a
lot of like blog posts and like talks
and like slides and like things so
within that I try to put my touch by
hand draw
diagrams of things I do I use a lot of
metaphors and drawings to communicate
technical concepts so I’ve been trying
to sneak in like whenever they say oh
we’re having conference and we need your
bio
I just acne Kinross planar and then see
if they accept me they did yeah what was
it you that had like some kind of
knitting project as well so the way this
job happened was so so I was a front-end
engineer well I was JavaScript engine
you I was suing server-side and then
also from inside I was doing solar
engineering like excited or startup
thing and while there I started doing
knitting with JavaScript side project
that would involves a lot of things with
what like what is that so is there a
machine that doesn’t it so there is a
knitting thing called knitting machine
from 80s that has microchip that talks
binary somebody reverse engineered how
to send the data to them so there was
there was a Python code to do that and I
was like oh I can write a node occurred
to do that same thing because it’s just
sending a buffer so that was one project
of me saying like hey so I have this
image I like do image filters and I’m
changing the color and things because
like yarn doesn’t come in twenty two
hundred and five fifty two colors right
yeah like you only have like limited
choice of color so if you have like you
know a cat picture that you want the
sweater the sweater
not exactly the sweater but I did figure
out how to draw this precisely by using
my software so like you know you can do
like Miss processing that was part of my
kneading project putting data into the
knitting machine that was part of my
knitting project and then I like there’s
like to two schools of knitting
I’m very I go to I I’m very experienced
with rabbit holes a lot of time so a lot
of people go into the deep into hardware
and figure out they’re like DIY hardware
and stuff but I went into a softer size
so I have project costs verify which is
just kind of like started as a joke like
Oh make your own ugly sweater so it’s a
combat up saying I pulled your selfie
and then I will put it into a like
sweater a template image and then you
can download to tweet it but also like
you you have a holiday mode and I put
like cheeky like reindeer and things
we’re gonna get one of these why wanted
that’s because I’m planning a holiday
special yeah I actually my my my
brother’s girlfriend has this thing with
ugly holiday sweaters if she has one you
I’ll never get every year like I would
love this so so for my personal
production I can I have made a
production for Paul Lewis who you did oh
yeah so he has a scarf of like his show
logo thing that I have made him when he
left our team I was like a gift but yeah
it takes time takes time so a lot of
people ask me oh you should do it see
shop and I’m like no no you don’t
understand
making those sweater takes like a week
yeah pricing point software engineering
job third party that would take care of
all of that for you you just send them
like the design and they will like knit
ship etc or like it’s just a software
engineering prize it’s like yeah made by
real software like a bicep tendon
there’s like grass fed by yeah either
so yeah that’s not like to Wall Street
people who look down on software
engineering so so so that that was like
for a while my side project and that’s
how I kind of gone into speaking at a
conference or writing blog about it
because I was just doing just like
projects and people gave me opportunity
talk about thanks and when the timing
was way
team was hiring and I was like I talk
about JavaScript in the form of knitting
but I can talk about JavaScript like I’m
interested in like browsers like
performant canvas and like you know how
to write performing JavaScript code with
a lot of data so yeah that’s how this
job happened brilliant so how long have
you been here almost two years two years
in April what it like utes
you also work with Jake Archibald yes
sometimes so that’s a medium yeah and
you do like I see you doing a bunch of
devrel videos and shooting shooting
videos do you do that in this building
or like ah so we have one studio in this
building that we’ll shoot but London
where Jake is based they also have a
studio there so all of their HTTP 203
and super choice video they’re not
shooting at wlky even though it looks
like it they have a fake set in a tiny
corner yeah but that’s how it works
it’s very unfortunate that like there is
a support to do those things yeah cool
so I guess we’ll link to whatever your
channel gone I keep forgetting like it’s
there that that what what the card where
are you oh the car Tube cards I will
link in one of those corners no no it’s
it’s definitely this corner right yeah
is that mirrored that’s mirrors yeah
maybe it’s actually this exactly that’s
like man it’s harder than you think to
point one thing that I like while you
were talking that I wanted to get back
to is like Python versus JavaScript as
data languages okay like because right
now it feels to me like Python is like
it’s the it’s a language that data
scientists tend to use and like but
whenever I abuse it I’ve kind of like
this would be so easy to make in
JavaScript I keep coming back to that I
mean it seems it seems to me like the
the what javascript would need in order
to like start grabbing that share is
some kind of equivalent
pi because it feels like numpy is what’s
really keeps yeah I was just going to
say like so actually in some sense that
Google’s JavaScript version of
tensorflow has that built into it oh yes
actually what it what it most but it but
but it essentially is underneath the
hood is a highly optimized you know
WebGL compatible numpy of course you can
then do all sorts of machine learning
stuff with that and that’s really what
people are using it for but the core the
core library is really just math with
the really cool sue I was going to say
that proper data engineers I think like
actually proper data people I think use
Python I’m not an actual proper data
engineer so who’s a proper engineer like
heard about Netflix has actual data
engineers
it’s like Netflix like uber I’m gonna be
here in this building somewhere properly
what do they look like what do they
either like oh you’re not a real
developer like oh yeah I’ve been told
I’m not an engineer my boyfriend my
fiance is a mechanical engineering oh
yeah sooo I don’t actually use Python
and for my own purposes of like you know
fetching data so like when I program
like it like and grab from api’s or when
I’m like cleaning data I find like
JavaScript to be perfectly fine and I
actually think a lot of my friends out
do data science they actually they
actually use our oh yeah yeah so but I’m
I’m feeling like if you just give
JavaScript another year or two it feel
like but I think it also depends like if
what you’re trying to do is more it’s
sort of like pure data science where the
end result is kind of like I need to get
this like number out versus trying to
communicate an idea or make something
interactive or easily shareable like the
like be able to do this stuff in the
browser with the javascript that gets
you that extra piece so I don’t view one
is this a better or worse but it’s just
a different okay well I can’t wait to
read the comments of this video
but um but I do i you know for me it’s
like I’m excited about JavaScript
because with students that I work with
if I can just send a URL and start
recording at 30 minutes maybe does maybe
does that would be an interesting like
know but it’s only been 90 minutes so
it’s probably like it didn’t stop maybe
it was it was it not recording from the
beginning no it probably no no God like
that is that it’s the nasty thing about
like not using here that you are not
completely used to all the time it’s
like this is this is camera you sort of
it’s kind of like I still haven’t
learned all the nooks and crannies of it
but it’s fine I will you just replace it
with us with like picture you still have
the sound yeah but I perhaps I can
usually animator yeah animated versions
of ourselves
oh no no yeah I just do the thing where
draw splaining know the multipotent
thing like where you just do the floppy
heads no I love that one was not better
than the other one time went into like
long Twitter conversation of like if you
should use Python or JavaScript for data
like analysis or whatever because uh so
I I started as a like internal tools
engineer was my first kind of job as a
programmer
before that I was product product
manager so like I learned to code and
then kind of switched the team and
things so like it was like you know
alright you have a lot of data business
people let me make data visualization
for you and then that’s around the time
I met early and then I switched job that
was Python shop and then they gave me
opportunity to live person on the job so
it’s like a great and then I was just
like on Python like how Python is
so good like pison hood like you know
data great so I was just like had a big
JSON files a CSV of someplace
data source and I was trying to get to a
minimize data that I just needed for
this visualization and I’m basically at
one time
operation of like taking the data out of
it I was just like writing Python code I
was like tweeting like a Python is so
nice like I read that they’re like you
terrible a psycho it’s so nice and then
somebody’s hit that like all jobs good
can do the same thing and I was it feels
like slow I had no idea
like I had no like was so new to me I
had no like you know but information two
pockets and I went into like some people
just sending me just stuff like you can
do this way it should be as fast thank
you but it came down at the end it is
overrated
like I only have that Python job show
for like six months yeah and then after
that like I kept doing the other
visualization project but it came down
to me that it’s just like what’s
familiar with you right is like most of
the time what you only doing is like
iterating through the line items and
then finding the information you want
the key value yes and then put it into
different JSON yeah and then for that
like Python or JavaScript like doesn’t
matter like whichever were comfortable
and I was crew way more comfortable to
write for loop in Java Script and
figuring out like Python and loopy
things I just like one box you’re like
I’m just gonna use Java
I feel like unless I’m doing like really
really intense data analysis I really
don’t think I need Python and which is
probably also chicken and egg thing
where I haven’t done the really really
intense data analysis because I don’t
want to go into but it’s sometimes also
depends on like there are certain
packages and libraries for Python that
may not have like an obvious easy
equivalent like NLT kay not that I’m
some expert nail 2k by the natural
language toolkit that’s in Python is
just very like you know sort of like the
the ubiquitous ubiquitous like
historical like standard for doing
natural language process and there are
like I’ve used because I teach like my
classes which I was like look there’s
like no natural and there’s this other
text analysis thing and now maybe you
could use attention flow Jas and the
node version others but there isn’t that
like historical standard of years and
years and years and years of material
not like sense that kind of tool but I
agree like if you’re big
for me like if I’m just iterating over
the array like I’m gonna do that where I
don’t know how to do that one thing I
did when I was sort of learning to use
JavaScript was like so I learned jobs
good first I was like a coding language
but then there was it was just like oh I
have as a knowledge of like how code
works and then I was still doing like
business job and I thought like Oh it’ll
be cool like I already deal with data
and I’m comfortable with code it’ll be
cool my next step would be like data
science right like business intelligence
so I did like one of like our Casella
data science class and I loaned our and
you know or has a lot of good features
for like having like you know that one
quantile data or like plotting and then
all of that and then I went into data
visualization so I was dealing with a
lot of JavaScript and I were using
underscore do you just luck be done if
yeah which is really nice like when I do
like when I need to do any kind of data
president I just fall back to like
underscore or our way around our or any
of those languages which is like makes
it so easy to process things so one of
like practice project that I did was
taking ARDS nice functions like
quantiles
or median and underscore didn’t support
but underscore support a mix-ins so you
can write eight basically pure like loop
functions oh yeah I have my own time
yeah yeah you I have my contour not seen
but there is also I’m so I use a little
– now but there’s also like I’m simple
statistics is a JavaScript like MA or
library that does all of it like
everything for statistics and then
deeply actually also has I’m a lot of
like data like array methods that will
help for like what you need to visualize
the date it’s there is a liova script
library called simple statistics yeah
what there’s basically so I also give
some talks and I give talks two types of
talks one is about like general things
I’ve learned doing data visualization
and then
and then the other type of talk is using
d3 would like react or I’m actually
gonna I’m actually gonna stop talking
about react because I stopped using yet
yeah Daniel was like the idea was that
Daniel teaches me d3 but then it’s very
simple it’s a letter and a number
yeah next week will do and you have an
m5 now ml5 yeah I mean I would say I
have an ml like you’re very good at
naming things why these things are not
mine along for the ride it’s an engineer
so get it but either way like we ended
up like oh but how do we like get
automatic reloading and obvious like yes
yes you create react up and then we just
like the episode turn into like an
exploration of how you would integrate
p5 and yeah and how they would function
well together so we taught each other’s
things back and forth it turned out real
nice you can find the episode at this
point to all the director I actually
tried to do that once I’m so I used I
used to do a lot of d3 with react and
now I do d3 with view I’m and I can talk
about how much I like view I’m we can
you can do that in a little bit I try I
do try to do p5 within react and I found
that p5 I couldn’t figure out how to
make PFI modular to import it into RIA
and and then I stopped using it right
looks like I have an answer to this
question but I’m like not sure like we
but yep though I think just the in
summary to summarize that what was sort
of interesting about that exploration is
that p5 and its core is really designed
to be a beginner friendly educational
environment it scales to doing many
exciting and amazing like animation you
can build real projects with that I
don’t mean it’s like some sort of toy
library but the way it does some certain
things are non-standard in JavaScript
yeah so in order to be extra beginner
friendly so mixing that with things that
are I guess like really strictly
standard
can get get blood funny business because
I was trying to use I’m the p5 noise
function for um for that SFMOMA project
right and then I think and then I use
and then I also use like random Gaussian
and like these kind of things that like
you know isn’t in low – isn’t and III
isn’t in simple statistics and I was
trying to like load that in and I think
I ended up having to be like import p5
and then I have to access with like p5
dot prototype dot like random Gaussian
is that what the uh yeah it’s like this
is probably not yeah well we can we can
have a side but I would not feel like
actually have a separate like technical
discussion about this because I’m very
curious about these kinds of
integrations but so so what about d3 and
view and react and how they how they
integrate talk a little bit about that
because like I really haven’t gotten
into view because I just looked at it it
seems like yeah it seems like it
approves a little bit on react but I’m
really good at react I figure that yeah
I’m gonna next wait for the next cool
thing after view because before I like
make the effort just because I don’t
know anything view react came before
view yeah and then view is view react I
know is maintained by Facebook is view
an independent open-source project or is
there an entity that maintains it as
well independent yeah yeah so view is
originally I think so okay historically
okay so I think d3 was 2011 and I’m just
putting d3 in there cuz that’s the thing
but that’s a completely different thing
than like reactive view which other what
do you call those things content
frameworks
yes yes there like sue okay so I think
there’s two to answer your question but
did we react view
I’m d3 was 2011 it’s not one of those I
think people call it like MV star
frameworks but but react is have we
talked about what d3 is for the people
that don’t know my house was shut yeah
so d3 is stands for data-driven
documents that’s why it’s it’s three DS
and that’s why it’s d3 um and what it
does is it’s this really really large
library to help you translate your raw
data into what you need to draw on to
the screen because the web is kind of
it’s it’s kind of weird for drawing
shapes on the web UI you either use SVG
or canvas and with SVG you have like
elements like rectangles and circles but
you also have to specify you know the XY
positions where everything should be
drawn or the the radii or something and
raw data wouldn’t have XY positions or
radii so d3 is the library that helps
you kind of like translate like let’s
say you have you know temperature data
so you might want to translate like
temperature into the Y positions or so
it really it’s really helpful for doing
that it’s really helpful for calculating
like you know layouts like trees or like
network diagrams so that’s what d3 is
and just just it’s also the important to
note that d3 is SVG rendering and p5 is
entirely canvas based so there’s like
some similarities between these library
sources or huge amount of differences
obviously but that’s one kind of
important distinction yeah well actually
d3 since version 4 has actually also
supported count interesting ok so what
do I know
oh yeah I think version 4 was 2016 and
that was when it became modular and that
was when I Mike Bostock who’s kind of
like the core person that does it I
guess he was like canvas is really
important for performance because when
you do data visualization you might have
thousands of data points or like tens of
thousands and um you can’t do SVG is not
performant after about a thousand yeah
about three thousand and then if you’re
animating and you’ve got SVG it’s not
performant after like a thousand I’m so
so it has really great canvas support so
interesting yeah so side note when I was
building that ugly sweater generator
my uploads was so basically it just your
your binary like to color pictures
translate it into a v-shaped knit stitch
and then I just lay them so like each
stitch is one pixel in the image my
first iteration was each of them or SVG
object and then I just like lay them out
of course it’s kind of going like even
like 100 500 pixel image gonna go into
so many elements and it was just like
unusable oh I need to use canvas for
this and then change it to canvas senses
yes Wow didn’t know that SVG had such
performance problems
yeah because SVG is essentially like
everything you want to draw is a single
Dom element so if we can imagine like if
you have a thousand things you’re
drawing there’s a thousand dominants and
you even with modern browsers especially
if you’re going to be transforming that
like I’m manipulating that in any way
like most browsers would be like what
are you doing to me I have like this
grass is greater thing of like I always
do everything in canvas and I keep
thinking Oh SVG that must be the most
wonderful place to be because everything
is like the data itself and it’s not
just render to pixels in there I can’t
so I think I aspire to be an SVG is so
like elegant form a react standpoint
because everything becomes like these
domino meds and like the diffing thing
like mesh is really well it’s like super
alluring you use it and then you like
build your entire application and then
you hit that limit then you’re like no
in some ways I think that I’m a canvas
is actually easier to use I’m at least
from a data visualization problem like
perspective so a lot of times what we
might do with data vis is like you might
want to translate from like one state to
the next like if you have a data set
about I don’t knew I’m the state of the
economy in one year and then you want to
animate that to the next year or
something traditionally when it was just
d3 d3 has a way where you have to like
you know give it the first set of data
and then give it the second of set of
data and d3 will tell you okay so um
these are the new things that you have
to create you know to do this these are
the things that you want to like you
know take out of the Dom these are the
things that you want to update to the
Dom and that’s what it’s called the
enter update exit pattern um and that’s
like a really should turn himself to
keep track of as soon as you have like
you know thousands of things you need to
like keep track of so canvas is really
nice because all you have to do is like
and at this millisecond draw this at
this next millisecond draw this I am et
cetera et cetera one thing I always
think about canvas though is that it’s I
always think of canvas design-wise was
like oh it’s this big block that sits on
the page whereas I think about like when
I think of d3 and SVG’s I think about it
it’s like little drawings and animations
and shapes all integrated within the
content itself but in sort of like in
you know what
things I love to look at are these like
explorable explanations Nikki cases work
for example and so for some reason I
imagine that it would be that SVG and
dative is having like an essay and
interactive essay about data that doing
that sort of stuff would be harder or
somehow to do with canvas in terms of
but I guess that’s not actually true I
may have just lost a little canvasses
everywhere yes so actually I think the
hardest thing about canvas is
interactivity because like that’s
probably also what you’re alluding to
right yes it’s just one big thing so if
you can’t there’s no built-in support
for like clicking on last circle or oh
yeah this is what I was thinking about
but the really nice thing is I’m d3 also
has a functionality like I’m quadtree
the it like has this yeah this module
called country where you feed it all of
the positions of your data and it will
kind of quadtrees are like then there
was subdivide and subdivide and it’s
just like tree that it’s really fast
like searches like n log n right um and
so it’s a it’s a fast way of searching
so all you have to do all I do is I have
a canvas limit
I’ll draw everything and because when
I’m drawing I’ve already calculated the
data and the XY positions of the data
I’ll then take that to your data passing
into d3 quadtree and then any time a
person like clicks I’m and d3 also has
like really like helpful is call like d3
Mouse and then it will gives you exactly
the XY position relative to the
container so you know what to do and
you’ve got like the top of the window
document open my like thing
yes okay yeah like this a lot of Mormons
in the JavaScript community they
complain about it but they know but
they’ve already left my channel binary
gives you is like nice waiting truck
waits comp yeah but yeah tell my voice
PG it’s like you know in mine okay I
think you know my knitting things I have
exact same thing of like Oh somebody
like click event well somebody hit this
coordinate
figure out where that was figure out
what should be underneath and figure out
what I need to be drawing your one thing
I’ve always wanted about d3 is its looks
when I see people use it it’s so
powerful and beautiful and all the
different examples and project I’m
amazed by but it doesn’t strike me as
necessary a library that you would start
with if you’re like learning to code or
if you’re like first doing database or
is that not actually true there are ways
to kind of like learn use is it beginner
Frank that it’s actually like d3 like
it’s almost harder to learn if you know
a lot of program because rate paradigm
it’s guys only thing completely so I had
a basic idea of what JavaScript is I was
able to like you know buttons count up
quick thing and then I went into oh I
know JavaScript now and I can do like
you know let’s do data visualization and
I go there’s like this week and what’s
great about d3 is that they have a lot
of examples yeah so – at the beginning
of my career I went into these were you
thinking like oh they have a lot of code
I can copy it’ll be fine easy and it was
easy the first one was like oh I can
just have like circles and bar chart
next up was really hard you realize that
you need to do something creative that
you you want to do something outside of
the box and then you have to figure out
this d3 language yep that is very
particular flavor of JavaScript hi I
have two parts two parts to the answer
to that question and the first part is I
like so I uh I went in and d3 was the
first JavaScript library I learned other
than jQuery um and I just thought that
all JavaScript libraries worth as hard
and I just kind of like barreled through
it would like purely no idea that this
was like kind of weird and not like a
standard JavaScript library and but like
I was I had to do it for work so I was
like this must be how it’s always like
and so I kind of just barreled through
it I’m but the second part of that
answer is actually I’ve been thinking a
lot about that problem that like you
know d3 does have a little bit of a
learning curve when it starts cuz
because the way you think about it is
different um so I actually now have a
workshop on front-end masters
that’s called I’m d3 for yak developers
so if you already know
a lot of fiat and then this actually
makes getting into t3 a lot easier and
the reason fray is that I think the main
learning curve for d3 is kind of the
enter update exit pattern uh-huh I’m but
that’s actually basically how react
works under the hood and so if you
realize that then you only need d3 to do
the data calculations of like you know
the XY coordinates or like that d3 mouse
or like you know that the quadtree kind
of things but you don’t have to use d3
for like the actual Dom you just told
reacts like these are my like SVG
elements I’m this is the data like this
is the state I want you to render in
that state this is the stay I want you
to render I’m and you don’t have to do
the actual like hard part of Gabriel so
that’s another thing I’ve kind of
wondered about libraries like this is
like d3 is so it’s one thing if what you
let’s say you get hired for a job like
please here’s our data I would like to
see any bar chart please yeah but III
has that put in the code you find the
example you adapt it you have the bar
chart of your data but for something
like what you worked on with like SFMOMA
for example if you’re trying to kind of
innovate we’re talking about that maybe
not but so so now all right finishing
thing but then we’re gonna talk about
yeah I guess that was so cool
so I didn’t mean that so I but it lured
more broadly if you’re trying to invent
your own language for data visualization
and come up with new ideas with your own
design that’s unique or creative do just
the library sometimes limit that those
or do does something like one thing
we’ve been always conscientious about
with processing in p5 is we try to build
in such a way that you don’t end up with
people making things that look like oh
that was made with this tool like it has
the there’s not like an automatic effect
that like really it’s really
cool-looking that everybody’s using and
every project kind of things so do you
do you struggle with that in using
libraries or no because it’s really yeah
go ahead
actually you probably also have an input
I’m I really dislike libraries built on
top of t3 that are like charting
libraries or and because I’m to me I
think d3
the right amount of flexible and the
right amount of open-ended kind of like
what you were saying I’m where I can
literally just pick and choose the
things that I want out of that library
and I don’t feel any obligation to use
other parts of the library um whereas
with a lot of charting libraries they’re
like you can build a bar chart but
that’s all you can do for me I do a lot
of like very non-standard visualizations
I don’t strictly just do a bar chart I
don’t strictly just do like a line chart
those I’m and it’s it’s all very fluid
so I only ever ever use like d3 itself
and I’ll use I use view these days and
how you get green sock for animation and
that’s all I use
so like talk a little bit about your
other V like you’re weird visualizations
I wanna hear about the project because
that was that the one with trees yeah
yeah flowers and trees yeah so tell me
about that how did that came to be and
what was it and what was I a lot of my
stuff are more in that direction than
like what you might think of as like
data visualization I’m I don’t know if
like I should be showing you or like how
do you want to do it wait what is that
project oh do you want to see this yeah
oh screen record so that people know
what we look at I need to replace this
get rid of all your secret passwords for
CD SFMOMA one and this one was I’m
basically what they asked me to do I
noticed this bugs me this is a bug right
here let’s ignore it and after I pointed
out and so what they asked me to do is
last year summer time they release this
program called semi SFMOMA
and you can text this number and you can
say like send me art sent me love send
me this emoji like send me I’m and they
have they got like 5 million texts and
what they would do is then they would
send back a artwork that they think
matched that request and then they came
to me and another creative agency and
asked us to like do something with that
data
I’m and so what I did was I’m five
million is a lot so I decided to
constant
on just five individuals and kind of
their interaction with the service for a
week and the way that you read it is I’m
each one of these leaves or flowers are
an individual text that they sent so a
flower means that the requests that they
sent was something positive like send me
love some me happiness a leaf meant it
was something neutral like semi-abstract
or send me re m and then when it’s one
of these kind of like empty leaves and
means that they asked for something
negative so it’s like something like
semi deaths that mean I don’t know I’ve
seen the there isn’t that they don’t
actually ask for that many negative
things which is really cool and it’s
colored by basically the artwork that
they received back so I extracted I
think the top few colors and then I’ll
like kind of watercolor effect it in I’m
and then you can also interact with it
to kind of see the order in which they
asked for things so it’s really cool to
see like for example this person asked
for something bakery and then they got
this thing that was the bakery which
somehow inspired them to ask for you for
like semi-smile
and they got this kind of like sorry to
the artist like slightly creepy looking
thing that looks like it’s crying and
then this person asked for semi tears
and and so like this kind of like the
way that this the text interactions
would send me SFMOMA
I’m kind of affected their mood it’s
kind of what I was going for so you can
so each of them are drawn in d3 no so
this one was an interesting exploration
in tech I’m so this is essentially I’m
what did I do
I have actually a really long write-up
for data sketches which is this project
I do with my friend Noddy and Doddy
Bremer and I and we have I’m basically
this is so cool so we you know have a
basically a topic and then we’ll
basically go out and find a data set for
that topic and then
do the whole visualization from start to
finish and then we do write-ups about
the whole process so you can actually
read the whole technical like how we
went and got the data so this is kind of
like um all the data I had on hand um
how I decided to go about like
prototyping so then and this is kind of
like you know I saw Tyler Hobbs like
watercolor effect and I really wanted to
recreate that and this is like
expectation versus reality and then so
then you know then he helped me various
people helped me and I got it right and
this is how it looks like with actual
data I’m and then I was in Japan at the
time so then I’m I started being like I
want to make flowers and so I did so
this is actually when I knew when I was
looking through Perlin noise because if
you notice I’m at each of these strokes
are varied and like kind of like
thickness so it looks like it’s hand
drawn yeah I’m and so what I did was
yeah I did SVG paths and then went
through every path with like carlin
noise and did this in canvas I’m so
there’s a lot of things overlaid I think
I only use d3 for this one to do like
basic like scaling um these actual
shapes are basically hand calculated and
then I have links to code like every
single step of the way I’m but this is a
combination of um it’s just pure like
you know a canvas Bezier curve commands
I’m used green sock to do the actual
kind of timing a when they’ll show up
when they’ll draw and I used view to
kind of just manage like like very basic
interactions I’m yeah I we are a bound
at fifty minutes and that’s not probably
problem is that the the battery is
running out on that camera but it’s
like I think it’s kind of like a good
place to wrap up and I think we should
finish up where people can find you
online oh yeah um so my twitter handle
is sxyw you I’m yeah yeah you do still
do twitch streaming no yes yeah yes well
I’m trying to get back into it like my
personal life has like exploded in the
last year yeah so people should follow
you social pressure like 600 followers
on Twitch in the light three months like
since in the three months I was not
doing any streams and I’m like what what
happened here yeah I’m yeah so I’m
hoping to get back into it like January
February or so I’m but yeah so I’m on
I’m not handle everywhere twitched
Twitter github on my own website is calm
and alright brain to to just look for
the train stuff so coding train on
YouTube and I’m at Schiffman which is no
seat Shi FF MN on Twitter all right and
I’m sorry ko and say ma RI on Twitter I
do not have YouTube although yeah I
think we had like a long three-hour
conversation while you were telling me
to start you’re always welcome to come
if you all both you will always want to
do like a guest spot if you’re in New
York again or anytime you want to we had
a lot of fun yeah whenever I do a like
girl splaining I usually end up I call
it tweeting like one time I call it
tweet Dean with like four pictures and
then like it had like starting to end
like story of like explaining a
technology and then somebody said like
can you do your drawers play anything on
this topic and I’m like I like that I
was doing yeah every time I do draw
spinning it always a nothing Twitter so
Oh Oh brilliant thank you so much
[Laughter]
this is still recording it is and here’s
me again with a battery in the camera
fantastic
I’m here to remind you that today’s
sponsor was brilliant if you are feeling
like you should probably stop
procrastinating on learning machine
learning brilliant is your friend
brilliant org slash
FFF also the same link is in the episode
description if you’re new to the show
welcome this was an episode of fun fun
function I release these every Monday
morning oh wait hundred GMT but you will
forget that so you can subscribe by
clicking here or watch another episode
right now by clicking here I am mpj until next Monday morning thank you
Please follow and like us: