Press "Enter" to skip to content

Cython Tutorial – Bridging between Python and C/C++ for performance gains


what’s going on everybody and welcome to
a site on introduction tutorial we’ve
got a lot of things to cover I’m going
to try to get everything fit into one
single video so let’s get to it so first
of all what is scythe on so I thought is
a superset of the Python language so
what this basically means is it’s going
to act as a bridge between Python and C
/ C++ and this means we’re going to go
both ways we can go from Python to C C++
or C++ back to Python and a lot of times
we’re going to do this all in one fell
swoop now the glory of site ton is you
can do this without really knowing
anything about C C++ it’s just going to
generate that code for you now why would
you want to do this well there’s quite a
few reasons like there’s some kind of
objects and like classes and attributes
stuff like that that are just going to
be more beneficial if we did it and C
but actually some of the main gains
you’re going to see or just from typing
information and that’s it so cases are
if you’ve been programming Python long
enough and you’ve been doing things in
Python you might have heard something
like I’ve heard where it’s like hey
syntax why are you using Python for
machine learning when you could use C or
C++ and it would be so much faster well
that’s because I’m actually not just
using Python I’m using Python and which
is have just so happens to be wrapping C
or C++ right that’s what we do so a lot
of libraries are actually C optimized
libraries like numpy pandas these are c
optimized libraries they just happen to
have basically a python api for you so
in most most cases you can actually find
but already made c optimized library for
whatever you’re trying to do but many
times you can’t how or at least
sometimes you can’t so one way that you
can it still improve the speed of your
Python program is to use something like
scythe on so the the main thing that at
least at the very basic level that you
can very simply do a scythe on there’s a
lot of really advanced stuff that you
can do size on as well but the basic
thing that you can do is just add typing
information so python is a dynamically
typed programming
language what that means is when you
type in a variable you just say x equals
five but later on X could equal Gary and
you’re not going to have any you know at
least you’re not gonna have any super
errors relating to at one point it was a
integer and now it’s a string you might
have other errors in your code because
of that but because it’s a dynamically
typed language the variable we don’t
have to actually tell the programming
language hey this is going to be an
integer or this is going to be a float
the problem is every time that variable
is referenced python has to say wait
hold on a second let’s see is this an
INT is this a full stream okay okay here
it is yeah it’s definitely a string
right this takes time now if you’re only
doing this once or something in a whole
program you don’t really have to add
typing information but when you start to
scale out the usage of variables either
in for loops or maybe you’re just a very
large website that has a ton of traffic
and you’re scaling just in that way
adding typing information can help you
so much so that’s what we’re going to be
showing here so anyways to get seith on
what you’re going to need is first of
all you’re going to need to where why
are you doing this to me right now
there we go so pip install or pip 3
install scythe on now I already have it
although this might try to put it to
Python 2 yeah anyway pip install scythe
on this is actually a Python 3 tutorials
so actually I would do three install
size on anyway yeah I already have it
the other thing you need is a compiler
or C compiler so if you’re on Linux you
already have one yay GCC should be
already here if you’re on Mac it should
be like Apple’s Xcode I think you can
grab and that has GCC I don’t know much
about Mac’s so if you’re on a Mac and
that doesn’t work for you sorry
go to scythe ons website though and they
have like a bunch of blurbs on there if
you’re on Windows you can use mingw
that’s probably the easiest one to use
but it’s still a pain in the butt that’s
why I’m filming this on a butsu anyway
but yeah it can be done it’s just a
massive pain so anyway
once you have sight on once you have a
compiler you’re ready to rumble so what
I’m going to do is make a new folder I’m
going to call it sites on tutorials and
we’re going to go ahead go on in there
and the first thing that we’re going to
do is let’s just let me just show us to
a new document example underscore PI dot
pi and let me just open this so what I
mean by like static and dynamic type
variables it’s like in Python you could
say x equals 5 right but in other
languages you might have like int x
equals 5
now what’s icons going to expect of us
is either see def is x equals whatever
or not or it can be like see PDF can’t x
equals 5 something like that now I’ll
talk a little bit more about those when
we get there but that’s kind of what it
is like so this is dynamic right X
actually doesn’t have to be an integer
like later on like I was saying before
we could say X is actually Gary screw
this autocomplete anyway um cool so now
let’s go ahead and example pie
let’s do define tests tests will take
one parameter that will be X and mental
retains y equals zero for now and then
we’re going to stay cool on in whole
range of X : y plus equals I and then
we’re going to return Y we also could
say Y plus equal never mind anyway
that’s good enough we’ll go ahead and
say that and that’s our regular Python
program now what if we wanted to compile
is with scythe on we haven’t added any
typing information or anything like that
but any time we do do that we have to
compile it to Saipan so first let me
show you just how simple it is to
compile to cycle so and create like a
shared object and all that so first
let’s go ahead and go to file save as
example sigh dot pi X so dot py x
that’s it save it done so now what we
need to do is create the setup PI so
let’s go ahead and new document setup
sup PI let’s open that in sublime and
now we’re going to do is from this utils
core imports setup just like a typical
stuffs up high and then what we’re going
to say is from scythe on bill that’s
capital C capital B import scythe on
eyes not typical so now we’re going to
say is setup and then the extension
modules that we’re going to be using EXT
underscore modules is equal to syphon
eyes and then whatever our actual pyx
file is so that’s going to be example
dot on earth rather example size dot d Y
X and that’s it for the setup pipe so
really there’s other things we can pass
in here but just for super simple
purposes this is really it
that’s all you got to do so now all we
have to do is actually run this so we’ll
open in terminal bring this over here
can I do this you know can I do this I’d
like to zoom in I know there’s a command
for it sorry anyway hopefully this is a
simple enough anyway so python 3 set up
high build underscore ext – – in place
we wish I could make that bigger you can
go ahead and run that let’s see if it
could control put I tried ctrl + it must
be because I’m in a virtual machine cool
anyway we’ll run that looks great bunch
of stuff we don’t understand but that’s
ok the thing that we care about is right
here so it’s this this s oh this shared
object so what’s created for us is our C
file some build information in here but
we don’t really worry about that and but
actually what we care about is right
here this is actual examples sign now
what we could do now is new folder
I call this step for deprecated and then
I’m going to take example sigh what toss
it in debt I didn’t mean to go into
death as well anyway now what we can do
is we can go and do new document testing
dot hi Wow got a little excited there
anyway and now what we can do is we can
import example underscore sign now again
I moved example signup pyx just so you
can see that it doesn’t import that file
but it won’t if you had that file
locally it’s still not going to import
it like dot pyx is not going to get
imported I hope I’m not mistaken on that
but pretty sure right example underscore
side dot test five so save that we’ll
come over here and Python 3 testing dot
PI is it not oh it doesn’t won’t print
it that doesn’t matter anyway it ran
that’s all that matters I don’t want to
print it out right now we will get to
some more output information later
anyway that’s how simple it is to at
least build and all that with sight on
right so now all we have to do is add
typing information you don’t have to you
can add as much or as little typing
information as you want okay so it’s
really up to you how deep down the
rabbit hole of scythe on you want to go
so let’s start slowly going down the
hole now I put the declarations in fact
I think I’m just going to pull it out
because I don’t really feel like
that way to type them all out you can go
to Python program internet to get all
these as well but I’ll just pull this
over cool so these are just your
declaration so if you’ve got integers
hopefully you can see that well here we
go so if you have an integer you can
it’s just see that int characters car
float double right single precision
double precision list Dix objects okay
and then you’ve got def CDF and CP def
for like your actual functions def it’s
just a regular Python function it’s in
Python only can’t be referenced within C
you’ve got C deaths this is a pure C
function and then you’ve got CP def
which is C in Python so it goes both
ways
so a common question is um you know
since basically because what CP that’s
going to do is also generate a way to do
it in Python for you whereas C death is
only in scythe on so you could only
reference that internally you wouldn’t
be able to reference that in any of your
Python so then some people might ask
like okay why don’t we always just use
CP def right well there might be some
things of see that you just can’t do in
Python so you have to do see death okay
so it’s there if you need it I’ve never
done anything because I don’t know
enough about C I’ve never done anything
that I needed to use G def I’ve just
always use CP def but if you’re like
real talented and C and you know a lot
of tricks to the trade see def you might
use but not me
so anyway there’s that so so those are
the various declarations that we can use
so now we can do is go back to example
scipy YX and what are all the things
that we can do here so one thing that we
can do is like X we know X is going well
we’re going to we’re going to enforce
that X is going to be can’t twice so int
X so in the parameters you don’t have to
do seed of CP def you don’t do anything
like that really you’re doing it right
here so
death in this case is going to just
simply be a Python function but we can
make it C PDF if we want to we can also
specify what the return is going to be
in our case it’s going to be an int so
this is kind of confusing so normally I
wouldn’t necessarily cover that
immediately so first of all just we’ll
leave this alone let’s just give typing
information to X well that’s possible
and hopefully by the end I’ll come back
and throw that in there but then we’re
going to come down here Y so Y again is
just going to be in can’t so we’re going
to do we’re going to say see death why
because um so I guess I’ll just go ahead
and contradict myself but in there’s
really no reason to what we won’t ever
need to access Y again so we can just
use C def here so I guess probably what
I should the way I should have said it
is my functions like the actual
functions up here I would never use most
likely see dev I don’t think I’ve ever
actually defined a whole function at CF
but anyway that’s what I was thinking
when I was going through it I’m sorry
anyway see def why because we’re not
going to access Y anywhere but this
function also I so this is just think
this can get kind of tricky but yet you
want to pay attention to really all of
the variables so in this case I is a
variable and believe it or not Python is
going to ask every single time we hit
this I bar and here as well what’s the
type of I well we can help it and we can
say see this int I and we forgot to give
into one here so anyway I was too busy
explaining why I just kind of true to
myself so see deaf int I and we don’t
have to give it a starting value there’s
no no point so there’s that then you’ve
got I eyes already been defined X has
already been or rather I already has a
type X already has a type why are the
other type I has a type Y has a type now
now that we’re at the end though Y is
actually the return so instead what I’m
going to say is see PDF
in because we return an int so aunt is
like what’s the value of the function so
that’s why you’ve got int coming up here
because this function is returning an
int kind of like this bat this variable
right isn’t so right before the function
and you give what it’s going to return
you don’t have to do this by the way we
could just use def it would be totally
fine anyway CP def int test into X yeah
I think this is all good so we can save
that and we really just moved to run
setup PI again hopefully we’ll get away
with that yes and then we can test it
again okay but that’s relatively um not
exciting right now so what we’re going
to do now is come back over to testing
and let’s go ahead and let’s import Tyne
it will say sine equals time it dot
climate assignment and we’re going to
run I think we can do just one line so
we’re going to say example underscore
owes its I dot or example side dot test
let’s say five and then this means a set
up
so we’ll say set up equals and this will
be import example sigh kind of running
off the screen hopefully I can get this
to fit
cool and then we’re going to say PI it
will we need one more variable here
let’s do a number equals 100 and then
what we’re going to do is go ahead and
take this line copy paste we’re going to
call this now PI and example PI import
example underscore PI cool all right so
those are the values so none of those
print let’s print psych on opine for the
x so basically what this does if you
don’t know what time it does I do have a
tutorial that you can go python permanet
search time it and you’ll get a brief
tutorial on it it just times how long
these take to to go and you can do like
whole chunks of them set up we’ll just
import it first so we don’t calculate
how long did the import take and it’s
just a really handy thing to actually
time operations because these operations
are really really fast and tiny changes
in what’s going on just in the
background on your computer kind of
seriously effect if you just use like
import time to time these operations and
probably number 100 is too low as well
but anyway let’s just test it so print
and let’s say Saipan is and it will be
some variable X faster and then we’ll
say dot format at what spy out of sign
so let’s do
[Music]
pi /
yeah basically how many times a psyphon
go into Python sorry my brain is I think
that’s right
anyway now so now we’re going to go
ahead and do is just run this in the
terminal so come over here
and now let’s run Python 3 testing dot
pi for 10 minutes up let’s go over here
where did we say – oh right here time it
try again
cool so in this case we can I mean this
is a little hard to read but we’ve got
the multiplier here and so we see
saikhan is actually 8 times faster
around one more time this can still vary
yeah 12 times faster now eight times
right so because let’s go ahead and do
let’s look at thousand maybe ten
thousand hopefully we can run that
pretty quickly oh my god still a huge
variance okay somewhere between five and
fifteen times faster anyway bottom line
is it’s clearly faster like we haven’t
actually been slower so that’s not too
bad now what I like to do is is try to
drive home why is it faster though so
coming back to here here where’s the the
largest gain being made so for example
what if we said you know if we didn’t do
this like what if we just got rid of I
here and is just remove that save and
then now let’s just rebuild retest and
then hmm
that’s um not much faster now it’s
actually slower whoops oh shoot what did
I do
I think I
rebuilt it there we go anyway slower a
little faster little faster slower
slower right so just that line made a
huge difference right
what if okay so clearly into eye matters
what about what oh I’m not appreciate
commenting should work it yeah there’s
no reason why that won’t work but I’m
just going to leave it anyways let’s
rebuild test it again
Oh exception
why reference before oh yeah what a noob
try again there we go so um so now it’s
like three four nine three four four
four four okay so clearly that one also
matters right that made a big difference
okay
trying to think of what else we could do
we could we could take away the typing
information for X now this should be
checked only once right it’ll it’ll go
through a range but it should only be
checked once so I’m going to wager
that’s not going to have a huge impact
but let’s see so we can build let’s test
right
test test test right not a big impact
because it was only referenced one time
okay
tracked I think we’re all set on all the
other variables at this point so so that
wasn’t really truly necessary so what
I’m trying to point to though is that
the huge gain happened basically
everything that was involved in this for
loop because every time we go through it
we’ve got to reference that variable
except for X because X is only value X
isn’t being reassessed every single time
we go through this loop how do I know
that because we didn’t gain games by you
know not chain you know like like object
we all need the reference value of X
once before we know okay how many loops
or interations are we going to do we
don’t need to read you know keep
rear-ending what is X what is X that’s
not that we don’t have to do that so
anyway okay so that’s one thing now
knowing what we know
in this loop so you know maybe you want
to impress all your friends who aren’t
Python programmers and you want to say
this is why I use Python and I know how
to make Python so fast so what’s one way
that we could make this better
because a lot of times you know maybe
you’ve heard that scythe on can help you
be like 200 times faster right so what
could we do well we could just make that
that four leaf larger so like what if we
said rather than five what if we said
5000 we might have to change this number
of iterations let’s do a thousand save
we don’t need to rebuild we just test
again boom 119 times faster they’re out
36 that’s ugly let’s go back to get one
1801 24 okay so we’re pretty we’re
pretty ballin at this point on stripe
50,000 Oh taking forever come on it’s
going to be faster Wow
I think we went down in speed 140 times
we’re going with that one
okay so we’ve written code that’s 140
times faster anyway okay so there are at
least a few examples or at least one
example of some simple code but adding
some simple typing information which
ones actually mattered which ones really
didn’t matter for example two let’s get
rid of the definition here so let’s just
take away that entire top line again
this shouldn’t have really a huge effect
I hope I’m not wrong but Python 3
testing dot pi obvious we left our loop
gigantic but anyway cool yeah so but
they’re going to have like a huge impact
in before like 36
yeah cool so so that’s not really
essential that you give it this the only
time I may be essential really prime in
my opinion is if it needed to be a seed
if but anyway or if you’re calling the
function whole bunch of times in a loop
make it CP desc
give the return of it and then give this
of
okay now let me go ahead and build that
one more time let’s test it and I want
to show one more thing for I let you all
go and that is the the HTML kind of
conversion that you can get from this so
so what I’d like to do is let’s take and
try and set how I want to do this
because we can’t let’s go to example dot
pi so let’s copy this paste so weird is
pasting the original example dot PI into
the new example side out pyx kind of
what we had initially I saved it and
then now I’m going to do is go into the
terminal and I am going to do I think
it’s just Saipan – a and then let’s call
this what was the example it’s just
example underscore Sun vample underscore
its size dot p YX will run that and what
this does is it just creates your C file
and ah even though this is here it
doesn’t create your s o file I don’t
know why I wish if it was a single file
you could just do size on a rather and
have the setup PI or some form of saipan
if somebody knows that you can do that
without setup pi let me know because it
doesn’t make any sense like why wouldn’t
you just be able to build it with just
the size on call but for whatever reason
you can but what I’m interested in is
the HTML file it produces so what we can
do is we can open this in Firefox and
basically as it tells you right here the
yellow lines hint at your Python
interaction interestingly enough y
equals zero
apparently isn’t super close I think
it’s there it’s just the least
interactive to Python but anyway you can
see all of you know how close are we in
in Python so clearly the definition is
just pure Python and
then the for loop is very heavy in
Python so now we can close this and
let’s go back here let’s just do ctrl Z
save and then let’s now do the same
thing again we’ll get a new HTML file
let’s open that one and now we get
basically only one line that’s the first
line which is CP def because we wanted
to be able to reference it and use it in
Python but other than that the rest of
this is nowhere near Python okay so
anyway pretty cool so so what has it
done for you basically like this for
loop has been written for you in C so
that’s in this file here and I don’t
know about you but I’m not going to
write like especially if you have like a
large Python program I mean it’s going
to generate like yeah like let’s say 50
lines of Python then generate like
hundreds of lines in C for you which
would be very annoying to to write
yourself and I probably would not be
successful but obviously this is it’s
pretty successful super tested I like I
said before it’s used in a lot of large
libraries like numpy pandas um’s and so
on it’s a huge list of packages that use
scythe on okay I think that we’ve made
it through everything I wanted to cover
at the the most basic level hopefully
that was simple enough
I personally was really surprised I had
looked into scythe on quite a few times
in the past and I just kind of found the
documentation to be not so great I found
a lot of kind of like tutorials on
Saipan but it was really like talks on
Saipan without actually showing exactly
the code and so for the longest time I
just couldn’t figure out Saipan but then
when I figured it out and I realized oh
my gosh it’s basically at least a
beginner insight on all it is just
typing information that’s it there are
more interesting things that you can do
obviously if you’re C programmer or C++
there’s some other things you should
look into also we can’t do it in this
for loop but there are certain things
that we can add with like with no guild
so this will release the Gil for you
you can do things unfortunately you’re
not going to be able to do this you’re
not going to be able to modify this
without the Gil obviously you couldn’t
do that and see anyways like you
couldn’t
you’d have to handle for that but
there’s some things that you can do
without the Gil as long as they’re not
you know changing some sort of shared
variable or something like that but yeah
there’s there’s quite a few other things
that we can do with scythe on and I
might cover them more in the future but
at even the most basic level just adding
typing information can give you huge
improvements so anyways that’s all for
now
if you have questions comments concerns
something specific you’d like me to
cover in scythe on feel free to let me
know below otherwise as always thanks
for watching things for all those for
subscriptions and I will see you in
another video
Please follow and like us:

Be First to Comment

Leave a Reply