Speeding Up Python¶

There are a couple of options for speeding up Python.

psyco¶

(Taken almost verbatim from the psyco introduction!)

psyco is a specializing compiler that lets you run your existing Python code much faster, with absolutely no change in your source code. It acts like a just-in-time compiler by rewriting several versions of your code blocks and then optimizing them by specializing the variables they use.

The main benefit is that you get a 2-100x speed-up with an unmodified Python interpreter and unmodified source code. (You just need to import psyco.)

The main drawbacks are that it only runs on i386-compatible processors (so, not PPC Macs) and it’s a bit of a memory hog.

For example, if you use the prime number generator generator code (see Idiomatic Python) to generate all primes under 100000, it takes about 10.4 seconds on my development server. With psyco, it takes about 1.6 seconds (that’s about a 6x speedup). Even when doing less numerical stuff, I see at least a 2x speedup.

Installing psyco¶

(Note: psyco is an extension module and does not come in pre-compiled form. Therefore, you will need to have a Python-compatible C compiler installed in order to install psyco.)

Grab the latest psyco snapshot from here:

http://psyco.sourceforge.net/psycoguide/sources.html

unpack it, and run ‘python setup.py install’.

Using psyco¶

Put the following code at the top of your __main__ Python script:

try:
   import psyco
   psyco.full()
except ImportError:
   pass

...and you’re done. (Yes, it’s magic!)

The only place where psyco won’t help you much is when you have already recoded the CPU-intensive component of your code into an extension module.

pyrex¶

pyrex is a Python-like language used to create C modules for Python. You can use it for two purposes: to increase performance by (re)writing your code in C (but with a friendly extension language), and to make C libraries available to Python.

In the context of speeding things up, here’s an example program:

def primes(int maxprime):
  cdef int n, k, i
  cdef int p[100000]
  result = []
  k = 0
  n = 2
  while n < maxprime:
    i = 0

    # test against previous primes
    while i < k and n % p[i] <> 0:
      i = i + 1

    # prime? if so, save.
    if i == k:
      p[k] = n
      k = k + 1
      result.append(n)
    n = n + 1

  return result

To compile this, you would execute:

pyrexc primes.pyx
gcc -c -fPIC -I /usr/local/include/python2.5 primes.c
gcc -shared primes.o -o primes.so

Or, more nicely, you can write a setup.py using some of the Pyrex helper functions:

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext                # <--

setup(
  name = "primes",
  ext_modules=[
    Extension("primes", ["primes.pyx"], libraries = [])
    ],
  cmdclass = {'build_ext': build_ext}
)

A few notes:

‘cdef’ is a C definition statement

this is a “python-alike” language but not Python, per se ;)

pyrex does handle a lot of the nasty C extension stuff for you.

There’s an excellent guide to Pyrex available online here: http://ldots.org/pyrex-guide/.

I haven’t used Pyrex much myself, but I have a friend who swears by it. My concerns are that it’s a “C/Python-alike” language but not C or Python, and I have already memorized too many weird rules about too many languages!

We’ll encounter Pyrex a bit further down the road in the context of linking existing C/C++ code into your own code.

Navigation