Playing with Pyrex

According to its author, Greg Ewing, Pyrex "lets you write code that mixes Python and C data types any way you want, and compiles it into a C extension for Python". It's a tremendous addition to Python's vast arsenal of tools.

Pyrex and Distutils

Update -- July 10: after some valuable feedback from Thomas Heller, I've modified the distribution of Pyrex available at this site. See this discussion thread for details.

Distutils is the standardized distribution/installation system for Python modules. In order to make Pyrex 0.3.3 work nicely with Distutils, I've created an experimental distribution of Pyrex 0.3.3 that integrates it nicely with the Distutils system. So far I have tested the modified distribution on Win32 (msvc and mingw32 compilers).

What did I change? Well, since Pyrex is Python extension language, I adapted the build_ext Distutil command, which is responsible for building extension modules. I added a subpackage, Pyrex.Distutils, that, when used in a setup.py script, causes the standard build_ext command to be overridden with a Pyrex-specific command. This modified command compiles any specified .pyx files into .c files before continuing with the standard build_ext, which finishes the task of building your .pyd (or .so, on Unix) module files.

Note that Pyrex.Distutils is a proper sub-package of Pyrex. No files have to be copied into the distutils.command directory for this to work.

Also included in Pyrex.Distutils is a hack to allow compilation of Pyrex-generated C files when you use the mingw32 compiler. See the discussion at comp.lang.python (and this discussion thread too). Basically, when using mingw32, Distutils will compile in C++ mode.

Example

Here's the example I included in my distribution:

foo.pyx – a straightforward extension class written in Pyrex.

"""A simple extension module example"""

cdef class Foo:
	"""A foo, like any other.
	Create one with Foo(s), where s is the name to assign to your foo."""

	cdef char *name

	def __init__(self, name):
		self.name = name  # don't do this in real code, it's unsafe.

	def __repr__(self):
		return "A Foo named %s." % (self.name)

setup.py – note the import of Pyrex.Distutils.build_ext, its placement via the cmdclass argument, and the specification of .pyx files (not .c files) in the ext_modules parameter.

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext

setup(name='foo', ext_modules=[Extension("foo", ["foo.pyx"])],
	cmdclass = {'build_ext': build_ext}
)

To build the extension module, fod, call the setup script like so:

python setup.py build_ext --inplace

The --inplace is just there so that foo.pyd ends up in the current directory (so that we can test it).

c:\testdir>python
ActivePython 2.2.1 Build 222 (ActiveState Corp.) based on
Python 2.2.1 (#34, Apr 15 2002, 09:51:39) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
>>> f = foo.Foo('Fred')
>>> f.__class__
<type 'foo.Foo'>
>>> f
A Foo named Fred.

References

The Pyrex project page
my modified distribution of Pyrex 0.3.3
Python docs – Distributing Python Modules
Python docs – Extending and Embedding the Python Interpreter

Pyrex and Win32

These are some basic notes I compiled while learning how to use Pyrex, and getting it to run on my Win32 box. Most of these notes are obsolete in the light of the Distutils approach above.

Installation

What you'll need:

Python 2.2

Python 2.2 or higher is required. I used an ActiveState distribution.

C compiler

MS Visual C will work fine but you can also use a number of other (free) compilers. You must, however, set up your environment to correctly build extension modules. See Using non-Microsoft compilers on Windows for details.

If you are using GCC/mingw32, the GNU compiler targetted against the native MSVCRT.DLL runtime, I have the import library libpython22.a available for download.

Pyrex

Download and unpack Pyrex itself, from the project page. If you already have it, ensure that it is version 0.3.3 or higher, otherwise you will also need Greg Ewing's Plex library.

Where you unpack Pyrex to is up to you; I unpacked it into \python22\tools and then renamed the toplevel folder from pyrex0.3.3 to pyrex.
In the toplevel folder, you'll find a file called pyrexc. This is the script that runs the compiler. Rename this file to pyrexc.py.
- Warning: in the 0.3.3 version of Pyrex, the script will not properly build (on Win32) extension classes that you write. I've got a hacked pyrexc.py that you can use as a replacement; this will fix the problem. Greg may correct the underlying issue in a future release; see the discussion at comp.lang.python.

Building your first extension class

Pyrex can be used to build native functions as well as extension classes. Here's a simple example based on Greg's introductory material that you can try.

Create a new folder for your project.
If you want to make life easier, download the file pyrex.py and copy it into your project folder. This script automates the steps needed to compile your .pyx files into .pyd modules. You may need to adjust the constants in the script to properly locate your GCC, Python and Pyrex installations. As the script runs, it shows you the shell commands that it executes so that you can learn to do it yourself.

Create a sample Pyrex source file, shrubbery.pyx :

cdef class Shrubbery:
    cdef int width, height

    def __init__(self, w, h):
        self.width = w
        self.height = h

    def describe(self):
        print "This shrubbery is", self.width, "by", self.height, "cubits."

compile this file into a module, using the pyrex.py script:

C:\testpyrex>python pyrex.py
pyrex.py. For options, use --help

build

clean

pyrexc

python \python22\tools\Pyrex\pyrexc.py shrubbery.pyx
Unpickling lexicon...
Done (0.17 seconds)

compile

gcc -I/python22/include -c shrubbery.c -o shrubbery.o

makedef

wrap

dllwrap --dllname shrubbery.pyd --driver-name gcc --def shrubbery.def -o shrubbe
ry.pyd shrubbery.o -s --entry _DllMain@12 --target=i386-mingw32 -L/python22/libs
 -lpython22

testimport
testing import on package shrubbery

python -c "import shrubbery; print dir(shrubbery)"
['Shrubbery', '__builtins__', '__doc__', '__file__', '__name__']

C:\testpyrex>

At this point, you should be able to use your new extension class!

C:\testpyrex>python
ActivePython 2.2.1 Build 222 (ActiveState Corp.) based on
Python 2.2.1 (#34, Apr 15 2002, 09:51:39) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import shrubbery
>>> shrub = shrubbery.Shrubbery(25, 50)
>>> shrub
<shrubbery.Shrubbery object at 0x007689B8
>>> shrub.describe()
This shrubbery is 25 by 50 cubits.

Final words

If you haven't done so already, read the Python document, Extending and Embedding the Python Interpreter, the definitive source on how to write extension modules.

Of course, you'll want the Pyrex documentation as well, it's at the Pyrex project site, as well as in the installation archive.

Special thanks to Greg Ewing for writing the Pyrex software.

Last modified July 2002 by Graham Fawcett.

home