Developer Guide

Do not hang around in Pythran code base without your developer guide! It is the compass that will guide you in the code jungle!

Disclaimer

This document is a never ending work-in-progress draft. Please contribute!

Additional Packages

Additional packages may be needed to use pythran to its fullest: colored logs, tests on several cores, doc generation

$> sudo apt-get install python-setuptools python-pytest-xdist python-docutils
$> sudo pip install colorlog pytest-pep8

Configuration

Pythran can be configured with a rc file. An example is found in pythran/pythran.cfg. Look at it! To customize it:

$> cp pythran/pythran.cfg ~/.pythranrc

In particular, you may want to add -g -O0 to the cxxflags.

NT2 is required to compile the C++ code generated by Pythran. It is automatically installed along Pythran. If you don’t have NT2 installed in a system directory and you try to run Pythran directly from the source directory, the NT2 includes won’t be found.

A trick to ease the development and to enable running directly from the source is to install Pythran once and to add -I<install_prefix>/lib/<python-version>/site-packages/pythran to the cppflags in order to point to the NT2 headers.

Coding Style

All Python code must be conform to the PEP 8, and the pep8 command must not yield any message when run on our database. Additionally, avoid backslashes, and try to make your code as concise as possible.

C++ code use spaces (no tabs) and a tab width of 4.

File Hierarchy

Listing the top level directory yields the following entries:

setup.py
The files that describes what gets installed, that holds PyPI entries and such.
doc/
If you’re reading this document, you know what it’s all about! MANUAL is the user documentation and DEVGUIDE is the developer documentation.
LICENSE
Boring but important stuff.
MANIFEST.in
Describe additional stuff to package there.
README
Quick introduction and description of _pythran_. The README.rst file is just a symbolic link that pleases github and PyPI.
pythran/
The source of all things.
pythran/tests/
The source of all issues.
pythran/pythonic/
Where C++ back-end lies.
scripts/
Where Python scripts calling the pythran module lies.

Validation

pythran uses the unittest module to manage test cases. The whole validation suite is run through the command:

$> python setup.py test

It is tested using the pytest <http://pytest.org/latest/> package:

$> apt-get install python-pytest

To run it faster we use the pytest extension xdist from debian package python-pytest-xdist , the test suite will run using all available cores. Otherwise it might run very slowly, something like four hours on a decent laptop :’(.

pep8 is checked using the pytest extension pytest-pep8. It can be installed using:

$> sudo pip install pytest-pep8

Note that it is still possible to use the pytest module directly, for instance to pass a subset of the test suite:

$> PYTHONPATH=. py.test -n 8 pythran/tests/test_list.py

runs all the tests found in pythran/tests/test_list.py.

Only compiler tests can be check using test filtering:

$> PYTHONPATH=. py.test -n 8 pythran/tests -m "not module"

There are two kinds of tests in pythran:

  1. unit tests that test a specific feature of the implementation. Such tests are listed as method of a class deriving from test_env.TestEnv and must call the run_test(function_to_translate, *effective_parameters, **name_to_signature) method [1]. It translates function_to_translate into a native function using the type annotations given in the name_to_signature dictionary, runs both the python and the native version with effective_parameters as arguments and asserts the results are the same.

    [1]

    See examples in pythran/tests/test_base.py for more details.

  2. test cases that are just plain python modules to be converted in native module by pythran. It is used to test complex situations, codes or benchmarks found on the web etc. They are just translated, not run. These test cases lie in pythran/tests/cases/ and are listed in pythran/tests/test_cases.py.

C++ runtime

The C++ code generated by pythran relies on a specific back-end, pythonic. It is a set of headers that mimics Python’s intrinsics and collections behavior in C++. It lies in pythran/pythonic/. There is one directory per module, e.g. pythran/pythonic/numpy for the numpy module, and one file per function, e.g. pythran/pythonic/numpy/ones.hpp for the numpy.ones function. Type definitions are stored in the seperate pythran/pythonic/types directory, one header per type. Each function header must be #includ-able independently, i.e. it itself includes all the type and function definition it needs. This helps keeping compilation time low.

All Pythran functions and types live in the pythonic namespace. Each extra module defines a new namespace, like pythonic::math or pythonic::random, and each type is defined in the pythonic::types namespace. The DECLARE_FUNCTOR and DEFINE_FUNCTOR macros from pythonic/utils/functor.hpp is commonly used to convert functions into functors and put them into the mandatory functor namespace.

The pythonic runtime can be used without Python support, so it is important to protect all Python-specific stuff inside ENABLE_PYTHON_MODULE guard.

All methods are represented by functions in Pythran. The associated pseudo-modules are prefixed and suffixed by a double underscore __, as in pythran/pythonic/__list__.

Benchmarking and Testing

Stand-alone algorithms are put into pythran/tests/cases. They must be valid Pythran input (including spec annotations). To be taken into account by the validation suite, they must be listed in pythran/tests/test_cases.py. To be taken into account by the benchmarking suite, they must have a line starting with the #runas directive. Check pythran/tests/matmul.py for a complete example.

To run the benchmark suite, one can rely on:

$> python setup.py bench --mode=<mode>

where <mode> is one among:

python
Uses the interpreter used to run setup.py.
pythran
Uses the Pythran compiler.
pythran+omp
Uses the Pythran compiler in OpenMP mode.

All measurements are made using the timeit module. The number of iterations is customizable through the --nb-iter switch.

How to

Add support for a new module:
 
  1. Provide its C++ implementation in pythran/pythonic++/<mymodule>.
    pythran/pythonic++/math/*.hpp and pythran/pythonic++/__list__/*.hpp are good example to referer to.
  2. Provide its description in pythran/tables.py. Each function, method
    or variable must be listed there with the appropriate description.
  3. Provide its test suite in pythran/tests/ under the name
    test_my_module.py. One test case per function, method or variable is great.
Add a new analysis:
 
  1. Subclass one of ModuleAnalysis, FunctionAnalysis or NodeAnalysis.
  2. List analysis required by yours in the parent constructor, they will be built automatically and stored in the attribute with the corresponding uncameled name.
  3. Write your analysis as a regular ast.NodeVisitor. The analysis result must be stored in self.result.
  4. Use it either from another pass’s constructor, or through the passmanager.gather function.
Push changes into the holy trunk:
 
  1. Use the github interface and the pull/push requests features
  2. Make your dev available on the web and asks for a merge on the IRC
    channel #pythran