Quickimport

The module quickimport provides an improved importer for python. If you ever started a Python application from a lame file server (like a CIFS server) you know the problem of long startup times. Usually the long startup time is caused by the enormous number of stat() calls required to locate a module. For each directory on sys.path (or on the package specific search path) and for each possible suffix (see function imp.get_suffixes()) Python probes the existence of a module or a package using the C-function call stat(). If your search path contains several directories (if you use easy_install, it usually contains many directories), you end up with several thousands stat() calls. Most of these calls fail, because the probed file does not exist.

The basic idea of Quickimport is to avoid those stat() calls that will fail for sure, because there is no matching file in the directory in question. If we assume, that the content of the directory does not change while our application is running, we can read the directory content once. A perfect implementation of this concept requires a patch for python. Fortunately we can use the PEP-302 import hooks to create a nearly perfect version, that skips directories not containing any candidate for the wanted module or package. You only need to invoke the function install() early enough. This function installs a PEP-302 importer and optionally eliminates directories without any Python modules or packages from sys.path.

Another option to speed up lame imports is to store many modules into a zip-archive. The time required to extract modules from a zip-archive is usually much lower than the time to locate the modules on the file system and to read them. Therefore sys.path already contains an entry for a zip-archive pythonXY.zip. The function buildZip() creates such a zip-archive for you.

Warning

Although the author is using the module quickimport in production, it is more or less untested outside the specific environment it was written for.

quickimport.install(flags=None, dirs=None)

Install the Quickimport importer.

This function optionally

  • removes directories without any modules or packages from sys.path.
  • appends the PEP-302 importer QuickimportFinder to the end of the sys.path_hooks list.
Parameters:
  • flags (str or a sequence of strings) – A string that contains keywords to control the Quickimport behavior.
  • dirs (sequence of strings) – A list of directories to cache. If this parameter is not given, sys.path is used instead.

Known keywords in flags are are:

off
Disable Quickimport entirely. Useful if you encounter import problems and want to rule out any influence from Quickimport.
noCache
Disable the caching of directories and the Quickimport PEP-302 finder.
noAutocache
Disable the automatic addition of directories to the directory cache. Using this flags limits the usage of Quickimport to the directories given by the dirs argument.
filterDirs
Remove non relevant items from dirs. An item is not relevant, if it is a directory, that does not contain any Python modules or packages.
quickimport.uninstall()

Uninstall Quickimport

quickimport.isDirRelevant(dir)

Test is a given directory contains any entries that could be modules or packages.

Parameters:dir (str or unicode) – the directory to inspect
Returns:True if the directory might contain Python modules or packages. False otherwise.
Return type:bool
quickimport.prepareCache(path=None, cache=None)

Create or update the directory cache for QuickimportFinder.

Parameters:
  • path (list) – a list of path entries to process
  • cache (dict) – the cache dictionary. If not given, a new dictionary is used
Returns:

the cache dictionary
Return type:

dict
quickimport.buildZip(zipname=None)

Build a zip-archive containing all suitable top-level modules.

This function creates the zip-archive zipname and adds a compiled version of every suitable top-level module to the archive. A module is suitable, if the module’s Python source code is available and does not reference the __path__ or __file__ members.

This function uses the python logging system to report its progress. Set the log level to INFO or DEBUG to get progress information.

You can run this module as a pathon script to invoke this function:

python -m quickimport [zipname]
Parameters:zipname (str or unicode) – the name of the archive. If not given, the function uses the name of the pythonXY.zip-item of sys.path.
quickimport.newQuickimportFinder(dir)

A PEP-302 finder factory function for the import hook sys.path_hooks.

This function returns a new finder object of class QuickimportFinder or NullFinder, if dir denotes a regular directory. Class NullFinder is used, if the directory does not contain any Python module or package.

If dir does not denote a regular directory, this function raises ImportError.

class quickimport.NullFinder

Bases: object

A PEP-302 finder class for the sys.path_hooks hook.

This class never finds anything.

class quickimport.QuickimportFinder(dir)

Bases: pkgutil.ImpImporter

A PEP-302 finder class for the sys.path_hooks hook.

This class uses the directory cache sys.quickimport_cache to store the content of directories from sys.path or from package specific search path lists.