.. _overview:
Overview
--------
The ``distil`` tool is a testbed for ``distlib``, which is a low-level library
that implements basic packaging functionality, and is intended for use by
packaging tools.
Why ``distil``?
===============
While maintainers of the `pip `_ project have
expressed an interest in the possibility of implementing some parts of ``pip``
functionality using ``distlib``, this is dependent on available volunteer time
and overall project priorities, and so no specific timescales for any adoption
of particular features of ``distlib`` can be expected. Of course, this also
applies to packaging tools other than ``pip``.
So, ``distil`` aims to fill the gap by providing a packaging tool, based only
on ``distlib`` and the Python standard library. If it is to demonstrate that
``distlib`` is useful for and usable by packaging tools, ``distil`` cannot be a
toy -- it must aim to provide a large part of the functionality of tools like
``pip``, and it must show, where possible, improvements over existing tools
that are possible through the use of ``distlib``.
.. note:: ``distil`` is not intended to supplant any existing packaging tool,
but rather to act as a testbed for ``distlib`` and to explore useful areas
in packaging. It should be used in virtual environments set up specifically
for the purpose of experimenting with it. Remember, it is alpha software.
Features
========
The initial release of ``distil`` provides the following features:
* Install projects from PyPI and wheels
(see `PEP 427 `_).
Distil does not invoke ``setup.py``, so projects that do significant
computation in ``setup.py`` may not be installable by ``distil``. However, a
large number of projects on PyPI *can* be installed, and dependencies
are detected, downloaded and installed.
* Optionally upgrade installed distributions, whether installed by ``distil``
or installed by ``pip``.
* Uninstall distributions installed by ``distil`` or ``pip``.
* Install links to local projects which remain editable so that changes are
immediately reflected in an environment (like ``pip install -e dir``).
* Build source distributions in ``.tar.gz``, ``.tar.bz2``, and ``.zip``
formats.
* Build binary distributions in wheel format. These can be pure-Python, or
have C libraries and extensions. Support for Cython and Fortran (using
``f2py``) is possible, though currently ``distil`` cannot install ``Cython``
or ``Numpy`` because of how they use ``setup.py``. You can optionally use
``pip`` to build wheels, which can then be installed using ``distil``.
* Run tests on built distributions.
* Register projects on PyPI.
* Upload distributions to PyPI.
* Upload documentation to http://pythonhosted.org/ (formerly
http://packages.python.org/).
* Very simple deployment -- just copy ``distil.py`` to a location on your path,
optionally naming it to ``distil`` on POSIX platforms. There's no need to
install ``distlib`` -- it's all included.
* Display dependencies of a distribution -- either as a list of what would be
downloaded (and a suggested download order), or in Graphviz format suitable
for conversion to an image.
* Uses either a system Python or one in a virtual environment, but by default
installs to the user site rather than system Python library locations.
* Tab-completion of commands and parameters on Bash-compatible shells.
Logically, packaging activities can be divided into a number of categories or
roles:
* *Archiver* -- builds source distributions from a source tree
* *Builder* -- builds binary distributions from source
* *Installer* -- installs source or binary distributions
This version of ``distil`` incorporates (for convenience) all of the above
roles. There is a school of thought which says that that these roles should be
fulfilled by separate programs, and that's fine for production quality tools --
it's just more convenient for now to have everything in one package for an
experimental tool like ``distil``.
Actual Improvements
===================
Despite the fact that ``distil`` is in an alpha stage of development and has
received no real-world exposure like the existing go-to packaging tools, it
does offer some improvements over them:
* Dependency resolution can be performed without downloading any distributions.
Unlike e.g. ``pip``, you are told of which additional dependencies will be
downloaded and installed, before any download occurs.
* Better information is stored for uninstallation. This allows better feedback
to be given to users during uninstallation.
* Dependency checking is done during uninstallation. Say you've installed a
distribution A, which pulled in dependencies B and C.
* If you request an uninstallation of B (or C), ``distil`` will complain that
you can't do this because A needs it.
* When you uninstall A, you are offered the option to uninstall B and C as
well (assuming you didn't install something *else* that depends on B or C,
after installing A).
* By default, installation is to the user site and not to the system Python,
so you shouldn't need to invoke ``sudo`` to install distributions for
personal use which are not for specific projects/virtual environments.
* There's no need to install ``distil`` -- the exact same script will run with
any system Python or any venv (subject to Python version constraints of 2.6,
2.7, 3.2 or greater). There's no need to have umpteen copies of
``setuptools``, ``distribute`` and ``pip`` lying about! Not that disk space
is a particular constraint, of course.
Unlike the ``pysetup`` tool that is part of ``distutils2``, ``distil`` can
install many source distributions that are already on PyPI without the need to
migrate them to ``setup.cfg`` first (or the need to convert them to wheels).
This allows a more gradual transition to new packaging norms, by not requiring
root-and-branch conversion of existing distributions.
More "actual" improvements can be achieved, dependent on feedback from you!
Possible Improvements
=====================
In line with recent thinking in packaging, ``distlib`` and ``distil`` rely on
a declarative format for distribution metadata, as opposed to the *ad hoc* code
approach (i.e. ``setup.py``) taken by
``distutils``/``setuptools``/``distribute``. This offers a means for different
packaging tools to interoperate -- because the declarative format is
standardised -- and this can lead to innovations in packaging and improved
features for Python package developers and users. There should be no more "my
way or the highway" in Python packaging, which the design of ``distutils`` has
fostered.
Limitations
===========
As already mentioned above, ``distil`` is not a replacement for ``pip`` or
``easy_install``. The main reason why ``distil`` cannot do things that ``pip``
or ``easy_install`` can do is that some projects use code in ``setup.py`` to
do things like creating new files and moving files around before invoking the
actual setup code in ``distutils``/``setuptools``/``distribute``. Because
``distlib`` and ``distil`` are declarative and don't actually execute
``setup.py``, the code therein cannot be taken advantage of, and so ``distil``
can fail when trying to build or install such projects. However, Python
packaging standards are expected to provide a migration path such that such
code as currently lives in ``setup.py`` can be replaced by other mechanisms
which allow better interoperability between packaging tools.
Distil *can* use ``pip`` to build wheels, for those cases where ``setup.py``
*must* be executed for a correct build. This only exercises the wheel-building
functionality of ``distlib``, but is provided as a convenience when
experimenting with wheels.
There are other areas of functionality which ``distil`` does not aim to
provide, such as direct installation from VCS repositories. These *could* be
provided, but are only really appropriate in a production-quality packaging
tool. Note that ``distil`` and ``distlib`` are free of dependencies other than
the Python standard library, and are expected to remain so for the foreseeable
future. (Although ``distil`` provides an option to use ``pip``, it is not a
hard dependency, and ``distil`` is useful even without ``pip`` being
installed.)
Change Log
==========
0.1.2
~~~~~
Released: Not yet.
0.1.1
~~~~~
Released: 2013-04-30
- Added ``distil init`` to support creating an initial version of
``package.json`` metadata, which can then be added to during development.
- Added ``distil link`` to support "editable" installations, similar to
``pip install -e local_dir``.
- Take into account pre-confirmation (``-y``) during uninstallation when dists
that are no longer needed are found. These are now removed automatically when
``-y`` is specified.
- Fixed error in setting up SSL certificate verification, and adjusted PyPI
URLs to be ``https://`` where specified as ``http://`` in metadata.
Successful SSL verification is now logged.
- Added ``--local-dists`` option to allow local wheels and sdists to be used
for installation.
- Fixed a bug in the handling of local archives (e.g. those returned through a
configured ``DirectoryLocator``). Local archives shouldn't be deleted after
unpacking.
- Added ``--python-tags`` argument to ``distil package`` when building wheels
to configure the tags for the built wheel.
- Added ``--no-unpack`` option to ``distil download``.
- Fixed problem with rollback on error caused by not recording ``SHARED`` and
``RECORD`` correctly.
- Fixed bug in writing entry points (EXPORTS) file.
- Use of ``2to3`` now defaults to ``True`` when run under 3.x.
- Fixed bug when run in venvs which caused e.g. ``dist-packages`` to be used
instead of ``site-packages``.
- Improved error message when run with Python 2.5 (not supported, but this is
now clear from the error message).
- Numerous documentation updates.
0.1.0
~~~~~
Released: 2013-03-22
- Initial release.