Using SCons for processing pipelines
====================================


.. deprecated:: 0.12
   This was very experimental (and possibly not working),
   and is deprecated anyway.
   

.. note:: 
   This requires scons installed.


No plumbing happens without pipes at some point; this package
also provides tools for handling pipelines.

`SCons` is a system to build software, akin to `make` and `Makefile`,
but it also be used to run any sequence of computing steps depending
on one another such as in an analysis pipeline.
This is of interest for implementing some kind of fault tolerance and
be able to resume a pipeline after interruption without re-runnning everything.
Causes for an interrupted pipeline go from hardware failure, to running out
of ressources (storage, or memory), to
unavailable third-party ressources, 
and to a queuing system with defined timeout killing tasks.

An `SConstruct` file taking FASTAQ files to sorted and indexed files
ready to use with :mod:`pysam` would look like:

.. code-block:: python

   from ngs_plumbing.scons import Bowtie_5500_ECC

   # path to the reference index file
   reference = '/path/to/index'

   bowtie_5500_ecc = Bowtie_5500_ECC(mapper = 'bowtie',
                                     reference = reference,
				     nproc = 3)
   env = Environment(BUILDER = {'bowtie': bowtie_5500_ecc,
                                'st_view': samtools_view,	
                                'st_sort': samtools_sort,
                                'st_index': samtools_index})
   Decider('timestamp-match')
   # loop over the file names (no extension)
   my_samples = ('foo', 'bar')
   for fn in my_files:
       env.bowtie(fn)
       env.st_view(fn)
       env.st_sort(fn)
       env.st_index(fn)

Running the pipeling can now go as simply as:

.. code-block:: sh

   scons

It is easy to create one's own functions.

.. automodule:: ngs_plumbing.scons
   :members:
   :undoc-members: