.. vim: set fileencoding=utf-8 :
.. Thu 13 Oct 2016 16:01:27 CEST

.. testsetup:: *

   import os
   import numpy
   import tempfile
   import bob.learn.linear
   import bob.io.base

   numpy.set_printoptions(precision=3, suppress=True)

   current_directory = os.path.realpath(os.curdir)
   temp_dir = tempfile.mkdtemp(prefix='bob_doctest_')
   os.chdir(temp_dir)

==============================
 Linear Machines and Trainers
==============================

Machines are one of the core components of |project|. They represent
statistical models or other functions defined by parameters that can be learnt
or manually set. The simplest of |project|'s machines is a
:py:class:`bob.learn.linear.Machine`. This package contains the definition of
this class as well as trainers that can learn linear machine parameters from
data.

Linear machines
---------------

Linear machines execute the simple operation :math:`y = \mathbf{W} x`, where
:math:`y` is the output vector, :math:`x` is the input vector and :math:`W` is
a matrix (2D array) stored in the machine. The input vector :math:`x` should be
composed of double-precision floating-point elements. The output will also be
in double-precision. Here is how to use a
:py:class:`bob.learn.linear.Machine`:

.. doctest::

  >>> W = numpy.array([[0.5, 0.5], [1.0, 1.0]], 'float64')
  >>> W
  array([[ 0.5,  0.5],
         [ 1. ,  1. ]])
  >>> machine = bob.learn.linear.Machine(W)
  >>> machine.shape
  (2, 2)
  >>> x = numpy.array([0.3, 0.4], 'float64')
  >>> y = machine(x)
  >>> y
  array([ 0.55,  0.55])

As was shown in the above example, the way to pass data through a machine is to
call its :py:meth:`bob.learn.linear.Machine.forward` method, for which the
``__call__`` method is an alias.

The first thing to notice about machines is that they can be stored and
retrieved in :py:class:`bob.io.base.HDF5File`. To save the before
metioned machine to a file, just use the machine's
:py:meth:`bob.learn.linear.Machine.save` command. Because several machines
can be stored on the same :py:class:`bob.io.base.HDF5File`, we let the user
open the file and set it up before the machine can write to it:

.. doctest::

  >>> myh5_file = bob.io.base.HDF5File('linear.hdf5', 'w')
  >>> #do other operations on myh5_file to set it up, optionally
  >>> machine.save(myh5_file)
  >>> del myh5_file #close

You can load the machine again in a similar way:

.. doctest::

  >>> myh5_file = bob.io.base.HDF5File('linear.hdf5')
  >>> reloaded = bob.learn.linear.Machine(myh5_file)
  >>> numpy.array_equal(machine.weights, reloaded.weights)
  True

The shape of a :py:class:`bob.learn.linear.Machine` (see
:py:attr:`bob.learn.linear.Machine.shape`) indicates the size of the input
vector that is expected by this machine and the size of the output vector it
produces, in a tuple format like ``(input_size, output_size)``:

.. doctest::

  >>> machine.shape
  (2, 2)

A :py:class:`bob.learn.linear.Machine` also supports pre-setting
normalization vectors that are applied to every input :math:`x`. You can set a
subtraction factor and a division factor, so that the actual input :math:`x'`
that is fed to the matrix :math:`W` is :math:`x' = (x - s) ./ d`. The variables
:math:`s` and :math:`d` are vectors that have to have the same size as the
input vector :math:`x`. The operator :math:`./` indicates an element-wise
division. By default, :math:`s := 0.0` and :math:`d := 1.0`.

.. doctest::

  >>> machine.input_subtract
  array([ 0.,  0.])
  >>> machine.input_divide
  array([ 1.,  1.])

To set a new value for :math:`s` or :math:`d` just assign the desired machine
property:

.. doctest::

  >>> machine.input_subtract = numpy.array([0.5, 0.8])
  >>> machine.input_divide = numpy.array([2.0, 4.0])
  >>> y = machine(x)
  >>> y
  array([-0.15, -0.15])

.. note::

  In the event you save a machine that has the subtraction and/or a division
  factor set, the vectors are saved and restored automatically w/o user
  intervention.

Linear machine trainers
-----------------------

Next, we examine available ways to train a :py:class:`bob.learn.linear.Machine`
so they can do something useful for you.

Principal component analysis
============================

**PCA** [1]_ is one way to train a :py:class:`bob.learn.linear.Machine`. The
associated |project| class is :py:class:`bob.learn.linear.PCATrainer` as the
training procedure mainly relies on a singular value decomposition.

**PCA** belongs to the category of `unsupervised` learning algorithms, which
means that the training data is not labelled. Therefore, the training set can
be represented by a set of features stored in a container. Using |project|,
this container is a 2D :py:class:`numpy.ndarray`.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> data = numpy.array([[3,-3,100], [4,-4,50], [3.5,-3.5,-50], [3.8,-3.7,-100]], dtype='float64')
   >>> print(data)
   [[   3.    -3.   100. ]
    [   4.    -4.    50. ]
    [   3.5   -3.5  -50. ]
    [   3.8   -3.7 -100. ]]

Once the training set has been defined, the overall procedure to train a
:py:class:`bob.learn.linear.Machine` with a
:py:class:`bob.learn.linear.PCATrainer` is simple and shown below. Please note
that the concepts remains very similar for most of the other `trainers` and
`machines`.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> trainer = bob.learn.linear.PCATrainer() # Creates a PCA trainer
   >>> [machine, eig_vals] = trainer.train(data)  # Trains the machine with the given data
   >>> print(machine.weights)  # The weights of the returned (linear) Machine after the training procedure
   [[ 0.002 -0.706 -0.708]
    [-0.002  0.708 -0.706]
    [-1.    -0.003 -0.   ]]

Next, input data can be projected using this learned projection matrix
:math:`W`.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> e = numpy.array([3.2,-3.3,-10], 'float64')
   >>> print(machine(e))
   [ 9.999 0.47 0.092]


Linear discriminant analysis
============================

**LDA** [2]_ is another way to train a :py:class:`bob.learn.linear.Machine`.
The associated |project| class is
:py:class:`bob.learn.linear.FisherLDATrainer`.

In contrast to **PCA** [1]_, **LDA** [2]_ is a `supervised` technique.
Furthermore, the training data should be organized differently. It is indeed
required to be a list of 2D :py:class:`numpy.ndarray`\'s, one for each class.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> data1 = numpy.array([[3,-3,100], [4,-4,50], [40,-40,150]], dtype='float64')
   >>> data2 = numpy.array([[3,6,-50], [4,8,-100], [40,79,-800]], dtype='float64')
   >>> data = [data1,data2]

Once the training set has been defined, the procedure to train the
:py:class:`bob.learn.linear.Machine` with **LDA** is very similar to the one
for **PCA**. This is shown below.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> trainer = bob.learn.linear.FisherLDATrainer()
   >>> [machine,eig_vals] = trainer.train(data)  # Trains the machine with the given data
   >>> print(eig_vals)  # doctest: +SKIP
   [ 13.10097786 0. ]
   >>> machine.resize(3,1)  # Make the output space of dimension 1
   >>> print(machine.weights)  # The new weights after the training procedure
   [[ 0.609]
    [ 0.785]
    [ 0.111]]

Whitening
==========

This is generally used for i-vector preprocessing.

Let's consider a 2D array of data used to train the withening, and a sample to be whitened:

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> data = numpy.array([[ 1.2622, -1.6443, 0.1889], [ 0.4286, -0.8922, 1.3020], [-0.6613,  0.0430, 0.6377], [-0.8718, -0.4788, 0.3988], [-0.0098, -0.3121,-0.1807],  [ 0.4301,  0.4886, -0.1456]])
   >>> sample = numpy.array([1, 2, 3.])

The initialisation of the trainer and the machine:

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> t = bob.learn.linear.WhiteningTrainer()

Then, the training and projection are done as follows:

.. doctest::
   :options: +NORMALIZE_WHITESPACE, +ELLIPSIS

   >>> m = t.train(data)
   >>> withened_sample = m.forward(sample)


Within-Class Covariance Normalisation
=====================================

This can also be used for i-vector preprocessing. Let's first put the training data into list of numpy arrays.

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> data = [numpy.array([[ 1.2622, -1.6443, 0.1889], [ 0.4286, -0.8922, 1.3020]]), numpy.array([[-0.6613,  0.0430, 0.6377], [-0.8718, -0.4788, 0.3988]]), numpy.array([[-0.0098, -0.3121,-0.1807],  [ 0.4301,  0.4886, -0.1456]])]


The initialisation of the trainer is done as follows:

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> t = bob.learn.linear.WCCNTrainer()

Then, the training and projection are done as follows:

.. doctest::
   :options: +NORMALIZE_WHITESPACE

   >>> m = t.train(data)
   >>> wccn_sample = m.forward(sample)


.. Place here your external references
.. [1] http://en.wikipedia.org/wiki/Principal_component_analysis
.. [2] http://en.wikipedia.org/wiki/Linear_discriminant_analysis