..
.. Copyright John Reid 2012
..
.. This is a reStructuredText document. If you are reading this in text format, it can be 
.. converted into a more readable format by using Docutils_ tools such as rst2html.
..

.. _Docutils: http://docutils.sourceforge.net/docs/user/tools.html


What are Gaussian processes?
============================

Often we have an inference problem involving :math:`n` data,

.. math:: \mathcal{D} = \{(\boldsymbol{x_i},y_i)|i=1,\ldots,n, \boldsymbol{x_i} \in \mathcal{X}, y_i \in \mathbb{R}\}

where the :math:`\boldsymbol{x_i}` are the inputs and the :math:`y_i`
are the targets. We wish to make predictions, :math:`y_*`\ , for new
inputs :math:`\boldsymbol{x_*}`\ . Taking a Bayesian perspective we
might build a model that defines a distribution over all possible
functions, :math:`f: \mathcal{X} \rightarrow \mathbb{R}`\ . We can
encode our initial beliefs about our particular problem as a prior over
these functions. Given the data, :math:`\mathcal{D}`\ , and applying
Bayes’ rule we can infer a posterior distribution. In particular, for
any given :math:`\boldsymbol{x_*}` we can calculate or approximate a
predictive distribution over :math:`y_*` under this posterior.

*Gaussian processes* (GPs) are probability distributions over functions
for which this inference task is tractable. They can be seen as a
generalisation of the Gaussian probability distribution to the space of
functions. That is a multivariate Gaussian *distribution* defines a
distribution over a finite set of random variables, a Gaussian *process*
defines a distribution over an infinite set of random variables (for example
the real numbers). GP domains are not restricted to the real numbers, any
space with a dot product is suitable. Analagously to a multivariate
Gaussian distribution, a GP is defined by its mean, :math:`\mu`\ , and
covariance, :math:`k`\ . However for a GP these are themselves
functions, :math:`\mu: \mathcal{X} \rightarrow \mathbb{R}` and
:math:`k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}`\ . In
all that follows we assume :math:`\mu(\boldsymbol{x}) = 0` as without loss
of generality as we can always shift the data to accommodate any given
mean.


.. samples

.. image:: _static/Images/samples_from_prior.*
.. image:: _static/Images/samples_from_posterior.*

Samples from two Gaussian processes with the same
mean and covariance functions. Here :math:`\mathcal{X} = \mathbb{R}`\ .
The prior samples are taken from a
Gaussian process without any data and the posterior samples are taken
from a Gaussian process where the data are shown as black squares. The
black dotted line represents the mean of the process and the gray shaded
area covers twice the standard deviation at each input, :math:`x`\ . The
coloured lines are samples from the process, or more accurately, samples
at a finite number of inputs, :math:`x`\ , joined by lines.

The code to generate the above figures:

.. literalinclude:: _static/Code/figure_samples.py
    :language: python
    :lines: 5-


 





The covariance function, :math:`k`
----------------------------------

Assuming the mean function, :math:`\mu`\ , is 0 everywhere then our GP
is defined by two quantities: the data, :math:`\mathcal{D}`\ ; and its
covariance function (sometimes referred to as its *kernel*), :math:`k`\ .
The data is fixed so our modelling problem is exactly that of choosing a
suitable covariance function. Given different problems we certainly wish
to specify different priors over possible functions. Fortunately we have
available a large library of possible covariance functions each of which
represents a different prior on the space of functions.

.. covariance-examples

.. image:: _static/Images/covariance_function_se.*
.. image:: _static/Images/covariance_function_matern_52.*
.. image:: _static/Images/covariance_function_se_long_length.*
.. image:: _static/Images/covariance_function_matern_32.*
.. image:: _static/Images/covariance_function_rq.*
.. image:: _static/Images/covariance_function_periodic.*

Samples drawn
from GPs with the same data and different covariance functions. Typical
samples from the posterior of GPs with different covariance functions
have different characteristics. The periodic covariance function’s
primary characteristic is self explanatory. The other covariance
functions affect the smoothness of the samples in different ways.

The code to generate the above figures:

.. literalinclude:: _static/Code/figure_covariance_examples.py
    :language: python
    :lines: 5-



Combining covariance functions and noisy data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Furthermore the point-wise product and sum of covariance functions are
themselves covariance functions. In this way we can combine simple
covariance functions to represent more complicated beliefs we have about
our functions.

Normally we are modelling a system where we do not actually have access
to the target values, :math:`y`\ , but only noisy versions of them,
:math:`y+\epsilon`\ . If we assume :math:`\epsilon` has a Gaussian
distribution with variance :math:`\sigma_n^2` we can incorporate this
noise term into our covariance function. This requires that our noisy
GP’s covariance function, :math:`k_{\textrm{noise}}(x_1,x_2)` is aware
of whether :math:`x_1` and :math:`x_2` are the same input as we may have
two noisy measurements at the same point in :math:`\mathcal{X}`\ .

.. math:: k_{\textrm{noise}}(x_1,x_2) = k(x_1,x_2) + \delta(x_1=x_2) \sigma_n^2

.. image:: _static/Images/noise_low.*
.. image:: _static/Images/noise_mid.*
.. image:: _static/Images/noise_high.*

GP predictions with varying levels of
noise. The covariance function is a squared exponential with additive
noise of levels 0.0001, 0.1 and 1.

The code to generate the above figures:

.. literalinclude:: _static/Code/figure_noise.py
    :language: python
    :lines: 5-





Learning covariance function parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Most of the commonly used covariance functions are parameterised. The
parameters
can be fixed if we are confident in our understanding of the problem.
Alternatively we can treat them as hyper-parameters in our Bayesian
inference task and optimise them through some technique such as maximum
likelihood estimation or conjugate gradient descent.

.. image:: _static/Images/learning_first_guess.*
.. image:: _static/Images/learning_learnt.*

The effects of learning covariance function
hyper-parameters. We see the predictions in the second figure seem
to fit the data more accurately.

The code to generate the above figures:

.. literalinclude:: _static/Code/figure_learning.py
    :language: python
    :lines: 5-