exp Package

exp Package

class infpy.exp.ConjugatePrior(likelihood, prior)

Bases: object

Represents a conjugate prior relationship between 2 exponential families, one for the likelihood and one for the prior.

likelihood = None

The exponential family whose parameters we have a conjugate prior for

likelihood_dimension = None

The dimension of the natural parameters and sufficient statistics of the likelihood.

log_conjugate_predictive(T, tau, n=1)

@arg T: the data drawn from the likelihood. @arg tau: the natural parameter for this conjugate prior distribution @arg n: the number of data drawn from the likelihood. @return: log of the predictive pdf for data, T, from the likelihood, given the prior’s parameters, tau.

prior = None

The exponential family that is a conjugate prior over the likelihood’s parameters

strength_dimension = None

The dimension of the strength part of the prior’s natural parameters.

class infpy.exp.DirichletExpFamily(k=2)

Bases: infpy.exp.ExponentialFamily

The dirichlet distribution in exponential family form.

• x = p1, ..., pk
• M{T(x) = log(x)}
• M{eta(theta) = theta-1}

Does not handle the case where x = 0.

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

cov_T(eta)

@arg eta: The natural parameters. @return: The covariance of T_i, T_j, the sufficient statistics, given eta.

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

plot(eta, scale=1.0, *plot_args, **plot_kwds)

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.DiscreteExpFamily(k=2)

Bases: infpy.exp.ExponentialFamily

The discrete distribution in exponential family form. Also known as the multinomial distribution.

• T(x) = delta(x=i)
• theta = (p1, ..., pk)
• eta = log(theta)
• A(theta) = 0

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.ExponentialFamily(dimension, normalisation_dimension=1, vectorisable=False)

Bases: object

An U{exponential family<http://en.wikipedia.org/wiki/Exponential_family>} of distributions over x.

M{log p(x|theta) = eta(theta).T(x) - A(theta) + h(T(x))}

Where

• x : the random variable
• T(x) : the random variable’s sufficient statistics
• theta : the canonical parameter of the distribution, e.g. (mean, covariance) for a gaussian
• eta(theta) : the natural parameter of the distribution
• A(theta) : the normalization factor (log partition function) (can be a vector)
• h(T) : commonly 0.0 and provides a measure for x

T and eta should be numpy arrays of shape (self.dimension,). Base classes that implement a specific exponential family must define T, eta, A and h such that the above equation holds. They should also define some of the methods/attributes in the following sections.

Conversion methods include:

• theta(eta) : converts from natural parameters to canonical parameters
• x(T) : converts from canonical parameters to natural parameters if it differs from 0.0
• dimension : returns the length of eta and T
• normalisation_dimension : returns the length of A

The following sampling methods are optional:

• sample_theta(theta, size=1) : returns a sample from the distribution parameterised by theta
• sample_eta(eta, size=1) : returns a sample from the distribution parameterised by eta

The following special methods are optional:

• exp_T(eta) : returns M{d(A)/d(eta)}, the derivative of the normalization factor with respect to

  the natural parameter, evaluated at eta. This is equivalent to the expectation of T.

Testing methods include:

• _p_truth(x, theta) : returns M{p(x|theta)} calculated by a different method

  (e.g. scipy.stats) for testing

• _entropy_truth(theta) I{optional} : returns entropy of M{p(x|theta)} calculated by a different method

  (e.g. scipy.stats) for testing

• _typical_xs : a sequence of typical x values for the family

• _typical_thetas : a sequence of typical theta values for the family

A(eta)

@return: The normalisation factor (log partition) for the exponential family (as a scalar).

See A_vec() for the vectorised version.

KL(eta_1, eta_2)

Returns the Kullback-Leibler divergence between the the distributions in this family parameterised by eta_1 and eta_2. @arg eta_1: Natural parameters. @arg eta_2: Natural parameters. @return: KL(eta_1||eta_2) = E[log p(x|eta_1)] - E[log p(x|eta_2)] where expectations are w.r.t. p(x|eta_1)

LL_fns(tau, nu)

@return: ll, ll_prime, ll_hessian : The log likelihood function and its derivative and hessian for the given tau and nu prior.

dimension = None

The dimension of the sufficient statistics, T, and the natural parameter, eta

entropy(eta)

The entropy of the distribution parameterised by eta. The entropy is calculated as the M{<-log p(T>|eta)>}, i.e. the expectation of the negative log probability. This relies on the family defining the exp_T(eta) and exp_h(eta) methods.

@arg eta: the exponential family parameters in natural form @return: The entropy of the family parameterised by eta

exp_h(eta)

The expectation of h(x) given the natural parameters, eta. @param eta: the natural parameters

h(T)

The default measure for x.

@param T: x’s sufficient statistic @return: 0.0

log_p_T(T, eta)

@param T: the random variable’s sufficient statistics @param eta: the family’s parameter in natural form @return: M{log p(T|eta)}

log_p_x(x, theta)

@param x: the random variable in canonical form @param theta: the parameter in canonical form @return: M{log p(x|theta)}

normalisation_dimension = None

The length of A(theta). Normally one when a scalar is returned, can be returned as a vector of the given dimension. This can be useful in conjugate analysis.

p_T(T, eta)

@return: M{p(T|eta)}.

p_x(x, theta)

@return: M{p(x|theta)}.

vectorisable = None

Can we pass more than one T or eta to the methods of this family?

class infpy.exp.GammaExpFamily

Bases: infpy.exp.ExponentialFamily

The U{gamma distribution<http://en.wikipedia.org/wiki/Gamma_distribution>} in exponential family form.

• theta = (a,b) where a is the shape and b is the rate (inverse scale)
• eta = (-b, a-1)
• T(x) = (x, log(x))
• A(theta) = log Gamma(a) - a log(b)

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.GaussianConjugatePrior(p=2)

Bases: infpy.exp.ConjugatePrior

Univariate Normal distribution with a Normal-Gamma conjugate prior.

exp_likelihood_log_normalisation_factor(tau)

@arg tau: the parameters of the prior in standard form @return: the expectation of the log normalisation factor of the likelihood given the prior’s parameters

class infpy.exp.GaussianExpFamily

Bases: infpy.exp.ExponentialFamily

The univariate gaussian distribution in exponential family form.

• theta = (mu,gamma) where mu is the mean and gamma is the precision
• T = (x, x*x)
• eta = (mu*gamma, -gamma/2)
• A = -.5 * (log(gamma) - gamma * mu * mu - log(2 * pi))

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

exp_h(eta)

The expectation of h(x) given the natural parameters, eta. @param eta: the natural parameters

h(T)

The default measure for x.

@param T: x’s sufficient statistic @return: 0.0

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.MultinomialConjugatePrior(k=2)

Bases: infpy.exp.ConjugatePrior

Multinomial distribution with a Dirichlet conjugate prior.

exp_likelihood_log_normalisation_factor(tau)

@arg tau: the parameters of the prior in standard form @return: the expectation of the log normalisation factor of the likelihood given the prior’s parameters

class infpy.exp.MvnConjugatePrior(p=2)

Bases: infpy.exp.ConjugatePrior

Multivariate Normal distribution with a Normal-Wishart conjugate prior.

exp_likelihood_log_normalisation_factor(tau)

@arg tau: the parameters of the prior in standard form @return: the expectation of the log normalisation factor of the likelihood given the prior’s parameters

class infpy.exp.MvnExpFamily(k=2)

Bases: infpy.exp.ExponentialFamily

The U{multi-variate normal distribution<http://en.wikipedia.org/wiki/Multivariate_normal_distribution>} in k dimensions in exponential family form.

• T(x) = (x, x.x)
• theta = (mu,W) where mu is the mean and W is the precision
• eta(theta) = (W.mu, W/2)
• A(theta) = .5 * (mu.W.mu - log|W| + k.log(2.pi))

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

k = None

The number of dimensions of x.

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.NormalGammaExpFamily(p=2)

Bases: infpy.exp.ExponentialFamily

A Normal-Gamma distribution in 1 dimension. Univariate version of Normal-Wishart multivariate. This is a conjugate prior for the univariate normal (Gaussian) distribution.

• M{gamma|alpha, beta ~ Gamma(alpha, beta)}
• M{mu|tau, mu_0, lambda ~ N(mu,lambda/tau)}

where the parameters are:

• M{mu_0} : mean
• M{lambda} :
• M{alpha} :
• M{beta} :

The exponential family parameterisation:

• x = (mu, gamma)
• T(x) = [log gamma, mu**2 gamma, mu*gamma, -gamma/2]
• theta = (alpha, beta, mu_0, lambda)
• eta(theta) = [alpha-1/2, 1/(2 lambda), mu_0/lambda, 2 beta + mu_0**2/lambda]
• A(theta) = alpha log beta + log Gamma(alpha) - (log 2 pi lambda)/2

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

gamma = None

The Gamma distribution

gaussian = None

The Gaussian distribution

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.NormalWishartExpFamily(p=2)

Bases: infpy.exp.ExponentialFamily

A Normal-Wishart distribution in p dimensions. This is a conjugate prior for the multivariate normal distribution.

• M{W ~ Wishart(nu,S)}
• M{mu|W ~ Normal(mu_0, inv(W)/kappa_0)}

where the parameters are:

• M{nu} : degrees of freedom of precision
• M{S} : precision
• M{mu_0} : mean
• M{kappa_0} : prior strength

The exponential family parameterisation:

• x = (mu, W)
• T(x) = [(log|W|-p.log(2.pi))/2, -mu’.W.mu/2, self.mvn.eta(x)]
• theta = (nu, S, kappa_0, mu_0)
• eta(theta) = [kappa_0, nu-p, -kappa_0.mu_0, kappa_0.mu_0.mu_0’-inv(S)]
• A(theta) = p/2[(p+1)log 2 - (nu-p-1)log(pi) - log kappa_0] + nu/2 log|S| + log Gamma_p(nu/2)

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

mvn = None

The Normal distribution

p = None

The dimension of mu

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

wishart = None

The Wishart distribution

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.PoissonExpFamily

Bases: infpy.exp.ExponentialFamily

The U{Poisson distribution<http://en.wikipedia.org/wiki/Poisson_distribution>} in exponential family form.

• theta = lambda where lambda is the rate parameter
• eta = log lambda
• T(x) = x
• A(theta) = lambda
• h(x) = -log x!

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

exp_h(eta)

The expectation of h(x) given the natural parameters, eta. @param eta: the natural parameters

h(T)

The measure for x in this exponential family.

@param T: x’s sufficient statistic

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

class infpy.exp.Variable(family, eta)

Bases: object

An exponential family variable has a family and natural parameters for that family.

eta = None

The natural parameters of this variable’s distribution.

family = None

The exponential family that this variable belongs to.

class infpy.exp.WishartExpFamily(p=2)

Bases: infpy.exp.ExponentialFamily

The U{Wishart distribution<http://en.wikipedia.org/wiki/Wishart_distribution>} in p dimensions in exponential family form.

• T(W) = [log(det(W)), W.flatten]

• theta = (n, V) where n is the degrees of freedom and V, the scale matrix,

  is typically regarded as the precision

• eta(theta) = [n-p-1, -inv(V).flatten()] / 2

• A(theta) = (n.p.log 2 + n.log det V) / 2 + log gamma_p (n/2)

A_vec(eta)

@return: The normalization factor (log partition)

T(x)

@return: T(x), the sufficient statistics of x

eta(theta)

@return: eta(theta), the natural parameter, eta, that corresponds to the canonical parameter, theta

exp_T(eta)

@arg eta: The natural parameters. @return: The expectation of T, the sufficient statistics, given eta.

exp_log_det_W(eta)

Return the expectation of the log of the determinant of W given eta @arg eta: Natural parameters @return: log|det(W)|

p = None

The number of dimensions.

sample(eta, size=1)

@param eta: the natural parameters @param size: the size of the sample @return: A sample of sufficient statistics

This uses the method detailed by U{Smith & Hocking<http://en.wikipedia.org/wiki/Wishart_distribution#Drawing_values_from_the_distribution>}.

theta(eta)

@return: theta(eta), the canonical parameter, theta, that corresponds to the natural parameter, eta

x(T)

@return: x(T), the x that has the sufficient statistics, T

infpy.exp.factorial(n)

@arg n: An integer >= 0 @return: n!

infpy.exp.log_factorial(n)

@arg n: An integer >= 0 @return: log n!

infpy.exp.log_multivariate_gamma(p, a)

The logarithm of the U{multivariate gamma function<http://en.wikipedia.org/wiki/Multivariate_gamma_function>}

Subpackages

• test Package
  • test Package
  • exp_family_test Module