Package dimer :: Package nnet :: Class SpeedLayer

[frames] | no frames]

Class SpeedLayer

object --+    
         |    
     Layer --+
             |
            SpeedLayer

Known Subclasses:

This layer provides an extra set of weights as a support the momentum algorithm for SGD. At time point t, we need weights at t-1 and the gradient at t to update weights. Namely

w(t+1) - w(t) = - rho * dE(w)/dw + p s(t), for t = 0, 1, ... and s(0) = 0

I call w(t) - w(t-1) = s(t) (speed)

Nested Classes

Inherited from Layer: __metaclass__

Instance Methods

__init__(self, ws, wn, rng, wdtype)
initialize by the given weight properties, all params are lists of the same length

source code

get_params(self)
get weights

source code

get_speeds(self)
get speeds

source code

set_params(self, params)
set weights

source code

set_speeds(self, v)
set speeds

source code

speed_update(self, gradient, mu, rho)
update speeds for the given gradients

source code

weight_update(self)
update speeds for the current speed

source code

Inherited from Layer: activation, get_flat_weights, get_weights, load_flat_weights, set_weights, weight_norm

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Class Methods

_speed_update_f(cls, s, g, mu, rho)
speed update formula

source code

_weight_update_f(cls, cur_weight, speed)
speed update for the given gradient

source code

Class Variables

__abstractmethods__ = frozenset(['activation'])

Inherited from Layer (private): _abc_cache, _abc_negative_cache, _abc_negative_cache_version, _abc_registry

Properties

Inherited from object: __class__

Method Details

init(self, ws, wn, rng, wdtype)
(Constructor)

initialize by the given weight properties, all params are lists of the same length

Parameters:

wshape_lst - shapes of weights
wnames - names of weights
fanin_lst - fan in of weights
rng - a random number generator or init value (see alloc_shared_weights_)
wdtype - data type of weights.

Overrides: object.__init__

(inherited documentation)

_speed_update_f(cls, s, g, mu, rho)
Class Method

speed update formula

new_speed = -rho * g + mu*s

Parameters:

s - cur speed
g - gradient
mu - momentum
rho - learning rate

Returns:

new speed

_weight_update_f(cls, cur_weight, speed)
Class Method

speed update for the given gradient

new_weight = cur_weight + speed

Parameters:

cur_weight - current weight (ndarray)
speed - speed (ndarray)

Returns:

new_weight (ndarray)

get_params(self)

get weights

Returns:: a tuple of (theano.tensor) weights
Overrides: Layer.get_params

get_speeds(self)

get speeds

Returns:: a tuple of (ndarray) speeds

set_params(self, params)

set weights

Parameters:

params - tuple of (ndarray) @return none

Overrides: Layer.set_params

set_speeds(self, v)

set speeds

Parameters:

v - tuple of (ndarray)

Returns:

none

speed_update(self, gradient, mu, rho)

update speeds for the given gradients

new_speed = -rho * gradient + mu*cur_speed

Parameters:

gradient - gradient (list of ndarray)
mu - momentum (float)
rho - learning rate (float)

Returns:

new_speed (ndarray)

weight_update(self)

update speeds for the current speed

new_weight = cur_weight + cur_speed