expr Package¶

`expr` Package¶

Lazy expression DAGs.

All expression operations are subclasses of the Expr class, which by default performs all operations lazily.

Operations are built up using a few high-level operations – these all live in their own modules:

Create a new distributed array spartan.expr.ndarray
Map over an array spartan.expr.map and spartan.expr.shuffle
Reduce over an array spartan.expr.reduce
Apply a stencil/convolution to an array spartan.expr.stencil
Slicing/indexing spartan.expr.index.

Optimizations on DAGs live in spartan.expr.optimize.

`backend` Module¶

Evalution for Expr nodes.

evaluate evaluates a nodes dependencies, caching results, then evaluates nodes themselves.

spartan.expr.backend.evaluate(ctx, prim)[source]¶

Evaluate an Expr.

Dependencies are evaluated prior to evaluating prim.

Dependencies may be either a list, a dictionary or a single value of type Expr. For convenience, we allow specifying non-Expr dependencies; these are left unaltered.

Parameters:	ctx – prim – `Expr`

`base` Module¶

Lazy arrays.

Expr operations are not performed immediately, but are set aside and built into a control flow graph, which is then compiled into a series of primitive array operations.

class spartan.expr.base.Expr(*args, **kw)[source]¶

Bases: spartan.node.Node

argmax(x, axis=None)¶

Compute argmax over axis.

See numpy.ndarray.argmax.

Parameters:	x – `Expr` to compute a maximum over. axis – Axis (integer or None).

argmin(x, axis=None)¶

Compute argmin over axis.

See numpy.ndarray.argmin.

Parameters:	x – `Expr` to compute a minimum over. axis – Axis (integer or None).

astype(x, dtype)¶

Convert x to a new dtype.

See numpy.ndarray.astype.

Parameters:	x – dtype –

compute_shape()[source]¶

Compute the shape of this expression.

If the shape is not available (data dependent), raises NotShapeable.

dag()[source]¶

dependencies()[source]¶

Return a dictionary mapping from name -> dependency.

Dependencies may either be a list or single value. Dependencies of type Expr are recursively evaluated.

evaluate(ctx, deps)[source]¶

force()[source]¶

glom()[source]¶

mean(x, axis=None)¶

Compute the mean of x over axis.

See numpy.ndarray.mean.

Parameters:	x – `Expr` axis – integer or `None`

node_init()[source]¶

outer(a, b)¶

ravel(v)¶

“Ravel” v to a one-dimensional array of shape (size(v),).

See numpy.ndarray.ravel. :param v:

shape[source]¶

Try to compute the shape of this DAG.

If the value has been computed already this always succeeds.

sum(x, axis=None)¶

Sum x over axis.

Parameters:	x – The array to sum. axis – Either an integer or `None`.

typename()[source]¶

visit(visitor)[source]¶: Apply visitor to all children of this node, returning a new Expr of the same type. :param visitor: OptimizePass

class spartan.expr.base.LazyCollection(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

LazyCollections wrap normal tuples, lists and dicts with Expr semantics.

visit() and evaluate() are supported; these thread the visitor through child elements as expected.

evaluate(ctx, deps)[source]¶

class spartan.expr.base.LazyDict(*args, **kw)[source]¶

Bases: spartan.expr.base.LazyCollection

visit(visitor)[source]¶

class spartan.expr.base.LazyList(*args, **kw)[source]¶

Bases: spartan.expr.base.LazyCollection

visit(visitor)[source]¶

class spartan.expr.base.LazyTuple(*args, **kw)[source]¶

Bases: spartan.expr.base.LazyCollection

visit(visitor)[source]¶

class spartan.expr.base.LazyVal(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

compute_shape()[source]¶

dependencies()[source]¶

evaluate(ctx, deps)[source]¶

visit(visitor)[source]¶

exception spartan.expr.base.NotShapeable[source]¶: Bases: exceptions.Exception

spartan.expr.base.dag(node)[source]¶

Compile and return the DAG representing this expression.

Parameters:	node – The node to compute a DAG for.

spartan.expr.base.eager(node)[source]¶

Eagerly evaluate node and convert the result back into an Expr.

Parameters:	node – `Expr` to evaluate.

spartan.expr.base.evaluate(node)[source]¶

Evaluate this expression.

Parameters:	node – `Expr` to evaluate.

spartan.expr.base.force(node)¶

Evaluate this expression.

Parameters:	node – `Expr` to evaluate.

spartan.expr.base.glom(node)[source]¶: Evaluate this expression and return the result as a numpy.ndarray.

spartan.expr.base.lazify(val)[source]¶

Lift val into an Expr node.

If val is already an expression, it is returned unmodified.

Parameters:	val –

spartan.expr.base.make_primitive(name, arg_names, evaluate_fn)[source]¶

`builtins` Module¶

Basic numpy style operations on arrays.

These include –

Array creation routines: (rand, randn, zeros, ones, arange)
Reductions: (sum, argmin, argmax, mean)
Shape/type casting: (reshape, ravel, astype, shape, size)
Other: (dot).

spartan.expr.builtins.arange(shape, dtype=<type 'float'>)[source]¶

spartan.expr.builtins.argmax(x, axis=None)¶

Compute argmax over axis.

See numpy.ndarray.argmax.

Parameters:	x – `Expr` to compute a maximum over. axis – Axis (integer or None).

spartan.expr.builtins.argmin(x, axis=None)[source]¶

Compute argmin over axis.

See numpy.ndarray.argmin.

Parameters:	x – `Expr` to compute a minimum over. axis – Axis (integer or None).

spartan.expr.builtins.astype(x, dtype)[source]¶

Convert x to a new dtype.

See numpy.ndarray.astype.

Parameters:	x – dtype –

spartan.expr.builtins.dot(a, b)[source]¶

Compute the dot product (matrix multiplication) of 2 arrays.

Parameters:	a – `Expr` or `numpy.ndarray` b – `Expr` or `numpy.ndarray`
Return type:	`Expr`

spartan.expr.builtins.mean(x, axis=None)[source]¶

Compute the mean of x over axis.

See numpy.ndarray.mean.

Parameters:	x – `Expr` axis – integer or `None`

spartan.expr.builtins.ones(shape, dtype=<type 'float'>, tile_hint=None)[source]¶

spartan.expr.builtins.rand(*shape, **kw)[source]¶

Parameters:	tile_hint – A tuple indicating the desired tile shape for this array.

spartan.expr.builtins.randn(*shape, **kw)[source]¶

spartan.expr.builtins.ravel(v)[source]¶

“Ravel” v to a one-dimensional array of shape (size(v),).

See numpy.ndarray.ravel. :param v:

spartan.expr.builtins.reshape(array, new_shape, tile_hint=None)[source]¶

Reshape/retile array.

Returns a new array with the given shape and tile size.

Parameters:	array – `Expr` new_shape – `tuple` tile_hint – `tuple` or None

spartan.expr.builtins.size(x, axis=None)[source]¶

Return the size (product of the size of all axes) of x.

See numpy.ndarray.size.

Parameters:	x – `Expr` to compute the size of.

spartan.expr.builtins.sum(x, axis=None)[source]¶

Sum x over axis.

Parameters:	x – The array to sum. axis – Either an integer or `None`.

spartan.expr.builtins.zeros(shape, dtype=<type 'float'>, tile_hint=None)[source]¶

`index` Module¶

Indexing operations (slicing and filtering).

class spartan.expr.index.IndexExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

evaluate(ctx, deps)[source]¶

node_init()[source]¶

spartan.expr.index.bool_index_mapper(ex, tile, src, idx)[source]¶

spartan.expr.index.eval_Index(ctx, prim, deps)[source]¶

spartan.expr.index.eval_Slice(ctx, prim, deps)[source]¶

spartan.expr.index.int_index_mapper(ex, tile, src, idx, dst)[source]¶: Map over the index array, fetching rows from the data array.

spartan.expr.index.slice_mapper(ex, val, region, matching_extents)[source]¶

`loop` Module¶

Implementation of the generic loop() expression.

loop([xrange(start1, stop2),
      xrange(start2, stop2),
       ... ], fn)

The loop function should expect as input the current indices being operated on: e.g. fn(i, j).

class spartan.expr.loop.LoopExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

evaluate(ctx, deps)[source]¶

spartan.expr.loop.loop(ranges, sources, arg_fn, mapper_fn, target)¶

`map` Module¶

class spartan.expr.map.MapExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

compute_shape()[source]¶

MapTiles retains the shape of inputs.

Broadcasting results in a map taking the shape of the largest input.

evaluate(ctx, deps)[source]¶

spartan.expr.map.map(inputs, fn, **kw)[source]¶

Evaluate fn over each tile of the input.

fn should be of the form ([inputs], **kw). :param v: Expr :param fn: callable taking arguments (inputs, **kw)

spartan.expr.map.tile_mapper(ex, data, children, map_fn, fn_kw)[source]¶

`ndarray` Module¶

class spartan.expr.ndarray.NdArrayExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

compute_shape()[source]¶

dependencies()[source]¶

evaluate(ctx, deps)[source]¶

visit(visitor)[source]¶

spartan.expr.ndarray.ndarray(shape, dtype=<type 'float'>, tile_hint=None, combine_fn=None, reduce_fn=None)[source]¶: Lazily create a new distributed array. :param shape: :param dtype: :param tile_hint:

`node` Module¶

Helper for constructing trees of objects.

Provides pretty printing, equality testing, hashing and keyword initialization.

class spartan.expr.node.Node(*args, **kw)[source]¶

Bases: object

clone(**kwds)[source]¶

eq_members(other)[source]¶

items()[source]¶

iteritems()[source]¶

itervalues()[source]¶

classmethod members()[source]¶

node_type()[source]¶

spartan.expr.node.get_members(klass)[source]¶: Walk through classes in mro order, accumulating member names.

spartan.expr.node.get_mro(klass)[source]¶

spartan.expr.node.get_reverse_mro(klass)[source]¶

spartan.expr.node.node_initializer(self, *args, **kw)[source]¶

spartan.expr.node.node_type(klass)[source]¶: Decorator to add node behavior to a class.

`optimize` Module¶

Optimizations over an expression graph.

class spartan.expr.optimize.LMap(*args, **kw)¶: Bases: spartan.expr.optimize.LocalOp

class spartan.expr.optimize.LReduce(*args, **kw)¶: Bases: spartan.expr.optimize.LocalOp

class spartan.expr.optimize.LVal(*args, **kw)¶

Bases: spartan.expr.optimize.LocalOp

A local value (a constant, or an input argument).

dependencies()¶

class spartan.expr.optimize.LocalOp(*args, **kw)¶

Bases: spartan.expr.base.Expr

evaluate(ctx)¶

class spartan.expr.optimize.MapMapFusion[source]¶

Bases: spartan.expr.optimize.OptimizePass

Fold sequences of Map operations together.

map(f, map(g, map(h, x))) -> map(f . g . h, x)

name = 'map_fusion'¶

visit_MapExpr(op)¶

class spartan.expr.optimize.NumexprFusionPass[source]¶

Bases: spartan.expr.optimize.OptimizePass

Fold binary operations compatible with numexpr into a single numexpr operator.

name = 'numexpr_fusion'¶

visit_MapExpr(op)¶

class spartan.expr.optimize.OptimizePass[source]¶

Bases: object

visit(op)[source]¶

class spartan.expr.optimize.ReduceMapFusion[source]¶

Bases: spartan.expr.optimize.OptimizePass

name = 'reduce_fusion'¶

visit_ReduceExpr(op)¶

spartan.expr.optimize.add_optimization(klass, default)[source]¶

spartan.expr.optimize.apply_pass(klass, dag)[source]¶

spartan.expr.optimize.map_like(v)[source]¶

spartan.expr.optimize.new_var()[source]¶

spartan.expr.optimize.optimize(dag)[source]¶

`outer` Module¶

class spartan.expr.outer.OuterProductExpr(*args, **kw)[source]¶: Bases: spartan.expr.base.Expr

spartan.expr.outer.outer(a, b)[source]¶

spartan.expr.outer.outer_product(a, b, map_fn, reduce_fn)[source]¶

Outer (cartesian) product over the tiles of a and b.

map_fn is applied to each pair; reduce_fn is used to combine overlapping outputs.

Parameters:	a – b –

`reduce` Module¶

class spartan.expr.reduce.ReduceExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

evaluate(ctx, deps)[source]¶

spartan.expr.reduce.reduce(v, axis, dtype_fn, local_reduce_fn, combine_fn, fn_kw=None)[source]¶

Reduce v over axis axis.

The resulting array should have a datatype given by dtype_fn(input).

For each tile of the input local_reduce_fn is called. The output is combined using combine_fn.

Parameters:	v – `Expr` axis – int or None dtype_fn – Callable: fn(array) -> `numpy.dtype` local_reduce_fn – Callable: fn(extent, data, axis) combine_fn – Callable: fn(old_v, update_v) -> new_v

`shuffle` Module¶

class spartan.expr.shuffle.ShuffleExpr(*args, **kw)[source]¶

Bases: spartan.expr.base.Expr

evaluate(ctx, deps)[source]¶

spartan.expr.shuffle.shuffle(v, fn, tile_hint=None, target=None, kw=None)[source]¶

Evaluate fn over each extent of the input.

fn should take arguments: (v, extent, **kw)

Parameters:	v – `Expr` to map over. fn – Callable with form `(v, extent, **kw)`

`stencil` Module¶

spartan.expr.stencil.maxpool(images, pool_size=2, stride=2)[source]¶

spartan.expr.stencil.stencil(images, filters, stride=1)[source]¶

spartan.expr.stencil.stencil_mapper(array, ex, filters=None, images=None, target_shape=None)[source]¶

spartan.expr.stencil.tiles_like(array, target_shape)[source]¶

expr Package¶

expr Package¶

backend Module¶

base Module¶

builtins Module¶

index Module¶

loop Module¶

map Module¶

ndarray Module¶

node Module¶

optimize Module¶

outer Module¶

reduce Module¶

shuffle Module¶

stencil Module¶

`expr` Package¶

`backend` Module¶

`base` Module¶

`builtins` Module¶

`index` Module¶

`loop` Module¶

`map` Module¶

`ndarray` Module¶

`node` Module¶

`optimize` Module¶

`outer` Module¶

`reduce` Module¶

`shuffle` Module¶

`stencil` Module¶