dense Package

distarray Module

class spartan.dense.distarray.Broadcast(base, shape)[source]

Bases: object

A broadcast object mimics the behavior of Numpy broadcasting.

Takes an input of shape (x, y) and a desired output shape (x, y, z), the broadcast object reports shape=(x,y,z) and overrides __getitem__ to return the appropriate values.

fetch(ex)[source]
foreach(fn)[source]
map_to_table(mapper_fn, combine_fn=None, reduce_fn=None, kw=None)[source]
class spartan.dense.distarray.DistArray(shape, dtype, tiles, reducer_fn)[source]

Bases: object

extent_for_blob(id)
fetch(region)[source]

Return a local numpy array for the given region.

If necessary, data will be copied from remote hosts to fill the region. :param region: Extent indicating the region to fetch.

foreach(mapper_fn, kw)[source]
glom()[source]
id()[source]
map_to_table(mapper_fn, kw=None)[source]
select(idx)[source]

Effectively __getitem__.

Renamed to avoid the chance of accidentally using a slow, local operation on a distributed array.

tile_shape()[source]
update(region, data)[source]
update_slice(slc, data)[source]
class spartan.dense.distarray.Slice(darray, idx)[source]

Bases: object

fetch(idx)[source]
foreach(mapper_fn, kw)[source]
glom()[source]
map_to_table(mapper_fn, kw)[source]
spartan.dense.distarray.best_locality(array, ex)[source]

Return the table shard with the best locality for extent ex. :param table: :param ex:

spartan.dense.distarray.broadcast(args)[source]
spartan.dense.distarray.broadcast_mapper(ex, tile, mapper_fn=None, bcast_obj=None)[source]
spartan.dense.distarray.compute_splits(shape, tile_hint=None, num_shards=-1)[source]

Split an array of shape shape into Extent`s containing roughly `TILE_SIZE elements.

Parameters:
  • shape – tuple
  • tile_hint – tuple indicating the desired tile shape
Return type:

list of Extent
spartan.dense.distarray.create(shape, dtype=<type 'float'>, sharder=None, combiner=None, reducer=None, tile_hint=None)[source]
spartan.dense.distarray.empty(shape, dtype=<type 'float'>, sharder=None, combiner=None, reducer=None, tile_hint=None)
spartan.dense.distarray.from_table(extents)[source]

Construct a distarray from an existing table. Keys must be of type Extent, values of type Tile.

Shape is computed as the maximum range of all extents.

Dtype is taken from the dtype of the tiles.

Parameters:table –
spartan.dense.distarray.largest_value(vals)[source]
spartan.dense.distarray.map_to_array(array, mapper_fn, kw=None)[source]
spartan.dense.distarray.take_first(a, b)[source]

extent Module

class spartan.dense.extent.TileExtent[source]

Bases: object

A rectangular tile of a distributed array.

These correspond (roughly) to a slice taken from an array (without any step component).

Arrays are indexed from the upper-left; for an array of shape (sx, sy, sz): (0,0...) is the upper-left corner of an array, and (sx,sy,sz...) the lower-right.

Extents are represented by an upper-left corner (inclusive) and a lower right corner (exclusive): [ul, lr). In addition, they carry the shape of the array they are a part of; this is used to compute global position information.

add_dim()[source]
clone()[source]
intersection(a, b)
Return type:The intersection of the 2 extents as a TileExtent, or None if the intersection is empty.
ndim[source]
ravelled_pos()[source]
shape[source]
size[source]
to_global(idx, axis)[source]

Convert idx from a local offset in this tile to a global offset.

to_slice()[source]
spartan.dense.extent.compute_slice(base, idx)[source]

Return a new TileExtent representing base[idx]

Parameters:
  • base – TileExtent
  • idx – int, slice, or tuple(slice,...)
spartan.dense.extent.create(ul, lr, array_shape)[source]

Create a new extent with the given coordinates and array shape.

Parameters:
  • ul – tuple:
  • lr –
  • array_shape –
spartan.dense.extent.drop_axis(ex, axis)[source]
spartan.dense.extent.find_overlapping(extents, region)[source]

Return the extents that overlap with region.

Parameters:
  • extents – List of extents to search over.
  • region – Extent to match.
spartan.dense.extent.find_shape(extents)[source]

Given a list of extents, return the shape of the array necessary to fit all of them. :param extents:

spartan.dense.extent.from_shape(shp)[source]
spartan.dense.extent.from_slice(idx, shape)[source]

Construct a TileExtent from a slice or tuple of slices.

Parameters:
  • idx – int, slice, or tuple(slice...)
  • shape – shape of the input array
Return type:

TileExtent corresponding to idx.
spartan.dense.extent.index_for_reduction(index, axis)[source]
spartan.dense.extent.intersection(a, b)[source]
Return type:The intersection of the 2 extents as a TileExtent, or None if the intersection is empty.
spartan.dense.extent.offset_from(base, other)[source]
Parameters:
Return type:

A new extent using this extent as a basis, instead of (0,0,0...)
spartan.dense.extent.offset_slice(base, other)[source]
Parameters:
Return type:

A slice representing the local offsets of other into this tile.
spartan.dense.extent.ravelled_pos(idx, array_shape)[source]
spartan.dense.extent.shape_for_reduction(input_shape, axis)[source]

Return the shape for the result of applying a reduction along axis to an input of shape input_shape. :param input_shape: :param axis:

spartan.dense.extent.shapes_match(offset, data)[source]

Return true if the shape of data matches the extent offset. :param offset: :param data:

spartan.dense.extent.unravelled_pos(idx, array_shape)[source]

Unravel idx into an index into an array of shape array_shape. :param idx: int :param array_shape: tuple :rtype: tuple indexing into array_shape

tile Module

class spartan.dense.tile.Tile[source]

Bases: object

A tile of an array: an extent (offset + size) and data for that extent.

data[source]
get(selector)[source]
update(update, reducer)
spartan.dense.tile.from_data(data)[source]
spartan.dense.tile.from_intersection(src, overlap, data)[source]

Return a tile for src, masked to update the area specifed by overlap.

Parameters:
  • src – TileExtent
  • overlap – TileExtent
  • data –
spartan.dense.tile.from_shape(shape, dtype)[source]
spartan.dense.tile.merge_tiles(old_tile, new_tile, reducer)