spartan Package

spartan Package

Spartan: A distributed array language.

cluster Module

spartan.cluster.start_cluster(num_workers, use_cluster_workers)[source]

Start a cluster with num_workers workers.

If use_cluster_workers is True, then use the remote workers defined in spartan.config. Otherwise, workers are all spawned on the localhost.

Parameters:
  • num_workers –
  • use_cluster_workers –

config Module

Configuration options and flags.

Options may be specified on the command line, or via a configuration file. Configuration files should be placed in $HOME/.config/spartanrc.

class spartan.config.AssignMode[source]

Bases: object

BY_CORE = 1
BY_NODE = 2
class spartan.config.Flags[source]

Bases: object

assign_mode = 2
cluster = False
config_file = ''
log_level = 'INFO'
num_workers = 3
oprofile = False
port_base = 10000
profile_kernels = False
profile_master = False
use_threads = False
spartan.config.add_bool_flag(name, default, **kw)[source]
spartan.config.add_flag(name, *args, **kw)[source]
spartan.config.parse_args(argv)[source]

util Module

class spartan.util.Assert[source]

Bases: object

Assertion helper functions.

a = 'foo'
b = 'bar'

Assert.eq(a, b) 
# equivalent to:
# assert a == b, 'a == b failed (%s vs %s)' % (a, b)
static all_eq(a, b)[source]
static eq(a, b, msg='')[source]
static ge(a, b)[source]
static gt(a, b)[source]
static isinstance(expr, klass)[source]
static iterable(expr)[source]
static le(a, b)[source]
static lt(a, b)[source]
static ne(a, b)[source]
static no_duplicates(collection)[source]
static true(expr)[source]
class spartan.util.EZTimer(name)[source]

Bases: object

Lazy timer.

Prints elapsed time when destroyed.

class spartan.util.FileWatchdog(file_handle=<open file '<stdin>', mode 'r' at 0x7f37e23950c0>, on_closed=<function <lambda> at 0x1fd5d70>)[source]

Bases: threading.Thread

Watchdog for a file (typically sys.stdin or sys.stdout).

When the file closes, terminate the process. (This occurs when an ssh connection is terminated, for example.)

run()[source]
spartan.util.as_list(x)[source]
spartan.util.count_calls(fn)[source]

Decorator: count calls to fn and print after each 100. :param fn:

spartan.util.divup(a, b)[source]
spartan.util.dump_stacks(out)[source]

Dump the stacks of all threads.

spartan.util.findCaller(obj)
spartan.util.flatten(lst, depth=1)[source]
spartan.util.iterable(x)[source]
spartan.util.join_tuple(tuple_a, tuple_b)[source]
spartan.util.rtype_check(typeclass)[source]

Function decorator to check return type.

Usage:

@rtype_check(int)
def fn(x, y, z):
  return x + y
spartan.util.stack_signal()[source]
spartan.util.synchronized(fn)[source]

Decorator: execution of this function is serialized by an threading.RLock. :param fn:

spartan.util.timeit(f, name=None)[source]

Run f and log the amount of time taken.

Parameters:
  • f –
  • name –
spartan.util.timer_ctx(*args, **kwds)[source]

Context based timer:

Usage:

with timer_ctx('LoopOp'):
  for i in range(10):
    my_op()
spartan.util.trace_fn(fn)[source]

Function decorator: log on entry and exit to fn.

worker Module

class spartan.worker.Worker(port, master)[source]

Bases: object

create(req, handle)
destroy(req, handle)
get(req, handle)[source]
get_worker(worker_id)
initialize(req, handle)[source]
run_kernel(req, handle)[source]
shutdown(req, handle)[source]
update(req, handle)
wait_for_shutdown()