pools

glimpse.pools.MakePool()

Return a new instance of the default worker pool.

Returns:A serializable worker pool.

glimpse.pools.GetClusterPackage(cluster_type=None)

Choose a cluster package by name.

If cluster_type is gearman, for example, then this method returns glimpse.pools.gearman_cluster. The returned module is guaranteed to contain the MakePool() and RunMain() functions.

Parameters:cluster_type (str) – Type of cluster to create (e.g., gearman, zmq, ipython). The default is read from the GLIMPSE_CLUSTER_TYPE environment variable, or is ipython if this variable is unset.

class glimpse.pools.SinglecorePool

A fall-back worker pool that uses a single core of a single machine.

imap(func, iterable, chunksize=1)

Apply a function to an iterable.

imap_unordered(func, iterable, chunksize=1)

Apply a function to an iterable, where elements are evaluated in any order.

map(func, iterable, chunksize=None)

Apply a function to a list.

class glimpse.pools.MulticorePool(*args)

A worker pool that utilizes multiple cores on a single machine.

This class delegates to the corresponding methods of multiprocessing.Pool. Thus, all restrictions — such as limitations on mapping an anonymous function — are preserved.

Note that there are limitations regarding the functions that can be mapped with the methods of this class. Specifically, note that the following example will fail:

>>> pool = glimpse.pools.MulticorePool()
>>> def f(x): return x * 10
>>> pool.map(f, [1, 2, 3])
AttributeError: 'module' object has no attribute 'f'

Here, the AttributeError is thrown because f() is not defined in the worker subprocess. Similarly, an anonymous function also throws an error:

>>> pool.map((lambda x: x * 10), [1, 2, 3])
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

Here, it becomes clear that communication between subprocesses uses pickled objects. The PicklingError results from the fact that an anonymous function can not be pickled.

imap(func, iterable, chunksize=1)

Apply a function to an iterable.

imap_unordered(func, iterable, chunksize=1)

Apply a function to an iterable, where elements are evaluated in any order.

map(func, iterable, chunksize=None)

Apply a function to a list.