pools.gearman_cluster.pool

glimpse.pools.gearman_cluster.pool.COMMAND_PING = 'ping'

Instructs the worker to respond with processing statistics.

glimpse.pools.gearman_cluster.pool.COMMAND_QUIT = 'quit'

Instructs the worker to exit with signal 0.

glimpse.pools.gearman_cluster.pool.COMMAND_RESTART = 'restart'

Instructs the worker to exit with signal -1.

class glimpse.pools.gearman_cluster.pool.ClusterPool(host_list=None, random_unique_bytes=16)

Client for the Gearman task farm.

chunksize = 8

Size of a map request (in number of input elements).

data_encoder

Encode communications as pickles.

alias of PickleDataEncoder

imap(func, iterable, chunksize=None)

Not implemented. This is currently an alias for map().

map(func, iterable, chunksize=None)

Apply a function to a list.

max_retries = 3

Resubmit a job up to three times.

poll_timeout = 60.0

Wait one minute for each job to complete.

glimpse.pools.gearman_cluster.pool.CommandHandlerTarget(worker, cmd_future, log_future, ping_handler)

Logic for process that handles command messages.

A quit or restart command is handled by setting the worker’s finish flag to True, setting the exit status, and returning. The exit status is 0 on quit, or -1 on restart.

Parameters:

• worker (Worker) – Instance for tracking worker state.
• cmd_future (FutureSocket) – Socket information for the command channel.
• log_future (FutureSocket) – Socket information for the log channel.
• ping_handler (callable) – Generates a response message for a ping command.

exception glimpse.pools.gearman_cluster.pool.ConfigException

Indicates that an error occurred while reading the cluster configuration.

glimpse.pools.gearman_cluster.pool.GEARMAN_TASK_MAP = 'glimpse_map'

Gearman task ID for the map task.

class glimpse.pools.gearman_cluster.pool.PickleDataEncoder

A data encoder that reads and writes using the Pickle format.

By default, a GearmanClient can only send byte-strings. If we want to be able to send Python objects, we must specify a data encoder. This will automatically convert byte strings to Python objects (and vice versa) for all commands that have the ‘data’ field. See http://gearman.org/index.php?id=protocol for client commands that send/receive ‘opaque data’.

glimpse.pools.gearman_cluster.pool.RunWorker(job_server_url, command_url, log_url, num_processes=None)

Launch a Gearman worker and wait for it to complete.

This worker processes batch requests using a MulticorePool.

Parameters:

• job_server_url (str) – URL for Gearman job server.
• command_url (str) – URL for command channel.
• log_url (str) – URL for logging channel.
• num_processes (int) – Number of concurrent processes to use when processing a batch request. Defaults to the number of available cores.

Returns:

Exit status of the worker processes.

class glimpse.pools.gearman_cluster.pool.Worker(host_list=None)

A Gearman worker that tracks exit status.

after_poll(any_activity)

Returns True unless the finish flag is set.

This method is called by the gearman.GearmanWorker to decide when to quit.

data_encoder

How to marshal data on the wire.

alias of PickleDataEncoder

exit_status = 0

Exit status of the worker.

finish = False

Indicates whether the worker has received a quit or restart command.