throttler — the module containing the throttlers

This module contains the throttlers.

BaseThrottler - the simplest requests throttler

class requests_throttler.throttler.ThrottlerStatusError(msg, current_status, previous_status=None)

Exception that occurs when something goes wrong while changing status

Parameters:

• msg (string) – the message
• current_status (string) – the status to set
• previous_status (string) – the previous status

class requests_throttler.throttler.FullRequestsPoolError(msg, pool)

Exception that occurs when an enqueue is tried in a full pool

Parameters:

• msg (string) – the message
• pool (collections.deque) – the pool

class requests_throttler.throttler.BaseThrottler(*args, **kwargs)

This class provides the base requests throttler

The base throttler guarantees that between each request a fixed amount of time between them elapsed. The pool can be limited by a maximum length and an exception is raised if a request is tried to be enqueued in the full pool.

Parameters:

• name (string) – the name of the throttler
• requests_pool (collections.dequeue) – the pool containing the requests (FIFO)
• delay (float) – the delay in seconds between each request
• status (string) – the current status of the thottler
• session (requests.Session) – the session to use to perform the requests
• executor (threading.ThreadPoolExecutor) – the executor responsable to start the throttler
• timer (utils.Timer) – the timer responsable to measure the time between each request
• successes (int) – the number of request that succeded
• failures – the number of request that failed
• wait_enqueued (boolean) – a flag that indicates if after the shutdown the requests enqueued have to be finished or aborted
• status_lock (threading.Condition) – the condition on which to wait on a specific status change
• not_empty (threading.Condition) – the condition on which to wait when the pool of requests is empty

__init__(*args, **kwargs)

Create a base throttler with the given delay time and pool size

When both delay and reqs_over_time are None, delay is set to 0.

Parameters:

• name (string) – the name of the throttler (default: None)
• session (requests.Session) – the sessions to use for each request
• delay (float) – the fixed positive amount of time that must elapsed bewteen each request in seconds (default: None)
• reqs_over_time ((float, float)) – a tuple of the form (number of requests, time) used to calculate the delay to use when it is None. The delay will be equal to time / number of requests (default: None)
• max_pool_size (int) – the maximum number of enqueueable requests (default: unlimited)

Raise:

ValueError:if delay or the value calculated from reqs_over_time is a negative number

name

The name of the throttler

Getter:Returns name Type:string

delay

The delay value between each request

Getter:Returns delay Type:float

status

The status of the throttler

Getter:

Returns status

Setter:

Sets the new status

Raise:

ThrottlerStatusError:  if the new status is invalid

Type:

string

successes

The number of successes

Getter:Returns successes Type:int

failures

The number of failures

Getter:Returns failures Type:int

start()

Start the throttler by starting the main loop

Raise:

ThrottlerStatusError:  if the throller has been already started

shutdown(wait_enqueued=True)

Shutdown the throttler by shutdowning the executor

If wait_enqueued is True then before stopping the throttlers consumes all the requests enqueued. Otherwise the throttler is forced to be shutdowned.

Parameters:

wait_enqueued – the flag that indicates if the already enqueued requests are to be processed or aborted

Raise:

ThrottlerStatusError:  if the throttler has been already shutdowned

pause()

Pause the throttler

Raise:

ThrottlerStatusError:  if the throttler is not running, waiting or already paused

unpause()

Unpause the throttler

Raise:

ThrottlerStatusError:  if the throttler is not paused

submit(req)

Submit a single request and return the corresponding throttled request

Parameters:

req (requests.Request) – the request to throttle

Returns:

the corresponding throttled request

Return type:

requests_throttler.throttled_request.ThrottledRequest

Raise:

ThrottlerStatusError:  if the throttler is not running, paused or waiting

multi_submit(reqs)

Submits a list of requests and return the corresponding list of throttled requests

Parameters:

reqs – the list of requests to throttle

Returns:

the corresponding list of throttled requests

Return type:

list(requests_throttler.throttled_request.ThrottledRequest)

Raise:

ThrottlerStatusError:  if the throttler is not running, paused or waiting

wait_end()

Wait until the throttler is ended