Rejected Companion Library

This library contains a number of utility and helper classes that were embedded in consumers that I have written recently. They add useful functionality on top of rejected.

	https://github.com/dave-shawley/vetoes
	https://pypi.org/project/vetoes/
	https://pythonhosted.org/vetoes/
	https://coveralls.io/github/dave-shawley/vetoes
	https://travis-ci.org/dave-shawley/vetoes

HTTP API Helpers

class vetoes.service.HTTPServiceMixin(*args, **kwargs)

Mix this in to add the call_http_service() method.

Parameters:

• service_map (dict) – mapping from logical “function” to HTTP service.
• args – these are passed to rejected.consumer.Consumer() as-is
• kwargs – these are passed to rejected.consumer.Consumer() as-is

The primary use of this mix-in is to make sending HTTP requests easier and safer. The call_http_service() method uses the get_service_url() to build the request URL for a named semantic function, send the request to a specific HTTP service, and perform some “opinionated” processing on the response. The separation of semantic function and HTTP service may seem a bit confusing at first. They are used to provide concise logging and well-named metrics. The semantic function describes the action being performed and the service is the actor performing the action.

The mapping from semantic function to HTTP service is handled by the service_map passed into the initializer. The mapping value is the HTTP service which is passed into get_service_url() to construct the request URL.

HTTP client behavior is controlled via consumer level configuration under the vetoes key. The following options are available:

Key	Description
max_clients	The max # of simultaneous requests that can be made
connect_timeout	Timeout for initial connection in seconds
request_time	Timeout for entire request in seconds

Example Configuration:

Application:
  Consumers:
    example:
      consumer: rejected.example.Consumer
      connections:
        - name: rabbitmq1
      qty: 2
      queue: generated_messages
      config:
        vetoes:
          max_clients: 10
          connect_timeout: 5.0
          request_timeout: 30.0

http_headers

tornado.httputil.HTTPHeaders instance of headers that are included in every request. This set is empty.

http

tornado.httpclient.AsyncHTTPClient used to make requests. The initializer sets the connect_timeout and request_timeout in self.http.defaults.

call_http_service(function, method, *path, **kwargs)

Send a HTTP request to a service.

Parameters:

• function (str) – the function to invoke. The service is determined based on the service_map established during initialization.
• method (str) – HTTP method to invoke.
• path – path elements to the HTTP resource.
• headers (dict) – optional set of headers to include in the message. These are in addition to http_headers.
• json (dict or list) – optional body to send in the message. If this keyword is included, then the value is JSON encoded before being used as the body.
• raise_error (bool) – if this keyword is included and set to False, then HTTP errors will be returned instead of raised as exceptions.
• url (str) – if this keyword is included then it is used as-is instead of doing a service lookup.
• kwargs – additional keyword arguments are passed to tornado.httpclient.AsyncHTTPClient.fetch().

Returns:

a tornado.httpclient.HTTPResponse instance

Return type:

tornado.httpclient.HTTPResponse

Raises:

tornado.httpclient.HTTPError if a HTTP error occurs

Raises:

rejected.consumer.ProcessingException if a low-level socket error occurs or a retry-able HTTP result is returned

get_service_url(service, *path, **kwargs)

Build a request URL for a specific service.

Parameters:

• service (str) – name of the service to invoke
• path – resource path elements
• query_args (dict or None) – optional query parameters to include in the URL

Returns:

the request URL

Return type:

str

Note

You are required to override this method in your consumer. The base implementation simply raises an exception.

Configuration Related

class vetoes.config.FeatureFlagMixin(*args, **kwargs)

Mix this in to parse self.settings['features'].

Each flag in the features consumer setting is parsed using distutils.util.strtobool() and if it is parseable, then self.feature_flags[name] is set to the parsed result.

For example, if your rejected configuration looks like:

Application:
  Consumers:
    my-consumer:
      config:
        features:
          frobinicate: on
          defenestrate: no

Then your consumer can expect feature_flags to be:

feature_flags = {'frobinicate': True,
                 'defenestrate': False}

class vetoes.config.TimeoutConfigurationMixin(settings, process, drop_invalid_messages=None, message_type=None, error_exchange=None, error_max_retry=None, drop_exchange=None)

Mix this in to parse self.settings['timeouts'].

The members from the timeouts dictionary is exposed by the get_timeout() method.

For example, if your rejected configuration looks like:

Applications:
  Consumers:
    my-consumer:
      config:
        timeouts:
          default: 0.1
          dynamodb: 1.5

Then your consumer can expect the following to hold true:

self.get_timeout('dynamodb') == 1.5
self.get_timeout('something-else') == 0.1

If you omit the “default” key, then it will fallback to 0.5s.

get_timeout(name, default=None)

Get the timeout named name from the consumer configuration.

Parameters:

• name (str) – the name of the timeout
• default – optional default value. If this is omitted then the get_timeout('default', 0.5) will be returned

Returns:

the appropriate timeout value in seconds

Return type:

float

Release History

0.4.0 (13-Apr-2017)

• Added support to correctly translate user:password@host style URLs into auth_username and auth_password keyword parameters in vetoes.service.HTTPServiceMixin.call_http_service().

0.3.0 (04-Apr-2017)

• Updated to work against rejected 3.17
• Change vetoes.service.HTTPServiceMixin.call_http_service() so that it honors timeouts set by vetoes.config.TimeoutConfigurationMixin.

0.2.0 (10-Jan-2017)

• Added url keyword to vetoes.service.HTTPServiceMixin.call_http_service()

0.1.1 (06-Jan-2017)

• Replaced readthedocs with pythonhosted.org.

0.1.0 (06-Jan-2017)

• Initial release including vetoes.service.HTTPServiceMixin, vetoes.config.FeatureFlagMixin, and vetoes.config.TimeoutConfigurationMixin