Tools

Documentation for utilities used by pulsar internals. This module is independent from all other pulsar modules and therefore can be used as a stand-alone library.

HTTP

This is a substantial module which imporfts several classes and functions from the standard library in a python 2.6 to python 3.3 compatible fashion. On top of that, it implements the HttpClient for handling synchronous and asynchronous HTTP requests in a pythonic way.

It is a thin layer on top of urllib2 in python2 / urllib in Python 3. Several opensource efforts have been used as source of snippets:

HTTP Headers

class pulsar.utils.httpurl.Headers(headers=None, kind='server', strict=False)[source]

Utility for managing HTTP headers for both clients and servers.

It has a dictionary like interface with few extra functions to facilitate the insertion of multiple header values. Header fields are case insensitive, therefore doing:

>>> h = Headers()
>>> h['Content-Length'] = '1050'

is equivalent to

>>> h['content-length'] = '1050'
Parameters:
  • headers – optional iterable over header field/value pairs.
  • kind – optional headers type, one of server, client or both.
  • strict – if True only valid headers field will be included.

This Headers container maintains an ordering as suggested by http://www.w3.org/Protocols/rfc2616/rfc2616.html:

The order in which header fields with differing field names are received is not significant. However, it is “good practice” to send general-header fields first, followed by request-header or response-header fields, and ending with the entity-header fields.

—rfc2616 section 4.2

The strict parameter is rarely used and it forces the omission on non-standard header fields.

update(iterable)[source]

Extend the headers with an iterable.

Parameters:iterable – a dictionary or an iterable over keys, vaues tuples.
override(iterable)[source]

Extend headers by overriding fields form iterable.

Parameters:iterable – a dictionary or an iterable over keys, vaues tuples.
get(key, default=None)[source]

Get the field value at key as comma separated values.

For example:

>>> from pulsar.utils.httpurl import Headers
>>> h = Headers(kind='client')
>>> h.add_header('accept-encoding', 'gzip')
>>> h.add_header('accept-encoding', 'deflate')
>>> h.get('accept-encoding')

results in:

'gzip, deflate'
get_all(key, default=None)[source]

Get the values at header key as a list rather than a string separated by comma (which is returned by the get() method).

For example:

>>> from pulsar.utils.httpurl import Headers
>>> h = Headers(kind='client')
>>> h.add_header('accept-encoding', 'gzip')
>>> h.add_header('accept-encoding', 'deflate')
>>> h.get_all('accept-encoding')

results in:

['gzip', 'deflate']
has(field, value)[source]

Check if value is avialble in header field.

clear()[source]

Same as dict.clear(), it removes all headers.

getheaders(key)[source]

Required by cookielib in python 2.

If the key is not available, it returns an empty list.

add_header(key, values)[source]

Add values to key header.

If the header is already available, append the value to the list.

Parameters:
  • key – header name
  • values – a string value or a list/tuple of strings values for header key
remove_header(key, value=None)[source]

Remove the header at key.

If value is provided, it removes only that value if found.

flat(version, status)[source]

Full headers bytes representation

HTTP Parser

class pulsar.utils.httpurl.HttpParser(kind=2, decompress=False, method=None)[source]

A python HTTP parser.

Original code from https://github.com/benoitc/http-parser

2011 (c) Benoit Chesneau <benoitc@e-engura.org>

recv_body()[source]

return last chunk of the parsed body

is_headers_complete()[source]

return True if all headers have been parsed.

is_partial_body()[source]

return True if a chunk of body have been parsed

is_message_begin()[source]

return True if the parsing start

is_message_complete()[source]

return True if the parsing is done (we get EOF)

is_chunked()[source]

return True if Transfer-Encoding header value is chunked

Websocket

WebSocket Protocol is implemented via the Frame and FrameParser classes.

To obtain a frame parser one should use the frame_parser() function.

frame parser

pulsar.utils.websocket.frame_parser(version=None, kind=0, extensions=None, protocols=None, pyparser=False)[source]

Create a new FrameParser instance.

Parameters:
  • version – protocol version, the default is 13
  • kind – the kind of parser, and integer between 0 and 3 (check the FrameParser documentation for details)
  • extensions – not used at the moment
  • protocols – not used at the moment
  • pyparser – if True (default False) uses the python frame parser implementation rather than the much faster cython implementation.

Frame

class pulsar.utils.websocket.Frame(opcode, final, payload_length)[source]

Frame Parser

class pulsar.utils.websocket.FrameParser(version, kind, ProtocolError, extensions=None, protocols=None, close_codes=None)[source]

Decoder and encoder for the websocket protocol.

version

Optional protocol version (Default 13).

kind
  • 0 for parsing client’s frames and sending server frames (to be used in the server)
  • 1 for parsing server frames and sending client frames (to be used by the client)
  • 2 Assumes always unmasked data
ping(body=None)[source]

return a ping Frame.

pong(body=None)[source]

return a pong Frame.

close(code=None)[source]

return a close Frame.

continuation(body=None, final=True)[source]

return a continuation Frame.

encode(message, final=True, masking_key=None, opcode=None, rsv1=0, rsv2=0, rsv3=0)[source]

Encode a message for writing into the wire.

To produce several frames for a given large message use multi_encode() method.

multi_encode(message, masking_key=None, opcode=None, rsv1=0, rsv2=0, rsv3=0, max_payload=0)[source]

Encode a message into several frames depending on size.

Returns a generator of bytes to be sent over the wire.

parse_close

pulsar.utils.websocket.parse_close(data)[source]

Parse the body of a close Frame.

Returns a tuple (code, reason) if successful otherwise raise ProtocolError.

Configuration

Configuration utilities which provides pulsar with configuration parameters which can be parsed from the command line. Parsing is implemented using the python argparser standard library module.

Config

class pulsar.utils.config.Config(description=None, epilog=None, version=None, apps=None, include=None, exclude=None, settings=None, prefix=None, name=None, log_name=None, **params)[source]

A dictionary-like container of Setting parameters for fine tuning pulsar servers.

It provides easy access to Setting.value attribute by exposing the Setting.name as attribute.

Parameters:
settings

Dictionary of all Setting instances available in this Config container.

Keys are given by the Setting.name attribute.

params

Dictionary of additional parameters which cannot be parsed on the command line

update(data, default=False)[source]

Update this Config with data.

Parameters:
  • data – must be a Mapping like object exposing the item method for iterating through key-value pairs.
  • default – if True the updated settings will also set their default attribute with the updating value (provided it is a valid one).
copy_globals(cfg)[source]

Copy global settings from cfg to this config.

The settings are copied only if they were not already modified.

get(name, default=None)[source]

Get the value at name for this Config container

The returned value is obtained from:

  • the value at name in the settings dictionary if available.
  • the value at name in the params dictionary if available.
  • the default value.
set(name, value, default=False, imported=False)[source]

Set the Setting at name with a new value.

If default is True, the Setting.default is also set.

parser()[source]

Create the argparser for this configuration by adding all settings via the Setting.add_argument() method.

Return type:an instance of ArgumentParser.
add_to_parser(parser)[source]

Add this container settings to an existing parser.

parse_command_line(argv=None)[source]

Parse the command line

on_start()[source]

Invoked by a Application just before starting.

workers

Number of workers

address

An address to bind to, only available if a bind setting has been added to this Config container.

copy(name=None, prefix=None)[source]

A copy of this Config container.

If prefix is given, it prefixes all non global settings with it. Used when multiple applications are loaded.

configured_logger(name=None)[source]

Configured logger.

Setting

class pulsar.utils.config.Setting(name=None, flags=None, action=None, type=None, default=None, nargs=None, desc=None, validator=None, app=None, meta=None, choices=None, const=None)[source]

Class for creating pulsar settings.

Most parameters can be specified on the command line, all of them on a config file.

virtual = None

If set to True the settings won’t be loaded.

It can be only used as base class for other settings.

validator = None

A validating function for this setting.

It provided it must be a function accepting one positional argument, the value to validate.

imported = False

The actual value for this setting.

const = None

A constant value required by some action and nargs selections

choices = None

Restrict the argument to the choices provided.

type = None

The type to which the command-line argument should be converted

is_global = False

True only for global settings.

app = None

Setting for a specific Application.

default = None

The default value for this setting.

flags = None

List of options strings, e.g. [-f, --foo].

action = None

The basic type of action to be taken when this argument is encountered at the command line

meta = None

Same usage as metavar in the python argparse module. It is the name for the argument in usage message.

name = None

The key to access this setting in a Config container.

nargs = None

The number of command-line arguments that should be consumed

short = None

Optional shot description string

desc = None

Description string

section = None

Setting section, used for creating the settings documentation.

on_start()[source]

Called when pulsar server starts.

It can be used to perform custom initialization for this Setting.

get()[source]

Returns value

set(val, default=False, imported=False)[source]

Set val as the value for this Setting.

If default is True set also the default value.

add_argument(parser, set_default=False)[source]

Add this Setting to the parser.

The operation is carried out only if flags or nargs and name are defined.

copy(name=None, prefix=None)[source]

Copy this SettingBase

Socket and addresses

pulsar.utils.internet.close_socket(sock)[source]

Shutdown and close the socket.

pulsar.utils.internet.is_socket_closed(sock)[source]

Check if socket sock is closed.

pulsar.utils.internet.is_tls(sock)[source]

Check if sock is a socket over transport layer security

pulsar.utils.internet.parse_address(netloc, default_port=8000)[source]

Parse an internet address netloc and return a tuple with host and port.

pulsar.utils.internet.parse_connection_string(connection_string, default_port=8000)[source]

Converts the connection_string into a three elements tuple (scheme, host, params) where scheme is a string, host could be a string or a two elements tuple (for a tcp address) and params a dictionary of parameters. The default_port parameter can be used to set the port if a port is not available in the connection_string.

For example:

>>> parse_connection_string('http://127.0.0.1:9080')
('http', ('127.0.0.1', 9080), {})

and this example:

>>> parse_connection_string('redis://127.0.0.1:6379?db=3&password=bla')
('redis', ('127.0.0.1', 6379), {'db': '3', 'password': 'bla'})

Internals

System info

pulsar.utils.system.process_info(pid=None)[source]

Returns a dictionary of system information for the process pid.

It uses the psutil module for the purpose. If psutil is not available it returns an empty dictionary.

checkarity

pulsar.utils.tools.checkarity(func, args, kwargs, discount=0)[source]

Check if arguments respect a given function arity and return an error message if the check did not pass, otherwise it returns None.

Parameters:
  • func – the function.
  • args – function arguments.
  • kwargs – function key-valued parameters.
  • discount – optional integer which discount the number of positional argument to check. Default 0.

Structures

Collection of data structures and function used throughout the library.

MultiValueDict

class pulsar.utils.structures.misc.MultiValueDict(data=None)[source]

A subclass of dictionary customized to handle multiple values for the same key.

getlist(key)[source]

Returns the list of values for the passed key.

extend(key, values)[source]

Appends an item to the internal list associated with key.

items()[source]

Returns a generator of (key, value) pairs.

lists()[source]

Returns a list of (key, list) pairs.

values()[source]

Returns a list of the last value on every key list.

AttributeDictionary

class pulsar.utils.structures.misc.AttributeDictionary(*iterable, **kwargs)[source]

A Mapping structures which exposes keys as attributes.

FrozenDict

class pulsar.utils.structures.misc.FrozenDict(*iterable, **kwargs)[source]

A dictionary which cannot be changed once initialised.

Skiplist

class pulsar.utils.structures.skiplist.Skiplist(data=None, unique=False)[source]

Sorted collection supporting O(log n) insertion, removal, and lookup by rank.

clear()[source]

Clear the container from all data.

extend(iterable)[source]

Extend this skiplist with an iterable over score, value pairs.

update(iterable)

Extend this skiplist with an iterable over score, value pairs.

rank(score)[source]

Return the 0-based index (rank) of score.

If the score is not available it returns a negative integer which absolute score is the right most closest index with score less than score.

remove_range(start, end, callback=None)[source]

Remove a range by rank.

This is equivalent to perform:

del l[start:end]

on a python list. It returns the number of element removed.

remove_range_by_score(minval, maxval, include_min=True, include_max=True, callback=None)[source]

Remove a range with scores between minval and maxval.

Parameters:
  • minval – the start value of the range to remove
  • maxval – the end value of the range to remove
  • include_min – whether or not to include minval in the values to remove
  • include_max – whether or not to include maxval in the scores to to remove
  • callback – optional callback function invoked for each score, value pair removed.
Returns:

the number of elements removed.

count(minval, maxval, include_min=True, include_max=True)[source]

Returns the number of elements in the skiplist with a score between min and max.

Zset

class pulsar.utils.structures.zset.Zset(data=None)[source]

Ordered-set equivalent of redis zset.

items()[source]

Iterable over ordered score, value pairs of this zset

score(member, default=None)[source]

The score of a given member

update(score_vals)[source]

Update the zset with an iterable over pairs of scores and values.

remove(item)[source]

Remove item for the zset it it exists. If found it returns the score of the item removed.

remove_range(start, end)[source]

Remove a range by score.

remove_range_by_score(minval, maxval, include_min=True, include_max=True)[source]

Remove a range by score.

clear()[source]

Clear this zset.

rank(item)[source]

Return the rank (index) of item in this zset.

HTML & Text

Utilities for HTML and text manipulation.

pulsar.utils.html.HTML_AMPERSAND = '&amp;'

HTML & symbol.

pulsar.utils.html.HTML_EMDASH = '&mdash;'

HTML – symbol.

pulsar.utils.html.HTML_ENDASH = '&ndash;'

HTML - symbol.

pulsar.utils.html.HTML_GREATER_THEN = '&gt;'

HTML > symbol.

pulsar.utils.html.HTML_LESS_THEN = '&lt;'

HTML < symbol.

pulsar.utils.html.HTML_NON_BREACKING_SPACE = '&nbsp;'

HTML non breaking space symbol.

pulsar.utils.html.NOTHING = ('', b'', None)

Tuple of elements considered as null.

pulsar.utils.html.capfirst(x)[source]

Capitalise the first letter of x.

pulsar.utils.html.child_tag(tag)[source]

The default children tag for a given tag.

pulsar.utils.html.escape(html, force=False)[source]

Returns the given HTML with ampersands, quotes and angle brackets encoded.

pulsar.utils.html.mark_safe(v)[source]

Mar a string v as safe. A safe string won’t be escaped by the escape() function.

pulsar.utils.html.nicename(name)[source]

Make name a more user friendly string.

Capitalise the first letter and replace dash and underscores with a space

Slugify

A slugify function which handle unicode

pulsar.utils.slugify.slugify(value, separator='-', max_length=0, word_boundary=False, entities=True, decimal=True, hexadecimal=True)[source]

Normalizes string, removes non-alpha characters, and converts spaces to separator character

Logging

Module containing utilities and mixins for logging and serialisation.

class pulsar.utils.log.LocalMixin[source]

Defines the local attribute.

Classes derived from a LocalMixin can use the local_method() and local_property() decorators for managing attributes which are not picklable.

local

A lazy pulsar.utils.structures.AttributeDictionary.

This attribute is removed when pickling an instance.

lock

A local threading.Lock.

class pulsar.utils.log.WritelnDecorator(stream)[source]

Used to decorate file-like objects with a handy ‘writeln’ method. taken from python.

pulsar.utils.log.configured_logger(name=None, config=None, level=None, handlers=None)[source]

Configured logger.

pulsar.utils.log.local_method(f)[source]

Decorator to be used in conjunction with LocalMixin methods.

pulsar.utils.log.local_property(f)[source]

Decorator to be used in conjunction with LocalMixin methods.

pulsar.utils.log.process_global(name, val=None, setval=False)[source]

Access and set global variables for the current process.

Path

Stand alone compact module for managing python paths.

class pulsar.utils.path.Path[source]

A lightweight utility for the filesystem and python modules.

add2python(module=None, up=0, down=None, front=False, must_exist=True)[source]

Add a directory to the python path.

Parameters:
  • module – Optional module name to try to import once we have found the directory
  • up – number of level to go up the directory three from local_path.
  • down – Optional tuple of directory names to travel down once we have gone up levels.
  • front – Boolean indicating if we want to insert the new path at the front of sys.path using sys.path.insert(0,path).
  • must_exist – Boolean indicating if the module must exists.
classmethod cwd()[source]

Return the current working directory as a path object.

ispymodule()[source]

Check if this Path is a python module.