API Reference

Classes

class HTTPConnection(hostname, port=None, ssl_context=None, proxy=None)

This class encapsulates a HTTP connection.

Methods whose name begin with co_ return coroutines. Instead of blocking, a coroutines will yield a PollNeeded instance that encapsulates information about the IO operation that would block. The coroutine should be resumed once the operation can be performed without blocking.

HTTPConnection instances can be used as context managers. The disconnect method will be called on exit from the managed block.

co_discard()

Read and discard current response body

This method returns a coroutine. discard is a regular method implementing the same functionality.

co_read(len_=None)

Read up to len_ bytes of response body data

This method may return less than len_ bytes, but will return b'' only if the response body has been read completely.

If len_ is None, this method returns the entire response body.

This method returns a coroutine. read is a regular method implementing the same functionality.

co_read_response()

Read response status line and headers

Return a HTTPResponse instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. using read or readall).

This method returns a coroutine. read_response is a regular method implementing the same functionality.

co_readall()

Read and return complete response body

This method returns a coroutine. readall is a regular method implementing the same functionality.

co_readinto(buf)

Read response body data into buf

Return the number of bytes written or zero if the response body has been read completely.

buf must implement the memoryview protocol.

This method returns a coroutine. readinto is a regular method implementing the same functionality.

co_send_request(method, path, headers=None, body=None, expect100=False)

Send a new HTTP request to the server

The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a BodyFollowing instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.

headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a CaseInsensitiveDict instance, other mappings will be converted to CaseInsensitiveDict automatically.

If body is a provided as a bytes-like object, a Content-MD5 header is generated automatically unless it has been provided in headers already.

This method returns a coroutine. send_request is a regular method implementing the same functionality.

co_write(buf)

Write request body data

ExcessBodyData will be raised when attempting to send more data than required to complete the request body of the active request.

This method returns a coroutine. write is a regular method implementing the same functionality.

connect()

Connect to the remote server

This method generally does not need to be called manually.

discard()

Read and discard current response body

This method may block. co_discard provides a coroutine implementing the same functionality without blocking.

disconnect()

Close HTTP connection

get_ssl_cipher()

Get active SSL cipher

If plain HTTP is used, return None. Otherwise, the call is delegated to the underlying SSL sockets cipher method.

get_ssl_peercert(binary_form=False)

Get peer SSL certificate

If plain HTTP is used, return None. Otherwise, the call is delegated to the underlying SSL sockets getpeercert method.

read(len_=None)

Read up to len_ bytes of response body data

This method may return less than len_ bytes, but will return b'' only if the response body has been read completely.

If len_ is None, this method returns the entire response body.

This method may block. co_read provides a coroutine implementing the same functionality without blocking.

read_raw(size)

Read size bytes of uninterpreted data

This method may be used even after UnsupportedResponse or InvalidResponse has been raised. It reads raw data from the socket without attempting to interpret it. This is probably only useful for debugging purposes to take a look at the raw data received from the server. This method blocks if no data is available, and returns b'' if the connection has been closed.

Calling this method will break the internal state and switch the socket to blocking operation. The connection has to be closed and reestablished afterwards.

Don’t use this method unless you know exactly what you are doing.

read_response()

Read response status line and headers

Return a HTTPResponse instance containing information about response status, reason, and headers. The response body data must be retrieved separately (e.g. using read or readall).

This method may block. co_read_response provides a coroutine implementing the same functionality without blocking.

readall()

Read and return complete response body

This method may block. co_readall provides a coroutine implementing the same functionality without blocking.

readinto(buf)

Read response body data into buf

Return the number of bytes written or zero if the response body has been read completely.

buf must implement the memoryview protocol.

This method may block. co_readinto provides a coroutine implementing the same functionality without blocking.

reset()

Reset HTTP connection

This method resets the status of the HTTP connection after an exception has occured. Any cached data and pending responses are discarded.

response_pending()

Return True if there are still outstanding responses

This includes responses that have been partially read.

send_request(method, path, headers=None, body=None, expect100=False)

Send a new HTTP request to the server

The message body may be passed in the body argument or be sent separately. In the former case, body must be a bytes-like object. In the latter case, body must be an a BodyFollowing instance specifying the length of the data that will be sent. If no length is specified, the data will be send using chunked encoding.

headers should be a mapping containing the HTTP headers to be send with the request. Multiple header lines with the same key are not supported. It is recommended to pass a CaseInsensitiveDict instance, other mappings will be converted to CaseInsensitiveDict automatically.

If body is a provided as a bytes-like object, a Content-MD5 header is generated automatically unless it has been provided in headers already.

This method may block. co_send_request provides a coroutine implementing the same functionality without blocking.

write(buf)

Write request body data

ExcessBodyData will be raised when attempting to send more data than required to complete the request body of the active request.

This method may block. co_write provides a coroutine implementing the same functionality without blocking.

proxy = None

a tuple (hostname, port) of the proxy server to use or None. Note that currently only CONNECT-style proxying is supported.

timeout = None

If a regular HTTPConnection method is unable to send or receive data for more than this period (in seconds), it will raise ConnectionTimedOut. Coroutines are not affected by this attribute.

class HTTPResponse(method, path, status, reason, headers, length=None)

This class encapsulates information about HTTP response. Instances of this class are returned by the HTTPConnection.read_response method and have access to response status, reason, and headers. Response body data has to be read directly from the HTTPConnection instance.

headers = None

HTTP Response headers, a email.message.Message instance

length = None

Length of the response body or None if not known. This attribute contains the actual length of the transmitted response. That means that for responses where RFC 2616 mandates that no request body be sent (e.g. in response to HEAD requests or for 1xx response codes) this value is zero. In these cases, the length of the body that would have been send can be extracted from the Content-Length response header.

method = None

HTTP Method of the request this was response is associated with

path = None

Path of the request this was response is associated with

reason = None

HTTP reason phrase returned by the server

status = None

HTTP status code returned by the server

class BodyFollowing(length=None)

Sentinel class for the body parameter of the send_request method. Passing an instance of this class declares that body data is going to be provided in separate method calls.

If no length is specified in the constructor, the body data will be send using chunked encoding.

length

the length of the body data that is going to be send, or None to use chunked encoding.

class CaseInsensitiveDict(data=None, **kwargs)

A case-insensitive dict-like object.

Implements all methods and operations of collections.abc.MutableMapping as well as copy.

All keys are expected to be strings. The structure remembers the case of the last key to be set, and iter(), keys() and items() will contain case-sensitive keys. However, querying and contains testing is case insensitive:

cid = CaseInsensitiveDict()
cid['Accept'] = 'application/json'
cid['aCCEPT'] == 'application/json' # True
list(cid) == ['Accept'] # True

For example, headers['content-encoding'] will return the value of a 'Content-Encoding' response header, regardless of how the header name was originally stored.

If the constructor, update(), or equality comparison operations are given multiple keys that have equal lower-case representions, the behavior is undefined.

lower_items()

Like items(), but with all lowercase keys.

class PollNeeded

This class encapsulates the requirements for a IO operation to continue. PollNeeded instances are typically yielded by coroutines.

poll(timeout=None)

Wait until fd is ready for requested IO

This is a convenince function that uses poll to wait until fd is ready for requested type of IO.

If timeout is specified, return False if the timeout is exceeded without the file descriptor becoming ready.

fd

File descriptor that the IO operation depends on

mask

Event mask specifiying the type of required IO

This attribute defines what type of IO the provider of the PollNeeded instance needs to perform on fd. It is expected that, when fd is ready for IO of the specified type, operation will continue without blocking.

The type of IO is specified as a poll compatible event mask, i.e. a bitwise combination of select.POLLIN and select.POLLOUT.

class AioFuture(crt, loop=None)

This class wraps a coroutine that yields PollNeeded instances into an asyncio compatible Future.

This is done by registering a callback with the event loop that resumes the coroutine when the requested IO is available.

Functions

is_temp_network_error(exc)

Return true if exc represents a potentially temporary network problem

DNS resolution errors (socket.gaierror or socket.herror) are considered permanent, because HTTPConnection employs a heuristic to convert these exceptions to HostnameNotResolvable or DNSUnavailable instead.

Exceptions

Dugong functions may pass through any exceptions raised by the socket and ssl.SSLSocket methods. In addition to that, the following dugong-specific exceptions may be raised as well:

exception ConnectionClosed(msg=None)

Raised if the connection was unexpectedly closed.

This exception is raised also if the server declared that it will close the connection (by sending a Connection: close header). Such responses can still be read completely, but the next attempt to send a request or read a response will raise the exception. To re-use the connection after the server has closed the connection, call HTTPConnection.reset before further requests are send.

This behavior is intentional, because the caller may have already issued other requests (i.e., used pipelining). By raising an exception, the caller is notified that any pending requests have been lost and need to be resend.

exception InvalidResponse(msg=None)

Raised if the server produced an invalid response (i.e, something that is not proper HTTP 1.0 or 1.1).

exception UnsupportedResponse(msg=None)

This exception is raised if the server produced a response that is not supported. This should not happen for servers that are HTTP 1.1 compatible.

If an UnsupportedResponse exception has been raised, this typically means that synchronization with the server will be lost (i.e., dugong cannot determine where the current response ends and the next response starts), so the connection needs to be reset by calling the disconnect() method.

exception ExcessBodyData(msg=None)

Raised when trying to send more data to the server than announced.

exception StateError(msg=None)

Raised when attempting an operation that doesn’t make sense in the current connection state.

exception ConnectionTimedOut(msg=None)

Raised if a regular HTTPConnection method (i.e., no coroutine) was unable to send or receive data for the timeout specified in the HTTPConnection.timeout attribute.

exception HostnameNotResolvable(hostname)

Raised if a host name does not resolve to an ip address.

Dugong raises this exception if a resolution attempt results in a socket.gaierror or socket.herror exception with errno socket.EAI_AGAIN or socket.EAI_NONAME, but at least one of the hostnames in DNS_TEST_HOSTNAMES can be resolved.

exception DNSUnavailable(hostname)

Raised if the DNS server cannot be reached.

Dugong raises this exception if a resolution attempt results in a socket.gaierror or socket.herror exception with errno socket.EAI_AGAIN or socket.EAI_NONAME, and none of the hostnames in DNS_TEST_HOSTNAMES can be resolved either.

Constants

MAX_LINE_SIZE = 65535

Maximal length of HTTP status line. If the server sends a line longer than this value, InvalidResponse will be raised.

MAX_HEADER_SIZE = 65535

Maximal length of a response header (i.e., for all header lines together). If the server sends a header segment longer than this value, InvalidResponse will be raised.

DNS_TEST_HOSTNAMES = (('www.google.com', 80), ('www.iana.org', 80), ('C.root-servers.org', 53))

Sequence of (hostname, port) tuples that are used by dugong to distinguish between permanent and temporary name resolution problems.

Thread Safety

Dugong is not generally threadsafe. However, simultaneous use of the same HTTPConnection instance by two threads is supported if once thread is restricted to sending requests, and the other thread restricted to reading responses.

Avoiding Deadlocks

The HTTPConnection class allows you to send an unlimited number of requests to the server before reading any of the responses. However, at some point the transmit and receive buffers on both the ends of the connection will fill up, and no more requests can be send before at least some of the responses are read, and attempts to send more data to the server will block. If the thread that attempts to send data is is also responsible for reading the responses, this will result in a deadlock.

There are several ways to avoid this:

  • Do not send a new request before the last response has been read. This is the easiest solution, but it means that no HTTP pipelining can be used.
  • Use different threads for sending requests and receiving responses.
  • Use the coroutine based API (see Pipelining with Coroutines in the tutorial).

Table Of Contents

Previous topic

Tutorial

Next topic

Coroutine API