Working with queues

This module contains classes which help you work with queues. A typical application is when you want to log from performance-critical threads, but where the handlers you want to use are slow (for example, SMTPHandler). In that case, you can create a queue, pass it to a QueueHandler instance and use that instance with your loggers. Elsewhere, you can instantiate a QueueListener with the same queue and some slow handlers, and call start() on it. This will start monitoring the queue on a separate thread and call all the configured handlers on that thread, so that your logging thread is not held up by the slow handlers.

Note that as well as in-process queues, you can use these classes with queues from the multiprocessing module.

N.B. This is part of the standard library since Python 3.2, so the version here is for use with earlier Python versions.

class logutils.queue.QueueHandler(queue)

This handler sends events to a queue. Typically, it would be used together with a multiprocessing Queue to centralise logging to file in one process (in a multi-process application), so as to avoid file write contention between processes.

Parameters:queue – The queue to send LogRecords to.
emit(record)

Emit a record.

Writes the LogRecord to the queue, preparing it for pickling first.

Parameters:record – The record to emit.
enqueue(record)

Enqueue a record.

The base implementation uses put_nowait(). You may want to override this method if you want to use blocking, timeouts or custom queue implementations.

Parameters:record – The record to enqueue.
prepare(record)

Prepares a record for queuing. The object returned by this method is enqueued.

The base implementation formats the record to merge the message and arguments, and removes unpickleable items from the record in-place.

You might want to override this method if you want to convert the record to a dict or JSON string, or send a modified copy of the record while leaving the original intact.

Parameters:record – The record to prepare.
class logutils.queue.QueueListener(queue, *handlers, **kwargs)

This class implements an internal threaded listener which watches for LogRecords being added to a queue, removes them and passes them to a list of handlers for processing.

Parameters:
  • record – The queue to listen to.
  • handlers – The handlers to invoke on everything received from the queue.
dequeue(block)

Dequeue a record and return it, optionally blocking.

The base implementation uses get(). You may want to override this method if you want to use timeouts or work with custom queue implementations.

Parameters:block – Whether to block if the queue is empty. If False and the queue is empty, an Empty exception will be thrown.
enqueue_sentinel()

Writes a sentinel to the queue to tell the listener to quit. This implementation uses put_nowait(). You may want to override this method if you want to use timeouts or work with custom queue implementations.

handle(record)

Handle a record.

This just loops through the handlers offering them the record to handle.

Parameters:record – The record to handle.
prepare(record)

Prepare a record for handling.

This method just returns the passed-in record. You may want to override this method if you need to do any custom marshalling or manipulation of the record before passing it to the handlers.

Parameters:record – The record to prepare.
start()

Start the listener.

This starts up a background thread to monitor the queue for LogRecords to process.

stop()

Stop the listener.

This asks the thread to terminate, and then waits for it to do so. Note that if you don’t call this before your application exits, there may be some records still left on the queue, which won’t be processed.