zc.async

What is it?

The zc.async package provides an easy-to-use Python tool that schedules durable tasks across multiple processes and machines.

For instance...

  • Web apps:

    maybe your web application lets users request the creation of a large PDF, or some other expensive task.

  • Postponed work:

    maybe you have a job that needs to be done at a certain time, not right now.

  • Parallel processing:

    maybe you have a long-running problem that can be made to complete faster by splitting it up into discrete parts, each performed in parallel, across multiple machines.

  • Serial processing:

    maybe you want to decompose and serialize a job.

High-level features include the following.

  • Easy to use.

    At its simplest, put a function in a zc.async queue and commit a transaction. See the quick-starts for examples.

  • Flexible configuration, changeable dynamically in production.

    Add and remove worker processes on the fly, with configurable policy on how to handle interrupts. Let processes decide how many of which tasks to perform. Configuration for each process is stored in the database so no restarts are needed and a change can happen for any process from any database client.

  • Reliable and fault tolerant, supporting high availability.

    Configurable policy lets zc.async know when, how, and under what circumstances to retry jobs that encounter problems. Multiple processes and machines can be available to work on jobs, and a machine or process that suddenly dies lets siblings decide what to do with incomplete jobs, with policy on a per-job basis. The central ZODB database server can be replicated with commercial tools (ZRS) or open-source tools (RelStorage plus, for instance PostgreSQL and slony; or gocept.zeoraid).

  • Good debugging tools.

    Exceptions generate persistent Failure objects (from the Twisted project) for analysis, and verbose log messages.

  • Well-tested.

    The package has good automated tests and is in use in mission-critical applications for large software deployments.

  • Friendly to testing.

    The package exposes testing helpers for a variety of circumstances, to make writing automated tests for zc.async-enabled software fairly painless.

While developed as part of the Zope project, zc.async can be used stand-alone, as seen in the quick-starts and the majority of the tests.

How does it work?

The system uses the Zope Object Database (ZODB), a transactional, pickle-based Python object database, for communication and coordination among participating processes.

zc.async participants can each run in their own process, or share a process (run in threads) with other code.

The Twisted framework supplies some code (failures and reactor implementations, primarily) and some concepts to the package.

Quick starts

These quick-starts can help you get a feel for the package. Please note: the Grok quickstart is only just begun, and should be regarded mostly as a placeholder.

Indices and tables