Flask-CeleryExt

https://img.shields.io/travis/inveniosoftware/flask-celeryext.svg https://img.shields.io/coveralls/inveniosoftware/flask-celeryext.svg https://img.shields.io/github/tag/inveniosoftware/flask-celeryext.svg https://img.shields.io/pypi/dm/flask-celeryext.svg https://img.shields.io/github/license/inveniosoftware/flask-celeryext.svg

About

Flask-CeleryExt is a simple integration layer between Celery and Flask.

Installation

Flask-CeleryExt is on PyPI so all you need is:

pip install flask-celeryext

Usage

Simple integration layer between Celery and Flask.

This extension adds a small integration layer between Celery and Flask based on https://flask.pocoo.org/docs/0.10/patterns/celery/. In addition to support execution of tasks in a Flask application context, the extension also supports execution of task in a test request context (e.g. to ensure that before first request functions have been executed).

Extension initialization

Initialize the extension like this:

>>> from flask import Flask
>>> from flask_celeryext import FlaskCeleryExt
>>> app = Flask('myapp')
>>> ext = FlaskCeleryExt(app)

or alternatively using the factory pattern:

>>> app = Flask('myapp')
>>> app.config.update(dict(
...     CELERY_ALWAYS_EAGER=True,
...     CELERY_RESULT_BACKEND='cache',
...     CELERY_CACHE_BACKEND='memory',
...     CELERY_EAGER_PROPAGATES_EXCEPTIONS=True))
>>> ext = FlaskCeleryExt()
>>> ext.init_app(app)

The extension will create a Celery application. Configuration is by default loaded from the Flask application:

>>> celery = ext.celery

Celery v3 and v4 configuration

The extension supports both Celery v3 and v4. The configuration for v3 and v4 is however significantly different. Above configuration is for Celery v3 while in v4 you should use:

>>> v4conf = dict(
...     CELERY_TASK_ALWAYS_EAGER=True,
...     CELERY_RESULT_BACKEND='cache',
...     CELERY_CACHE_BACKEND='memory',
...     CELERY_TASK_EAGER_PROPAGATES=True)

If you have Celery v4 installed but use v3 configuration the extension will automatically translate your configuration from v3 to v4 and issue a warning.

All configuration variables for v4 are prefixed with CELERY_ and in upper case.

Defining tasks

You can now define tasks using the Celery application. By default the tasks will be run inside a Flask application context, and thus have access to e.g. current_app:

>>> from flask import current_app
>>> @celery.task
... def apptask():
...     return current_app.name

>>> r = apptask.delay()
>>> r.result
'myapp'

If you need to run tasks inside a Flask request context, simply change the task base class:

>>> from flask import has_request_context, request
>>> from flask_celeryext import RequestContextTask
>>> @celery.task(base=RequestContextTask)
... def reqtask():
...     return has_request_context()

>>> r = reqtask.delay()
>>> r.result
True

Application factory

The Celery application is created by a default application factory, which you can also use separately:

>>> from flask_celeryext import create_celery_app
>>> app = Flask('myapp')
>>> celery = create_celery_app(app)
>>> @celery.task
... def appctx():
...     return 'test'

It’s also possible to provide a custom Celery application factory to the extension:

>>> from celery import Celery
>>> from flask_celeryext import AppContextTask
>>> def make_celery(app):
...     celery = Celery(app.import_name)
...     celery.flask_app = app
...     celery.Task = AppContextTask
...     return celery
>>> app = Flask('myapp')
>>> ext = FlaskCeleryExt(app, create_celery_app=make_celery)

Larger applications

In a front-end Flask application you will usually only need a minimal Celery application configured in order for Celery to know which broker to use etc.

Note

There is a difference with the Celery tutorial in Flask documentation. One should use BROKER_URL configuration option instead of CELERY_BROKER_URL.

This minimal application however does not need to load all tasks upfront, as especially for larger applications loading many tasks can cause startup time to increase significantly. The background worker on the other hand usually needs to load tasks upfront in order to know which tasks it can handle.

See https://flask-appfactory.readthedocs.io for a solution on using Flask-CeleryExt in larger applications.

Testing

Testing your celery tasks is rather easy. First ensure that you Celery is configured to execute tasks eagerly and stores results in local memory:

>>> app = Flask('myapp')
>>> app.config.update(dict(
...     CELERY_ALWAYS_EAGER=True,
...     CELERY_RESULT_BACKEND='cache',
...     CELERY_CACHE_BACKEND='memory',
...     CELERY_EAGER_PROPAGATES_EXCEPTIONS=True))
>>> celery = create_celery_app(app)

You can now create your task:

>>> from celery import current_task
>>> @celery.task(name='myapp.test_name')
... def test():
...     return current_task.name

And finally execute your task:

>>> r = test.delay()
>>> r.result
'myapp.test_name'

Changes

Version 0.3.0 (released 2017-03-24)

  • Adds support for Celery v4.

Version 0.2.2 (released 2016-11-07)

  • Forces celery version to v3.1-4.0 due to problem with 4.x.

Version 0.2.1 (released 2016-07-25)

Improved features

  • Improves documentation structure and its automatic generation.

Version 0.2.0 (released 2016-02-02)

Incompatible changes

  • Changes celery application creation to use the default current celery application instead creating a new celery application. This addresses an issue with tasks using the shared_task decorator and having Flask-CeleryExt initialized multiple times.

Version 0.1.0 (released 2015-08-17)

  • Initial public release

Contributing

Bug reports, feature requests, and other contributions are welcome. If you find a demonstrable problem that is caused by the code of this library, please:

  1. Search for already reported problems.
  2. Check if the issue has been fixed or is still reproducible on the latest master branch.
  3. Create an issue with a test case.

If you create a feature branch, you can run the tests to ensure everything is operating correctly:

$ ./run-tests.sh

You can also test your feature branch using Docker:

$ docker-compose build
$ docker-compose run --rm web /code/run-tests.sh

License

Flask-CeleryExt is free software; you can redistribute it and/or modify it under the terms of the Revised BSD License quoted below.

Copyright (C) 2015 CERN.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

Authors

Contact us at info@inveniosoftware.org