“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”- Abraham Lincoln
FireWorks is a free, open-source code for defining, managing, and executing scientific workflows. It can be used to automate calculations over arbitrary computing resources, including those that have a queueing system.
Some features that distinguish FireWorks are dynamic workflows, failure-detection routines, and built-in tools and execution modes for running high-throughput computations at large computing centers.
FireWorks is intended to be a friendly workflow software that is easy to get started with, but flexible enough to handle complicated use cases.
Some (but not all) of its features include:
While FireWorks provides many features, its basic operation is simple. You can run FireWorks on a single laptop or at a supercomputing center.
There are essentially just two components of a FireWorks installation:
The basic infrastructure looks like this:
The components are largely decoupled, which makes FireWorks easier to use. End users can add new workflows to the LaunchPad without worrying about the details of how and where the workflows will be run (unless they really want to tailor the details of job execution). This keeps the workflow specifications lightweight, tidy, and easy to learn and use (if you’ve ever seen lengthy XML-based specifications in other workflow software, you’ll notice the difference in FireWorks right away).
On the opposite end, administrators can configure worker computers without worrying about where workflows are coming from or what they look like (although you can assign jobs to certain resources if desired). Running on a heterogeneous set of worker computers is simple because essentially the same code is used internally by FireWorks for running on simple workstations or a large supercomputing center, submitting to a traditional or web-based queue system, or packing together many jobs into a single queue submission.
Workflows in FireWorks are made up of three main components:
Between FireWorks, you can return a FWAction that can store data or modify the Workflow depending on the output (e.g., pass data to the next step, cancel the remaining parts of the Workflow, or even add new FireWorks that are defined within the object).
The FireWorks tutorials and FW design tips explain how to connect these components to achieve the desired behavior.
To get a first glimpse of FireWorks, we suggest that you follow our installation and quickstart tutorials.
After completing the quickstart, we suggest that you follow our core tutorials that cover the primary features of FireWorks. Depending on your application, you may not need to complete all the tutorials.
This series of tutorials cover how to manage your jobs and deploy FireWorks in a production environment.
A paper for FireWorks is in preparation. In the meantime, you can cite FireWorks as follows:
FireWorks workflow software, http://pythonhosted.org/FireWorks.
Want to see something added or changed? There are many ways to make that a reality! Some ways to get involved are:
Please submit questions, issues / bug reports, and all other communication through the FireWorks Google Groups.
The list of contributors to FireWorks can be found here.