A job consists of a name, a node/node pool, a list of actions, a schedule, and an optional cleanup action. They are periodic events that do not interact with other jobs while running.
If all actions exit with status 0, the job has succeeded. If any action exists with a nonzero status, the job has failed.
If True run this job on each node in the node pool list. If a node appears more than once in the list, the job will be run on that node once for each appearance.
If False run this job on a random node from the node pool list. If a node appears more than once in the list, the job will be more likely to run on that node, proportionate to the number of appearances.
If node is not a node pool, this option has no effect.
A time interval (ex: “2 hours”) that limits the duration of each job run. If the job run is still running after this duration, all of it’s actions are sent SIGTERM.
Note: This requires an Action Runners to be configured. If action_runner is none max_runtime does nothing.
Actions consist primarily of a name and command. An action’s command is executed as soon as its dependencies (specified by requires) are satisfied. So if your job has 10 actions, 1 of which depends on the other 9, then Tron will launch the first 9 actions in parallel and run the last one when all have completed successfully.
If any action exits with nonzero status, the job will continue to run any actions which do not depend on the failed action.
jobs: - name: convert_logs node: node1 schedule: start_time: 04:00:00 actions: - name: verify_logs_present command: "ls /var/log/app/log_%(shortdate-1).txt" - name: convert_logs command: "convert_logs /var/log/app/log_%(shortdate-1).txt /var/log/app_converted/log_%(shortdate-1).txt" requires: [verify_logs_present]
Tron supports four methods for configuring the schedule of a job. Schedulers support a jitter parameter that allows them to vary their runtime by a random time delta.
Run the job every X seconds, minutes, hours, or days. The time expression is <interval> days|hours|minutes|seconds, where the units can be abbreviated.
schedule: "interval 20s"
schedule: type: "interval" value: "5 mins" jitter: "10 seconds" # Optional
schedule: type: "interval" value: "hourly"
Run the job on specific days at a specific time. The time expression is HH:MM:SS[ MTWRFSU].
schedule: "daily 04:00:00"
Short form with days:
schedule: "daily 04:00:00 MWF"
schedule: type: "daily" value: "07:00:00 MWF" jitter: "10 min" # Optional
Schedule a job using cron syntax. Tron supports predefined schedules, ranges, and lists for each field. It supports the L in day of month field only (which schedules the job on the last day of the month). Only one of the day fields (day of month and day of week) can have a value.
schedule: "cron */5 * * 7,8 *" # Every 5 minutes in July and August
schedule: "cron 0 3-6 * * *" # Every hour between 3am and 6am
schedule: # long form type: "cron" value: "30 4 L * *" # The last day of the month at 4:30am
More powerful version of the daily scheduler based on the one used by Google App Engine’s cron library. To use this scheduler, use a string in this format as the schedule:
("every"|ordinal) (days) ["of|in" (monthspec)] (["at"] HH:MM)
2nd,third mon,wed,thu of march 17:00 every monday at 09:00 1st monday of sep,oct,nov at 17:00 every day of oct at 00:00
In the config:
schedule: "every monday at 09:00"
schedule: type: "groc daily" value: "every day 11:22" jitter: "5 min"
Some system clocks are configured to track local time and may observe daylight savings time. For example, on November 6, 2011, 1 AM occurred twice. Prior to version 0.2.9, this would cause Tron to schedule a daily midnight job to be run an hour early on November 7, at 11 PM. For some jobs this doesn’t matter, but for jobs that depend on the availability of data for a day, it can cause a failure.
Similarly, some jobs on March 14, 2011 were scheduled an hour late.
To avoid this problem, set the Time Zone config variable. For example:
If a job is scheduled at a time that occurs twice, such as 1 AM on “fall back”, it will be run on the first occurrence of that time.
If a job is scheduled at a time that does not exists, such as 2 AM on “spring forward”, it will be run an hour later in the “new” time, in this case 3 AM. In the “old” time this is 2 AM, so from the perspective of previous jobs, it runs at the correct time.
In general, Tron tries to schedule a job as soon as is correct, and no sooner. A job that is schedule for 2:30 AM will not run at 3 AM on “spring forward” because that would be half an hour too soon from a pre-switch perspective (2 AM).
If you experience unexpected scheduler behavior, file an issue on Tron’s Github page.
Cleanup actions run after the job succeeds or fails. They are specified just like regular actions except that there is only one per job and it has no name or requirements list.
If your job creates shared resources that should be destroyed after a run regardless of success or failure, such as intermedmiate files or Amazon Elastic MapReduce job flows, you can use cleanup actions to tear them down.
The command context variable cleanup_job_status is provided to cleanup actions and has a value of SUCCESS or FAILURE depending on the job’s final state. For example:
- # ... cleanup_action: command: "python -m mrjob.tools.emr.job_flow_pool --terminate MY_POOL"
The following are the possible states for a Job and Job Run.
Job states are derived from the aggregate state of their actions. The following is a state diagram for an action.