rhessyscalibrator.calibration_runner.CalibrationRunnerQueue Class Reference

Abstract class for use with queue-based job management tools. More...

Inheritance diagram for rhessyscalibrator.calibration_runner.CalibrationRunnerQueue:

Public Member Functions
def	getRunCmd (self, args, kwargs)
	Get job submission command given selected options. More...
def	getRunStatusCmd (self, args, kwargs)
	Get job status command. More...
def	getRunCmdRegex (self)
	Get compiled regular expression for parsing run command output. More...
def	getRunStatusCmdRegex (self)
	Get compiled regular expression for parsing run status command output. More...
def	submitJob (self, job)
	Submit a job to the underlying queue system. More...
def	mapStatusCode (self, status_code)
	Map between status codes of the underlying queue system and calibrator status codes. More...
def	__init__ (self, basedir, session_id, queue, db_path, run_path, logger, restart_runs, submit_queue, polling_delay, mem_limit, max_active_jobs)
	Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called. More...
def	pollJobsStatus (self)
	Check status of jobs submitted. More...
def	run (self)
	Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process. More...
def	getRunCmd (self, args, kwargs)
	Get job submission command given selected options. More...
def	getRunStatusCmd (self, args, kwargs)
	Get job status command. More...
def	getRunCmdRegex (self)
	Get compiled regular expression for parsing run command output. More...
def	getRunStatusCmdRegex (self)
	Get compiled regular expression for parsing run status command output. More...
def	submitJob (self, job)
	Submit a job to the underlying queue system. More...
def	mapStatusCode (self, status_code)
	Map between status codes of the underlying queue system and calibrator status codes. More...
def	__init__ (self, basedir, session_id, queue, db_path, run_path, logger, restart_runs, submit_queue, polling_delay, mem_limit, max_active_jobs)
	Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called. More...
def	pollJobsStatus (self)
	Check status of jobs submitted. More...
def	run (self)
	Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process. More...
def	getRunCmd (self, args, kwargs)
	Get job submission command given selected options. More...
def	getRunStatusCmd (self, args, kwargs)
	Get job status command. More...
def	getRunCmdRegex (self)
	Get compiled regular expression for parsing run command output. More...
def	getRunStatusCmdRegex (self)
	Get compiled regular expression for parsing run status command output. More...
def	submitJob (self, job)
	Submit a job to the underlying queue system. More...
def	mapStatusCode (self, status_code)
	Map between status codes of the underlying queue system and calibrator status codes. More...
def	__init__ (self, basedir, session_id, queue, db_path, run_path, logger, restart_runs, submit_queue, polling_delay, mem_limit, max_active_jobs)
	Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called. More...
def	pollJobsStatus (self)
	Check status of jobs submitted. More...
def	run (self)
	Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process. More...
Public Member Functions inherited from rhessyscalibrator.calibration_runner.CalibrationRunner
def	__init__
def	__del__ (self)
def	run (self)
def	storeJobInDB (self, job)
	Store new job in ModelRunnerDB2. More...
def	jobCompleteCallback (self, args, kwargs)
	Called when a job is complete. More...
def	__init__
def	__del__ (self)
def	run (self)
def	storeJobInDB (self, job)
	Store new job in ModelRunnerDB2. More...
def	jobCompleteCallback (self, args, kwargs)
	Called when a job is complete. More...
def	__init__
def	__del__ (self)
def	run (self)
def	storeJobInDB (self, job)
	Store new job in ModelRunnerDB2. More...
def	jobCompleteCallback (self, args, kwargs)
	Called when a job is complete. More...

Public Attributes
	submit_queue
	mem_limit
	max_active_jobs
	run_cmd
	run_status_cmd
Public Attributes inherited from rhessyscalibrator.calibration_runner.CalibrationRunner
	basedir
	session_id
	queue
	db_path
	run_path
	logger
	restart_runs
	db
	numActiveJobs

Static Public Attributes
int	INIT_SLEEP_SECS = 15
int	JOB_STATUS_SLEEP_SECS = 60
int	JOB_SUBMIT_PENDING_THRESHOLD_SLEEP_SECS = 90
int	QUEUE_GET_TIMEOUT_SECS = 15
int	EXIT_SLEEP_SECS = 5

Detailed Description

Abstract class for use with queue-based job management tools.

Subclasses must be override the run() method to handle implementation- specific job queueing suystem. run methods should check that self.numActiveJobs never exceeds self.max_active_jobs.

Constructor & Destructor Documentation

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.__init__	(		self,
			basedir,
			session_id,
			queue,
			db_path,
			run_path,
			logger,
			restart_runs,
			submit_queue,
			polling_delay,
			mem_limit,
			max_active_jobs
	)

Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called.

Parameters

basedir	String representing the basedir of the calibration session
session_id	Integer representing the session ID of current calibration session
queue	Queue that jobs will be read from
db_path	String representing the path of sqlite DB to store the job (run) in
run_path	String representing the absolute path of the directory from which jobs should be run.
logger	logging.Logger to use to for debug messages
restart_runs	Boolean indicating that runs are to be restarted
submit_queue	String representing the name of the queue to submit jobs to
polling_delay	Integer representing the multiplier to applied to self.JOB_STATUS_SLEEP_SECS
mem_limit	Integer representing the memory limit for jobs. Units GB.
max_active_jobs	Integer representing the max active jobs permitted

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.__init__	(		self,
			basedir,
			session_id,
			queue,
			db_path,
			run_path,
			logger,
			restart_runs,
			submit_queue,
			polling_delay,
			mem_limit,
			max_active_jobs
	)

Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called.

Parameters

basedir	String representing the basedir of the calibration session
session_id	Integer representing the session ID of current calibration session
queue	Queue that jobs will be read from
db_path	String representing the path of sqlite DB to store the job (run) in
run_path	String representing the absolute path of the directory from which jobs should be run.
logger	logging.Logger to use to for debug messages
restart_runs	Boolean indicating that runs are to be restarted
submit_queue	String representing the name of the queue to submit jobs to
polling_delay	Integer representing the multiplier to applied to self.JOB_STATUS_SLEEP_SECS
mem_limit	Integer representing the memory limit for jobs. Units GB.
max_active_jobs	Integer representing the max active jobs permitted

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.__init__	(		self,
			basedir,
			session_id,
			queue,
			db_path,
			run_path,
			logger,
			restart_runs,
			submit_queue,
			polling_delay,
			mem_limit,
			max_active_jobs
	)

Concrete classes must use getRunCmd() and getRunStatusCmd() to initialize self.run_cmd and self.run_status_cmd before the run() method can be called.

Parameters

basedir	String representing the basedir of the calibration session
session_id	Integer representing the session ID of current calibration session
queue	Queue that jobs will be read from
db_path	String representing the path of sqlite DB to store the job (run) in
run_path	String representing the absolute path of the directory from which jobs should be run.
logger	logging.Logger to use to for debug messages
restart_runs	Boolean indicating that runs are to be restarted
submit_queue	String representing the name of the queue to submit jobs to
polling_delay	Integer representing the multiplier to applied to self.JOB_STATUS_SLEEP_SECS
mem_limit	Integer representing the memory limit for jobs. Units GB.
max_active_jobs	Integer representing the max active jobs permitted

Member Function Documentation

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmd	(		self,
			args,
			kwargs
	)

Get job submission command given selected options.

Returns

String representing job submission command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmd	(		self,
			args,
			kwargs
	)

Get job submission command given selected options.

Returns

String representing job submission command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmd	(		self,
			args,
			kwargs
	)

Get job submission command given selected options.

Returns

String representing job submission command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmdRegex

(

self

)

Get compiled regular expression for parsing run command output.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmdRegex

(

self

)

Get compiled regular expression for parsing run command output.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunCmdRegex

(

self

)

Get compiled regular expression for parsing run command output.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmd	(		self,
			args,
			kwargs
	)

Get job status command.

Returns

String representing job status command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmd	(		self,
			args,
			kwargs
	)

Get job status command.

Returns

String representing job status command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmd	(		self,
			args,
			kwargs
	)

Get job status command.

Returns

String representing job status command

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmdRegex

(

self

)

Get compiled regular expression for parsing run status command output.

Concrete classes must return a regex that matches the job ID in group 1 and the job status in group 2.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmdRegex

(

self

)

Get compiled regular expression for parsing run status command output.

Concrete classes must return a regex that matches the job ID in group 1 and the job status in group 2.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.getRunStatusCmdRegex

(

self

)

Get compiled regular expression for parsing run status command output.

Concrete classes must return a regex that matches the job ID in group 1 and the job status in group 2.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.mapStatusCode	(		self,
			status_code
	)

Map between status codes of the underlying queue system and calibrator status codes.

Parameters

status_code

String representing status code of underlying queue system

Returns

String representing calibrator status code

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.mapStatusCode	(		self,
			status_code
	)

Map between status codes of the underlying queue system and calibrator status codes.

Parameters

status_code

String representing status code of underlying queue system

Returns

String representing calibrator status code

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.mapStatusCode	(		self,
			status_code
	)

Map between status codes of the underlying queue system and calibrator status codes.

Parameters

status_code

String representing status code of underlying queue system

Returns

String representing calibrator status code

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.pollJobsStatus

(

self

)

Check status of jobs submitted.

Will update status for each job (run) in the DB.

Returns

Tuple (integer, integer, integer) that represent the number of pending, running, and retired jobs for the current invocation.

Exception if qstat output is not what was expected.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.pollJobsStatus

(

self

)

Check status of jobs submitted.

Will update status for each job (run) in the DB.

Returns

Tuple (integer, integer, integer) that represent the number of pending, running, and retired jobs for the current invocation.

Exception if qstat output is not what was expected.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.pollJobsStatus

(

self

)

Check status of jobs submitted.

Will update status for each job (run) in the DB.

Returns

Tuple (integer, integer, integer) that represent the number of pending, running, and retired jobs for the current invocation.

Exception if qstat output is not what was expected.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.run

(

self

)

Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.run

(

self

)

Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.run

(

self

)

Method to be run in a consumer thread/process to launch a run submitted by procuder thread/process.

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.submitJob	(		self,
			job
	)

Submit a job to the underlying queue system.

Note

Will add job to calibration DB.

Parameters

job	model_runner_db.ModelRun representing the job to run

Exception if run status command output is not what was expected

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.submitJob	(		self,
			job
	)

Submit a job to the underlying queue system.

Note

Will add job to calibration DB.

Parameters

job	model_runner_db.ModelRun representing the job to run

Exception if run status command output is not what was expected

def rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.submitJob	(		self,
			job
	)

Submit a job to the underlying queue system.

Note

Will add job to calibration DB.

Parameters

job	model_runner_db.ModelRun representing the job to run

Exception if run status command output is not what was expected

Member Data Documentation

int rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.EXIT_SLEEP_SECS = 5

static

int rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.INIT_SLEEP_SECS = 15

static

int rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.JOB_STATUS_SLEEP_SECS = 60

static

int rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.JOB_SUBMIT_PENDING_THRESHOLD_SLEEP_SECS = 90

static

rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.max_active_jobs

rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.mem_limit

int rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.QUEUE_GET_TIMEOUT_SECS = 15

static

rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.run_cmd

rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.run_status_cmd

rhessyscalibrator.calibration_runner.CalibrationRunnerQueue.submit_queue

---

The documentation for this class was generated from the following file:

• build/lib/rhessyscalibrator/calibration_runner.py