slippinj.emr package

Submodules

slippinj.emr.cluster module

class slippinj.emr.cluster.EmrCluster(self_, *args, **kwargs)[source]

Bases: object

Handle all EMR cluster information and configuration

exec_command(command, cluster_id, stop_on_error=False)[source]

Execute given command in the master of the selected cluster :param command: string :param cluster_id: string :param stop_on_error: boolean :return: string

get_cluster_information(cluster_id)[source]

Get the cluster information from the AWS API :param cluster_id: string :return: dict

get_pem_path(cluster_id)[source]

Get the path to the private key file to connect to the cluster :param cluster_id: string :return: string

open_sftp(cluster_id)[source]

Open an SFTP connection to the given cluster :param cluster_id: string :return: SFTP

slippinj.emr.deploy module

class slippinj.emr.deploy.EmrDeploy(self_, *args, **kwargs)[source]

Bases: object

Execute all the deploy process inside an EMR cluster

run_properties_file(properties_file, cluster_id)[source]

Try to execute the given properties file in the selected cluster. It returns the job id so can check the output log in Oozie :param properties_file: string :param cluster_id: string :return: string

upload_code(wf_folder, cluster_id, hdfs_deploy_folder, workflow_name=None)[source]

Upload given workflow code to HDFS connecting to given cluster :param wf_folder: string :param cluster_id: string :param hdfs_deploy_folder: string :param workflow_name: string :return: boolean

Module contents