slippinj.filesystem package

Submodules

slippinj.filesystem.excel_writer module

class slippinj.filesystem.excel_writer.ExcelWriter(self_, *args, **kwargs)

Bases: object

Generates the Excel files

generate_excel_file(tables_information, db_name, location)

Generate all the excel file content given tables list and save it into a location :param tables_information: dict :param db_name: string :param location: string :return: boolean

slippinj.filesystem.filesystem module

class slippinj.filesystem.filesystem.Filesystem(self_, *args, **kwargs)

Bases: object

Manage the filesystem

find_properties_file(wf_folder)

Find all the properties file inside the given folder :param wf_folder: string :return: list

generate_tar_file(wf_folder, workflow_name=None)

Generate a tar file and compress inside all the workflow code, and returns the path to the newly created file :param wf_folder: string :param workflow_name: string :return: string

generate_temp_dir()

Generate a temporary directory and return the path to it :return: string

mkdir(output_dir)

Create a directory into filesystem :param output_dir: string :return: boolean

remove_temp_dir()

Remove the temporary directory created using self.generate_temp_dir

write_file(output_file, content)

Write given content into file :param output_file: string :param content: string

slippinj.filesystem.hdfs module

class slippinj.filesystem.hdfs.HDFSFilesystem(self_, *args, **kwargs)

Bases: object

Run HDFS commands inside the cluster

mkdir(path, cluster_id)

put(src, dest, cluster_id)

rmdir(path, cluster_id)

slippinj.filesystem.yaml_configuration module

class slippinj.filesystem.yaml_configuration.WorkflowsYamlConfigurationWriter(self_, *args, **kwargs)

Bases: object

Write the YAML configuration files needed to run the workflows

generate_yaml_files(injector, tables_information, db_driver, db_host, db_port, db_user, db_name, db_password, location=None)

Given tables and database connection information generate the YAML files and save them into provided location :param injector: Injector :param tables_information: dict :param db_driver: string :param db_host: string :param db_port: integer :param db_user: string :param db_name: string :param db_password: string :param location: string

class slippinj.filesystem.yaml_configuration.YamlConfiguration(self_, *args, **kwargs)

Bases: object

Process YAML configuration files

autodiscover_config_files(root_dir)

Find all the YAML files inside the given directory :param root_dir: string :return: list

read_config_file(config_file)

Read the given configuration file :param config_file: string :return: dict

read_multiple_config_files(configuration_files)

Given multiple YAML files read all and merge there content in the given order, overriding same configuration keys :param configuration_files: list :return: dict

Module contents