Code References¶
Welcome to DataDuct
Subpackages and Modules¶
Definition Parser¶
Script that parses the pipeline definition from the yaml schema
- dataduct.definition_parser.activate_pipeline(etl)¶
Activate the pipeline that was created
Parameters: etl (EtlPipeline) – pipeline object that needs to be activated
- dataduct.definition_parser.create_pipeline(definition)¶
Creates the pipeline and add the steps specified to the pipeline
Parameters: definition (dict) – YAML definition parsed from the datapipeline
- dataduct.definition_parser.read_pipeline_definition(file_path)¶
Function reads the yaml pipeline definitions.
Function reads the yaml pipeline definitions. We also remove the variables key as that was only used for yaml placeholders.
Parameters: file_path (str) – Path to the pipeline definition. Returns: parsed yaml definition as dictionary. Return type: dict Raises: ETLInputError – If file_path extention is not yaml
- dataduct.definition_parser.validate_pipeline(etl, force_overwrite=False)¶
Validates the pipeline that was created
Parameters: - etl (EtlPipeline) – pipeline object that needs to be validated
- force_overwrite (bool) – delete if a pipeline of same name exists