Welcome to DataDuct
Subpackages and Modules¶
Script that parses the pipeline definition from the yaml schema
Activate the pipeline that was created
Parameters: etl (EtlPipeline) – pipeline object that needs to be activated
Creates the pipeline and add the steps specified to the pipeline
Parameters: definition (dict) – YAML definition parsed from the datapipeline
Function reads the yaml pipeline definitions.
Function reads the yaml pipeline definitions. We also remove the variables key as that was only used for yaml placeholders.
Parameters: file_path (str) – Path to the pipeline definition. Returns: parsed yaml definition as dictionary. Return type: dict Raises: ETLInputError – If file_path extention is not yaml
- dataduct.definition_parser.validate_pipeline(etl, force_overwrite=False)¶
Validates the pipeline that was created
- etl (EtlPipeline) – pipeline object that needs to be validated
- force_overwrite (bool) – delete if a pipeline of same name exists