Code References

Welcome to DataDuct

Subpackages and Modules

• ETLPipeline
• dataduct.pipeline package
• dataduct.s3 package
• dataduct.steps package
• dataduct.tests package
• dataduct.utils package

Definition Parser

Script that parses the pipeline definition from the yaml schema

dataduct.definition_parser.activate_pipeline(etl)

Activate the pipeline that was created

Parameters:etl (EtlPipeline) – pipeline object that needs to be activated

dataduct.definition_parser.create_pipeline(definition)

Creates the pipeline and add the steps specified to the pipeline

Parameters:definition (dict) – YAML definition parsed from the datapipeline

dataduct.definition_parser.read_pipeline_definition(file_path)

Function reads the yaml pipeline definitions.

Function reads the yaml pipeline definitions. We also remove the variables key as that was only used for yaml placeholders.

Parameters:file_path (str) – Path to the pipeline definition. Returns:parsed yaml definition as dictionary. Return type:dict Raises:ETLInputError – If file_path extention is not yaml

dataduct.definition_parser.validate_pipeline(etl, force_overwrite=False)

Validates the pipeline that was created

Parameters:

• etl (EtlPipeline) – pipeline object that needs to be validated
• force_overwrite (bool) – delete if a pipeline of same name exists