Code References

Welcome to DataDuct

Definition Parser

Script that parses the pipeline definition from the yaml schema

dataduct.definition_parser.activate_pipeline(etl)

Activate the pipeline that was created

Parameters:etl (EtlPipeline) – pipeline object that needs to be activated
dataduct.definition_parser.create_pipeline(definition)

Creates the pipeline and add the steps specified to the pipeline

Parameters:definition (dict) – YAML definition parsed from the datapipeline
dataduct.definition_parser.read_pipeline_definition(file_path)

Function reads the yaml pipeline definitions.

Function reads the yaml pipeline definitions. We also remove the variables key as that was only used for yaml placeholders.

Parameters:file_path (str) – Path to the pipeline definition.
Returns:parsed yaml definition as dictionary.
Return type:dict
Raises:ETLInputError – If file_path extention is not yaml
dataduct.definition_parser.validate_pipeline(etl, force_overwrite=False)

Validates the pipeline that was created

Parameters:
  • etl (EtlPipeline) – pipeline object that needs to be validated
  • force_overwrite (bool) – delete if a pipeline of same name exists