As a Data Engineer, Create a New Pipeline
Pipelines are structured, automated data flow procedures that can be scheduled or triggered.
To create a new pipeline, create a new file in the pipeline repository under the `orchestration` folder, named `pipeline_<name>.py`. Or you can copy the existing `pipeline_main.py` to the new filename and refactor it to your needs.
The important pieces of a pipeline are:
- The schedule and flow name (flow name should match the file name, as in `pipeline_<name>.py`
- Load the settings as the first task, so that your pipeline has access to things like security credentials and configuration information
- Configure the new flow by using the Datateer PipelineFlow:
With those pieces in place you can add whatever tasks necessary. These flows are Prefect flows, and PipelineFlow is simply a subclass of Prefect's Flow class.