Astro Python SDK for ETL
Use the Astro Python SDK to implement ELT use cases in Airflow.
When designing your data pipelines, you may encounter use cases that require more complex task flows than "Task A > Task B > Task C". For example, you may have a use case where you need to decide between multiple tasks to execute based on the results of an upstream task. Or you may have a case where part of your pipeline should only run under certain external conditions. Fortunately, Airflow has multiple options for building conditional logic and/or branching into your DAGs.
Access the Airflow context in your tasks.
How to implement dependencies between your Airflow DAGs.
Custom hooks and operators
How to correctly import custom hooks and operators.
When you're using a data orchestration tool, how do you know when something has gone wrong? Airflow users can check the Airflow UI to determine the status of their DAGs, but this is an inefficient way of managing errors systematically, especially if certain failures need to be addressed promptly or by multiple team members. Fortunately, Airflow has built-in notification mechanisms that can be leveraged to configure error notifications in a way that works for your organization.
DAG writing best practices
Because Airflow is 100% code, knowing the basics of Python is all it takes to get started writing DAGs. However, writing DAGs that are efficient, secure, and scalable requires some Airflow-specific finesse. In this guide, you'll learn how you can develop DAGs that make the most of what Airflow has to offer.
Troubleshoot Airflow DAGs
How to dynamically create tasks at runtime in your Airflow DAGs.
Templating allows you to pass dynamic information into task instances at runtime. For example, you can run the following command to print the day of the week every time you run a task:
Create and use DAG and task-level params in Airflow.
Pass data between tasks
Sharing data between tasks is a very common use case in Airflow. If you've been writing DAGs, you probably know that breaking them up into smaller tasks is a best practice for debugging and recovering quickly from failures. What do you do when one of your downstream tasks requires metadata about an upstream task, or processes the results of the task immediately before it?
Rerun DAGs and tasks
How to configure retries, catchup, backfill, and clear task instances in Airflow.
Use SubDAGs to create Airflow modular workflows.
TaskFlow API & decorators
An overview of Airflow decorators and how they can improve the DAG authoring experience.
Airflow task groups are a tool to organize tasks into groups within your DAGs. Using task groups allows you to: