Orchestrate Redshift queries from your Airflow DAGs.
Follow a step-by-step tutorial for using Airflow to orchestrate the training and testing of a SageMaker model.
How to produce to and consume from Kafka topics using the Kafka Airflow provider
Azure Container Instances
Learn how to orchestrate containers with Azure Container Instances from your Airflow DAGs.
Azure Data Explorer
Learn how to orchestrate Azure Data Explorer queries with your Apache Airflow DAGs.
Azure Data Factory
Learn how to orchestrate remote jobs in Azure Data Factory with your Apache Airflow DAGs.
Orchestrate Databricks jobs with your Airflow DAGs.
dbt Cloud is a managed service that provides a hosted architecture to run dbt, a tool that helps you build interdependent SQL models for in-warehouse data transformation.
dbt Core is an open-source library for analytics engineering that helps users build interdependent SQL models for in-warehouse data transformation, using ephemeral compute of data warehouses.
DuckDB is an open-source in-process SQL OLAP database management system. It allows you to run complex queries on relational datasets using either local, file-based DuckDB instances, or the cloud service MotherDuck. The ability to create a local DuckDB instance is useful for testing complex Airflow pipelines without the need to connect to a remote database.
Learn how to orchestrate Fivetran syncs using Airflow
Orchestrate Great Expectations data quality checks with your Airflow DAGs.
Run a parameterized Jupyter notebook using Airflow and the Astro CLI.
Use OpenLineage and Marquez to get lineage metadata locally from your Airflow DAGs.
How to use MLflow with Airflow in three different ways.
Learn how to load data into MongoDB with your Apache Airflow DAGs.
Get enhanced observability and compute savings while orchestrating Snowflake jobs from your Airflow DAGs.
Learn how to orchestrate Soda Core data quality checks with your Airflow DAGs.
Orchestrate remote jobs in Talend with your Apache Airflow DAGs.
Weights and Biases
Learn how to use Airflow and Weights and Biases to manage and visualize your ML model lifecycle.