Create an Astro project
Overview
To run Airflow pipelines on Astro, you first need to create an Astro project. An Astro project contains the set of files necessary to run Airflow, including dedicated folders for your DAG files, plugins, and dependencies. Once you've tested these files locally, the Astro project structure makes it easy to deploy your pipelines to Astro.
This guide provides instructions for creating a new Astro project, as well as information about the default Astro project structure.
Prerequisites
To create an Astro project, you need:
Step 1: Create an Astro project
To create a new Astro project:
Create a new directory for your Astro project:
mkdir <your-astro-project-name>
Open the directory:
cd <your-astro-project-name>
Run the following Astro CLI command to initialize an Astro project in the directory:
astrocloud dev init
This command generates the following files in the directory:
.
├── dags # Where your DAGs go
│ └── example-dag-basic.py # Example DAG that showcases a simple ETL data pipeline
| └── example-dag-advanced.py # Example DAG that showcases more advanced Airflow features, such as the TaskFlow API
├── Dockerfile # For the Astro Runtime Docker image, environment variables, and overrides
├── include # For any other files you'd like to include
├── plugins # For any custom or community Airflow plugins
| └── example-plugin.py
├── tests # For any DAG unit test files to be run with pytest
| └── test_dag_integrity.py # Test that checks for basic errors in your DAGs
├── airflow_settings.yaml # For your Airflow Connections, Variables and Pools (local only)
├── packages.txt # For OS-level packages
└── requirements.txt # For Python packagesThis set of files will build into a Docker image that you can both run on your local machine and deploy to Astro.
Astro Runtime
Your Dockerfile
includes a reference to Astro Runtime. Packaged into a Debian-based Docker image, Astro Runtime extends the Apache Airflow open source project to provide you with differentiated functionality that centers around reliability, efficiency, and performance. For more information on what's included in Runtime and how it's versioned, see Runtime Versioning.
By default, the Docker image in your Dockerfile is:
FROM quay.io/astronomer/astro-runtime:5.0.1
Step 2: Build Your Project Locally
To confirm that you successfully initialized an Astro project, run the following command from your project directory:
astrocloud dev start
This command builds your project and spins up 3 Docker containers on your machine, each for a different Airflow component:
- Postgres: Airflow's metadata database
- Webserver: The Airflow component responsible for rendering the Airflow UI
- Scheduler: The Airflow component responsible for monitoring and triggering tasks
- Triggerer: The Airflow component responsible for running Triggers and signaling tasks to resume when their conditions have been met. The Triggerer is used exclusively for tasks that are run with deferrable operators
As your project builds locally, you should see the following output:
% astrocloud dev start
Env file ".env" found. Loading...
Sending build context to Docker daemon 10.75kB
Step 1/1 : FROM quay.io/astronomer/astro-runtime:5.0.1
# Executing 5 build triggers
---> Using cache
---> Using cache
---> Using cache
---> Using cache
---> Using cache
---> 5160cfd00623
Successfully built 5160cfd00623
Successfully tagged astro-trial_705330/airflow:latest
INFO[0000] [0/4] [postgres]: Starting
INFO[0002] [1/4] [postgres]: Started
INFO[0002] [1/4] [scheduler]: Starting
INFO[0003] [2/4] [scheduler]: Started
INFO[0003] [2/4] [webserver]: Starting
INFO[0004] [3/4] [webserver]: Started
INFO[0003] [3/4] [triggerer]: Starting
INFO[0004] [4/4] [triggerer]: Started
Airflow Webserver: http://localhost:8080
Postgres Database: localhost:5432/postgres
The default credentials are admin:admin
info
By default, the Astro CLI uses port 8080
for the Airflow Webserver and port 5432
for the Airflow metadata database. If these ports are already in use on your local machine, you can change the default ports for these components by following these steps:
In your Astro project directory, open
.astrocloud/config.yaml
. This file might be hidden in graphical file browsers. You can show hidden files using⌘ + Shift + .
on Mac or by selecting View > Hidden items in Windows file explorer.Specify alternative ports for your Webserver and/or metadata database in
config.yaml
. For example, to use8081
for your Webserver port and5435
for your database port, you would specify the following:project:
name: <your-directory-name>
webserver:
port: 8081
postgres:
port: 5435Run
astrocloud dev restart
to rebuild and rerun your project.
Step 3: Access the Airflow UI
Once your project builds successfully, you can access the Airflow UI by going to http://localhost:8080/
and logging in with admin
for both your username and password.
info
It might take a few minutes for the Airflow UI to be available. As you wait for the Webserver container to start up, you may need to refresh your browser.
After logging in, you should see the DAGs from your dags
directory in the Airflow UI.

Next Steps
Running your project locally is the best way to test your DAGs before pushing them to Astro. For more information on running a local Airflow environment, read: