Astro CI/CD templates for GitLab
Use the following CI/CD templates to automate deploying Apache Airflow DAGs from a GitLab repository to Astro.
The templates for GitLab use the image-only deploy process. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation.
If you use the DAG-only deploy feature on Astro and are interested in a DAG-based CI/CD template, see Template overview to configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.
Prerequisites
- An Astro project hosted in a GitLab repository.
- An Astro Deployment.
- Either a Deployment API key ID and secret, a Workspace API token, or an Organization API token.
Each CI/CD template implementation might have additional requirements.
Image-only templates
Single branch implementation
Use this template to push code to from a GitLab repository to Astro.
In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Go to the Editor option in your project's CI/CD section and commit the following:
---
astro_deploy:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- main
Multiple branch implementation
Use this template to push code to both a development and a production Deployment on Astro using GitLab.
In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
When you create environment variables that will be used in multiple branches, you might want to protect where you use them. Otherwise, uncheck the Protect variable
flag when you create the variable in GitLab. For more information on protected branches, see GitLab documentation.
Go to the Editor option in your project's CI/CD section and commit the following:
---
astro_deploy_dev:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${DEV_ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${DEV_ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
- apk add jq
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- dev
astro_deploy_prod:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${PROD_ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${PROD_ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
- apk add jq
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- main
DAG-based templates
The DAG-based template uses the --dags
flag in the Astro CLI to push DAG changes to Astro. These CI/CD pipelines deploy your DAGs only when files in your dags
folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy DAGs only.
Single branch implementation
Use this template to push code to from a GitLab repository to Astro.
In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Go to the Editor option in your project's CI/CD section and commit the following:
publish-dags:
stage: publish
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
DAG_FOLDER: "/dags"
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
- apk add jq
script:
- (curl -sSL install.astronomer.io | bash -s)
- files=$(git diff --name-only HEAD^..HEAD)
- dags_only=1
- for file in $files; do
- if [[ $file != "$DAG_FOLDER"* ]]; then
- echo "$file is not a dag, triggering a full image build"
- dags_only=0
- break
- fi
- done
- if [ $dags_only == 1 ]
- then
- astro deploy --dags
- fi
- if [ $dags_only == 0 ]
- then
- astro deploy -f
- fi
only:
- main