Astro CI/CD templates for GitLab

Use the following CI/CD templates to automate deploying Apache Airflow DAGs from a GitLab repository to Astro.

The templates for GitLab use the image-only deploy process. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation.

If you use the DAG-only deploy feature on Astro and are interested in a DAG-based CI/CD template, see Template overview to configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a GitLab repository.
• An Astro Deployment.
• Either a Deployment API key ID and secret, a Workspace API token, or an Organization API token.

Each CI/CD template implementation might have additional requirements.

Image-only templates

Single branch implementation

Use this template to push code to from a GitLab repository to Astro.

1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

   • ASTRONOMER_KEY_ID = <your-key-id>
   • ASTRONOMER_KEY_SECRET = <your-key-secret>

2. Go to the Editor option in your project's CI/CD section and commit the following:

   ---
    astro_deploy:
      stage: deploy
      image: docker:latest
      services:
       - docker:dind
      variables:
         ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
         ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
      before_script:
       - apk add --update curl && rm -rf /var/cache/apk/*
       - apk add bash
      script:
       - (curl -sSL install.astronomer.io | bash -s)
       - astro deploy -f
      only:
       - main

Multiple branch implementation

Use this template to push code to both a development and a production Deployment on Astro using GitLab.

1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

   • DEV_ASTRONOMER_KEY_ID = <your-dev-key-id>
   • DEV_ASTRONOMER_KEY_SECRET = <your-dev-key-secret>
   • PROD_ASTRONOMER_KEY_ID = <your-prod-key-id>
   • PROD_ASTRONOMER_KEY_SECRET = <your-prod-key-secret>

caution

When you create environment variables that will be used in multiple branches, you might want to protect where you use them. Otherwise, uncheck the Protect variable flag when you create the variable in GitLab. For more information on protected branches, see GitLab documentation.

2. Go to the Editor option in your project's CI/CD section and commit the following:

   ---
      astro_deploy_dev:
        stage: deploy
        image: docker:latest
        services:
          - docker:dind
        variables:
            ASTRONOMER_KEY_ID: ${DEV_ASTRONOMER_KEY_ID}
            ASTRONOMER_KEY_SECRET: ${DEV_ASTRONOMER_KEY_SECRET}
        before_script:
          - apk add --update curl && rm -rf /var/cache/apk/*
          - apk add bash
          - apk add jq
        script:
          - (curl -sSL install.astronomer.io | bash -s)
          - astro deploy -f
        only:
          - dev
   
      astro_deploy_prod:
        stage: deploy
        image: docker:latest
        services:
          - docker:dind
        variables:
            ASTRONOMER_KEY_ID: ${PROD_ASTRONOMER_KEY_ID}
            ASTRONOMER_KEY_SECRET: ${PROD_ASTRONOMER_KEY_SECRET}
        before_script:
          - apk add --update curl && rm -rf /var/cache/apk/*
          - apk add bash
          - apk add jq
        script:
          - (curl -sSL install.astronomer.io | bash -s)
          - astro deploy -f
        only:
          - main

DAG-based templates

The DAG-based template uses the --dags flag in the Astro CLI to push DAG changes to Astro. These CI/CD pipelines deploy your DAGs only when files in your dags folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy DAGs only.

Single branch implementation

Use this template to push code to from a GitLab repository to Astro.

1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

   • ASTRONOMER_KEY_ID = <your-key-id>
   • ASTRONOMER_KEY_SECRET = <your-key-secret>

2. Go to the Editor option in your project's CI/CD section and commit the following:

   publish-dags:
     stage: publish
     image: docker:latest
     services:
      - docker:dind
     variables:
       ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
       ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
       DAG_FOLDER: "/dags"
     before_script:
       - apk add --update curl && rm -rf /var/cache/apk/*
       - apk add bash
       - apk add jq
     script:
       - (curl -sSL install.astronomer.io | bash -s)
       - files=$(git diff --name-only HEAD^..HEAD)
       - dags_only=1
       - for file in $files; do
       -  if [[ $file != "$DAG_FOLDER"* ]]; then
       -    echo "$file is not a dag, triggering a full image build"
       -    dags_only=0
       -    break
       -  fi
       - done
       - if [ $dags_only == 1 ]
       - then
       -   astro deploy --dags
       - fi
       - if [ $dags_only == 0 ]
       - then
       -   astro deploy -f
       - fi
     only:
       - main