Skip to main content

Astro CI/CD templates for GitLab

Use the following CI/CD templates to automate deploying Apache Airflow DAGs from a GitLab repository to Astro.

The templates for GitLab use the image-only deploy process. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation.

If you use the DAG-only deploy feature on Astro and are interested in a DAG-based CI/CD template, see Template overview to configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

Each CI/CD template implementation might have additional requirements.

Image-only templates

Single branch implementation

Use this template to push code to from a GitLab repository to Astro.

  1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Go to the Editor option in your project's CI/CD section and commit the following:

    ---
    astro_deploy:
    stage: deploy
    image: docker:latest
    services:
    - docker:dind
    variables:
    ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
    ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
    before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f
    only:
    - main

Multiple branch implementation

Use this template to push code to both a development and a production Deployment on Astro using GitLab.

  1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

    • DEV_ASTRONOMER_KEY_ID = <your-dev-key-id>
    • DEV_ASTRONOMER_KEY_SECRET = <your-dev-key-secret>
    • PROD_ASTRONOMER_KEY_ID = <your-prod-key-id>
    • PROD_ASTRONOMER_KEY_SECRET = <your-prod-key-secret>
caution

When you create environment variables that will be used in multiple branches, you might want to protect where you use them. Otherwise, uncheck the Protect variable flag when you create the variable in GitLab. For more information on protected branches, see GitLab documentation.

  1. Go to the Editor option in your project's CI/CD section and commit the following:

    ---
    astro_deploy_dev:
    stage: deploy
    image: docker:latest
    services:
    - docker:dind
    variables:
    ASTRONOMER_KEY_ID: ${DEV_ASTRONOMER_KEY_ID}
    ASTRONOMER_KEY_SECRET: ${DEV_ASTRONOMER_KEY_SECRET}
    before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    - apk add jq
    script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f
    only:
    - dev

    astro_deploy_prod:
    stage: deploy
    image: docker:latest
    services:
    - docker:dind
    variables:
    ASTRONOMER_KEY_ID: ${PROD_ASTRONOMER_KEY_ID}
    ASTRONOMER_KEY_SECRET: ${PROD_ASTRONOMER_KEY_SECRET}
    before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    - apk add jq
    script:
    - (curl -sSL install.astronomer.io | bash -s)
    - astro deploy -f
    only:
    - main

DAG-based templates

The DAG-based template uses the --dags flag in the Astro CLI to push DAG changes to Astro. These CI/CD pipelines deploy your DAGs only when files in your dags folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy DAGs only.

Single branch implementation

Use this template to push code to from a GitLab repository to Astro.

  1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Go to the Editor option in your project's CI/CD section and commit the following:

    publish-dags:
    stage: publish
    image: docker:latest
    services:
    - docker:dind
    variables:
    ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
    ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
    DAG_FOLDER: "/dags"
    before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    - apk add jq
    script:
    - (curl -sSL install.astronomer.io | bash -s)
    - files=$(git diff --name-only HEAD^..HEAD)
    - dags_only=1
    - for file in $files; do
    - if [[ $file != "$DAG_FOLDER"* ]]; then
    - echo "$file is not a dag, triggering a full image build"
    - dags_only=0
    - break
    - fi
    - done
    - if [ $dags_only == 1 ]
    - then
    - astro deploy --dags
    - fi
    - if [ $dags_only == 0 ]
    - then
    - astro deploy -f
    - fi
    only:
    - main

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.