Skip to main content

Automate code deploys with CI/CD

This guide provides setup steps for configuring a CI/CD pipeline to deploy DAGs on Astro.

There are many benefits to deploying DAGs and other changes to Airflow via a CI/CD workflow. Specifically, you can:

  • Deploy new and updated DAGs in a way that streamlines the development process amongst team members.
  • Decrease the maintenance cost of integrating changes, allowing your team to quickly respond in case of an error or failure.
  • Enforce continuous, automating testing, which increases code quality and protects your DAGs in production.

Prerequisites

CI/CD templates

The following section provides basic templates for configuring individual CI pipelines using popular CI/CD tools. Each template can be implemented as-is to produce a simple CI/CD pipeline, but Astronomer recommends reconfiguring the templates to work with your own directory structures, workflows, and best practices. More templates are coming soon.

At a high level, these CI/CD pipelines will:

  • Access Deployment API key credentials. These credentials must be set as OS-level environment variables named ASTRONOMER_KEY_ID and ASTRONOMER_KEY_SECRET.
  • Install the latest version of the Astro CLI.
  • Run astro deploy. This creates a Docker image for your Astro project, authenticates to Astro using your Deployment API key, and pushes the image to your Deployment.

This workflow is equivalent to the following bash script:

# Set Deployment API key credentials as environment variables
$ export ASTRONOMER_KEY_ID="<your-api-key-id>"
$ export ASTRONOMER_KEY_SECRET="<your-api-key-secret>"

# Install the latest version of Astro CLI
$ curl -sSL install.astronomer.io | sudo bash -s

# Build your Astro project into a Docker image and push the image to your Deployment
$ astro deploy
info

The following templates use Astro CLI v1.0+ to deploy via CI/CD. These templates will not work if you use the astrocloud executable. To upgrade, see Install the Astro CLI.

GitHub Actions

To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:

  1. Set the following as GitHub secrets:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

    name: Astronomer CI - Deploy code

    on:
    push:
    branches:
    - main

    env:
    ## Sets Deployment API key credentials as environment variables
    ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
    ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}

    jobs:
    build:
    runs-on: ubuntu-latest
    steps:
    - name: checkout repo
    uses: actions/checkout@v2.3.4
    - name: Deploy to Astro
    run: |
    curl -sSL install.astronomer.io | sudo bash -s
    astro deploy

Jenkins

To automate code deploys to a single Deployment using Jenkins, complete the following setup in a Git-based repository hosting an Astro project:

  1. In your Jenkins pipeline configuration, add the following parameters:

    • ASTRONOMER_KEY_ID: Your Deployment API key ID
    • ASTRONOMER_KEY_SECRET: Your Deployment API key secret
    • ASTRONOMER_DEPLOYMENT_ID: The Deployment ID of your production deployment

    Be sure to set the values for your API credentials as secret.

  2. At the root of your Git repository, add a Jenkinsfile that includes the following script:

    pipeline {
    agent any
    stages {
    stage('Deploy to Astronomer') {
    when {
    expression {
    return env.GIT_BRANCH == "origin/main"
    }
    }
    steps {
    script {
    sh 'curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.6.0/astro_1.6.0_linux_amd64.tar.gz'
    sh 'tar xzf astro_1.6.0_linux_amd64.tar.gz'
    sh "./astro deploy ${ASTRONOMER_DEPLOYMENT_ID} -f"
    }
    }
    }
    }
    post {
    always {
    cleanWs()
    }
    }
    }

    This Jenkinsfile triggers a code push to Astro every time a commit or pull request is merged to the main branch of your repository.

AWS CodeBuild

To automate code deploys to a single Deployment using AWS CodeBuild, complete the following setup in a Git-based repository hosting an Astro project:

  1. In your AWS CodeBuild pipeline configuration, add the following environment variables:

    • ASTRONOMER_KEY_ID: Your Deployment API key ID
    • ASTRONOMER_KEY_SECRET: Your Deployment API key secret
    • ASTRONOMER_DEPLOYMENT_ID: The Deployment ID of your production deployment

    Be sure to set the values for your API credentials as secret.

  2. At the root of your Git repository, add a buildspec.yml file that includes the following script:


    version: 0.2

    phases:
    install:
    runtime-versions:
    python: latest

    build:
    commands:
    - echo "${CODEBUILD_WEBHOOK_HEAD_REF}"
    - export ASTRONOMER_KEY_ID="${ASTRONOMER_KEY_ID}"
    - export ASTRONOMER_KEY_SECRET="${ASTRONOMER_KEY_SECRET}"
    - curl -sSL install.astronomer.io | sudo bash -s
    - astro deploy "${ASTRONOMER_DEPLOYMENT_ID}" -f

  3. In your AWS CodeBuild project, create a webhook event for the source provider where your Astro project is hosted, such as BitBucket or GitHub. When configuring the webhook, select an event type of PUSH.

Your buildspec.yml file now triggers a code push to an Astro Deployment every time a commit or pull request is merged to the main branch of your repository.

CircleCI

To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:

  1. Set the following environment variables in a CircleCI context:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Create a new YAML file in .circleci/config.yml that includes the following configuration:

    # Use the latest 2.1 version of CircleCI pipeline process engine.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
    docker: circleci/docker@2.0.1
    github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

    build_image_and_deploy:
    docker:
    - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
    - setup_remote_docker:
    version: 20.10.11
    - checkout
    - run:
    name: "Setup custom environment variables"
    command: |
    echo export ASTRONOMER_KEY_ID=${ASTRONOMER_KEY_ID} >> $BASH_ENV
    echo export ASTRONOMER_KEY_SECRET=${ASTRONOMER_KEY_SECRET} >> $BASH_ENV
    - run:
    name: "Deploy to Astro"
    command: |
    curl -sSL install.astronomer.io | sudo bash -s
    astro deploy -f

    # Invoke jobs via workflows
    # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
    workflows:
    version: 2.1
    build-and-deploy-prod:
    jobs:
    - build_image_and_deploy:
    context:
    - <YOUR-CIRCLE-CI-CONTEXT>
    filters:
    branches:
    only:
    - main

Drone

To automate code deploys to a Deployment using a Docker-based Drone CI pipeline, complete the following setup in a Git-based repository that hosts an Astro project.

Prerequisites

This pipeline configuration requires:

  1. Set the following environment variables as repository-level secrets on Drone:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. In your Drone server, open your Astro project repository and go to Settings > General. Under Project Settings, turn on the Trusted setting.

  3. In the top level of your Git repository, create a file called .drone.yml that includes the following configuration:

    ---
    kind: pipeline
    type: docker
    name: deploy

    steps:
    - name: install
    image: debian
    commands:
    - apt-get update
    - apt-get -y install curl
    - curl -sSL install.astronomer.io | sudo bash -s
    - name: wait
    image: docker:dind
    volumes:
    - name: dockersock
    path: /var/run
    commands:
    - sleep 5
    - name: deploy
    image: docker:dind
    volumes:
    - name: dockersock
    path: /var/run
    commands:
    - astro deploy -f
    depends on:
    - wait

    environment:
    ASTRONOMER_KEY_ID:
    from_secret: ASTRONOMER_KEY_ID
    ASTRONOMER_KEY_SECRET:
    from_secret: ASTRONOMER_KEY_SECRET

    services:
    - name: docker
    image: docker:dind
    privileged: true
    volumes:
    - name: dockersock
    path: /var/run

    volumes:
    - name: dockersock
    temp: {}

    trigger:
    branch:
    - main
    event:
    - push

GitLab

To automate code deploys to a Deployment using GitLab, complete the following setup in your GitLab repository that hosts an Astro project:

  1. In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Go to the Editor option in your project's CI/CD section and commit the following:

    ---
    astro_deploy:
    stage: deploy
    image: docker:latest
    services:
    - docker:dind
    variables:
    ASTRONOMER_KEY_ID: $ASTRONOMER_KEY_ID
    ASTRONOMER_KEY_SECRET: $ASTRONOMER_KEY_SECRET
    before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add bash
    script:
    - curl -sSL install.astronomer.io | bash -s
    - astro deploy -f
    only:
    - main

Bitbucket

To automate code deploys to a Deployment using Bitbucket, complete the following setup in a Git-based repository that hosts an Astro project:

  1. Set the following environment variables as Bitbucket pipeline variables:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Create a new YAML file in bitbucket-pipelines.yml at the root of the repository that includes the following configuration:


    pipelines:
    pull-requests: # The branch pattern under pull requests defines the *source* branch.
    dev:
    - step:
    name: Deploy to Production
    deployment: Production
    script:
    - curl -sSL install.astronomer.io | sudo bash -s
    - astro deploy
    services:
    - docker

Azure DevOps

To automate code deploys to a Deployment using Azure DevOps, complete the following setup in a Git-based repository that hosts an Astro project:

  1. Set the following environment variables as DevOps pipeline variables:

    • ASTRONOMER_KEY_ID = <your-key-id>
    • ASTRONOMER_KEY_SECRET = <your-key-secret>
  2. Create a new YAML file in astro-devops-cicd.yaml at the root of the repository that includes the following configuration:


    trigger:
    - main

    pr: none

    stages:
    - stage: deploy
    jobs:
    - job: deploy_image
    pool:
    vmImage: 'Ubuntu-latest'
    steps:
    - script: |
    curl -sSL install.astronomer.io | sudo bash -s
    astro deploy
    env:
    ASTRONOMER_KEY_ID: $(ASTRONOMER_KEY_ID)
    ASTRONOMER_KEY_SECRET: $(ASTRONOMER_KEY_SECRET)