Astro CI/CD templates for CircleCI

CircleCI is a continuous integration and continuous delivery platform that can be used to implement DevOps practices. This document provides sample CI/CD templates to automate deploying Apache Airflow DAGs from a GitHub repository to Astro using CircleCI.

If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.

Refer to Template overview to see generic templates expressed as simple shell scripts or configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a Git repository that CircleCI can access.
• An Astro Deployment.
• A Deployment API token, Workspace API token, or Organization API token.
• A CircleCI account.

Image deploy templates

Image deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.

Single branch

To automate code deploys to a Deployment using CircleCI for a single branch implementation, complete the following setup in a Git-based repository that hosts an Astro project:

Configuration requirements

• You have a main branch of an Astro project hosted in a single GitHub repository.
• You have a production Deployment on Astro where you want to deploy your main GitHub branch.
• You have a production CircleCI context that stores environment variables for your CI/CD workflows.

Implementation

1. Set the following environment variables in a CircleCI context:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

2. Create a new YAML file in .circleci/config.yml that includes the following configuration:

   # Use the latest CircleCI pipeline process engine version.
   # See: https://circleci.com/docs/2.0/configuration-reference
   version: 2.1
   
   orbs:
     docker: circleci/docker@2.0.1
     github-cli: circleci/github-cli@2.0.0
   
   # Define a job to be invoked later in a workflow.
   # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
   jobs:
   
     build_image_and_deploy:
       docker:
         - image: cimg/base:stable
       # Add steps to the job
       # See: https://circleci.com/docs/2.0/configuration-reference/#steps
       steps:
         - setup_remote_docker:
             version: 20.10.11
         - checkout
         - run:
             name: "Deploy to Astro"
             command: |
               curl -sSL install.astronomer.io | sudo bash -s
               astro deploy ${ASTRO_DEPLOYMENT_ID} -f
   
   # Invoke jobs with workflows
   # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
   workflows:
     version: 2.1
     wf-build-and-deploy:
       jobs:
         - build_image_and_deploy:
             context:
               - <YOUR-CIRCLE-CI-CONTEXT>
             filters:
               branches:
                 only:
                   - <YOUR-BRANCH-NAME>

Multiple branch

The following template can be used to create a multiple branch CI/CD pipeline using CircleCI. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.

Configuration requirements

• You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
• You have respective development and production Deployments on Astro where you deploy your GitHub branches to.
• You have respective development and production CircleCI contexts that store environment variables to use in your CI/CD workflows.

Implementation

1. Set the following environment variables in both your production and development CircleCI contexts:

   • ASTRO_API_TOKEN = <your-workspace-or-organization-api-token-with-access-to-prod-deployment>
   • ASTRO_DEPLOYMENT_ID = <your-prod-deployment-id>

2. Create a new YAML file in .circleci/config.yml that includes the following configuration:

   # Use the latest CircleCI pipeline process engine version.
   # See: https://circleci.com/docs/2.0/configuration-reference
   version: 2.1
   
   orbs:
     docker: circleci/docker@2.0.1
     github-cli: circleci/github-cli@2.0.0
   
   # Define a job to be invoked later in a workflow.
   # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
   jobs:
   
     build_image_and_deploy:
       docker:
         - image: cimg/base:stable
       # Add steps to the job
       # See: https://circleci.com/docs/2.0/configuration-reference/#steps
       steps:
         - setup_remote_docker:
             version: 20.10.11
         - checkout
         - run:
             name: "Deploy to Astro"
             command: |
               curl -sSL install.astronomer.io | sudo bash -s
               astro deploy ${ASTRO_DEPLOYMENT_ID} -f
   
   # Invoke jobs with workflows
   # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
   workflows:
     version: 2.1
     wf_build-and-deploy:
       jobs:
         - build_image_and_deploy:
             context:
               - <YOUR-PROD-CIRCLE-CI-CONTEXT>
             filters:
               branches:
                 only:
                   - <YOUR-PRODUCTION-BRANCH-NAME>
       jobs:
         - build_image_and_deploy:
             context:
               - <YOUR-DEV-CIRCLE-CI-CONTEXT>
             filters:
               branches:
                 only:
                   - <YOUR-DEVELOPMENT-BRANCH-NAME>

Read more about multiple workflows in the CircleCI documentation.

Custom Image

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments in docker build command and then use the image tag to deploy to Astro. See docker build for reference.

Configuration requirements

• You have a main branch of an Astro project hosted in a single GitHub repository.
• You have a production Deployment on Astro where you want to deploy your main GitHub branch.
• You have a production CircleCI context where you store environment variables to use in your CI/CD workflow.s

Implementation

1. Set the following environment variables in a CircleCI context:

   • ASTRO_API_TOKEN = <your-workspace-or-organization-api-token-with-access-to-prod-deployment>
   • ASTRO_DEPLOYMENT_ID = <your-deployment-id>

2. Create a new YAML file in .circleci/config.yml that includes the following configuration:

   # Use the latest CircleCI pipeline process engine version.
   # See: https://circleci.com/docs/2.0/configuration-reference
   version: 2.1
   
   orbs:
     docker: circleci/docker@2.0.1
     github-cli: circleci/github-cli@2.0.0
   
   # Define a job to be invoked later in a workflow.
   # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
   jobs:
   
     build_image_and_deploy:
       docker:
         - image: cimg/base:stable
       # Add steps to the job
       # See: https://circleci.com/docs/2.0/configuration-reference/#steps
       steps:
         - setup_remote_docker:
             version: 20.10.11
         - checkout
         - run:
             name: "Build image and deploy"
             command: |
               set -e
               echo "export image_tag=astro-$(date +%Y%m%d%H%M%S)" >> $BASH_ENV
               source "$BASH_ENV"
               docker build -t ${image_tag} --build-arg="<your-build-arg>=<your-build-arg-value>" .
               curl -sSL install.astronomer.io | sudo bash -s
               astro deploy --image-name ${image_tag} ${ASTRO_DEPLOYMENT_ID} -f
   
     # Invoke jobs with workflows
     # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
     workflows:
       version: 2.1
       build-and-deploy-prod:
         jobs:
           - build_image_and_deploy_prod:
               context:
                 - <YOUR-CIRCLE-CI-CONTEXT>
               filters:
                 branches:
                   only:
                     - <YOUR-BRANCH-NAME>
   info

   If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.

DAG deploy templates

A DAG deploy template uses the --dags flag in the astro deploy command in the Astro CLI to push only DAGs to your Deployment.

This CI/CD pipeline deploys your DAGs to Astro when one or more files in your dags folder are modified. It deploys the rest of your Astro project as a Docker image when other files or directories are also modified. For more information about the benefits of this workflow, see Deploy DAGs only.

Configuration requirements

For each Deployment that you use with DAG deploy templates, you must enable DAG deploys.

Single branch implementation

To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following environment variables in a CircleCI context:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

2. In your project repository, create a new YAML file in .circleci/config.yml that includes the following configuration:

   # Use the latest CircleCI pipeline process engine version.
   # See: https://circleci.com/docs/2.0/configuration-reference
   version: 2.1
   
   orbs:
     docker: circleci/docker@2.0.1
     github-cli: circleci/github-cli@2.0.0
   
   # Define a job to be invoked later in a workflow.
   # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
   jobs:
   
     build_image_and_deploy:
       docker:
         - image: cimg/base:stable
       # Add steps to the job
       # See: https://circleci.com/docs/2.0/configuration-reference/#steps
       steps:
         - setup_remote_docker:
             version: 20.10.11
         - checkout
         - run:
             name: "Build image and deploy"
             command: |
               curl -sSL install.astronomer.io | sudo bash -s
               files=($(git diff-tree HEAD --name-only --no-commit-id))
               echo ${files}
               find="dags"
               if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
                 echo "only deploying dags"
                 astro deploy ${ASTRO_DEPLOYMENT_ID} --dags -f;
               else
                 echo "image deploy"
                 astro deploy ${ASTRO_DEPLOYMENT_ID} -f;
               fi
   
   # Invoke jobs with workflows
   # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
   workflows:
     version: 2.1
     wf-build-and-deploy:
       jobs:
         - build_image_and_deploy:
             context:
               - <YOUR-CIRCLE-CI-CONTEXT>
             filters:
               branches:
                 only:
                   - <YOUR-BRANCH-NAME>

This script checks the diff between your current commit and the HEAD of your branch to which you are pushing the changes to. If the changes are only in dags then it executes a dag-only deploy. Otherwise, it executes an image-based deploy. Make sure to customize the script to use your specific branch and context.

You can customize this script to work for multiple branches as shown in the image-based multi-branch deploy template by creating separate job and workflow for each branch.