Astro CI/CD templates for GitHub Actions

Use the following CI/CD templates to automate deploying Apache Airflow DAGs from a GitHub repository using GitHub Actions.

The following templates for GitHub Actions are available:

• Image-only deploy templates
• DAG-based deploy templates
• Preview Deployment templates

Each template type supports multiple implementations. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.

GitHub Action templates use the Astronomer-maintained Deploy Action in the GitHub Marketplace. See the Deploy Action README to learn more about using and customizing this action. If you can't access public GitHub actions from your repository, see Private network templates.

For more information on each template or to configure your own, see Template overview. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a GitHub repository.
• An Astro Deployment.
• Either a Deployment API key ID and secret, a Workspace API token, or an Organization API token.
• Access to GitHub Actions.

Each CI/CD template implementation might have additional requirements.

Image-only templates

Image-only deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.

Single branch

To automate code deploys to a single Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following as GitHub secrets:

   • ASTRONOMER_KEY_ID = <your-key-id>
   • ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code
   
   on:
     push:
       branches:
         - main
   
   env:
     ## Sets Deployment API key credentials as environment variables
     ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
     ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
   
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2

Multiple branch

The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.

Configuration requirements

• You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
• You have respective dev and prod Deployments on Astro where you deploy your GitHub branches to.
• You have unique Deployment API keys and secrets for both of your Deployments.

Implementation

1. Set the following as GitHub secrets:

   • PROD_ASTRONOMER_KEY_ID = <your-prod-key-id>
   • PROD_ASTRONOMER_KEY_SECRET = <your-prod-key-secret>
   • DEV_ASTRONOMER_KEY_ID = <your-dev-key-id>
   • DEV_ASTRONOMER_KEY_SECRET = <your-dev-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code (Multiple Branches)
   
   on:
     push:
       branches: [dev]
     pull_request:
       types:
         - closed
       branches: [main]
   
   jobs:
     dev-push:
       if: github.ref == 'refs/heads/dev'
       env:
         ## Sets DEV Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
     prod-push:
       if: github.event.action == 'closed' && github.event.pull_request.merged == true
       env:
         ## Sets PROD Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2

Custom Image

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action.

Prerequisites

• An Astro project that requires additional build-time arguments to build the Runtime image.

Implementation

1. Set the following as GitHub secrets:

• ASTRONOMER_KEY_ID = <your-key-id>
• ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Additional build-time args
   
   on:
     push:
       branches:
         - main
   
   jobs:
     build:
       runs-on: ubuntu-latest
       env:
         ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Create image tag
         id: image_tag
         run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
       - name: Build image
         uses: docker/build-push-action@v2
         with:
           tags: ${{ steps.image_tag.outputs.image_tag }}
           load: true
           # Define your custom image's build arguments, contexts, and connections here using
           # the available GitHub Action settings:
           # https://github.com/docker/build-push-action#customizing .
           # This example uses `build-args` , but your use case might require configuring
           # different values.
           build-args: |
             <your-build-arguments>
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           image-name: ${{ steps.image_tag.outputs.image_tag }}

   For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:

   name: Astronomer CI - Custom base image
   
   on:
     push:
       branches:
         - main
   
   jobs:
     build:
       runs-on: ubuntu-latest
       env:
         ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Create image tag
         id: image_tag
         run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
       - name: Create SSH Socket
         uses: webfactory/ssh-agent@v0.5.4
         with:
           # GITHUB_SSH_KEY must be defined as a GitHub secret.
           ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
       - name: (Optional) Test SSH Connection - Should print hello message.
         run: (ssh git@github.com) || true
       - name: Build image
         uses: docker/build-push-action@v2
         with:
           tags: ${{ steps.image_tag.outputs.image_tag }}
           load: true
           ssh: |
             github=${{ env.SSH_AUTH_SOCK }
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           image-name: ${{ steps.image_tag.outputs.image_tag }}
   info

   If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.

DAG-based templates

The following templates show how to configure DAG-based deploys in GitHub Actions. They use the Deploy action dag-deploy-enabled option to implement a DAG-based deploy workflow.

Single branch

Implementation

To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following as GitHub secrets:

   • ASTRONOMER_KEY_ID = <your-key-id>
   • ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code
   
   on:
     push:
       branches:
         - main
   
   env:
     ## Sets Deployment API key credentials as environment variables
     ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
     ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
   
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           dag-deploy-enabled: true

This Github Actions script checks the diff between your current commit and your main branch when a commit is pushed to main. Make sure to customize the script for your specific use case.

Multiple branch

The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment. The finished pipeline deploys your code to Astro as demonstrated in the following diagram:

Configuration requirements:

• You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
• You have respective dev and prod Deployments on Astro where you deploy your GitHub branches to.
• You have unique Deployment API keys and secrets for both of your Deployments.

Implementation

1. Set the following as GitHub secrets:

   • PROD_ASTRONOMER_KEY_ID = <your-prod-key-id>
   • PROD_ASTRONOMER_KEY_SECRET = <your-prod-key-secret>
   • DEV_ASTRONOMER_KEY_ID = <your-dev-key-id>
   • DEV_ASTRONOMER_KEY_SECRET = <your-dev-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code (Multiple Branches)
   
   on:
     push:
       branches: [dev]
     pull_request:
       types:
         - closed
       branches: [main]
   
   jobs:
     dev-push:
       if: github.ref == 'refs/heads/dev'
       env:
         ## Sets DEV Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           dag-deploy-enabled: true
       
     prod-push:
       if: github.event.action == 'closed' && github.event.pull_request.merged == true
       env:
         ## Sets PROD Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           dag-deploy-enabled: true

Custom Image

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action.

Configuration requirements

• An Astro project that requires additional build-time arguments to build the Runtime image.

Implementation

1. Set the following as GitHub secrets:

• ASTRONOMER_KEY_ID = <your-key-id>
• ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Additional build-time args
   
   on:
     push:
       branches:
         - main
   
   jobs:
     build:
       runs-on: ubuntu-latest
       env:
         ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Create image tag
         id: image_tag
         run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
       - name: Build image
         uses: docker/build-push-action@v2
         with:
           tags: ${{ steps.image_tag.outputs.image_tag }}
           load: true
           # Define your custom image's build arguments, contexts, and connections here using
           # the available GitHub Action settings:
           # https://github.com/docker/build-push-action#customizing .
           # This example uses `build-args` , but your use case might require configuring
           # different values.
           build-args: |
             <your-build-arguments>
       - name: Deploy to Astro
         uses: astronomer/deploy-action@v0.2
         with:
           image-name: ${{ steps.image_tag.outputs.image_tag }}
           dag-deploy-enabled: true
   info

   If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.

Deployment preview templates

The Astronomer Deploy Action includes several sub-actions that can be used together to create a complete Deployment preview pipeline.

Prerequisites

• An Astro project hosted in a GitHub repository.
• A Workspace API token.
• A Deployment.

Deployment preview implementation

1. Copy and save the Deployment ID for your Astro deployment.
2. Set the following GitHub secret in the repository hosting your Astro project:

• Key: ASTRO_API_TOKEN
• Secret: <your-token>

3. In your project repository, create a new YAML file in .github/workflows named create-deployment-preview.yml that includes the following configuration:

   name: Astronomer CI - Create preview Deployment
   
   on:
     create:
       branches:
         - "**"
   
   env:
     ## Set your Workspace API key token as a GitHub secret
     ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
   
   jobs:
     deploy:
       runs-on: ubuntu-latest
       steps:
       - name: Create preview Deployment
         uses: astronomer/deploy-action@v0.2
         with:
           action: create-deployment-preview
           deployment-id: <main-deployment-id>

4. In the same folder, create a new YAML file named deploy-to-preview.yml that includes the following configuration:

   name: Astronomer CI - Deploy code to preview
   
   on:
     pull_request:
       branches:
         - main
   
   env:
     ## Set your Workspace API key token as a GitHub secret
     ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
   
   jobs:
     deploy:
       runs-on: ubuntu-latest
       steps:
       - name: Deploy code to preview
         uses: astronomer/deploy-action@v0.2
         with:
           action: deploy-deployment-preview
           deployment-id: <main-deployment-id>

5. In the same folder, create a new YAML file named delete-preview-deployment.yml that includes the following configuration:

   name: Astronomer CI - Delete Preview Deployment
   
   on:
     delete:
       branches:
         - "**"
   env:
     ## Set your Workspace API key token as a GitHub secret
     ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
   
   jobs:
     deploy:
       runs-on: ubuntu-latest
       steps:
       - name: Delete preview Deployment
         uses: astronomer/deploy-action@v0.2
         with:
           action: delete-deployment-preview
           deployment-id: <main-deployment-id>

6. In the same folder, create a new YAML file named deploy-to-main-deployment.yml that includes the following configuration:

   name: Astronomer CI - Deploy code to main Deployment
   
   on:
     push:
       branches:
         - main
   
   env:
     ## Set your Workspace API key token as a GitHub secret
     ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
   
   jobs:
     deploy:
       runs-on: ubuntu-latest
       steps:
       - name: Deploy code to main Deployment
         uses: astronomer/deploy-action@v0.2
         with:
           deployment-id: <main-deployment-id>

   All four workflow files must have the same Deployment ID specified. The actions use this Deployment ID to create and delete preview Deployments based on your main Deployment.

Private network templates

If you use GitHub Enterprise and can't use the public Astronomer Deploy Action in the GitHub Marketplace, use the following templates to implement CI/CD.

Single branch

To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following as GitHub secrets:

   • ASTRONOMER_KEY_ID = <your-key-id>
   • ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code
   
   on:
     push:
       branches:
         - main
   
   env:
     ## Sets Deployment API key credentials as environment variables
     ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
     ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
   
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
       - name: checkout repo
         uses: actions/checkout@v3
       - name: Deploy to Astro
         run: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy

Multiple branch

The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.

Prerequisites

• You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
• You have respective dev and prod Deployments on Astro where you deploy your GitHub branches to.
• You have unique Deployment API keys and secrets for both of your Deployments.

Setup

1. Set the following as GitHub secrets:

   • PROD_ASTRONOMER_KEY_ID = <your-prod-key-id>
   • PROD_ASTRONOMER_KEY_SECRET = <your-prod-key-secret>
   • DEV_ASTRONOMER_KEY_ID = <your-dev-key-id>
   • DEV_ASTRONOMER_KEY_SECRET = <your-dev-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Deploy code (Multiple Branches)
   
   on:
     push:
       branches: [dev]
     pull_request:
       types:
         - closed
       branches: [main]
   
   jobs:
     dev-push:
       if: github.ref == 'refs/heads/dev'
       env:
         ## Sets DEV Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: checkout repo
         uses: actions/checkout@v3
       - name: Deploy to Astro
         run: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy
     prod-push:
       if: github.event.action == 'closed' && github.event.pull_request.merged == true
       env:
         ## Sets PROD Deployment API key credentials as environment variables
         ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
       runs-on: ubuntu-latest
       steps:
       - name: checkout repo
         uses: actions/checkout@v3
       - name: Deploy to Astro
         run: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy

Custom Image

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action.

Prerequisites

• An Astro project that requires additional build-time arguments to build the Runtime image.

Setup

1. Set the following as GitHub secrets:

• ASTRONOMER_KEY_ID = <your-key-id>
• ASTRONOMER_KEY_SECRET = <your-key-secret>

2. In your project repository, create a new YAML file in .github/workflows that includes the following configuration:

   name: Astronomer CI - Additional build-time args
   
   on:
     push:
       branches:
         - main
   
   jobs:
     build:
       runs-on: ubuntu-latest
       env:
         ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Create image tag
         id: image_tag
         run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
       - name: Build image
         uses: docker/build-push-action@v4
         with:
           tags: ${{ steps.image_tag.outputs.image_tag }}
           load: true
           # Define your custom image's build arguments, contexts, and connections here using
           # the available GitHub Action settings:
           # https://github.com/docker/build-push-action#customizing .
           # This example uses `build-args` , but your use case might require configuring
           # different values.
           build-args: |
             <your-build-arguments>
       - name: Deploy to Astro
         run: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}

   For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:

   name: Astronomer CI - Custom base image
   
   on:
     push:
       branches:
         - main
   
   jobs:
     build:
       runs-on: ubuntu-latest
       env:
         ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
         ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Create image tag
         id: image_tag
         run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
       - name: Create SSH Socket
         uses: webfactory/ssh-agent@v0.5.4
         with:
           # GITHUB_SSH_KEY must be defined as a GitHub secret.
           ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
       - name: (Optional) Test SSH Connection - Should print hello message.
         run: (ssh git@github.com) || true
       - name: Build image
         uses: docker/build-push-action@v2
         with:
           tags: ${{ steps.image_tag.outputs.image_tag }}
           load: true
           ssh: |
             github=${{ env.SSH_AUTH_SOCK }
       - name: Deploy to Astro
         run: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}
   info

   If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.