Astro CI/CD templates for GitHub Actions
Use the following CI/CD templates to automate deploying Apache Airflow DAGs from a GitHub repository using GitHub Actions.
The following templates for GitHub Actions are available:
Each template type supports multiple implementations. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.
GitHub Action templates use the Astronomer-maintained Deploy Action in the GitHub Marketplace. See the Deploy Action README to learn more about using and customizing this action. If you can't access public GitHub actions from your repository, see Private network templates.
For more information on each template or to configure your own, see Template overview. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.
Prerequisites
- An Astro project hosted in a GitHub repository.
- An Astro Deployment.
- Either a Deployment API key ID and secret, a Workspace API token, or an Organization API token.
- Access to GitHub Actions.
Each CI/CD template implementation might have additional requirements.
Image-only templates
Image-only deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.
- Single branch
- Multiple branch
- Custom Image
To automate code deploys to a single Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.
Configuration requirements
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have unique Deployment API keys and secrets for both of your Deployments.
Implementation
Set the following as GitHub secrets:
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets PROD Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Prerequisites
- An Astro project that requires additional build-time arguments to build the Runtime image.
Implementation
- Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
image-name: ${{ steps.image_tag.outputs.image_tag }}For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:
name: Astronomer CI - Custom base image
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Create SSH Socket
uses: webfactory/ssh-agent@v0.5.4
with:
# GITHUB_SSH_KEY must be defined as a GitHub secret.
ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
- name: (Optional) Test SSH Connection - Should print hello message.
run: (ssh git@github.com) || true
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
ssh: |
github=${{ env.SSH_AUTH_SOCK }
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
image-name: ${{ steps.image_tag.outputs.image_tag }}infoIf you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.
DAG-based templates
The following templates show how to configure DAG-based deploys in GitHub Actions. They use the Deploy action dag-deploy-enabled
option to implement a DAG-based deploy workflow.
- Single branch
- Multiple branch
- Custom Image
Implementation
To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
dag-deploy-enabled: true
This Github Actions script checks the diff between your current commit and your main
branch when a commit is pushed to main
. Make sure to customize the script for your specific use case.
The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment. The finished pipeline deploys your code to Astro as demonstrated in the following diagram:
Configuration requirements:
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have unique Deployment API keys and secrets for both of your Deployments.
Implementation
Set the following as GitHub secrets:
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
dag-deploy-enabled: true
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets PROD Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
dag-deploy-enabled: true
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Configuration requirements
- An Astro project that requires additional build-time arguments to build the Runtime image.
Implementation
- Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
uses: astronomer/deploy-action@v0.2
with:
image-name: ${{ steps.image_tag.outputs.image_tag }}
dag-deploy-enabled: trueinfoIf you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.
Deployment preview templates
The Astronomer Deploy Action includes several sub-actions that can be used together to create a complete Deployment preview pipeline.
Prerequisites
- An Astro project hosted in a GitHub repository.
- A Workspace API token.
- A Deployment.
Deployment preview implementation
- Copy and save the Deployment ID for your Astro deployment.
- Set the following GitHub secret in the repository hosting your Astro project:
- Key:
ASTRO_API_TOKEN
- Secret:
<your-token>
In your project repository, create a new YAML file in
.github/workflows
namedcreate-deployment-preview.yml
that includes the following configuration:name: Astronomer CI - Create preview Deployment
on:
create:
branches:
- "**"
env:
## Set your Workspace API key token as a GitHub secret
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Create preview Deployment
uses: astronomer/deploy-action@v0.2
with:
action: create-deployment-preview
deployment-id: <main-deployment-id>In the same folder, create a new YAML file named
deploy-to-preview.yml
that includes the following configuration:name: Astronomer CI - Deploy code to preview
on:
pull_request:
branches:
- main
env:
## Set your Workspace API key token as a GitHub secret
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy code to preview
uses: astronomer/deploy-action@v0.2
with:
action: deploy-deployment-preview
deployment-id: <main-deployment-id>In the same folder, create a new YAML file named
delete-preview-deployment.yml
that includes the following configuration:name: Astronomer CI - Delete Preview Deployment
on:
delete:
branches:
- "**"
env:
## Set your Workspace API key token as a GitHub secret
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Delete preview Deployment
uses: astronomer/deploy-action@v0.2
with:
action: delete-deployment-preview
deployment-id: <main-deployment-id>In the same folder, create a new YAML file named
deploy-to-main-deployment.yml
that includes the following configuration:name: Astronomer CI - Deploy code to main Deployment
on:
push:
branches:
- main
env:
## Set your Workspace API key token as a GitHub secret
ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy code to main Deployment
uses: astronomer/deploy-action@v0.2
with:
deployment-id: <main-deployment-id>All four workflow files must have the same Deployment ID specified. The actions use this Deployment ID to create and delete preview Deployments based on your main Deployment.
Private network templates
If you use GitHub Enterprise and can't use the public Astronomer Deploy Action in the GitHub Marketplace, use the following templates to implement CI/CD.
- Single branch
- Multiple branch
- Custom Image
To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment.
Prerequisites
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have unique Deployment API keys and secrets for both of your Deployments.
Setup
Set the following as GitHub secrets:
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets PROD Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Prerequisites
- An Astro project that requires additional build-time arguments to build the Runtime image.
Setup
- Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
uses: docker/build-push-action@v4
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:
name: Astronomer CI - Custom base image
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Create SSH Socket
uses: webfactory/ssh-agent@v0.5.4
with:
# GITHUB_SSH_KEY must be defined as a GitHub secret.
ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
- name: (Optional) Test SSH Connection - Should print hello message.
run: (ssh git@github.com) || true
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
ssh: |
github=${{ env.SSH_AUTH_SOCK }
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}infoIf you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.