Automate code deploys with CI/CD
Continuous Integration and Continuous Delivery (CI/CD) is an industry term that refers to programmatic workflows that automate key parts of the software development lifecycle, including code changes, builds, and testing. CI/CD enables teams to develop faster, more securely, and more reliably.
On Astro, you can use Deployment API keys to automate deploying code changes to a Deployment. Astronomer recommends hosting your Astro project source code in a version control tool and setting up a CI/CD workflow for all production environments.
There are many benefits to configuring a CI/CD workflow on Astro. Specifically, you can:
- Avoid manually running
astro deploy
every time you make a change to your Astro project. - Ensure that all changes to your Astro project are reviewed and approved by your team before they get pushed to Astro.
- Automate promoting code across development and production environments on Astro when pull requests to certain branches are merged.
- Enforce automated testing, which increases code quality and allows your team to respond quickly in case of an error or failure.
- Configure more granular user permissions by managing access and changes to your Astro project source code in your version control tool.
Use the Astronomer CI/CD templates to automate deploying code to Astro with popular CI/CD management tools, including GitHub Actions and Circle CI.
Prerequisites
- A Deployment API key ID and secret.
- A CI/CD management tool, such as GitHub Actions.
- An Astro project that is hosted in a place that your CI/CD tool can access.
CI/CD templates
Templates allow you to easily configure automated workflows using popular CI/CD tools. Each template can be implemented as-is to produce a simple CI/CD pipeline. Astronomer recommends reconfiguring the templates to work with your own directory structures, tools, and best practices.
Astro supports the following CI/CD workflows:
- Image-only workflows: All files in your Astro project are built into a Docker image and pushed to Astro in a single step.
- DAG-based workflows: The DAG-only deploy feature is used to deploy DAGs in your Astro project separate from the Docker image that is built for all other project files.
The following templates are available to implement your CI/CD workflows:
- Single branch: Deploys a single branch from your version control tool to Astro. This is the default template for all CI/CD tools.
- Multiple branch: Deploys multiple branches to separate Deployments on Astro.
- Custom image: Deploys an Astro project with a customized Runtime image and additional build arguments.
Image-only workflows
The image-only workflow builds a Docker image and pushes it to Astro whenever you update any file in your Astro project. This type of template is simple to set up and works well for development workflows that include complex Docker customization or logic.
CI/CD templates for image-only workflows:
- Access Deployment API key credentials. These credentials must be set as OS-level environment variables named
ASTRONOMER_KEY_ID
andASTRONOMER_KEY_SECRET
. - Install the latest version of the Astro CLI.
- Run
astro deploy
. This creates a Docker image for your Astro project, authenticates to Astro using your Deployment API key, and pushes the image to your Deployment.
This workflow is equivalent to the following bash script:
# Set Deployment API key credentials as environment variables
$ export ASTRONOMER_KEY_ID="<your-api-key-id>"
$ export ASTRONOMER_KEY_SECRET="<your-api-key-secret>"
# Install the latest version of Astro CLI
$ curl -sSL install.astronomer.io | sudo bash -s
# Build your Astro project into a Docker image and push the image to your Deployment
$ astro deploy
All image-only templates use Astro CLI v1.0+ to deploy CI/CD pipelines. These templates will not work if you use the astrocloud
executable. To upgrade, see Install the Astro CLI.
DAG-based workflows
The features used in this workflow are in Public Preview.
The DAG-based workflow uses the --dags
flag in the Astro CLI to enable a faster way to push DAG changes to Astro. These CI/CD pipelines deploy your DAGs only when files in your dags
folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy DAGs only.
CI/CD templates that use the DAG-based workflow do the following:
- Access Deployment API key credentials. These credentials must be set as OS-level environment variables named
ASTRONOMER_KEY_ID
andASTRONOMER_KEY_SECRET
. - Install the latest version of the Astro CLI.
- Determine which files were updated by the commit:
- If only DAG files in the
dags
folder have changed, runastro deploy --dags
. This pushes yourdags
folder to your Deployment. - If any file not in the
dags
folder has changed, runastro deploy
. This triggers two subprocesses. One that creates a Docker image for your Astro project, authenticates to Astro using your Deployment API key, and pushes the image to your Deployment. A second that pushes yourdags
folder to your Deployment.
- If only DAG files in the
This workflow is equivalent to the following bash script:
# Set Deployment API key credentials as environment variables
export ASTRONOMER_KEY_ID="<your-api-key-id>"
export ASTRONOMER_KEY_SECRET="<your-api-key-secret>"
export DAG_FOLDER="<path to dag folder ie. dags/>"
# Install the latest version of Astro CLI
curl -sSL install.astronomer.io | sudo bash -s
# Determine if only DAG files have changes
files=$(git diff --name-only HEAD^..HEAD)
dags_only=1
for file in $files; do
if [[ $file != "$DAG_FOLDER"* ]]; then
echo "$file is not a dag, triggering a full image build"
dags_only=0
break
fi
done
# If only DAGs changed deploy only the DAGs in your 'dags' folder to your Deployment
if [ $dags_only == 1 ]
then
astro deploy --dags
fi
# If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and DAG changes
if [ $dags_only == 0 ]
then
astro deploy
fi
All CI/CD pipelines that use the DAG-based deployment method require Astro CLI v1.7 or later to deploy. All CI/CD templates in this document automatically install the latest version of the Astro CLI. If your CI/CD pipeline does not automatically install the latest version of the Astro CLI, make sure to specify version 1.7 or later.
GitHub Actions (Image-only deploys)
- Single branch
- Multiple branch
- Custom Image
To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment. The finished pipeline would deploy your code to Astro as demonstrated in the following diagram:
This setup assumes the following prerequisites:
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have unique Deployment API keys and secrets for both of your Deployments.
Set the following as GitHub secrets:
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets PROD Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Prerequisites
To complete this setup, you need:
- An Astro project that requires additional build-time arguments to build the Runtime image.
Setup
- Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}For example, to create a CI/CD pipeline that deploys a project which installs Python packages from a private GitHub repository, you would use the following configuration:
name: Astronomer CI - Custom base image
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
steps:
- name: Check out the repo
uses: actions/checkout@v3
- name: Create image tag
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Create SSH Socket
uses: webfactory/ssh-agent@v0.5.4
with:
# GITHUB_SSH_KEY must be defined as a GitHub secret.
ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
- name: (Optional) Test SSH Connection - Should print hello message.
run: (ssh git@github.com) || true
- name: Build image
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
ssh: |
github=${{ env.SSH_AUTH_SOCK }
- name: Deploy to Astro
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}infoIf you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.
GitHub Actions (DAG-based deploys)
The following templates are examples of how to implement DAG-only deploys in GitHub Actions. These templates can be modified to run on other CI/CD tools.
- Single branch
- Multiple branch
- Custom Image
To automate code deploys to a Deployment using GitHub Actions, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code
on:
push:
branches:
- main
env:
## Sets Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}
DAG_FOLDER: <path to dag folder ie. dags/>
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: checkout repo
uses: actions/checkout@v3
with:
# Checkout as many commits as needed for the diff
fetch-depth: 2
# Determine if only DAGs have changes
- name: Get Deployment Type
run: |
files=$(git diff --name-only HEAD^..HEAD)
dags_only=1
for file in $files; do
if [[ $file != "$DAG_FOLDER"* ]]; then
echo "$file is not a dag, triggering a full image build"
dags_only=0
break
fi
done
echo "DAGS_ONLY=$dags_only" >> $GITHUB_OUTPUT
id: deployment-type
# If only DAGs changed, do a DAG-only deploy
- name: DAG Deploy to Astro
if: steps.deployment-type.outputs.DAGS_ONLY == 1
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy --dags
# If any other files changed, deploy the entire Astro project
- name: Image and DAG Deploy to Astro
if: steps.deployment-type.outputs.DAGS_ONLY == 0
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy
This Github Actions script checks the diff between your current commit and your main
branch when a commit is pushed to main
. Make sure to customize the script for your specific use case.
The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test DAGs in a development Deployment and promote them to a production Deployment. The finished pipeline deploys your code to Astro as demonstrated in the following diagram:
This setup assumes the following prerequisites:
- You have both a
dev
andmain
branch of an Astro project hosted in a single GitHub repository. - You have respective
dev
andprod
Deployments on Astro where you deploy your GitHub branches to. - You have unique Deployment API keys and secrets for both of your Deployments.
Set the following as GitHub secrets:
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Deploy code (Multiple Branches)
on:
push:
branches: [dev]
pull_request:
types:
- closed
branches: [main]
jobs:
deployment-type:
runs-on: ubuntu-latest
env:
DAG_FOLDER: <path to dag folder ie. dags/>
outputs:
DAGS_ONLY: ${{ steps.deployment-type.outputs.DAGS_ONLY }}
steps:
- name: checkout repo
uses: actions/checkout@v3
with:
# Checkout as many commits as needed for the diff
fetch-depth: 2
# Determine if only DAGs have changes
- name: Get Deployment Type
id: deployment-type
run: |
files=$(git diff --name-only HEAD^..HEAD)
dags_only=1
for file in $files; do
if [[ $file != "$DAG_FOLDER"* ]]; then
echo "$file is not a dag, triggering a full image build"
dags_only=0
break
fi
done
echo "DAGS_ONLY=$dags_only" >> $GITHUB_OUTPUT
dev-push:
if: github.ref == 'refs/heads/dev'
env:
## Sets DEV Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.DEV_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.DEV_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
needs: deployment-type
steps:
- name: checkout repo
uses: actions/checkout@v3
# If only DAGs changed do a DAG Deploy
- name: DAG Deploy to Astro
if: needs.deployment-type.outputs.DAGS_ONLY == 1
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy --dags
# If any other files changed do a regular Deploy
- name: Image and DAG Deploy to Astro
if: needs.deployment-type.outputs.DAGS_ONLY == 0
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy
prod-push:
if: github.event.action == 'closed' && github.event.pull_request.merged == true
env:
## Sets PROD Deployment API key credentials as environment variables
ASTRONOMER_KEY_ID: ${{ secrets.PROD_ASTRONOMER_KEY_ID }}
ASTRONOMER_KEY_SECRET: ${{ secrets.PROD_ASTRONOMER_KEY_SECRET }}
runs-on: ubuntu-latest
needs: deployment-type
steps:
- name: checkout repo
uses: actions/checkout@v3
# If only DAGs changed do a DAG Deploy
- name: DAG Deploy to Astro
if: needs.deployment-type.outputs.DAGS_ONLY == 1
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy --dags
# If any other files changed do a regular Deploy
- name: Image and DAG Deploy to Astro
if: needs.deployment-type.outputs.DAGS_ONLY == 0
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy
If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's build-push-action
.
Prerequisites
- An Astro project that requires additional build-time arguments to build the Runtime image.
Setup
- Set the following as GitHub secrets:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your project repository, create a new YAML file in
.github/workflows
that includes the following configuration:name: Astronomer CI - Additional build-time args
on:
push:
branches:
- main
jobs:
build:
runs-on: ubuntu-latest
env:
ASTRONOMER_KEY_ID: ${{ secrets.ASTRO_ACCESS_KEY_ID_DEV }}
ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRO_SECRET_ACCESS_KEY_DEV }}
DAG_FOLDER: <path to dag folder ie. dags/>
steps:
- name: Check out the repo
uses: actions/checkout@v3
with:
# Checkout as many commits as needed for the diff
fetch-depth: 2
# Determine if only dags have changes
- name: Get Deployment Type
run: |
files=$(git diff --name-only HEAD^..HEAD)
dags_only=1
for file in $files; do
if [[ $file != "$DAG_FOLDER"* ]]; then
echo "$file is not a dag, triggering a full image build"
dags_only=0
break
fi
done
echo "DAGS_ONLY=$dags_only" >> $GITHUB_OUTPUT
id: deployment-type
# If only DAGs changed do a DAG Deploy
- name: DAG Deploy to Astro
if: steps.deployment-type.outputs.DAGS_ONLY == 1
run: |
curl -sSL https://install.astronomer.io | sudo bash -s
astro deploy --dags
# If any other files changed do a regular custom image Deploy
- name: Create image tag
if: steps.deployment-type.outputs.DAGS_ONLY == 0
id: image_tag
run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
- name: Build image
if: steps.deployment-type.outputs.DAGS_ONLY == 0
uses: docker/build-push-action@v2
with:
tags: ${{ steps.image_tag.outputs.image_tag }}
load: true
# Define your custom image's build arguments, contexts, and connections here using
# the available GitHub Action settings:
# https://github.com/docker/build-push-action#customizing .
# This example uses `build-args` , but your use case might require configuring
# different values.
build-args: |
<your-build-arguments>
- name: Deploy to Astro
if: steps.deployment-type.outputs.DAGS_ONLY == 0
run: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy --image-name ${{ steps.image_tag.outputs.image_tag }}infoIf you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.
Jenkins (Image-only deploys)
- Single branch
- Multiple branch
To automate code deploys to a single Deployment using Jenkins, complete the following setup in a Git-based repository hosting an Astro project:
In your Jenkins pipeline configuration, add the following parameters:
ASTRONOMER_KEY_ID
: Your Deployment API key IDASTRONOMER_KEY_SECRET
: Your Deployment API key secretASTRONOMER_DEPLOYMENT_ID
: The Deployment ID of your production deployment
Be sure to set the values for your API credentials as secret.
At the root of your Git repository, add a Jenkinsfile that includes the following script:
pipeline {
agent any
stages {
stage('Deploy to Astronomer') {
when {
expression {
return env.GIT_BRANCH == "origin/main"
}
}
steps {
checkout scm
sh '''
curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.9.0/astro_1.9.0_linux_amd64.tar.gz
tar -zxvf astro_1.9.0_linux_amd64.tar.gz astro && rm astro_1.9.0_linux_amd64.tar.gz
./astro deploy
'''
}
}
}
}
post {
always {
cleanWs()
}
}
}This Jenkinsfile triggers a code push to Astro every time a commit or pull request is merged to the
main
branch of your repository.
To automate code deploys across multiple Deployments using Jenkins, complete the following setup in a Git-based repository hosting an Astro project:
In Jenkins, add the following environment variables:
PROD_ASTRONOMER_KEY_ID
: Your Production Deployment API key IDPROD_ASTRONOMER_KEY_SECRET
: Your Production Deployment API key secretPROD_DEPLOYMENT_ID
: The Deployment ID of your Production DeploymentDEV_ASTRONOMER_KEY_ID
: Your Development Deployment API key IDDEV_ASTRONOMER_KEY_SECRET
: Your Development Deployment API key secretDEV_DEPLOYMENT_ID
: The Deployment ID of your Development Deployment
To set environment variables in Jenkins, on the Jenkins Dashboard go to Manage Jenkins > Configure System > Global Properties > Environment Variables > Add.
Be sure to set the values for your API credentials as secret.
At the root of your Git repository, add a Jenkinsfile that includes the following script:
pipeline {
agent any
stages {
stage('Set Environment Variables') {
steps {
script {
if (env.GIT_BRANCH == 'main') {
echo "The git branch is ${env.GIT_BRANCH}";
env.ASTRONOMER_KEY_ID = env.PROD_ASTRONOMER_KEY_ID;
env.ASTRONOMER_KEY_SECRET = env.PROD_ASTRONOMER_KEY_SECRET;
env.ASTRONOMER_DEPLOYMENT_ID = env.PROD_DEPLOYMENT_ID;
} else if (env.GIT_BRANCH == 'dev') {
echo "The git branch is ${env.GIT_BRANCH}";
env.ASTRONOMER_KEY_ID = env.DEV_ASTRONOMER_KEY_ID;
env.ASTRONOMER_KEY_SECRET = env.DEV_ASTRONOMER_KEY_SECRET;
env.ASTRONOMER_DEPLOYMENT_ID = env.DEV_DEPLOYMENT_ID;
} else {
echo "This git branch ${env.GIT_BRANCH} is not configured in this pipeline."
}
}
}
}
stage('Deploy to Astronomer') {
steps {
checkout scm
sh '''
curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.9.0/astro_1.9.0_linux_amd64.tar.gz
tar -zxvf astro_1.9.0_linux_amd64.tar.gz astro && rm astro_1.9.0_linux_amd64.tar.gz
./astro deploy
'''
}
}
}
post {
always {
cleanWs()
}
}
}
}This Jenkinsfile triggers a code push to an Astro Deployment every time a commit or pull request is merged to the
dev
ormain
branch of your repository.
Jenkins (DAG-based deploys)
Use the following template to implement DAG-only deploys with Jenkins.
In your Jenkins pipeline configuration, add the following parameters:
ASTRONOMER_KEY_ID
: Your Deployment API key IDASTRONOMER_KEY_SECRET
: Your Deployment API key secretASTRONOMER_DEPLOYMENT_ID
: The Deployment ID of your production deployment
Be sure to set the values for your API credentials as secret.
At the root of your Git repository, add a Jenkinsfile that includes the following script:
pipeline {
agent any
stages {
stage('Dag Only Deploy to Astronomer') {
when {
expression {
return env.GIT_BRANCH == "origin/main"
}
}
steps {
checkout scm
sh '''
curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.9.0/astro_1.9.0_linux_amd64.tar.gz
tar -zxvf astro_1.9.0_linux_amd64.tar.gz astro && rm astro_1.9.0_linux_amd64.tar.gz
files=($(git diff-tree HEAD --name-only --no-commit-id))
find="dags"
if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
./astro deploy --dags;
else
./astro deploy;
fi
'''
}
}
}
post {
always {
cleanWs()
}
}
}
AWS CodeBuild
- Single branch
- Multiple branch
To automate code deploys to a single Deployment using AWS CodeBuild, complete the following setup in a Git-based repository hosting an Astro project:
In your AWS CodeBuild pipeline configuration, add the following environment variables:
ASTRONOMER_KEY_ID
: Your Deployment API key IDASTRONOMER_KEY_SECRET
: Your Deployment API key secretASTRONOMER_DEPLOYMENT_ID
: The Deployment ID of your production deployment
Be sure to set the values for your API credentials as secret.
At the root of your Git repository, add a buildspec.yml file that includes the following script:
version: 0.2
phases:
install:
runtime-versions:
python: latest
build:
commands:
- echo "${CODEBUILD_WEBHOOK_HEAD_REF}"
- export ASTRONOMER_KEY_ID="${ASTRONOMER_KEY_ID}"
- export ASTRONOMER_KEY_SECRET="${ASTRONOMER_KEY_SECRET}"
- curl -sSL install.astronomer.io | sudo bash -s
- astro deploy "${ASTRONOMER_DEPLOYMENT_ID}" -fIn your AWS CodeBuild project, create a webhook event for the source provider where your Astro project is hosted, such as BitBucket or GitHub. When configuring the webhook, select an event type of
PUSH
.
Your buildspec.yml file now triggers a code push to an Astro Deployment every time a commit or pull request is merged to the main
branch of your repository.
To automate code deploys across multiple Deployments using AWS CodeBuild, complete the following setup in a Git-based repository hosting an Astro project:
In your AWS CodeBuild pipeline configuration, add the following environment variables:
PROD_ASTRONOMER_KEY_ID
: Your Production Deployment API key IDPROD_ASTRONOMER_KEY_SECRET
: Your Production Deployment API key secretPROD_DEPLOYMENT_ID
: The Deployment ID of your Production DeploymentDEV_ASTRONOMER_KEY_ID
: Your Development Deployment API key IDDEV_ASTRONOMER_KEY_SECRET
: Your Development Deployment API key secretDEV_DEPLOYMENT_ID
: The Deployment ID of your Development Deployment
At the root of your Git repository, add a buildspec.yml that includes the following script:
version: 0.2
phases:
install:
runtime-versions:
python: latest
build:
commands:
- |
if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/main" >/dev/null; then
export ASTRONOMER_KEY_ID="${PROD_ASTRONOMER_KEY_ID}"
export ASTRONOMER_KEY_SECRET="${PROD_ASTRONOMER_KEY_SECRET}"
curl -sSL install.astronomer.io | sudo bash -s
astro deploy "${PROD_DEPLOYMENT_ID}" -f
fi
- |
if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/dev" >/dev/null; then
export ASTRONOMER_KEY_ID="${DEV_ASTRONOMER_KEY_ID}"
export ASTRONOMER_KEY_SECRET="${DEV_ASTRONOMER_KEY_SECRET}"
curl -sSL install.astronomer.io | sudo bash -s
astro deploy "${DEV_DEPLOYMENT_ID}" -f
fi- In your AWS CodeBuild project, create a webhook event for the source provider where your Astro project is hosted, such as BitBucket or GitHub. When configuring the webhook, select an event type of
PUSH
.
- In your AWS CodeBuild project, create a webhook event for the source provider where your Astro project is hosted, such as BitBucket or GitHub. When configuring the webhook, select an event type of
Your buildspec.yml file now triggers a code push to an Astro Deployment every time a commit or pull request is merged to the main
branch of your repository.
CircleCI
To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following environment variables in a CircleCI context:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Create a new YAML file in
.circleci/config.yml
that includes the following configuration:# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1
orbs:
docker: circleci/docker@2.0.1
github-cli: circleci/github-cli@2.0.0
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:
build_image_and_deploy:
docker:
- image: cimg/base:stable
# Add steps to the job
# See: https://circleci.com/docs/2.0/configuration-reference/#steps
steps:
- setup_remote_docker:
version: 20.10.11
- checkout
- run:
name: "Setup custom environment variables"
command: |
echo export ASTRONOMER_KEY_ID=${ASTRONOMER_KEY_ID} >> $BASH_ENV
echo export ASTRONOMER_KEY_SECRET=${ASTRONOMER_KEY_SECRET} >> $BASH_ENV
- run:
name: "Deploy to Astro"
command: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy -f
# Invoke jobs with workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
version: 2.1
build-and-deploy-prod:
jobs:
- build_image_and_deploy:
context:
- <YOUR-CIRCLE-CI-CONTEXT>
filters:
branches:
only:
- main
Drone
To automate code deploys to a Deployment using a Docker-based Drone CI pipeline, complete the following setup in a Git-based repository that hosts an Astro project.
Prerequisites
This pipeline configuration requires:
- A functional Drone server and Docker runner.
- A user with admin privileges to your Drone server.
Set the following environment variables as repository-level secrets on Drone:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
In your Drone server, open your Astro project repository and go to Settings > General. Under Project Settings, turn on the Trusted setting.
In the top level of your Git repository, create a file called
.drone.yml
that includes the following configuration:---
kind: pipeline
type: docker
name: deploy
steps:
- name: install
image: debian
commands:
- apt-get update
- apt-get -y install curl
- curl -sSL install.astronomer.io | sudo bash -s
- name: wait
image: docker:dind
volumes:
- name: dockersock
path: /var/run
commands:
- sleep 5
- name: deploy
image: docker:dind
volumes:
- name: dockersock
path: /var/run
commands:
- astro deploy -f
depends on:
- wait
environment:
ASTRONOMER_KEY_ID:
from_secret: ASTRONOMER_KEY_ID
ASTRONOMER_KEY_SECRET:
from_secret: ASTRONOMER_KEY_SECRET
services:
- name: docker
image: docker:dind
privileged: true
volumes:
- name: dockersock
path: /var/run
volumes:
- name: dockersock
temp: {}
trigger:
branch:
- main
event:
- push
GitLab
If you use hosted runners, you might need certain tags or modifications based on your Gitlab runner. Speak with your Gitlab administrator and review the Gitlab Documentation to customize this template for your use case.
- Single branch
- Multiple branch
To automate code deploys to a Deployment using GitLab, complete the following setup in your GitLab repository that hosts an Astro project:
In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Go to the Editor option in your project's CI/CD section and commit the following:
---
astro_deploy:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- main
To automate code deploys to Astro across multiple environments using GitLab, complete the following setup in your GitLab repository that hosts an Astro project:
In GitLab, go to Project Settings > CI/CD > Variables and set the following environment variables:
DEV_ASTRONOMER_KEY_ID
=<your-dev-key-id>
DEV_ASTRONOMER_KEY_SECRET
=<your-dev-key-secret>
PROD_ASTRONOMER_KEY_ID
=<your-prod-key-id>
PROD_ASTRONOMER_KEY_SECRET
=<your-prod-key-secret>
When you create environment variables that will be used in multiple branches, you may want to protect the branch they are being used in. Otherwise, uncheck the Protect variable
flag when you create the variable in GitLab. For more information on protected branches, see GitLab documentation.
Go to the Editor option in your project's CI/CD section and commit the following:
---
astro_deploy_dev:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${DEV_ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${DEV_ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- dev
astro_deploy_prod:
stage: deploy
image: docker:latest
services:
- docker:dind
variables:
ASTRONOMER_KEY_ID: ${PROD_ASTRONOMER_KEY_ID}
ASTRONOMER_KEY_SECRET: ${PROD_ASTRONOMER_KEY_SECRET}
before_script:
- apk add --update curl && rm -rf /var/cache/apk/*
- apk add bash
script:
- (curl -sSL install.astronomer.io | bash -s)
- astro deploy -f
only:
- main
Bitbucket
To automate code deploys to a Deployment using Bitbucket, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following environment variables as Bitbucket pipeline variables:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Create a new YAML file in
bitbucket-pipelines.yml
at the root of the repository that includes the following configuration:pipelines:
pull-requests: # The branch pattern under pull requests defines the *source* branch.
dev:
- step:
name: Deploy to Production
deployment: Production
script:
- curl -sSL install.astronomer.io | sudo bash -s
- astro deploy
services:
- docker
Azure DevOps
To automate code deploys to a Deployment using Azure DevOps, complete the following setup in a Git-based repository that hosts an Astro project:
Set the following environment variables as DevOps pipeline variables:
ASTRONOMER_KEY_ID
=<your-key-id>
ASTRONOMER_KEY_SECRET
=<your-key-secret>
Create a new YAML file in
astro-devops-cicd.yaml
at the root of the repository that includes the following configuration:trigger:
- main
pr: none
stages:
- stage: deploy
jobs:
- job: deploy_image
pool:
vmImage: 'Ubuntu-latest'
steps:
- script: |
curl -sSL install.astronomer.io | sudo bash -s
astro deploy
env:
ASTRONOMER_KEY_ID: $(ASTRONOMER_KEY_ID)
ASTRONOMER_KEY_SECRET: $(ASTRONOMER_KEY_SECRET)