Skip to main content
Version: 0.29

Configure a custom registry for Deployment images

Overview

Astronomer Software includes access to a Docker image registry that is managed by Astronomer. Every time a user deploys to Astronomer Software, a Docker image is generated and pushed to this registry. Depending on your deployment method, these Docker images can include OS and Python dependencies, DAG code, and the Airflow service.

Using the Astronomer internal registry is often the best option for organizations getting started with Astronomer and when users are comfortable deploying code. However, using this registry might not be compatible with your organization's security requirements.

If your organization cannot use the default Astronomer internal registry, you can configure a custom container image registry. A custom container image registry is recommended for mature organizations who require additional control for security and governance reasons. Your organization can use a custom container registry to scan images for CVEs, malicious code, and unapproved Python and OS-level packages contained in the Docker images that are generated during the code deploy process.

Implementation considerations

Deploying code changes to a custom image registry requires triggering a GraphQL mutation to provide a Deployment release name, image name, and Airflow version to the registry. Because this process is difficult to manually trigger, Astronomer recommends configuring a custom image registry only if your DAG authors can deploy code changes using continuous integration and continuous delivery (CI/CD) pipelines. In this implementation, you use your CI/CD tool to:

  • Build your Astro project into a container image.
  • Deploy the image to your custom registry.
  • Run a query to push the image from your registry to Astronomer Software.

Prerequisites

  • Helm.
  • kubectl.
  • A custom container image registry.
  • A process for building and pushing your Astro projects as images to your custom registry.

Setup

  1. Create a secret for the container repository credentials in your Astronomer namespace:

    kubectl -n <your-namespace> create secret docker-registry <name-of-secret> --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password> --docker-email=<your-email>

    To have Astronomer Software sync the registry credentials to all Deployment namespaces, add the following annotation:

    kubectl -n <release-namespace> annotate secret <name-of-secret> "astronomer.io/commander-sync"="platform=astronomer"
    info

    To use different registries for each Deployment, create the same secret in each Deployment namespace instead of your Astronomer namespace. Make sure to specify different custom registries using --docker-server. If you don't need to synch your secrets between deployments, you don't need to add the secret annotation.

  2. Open your config.yaml file. See Apply a Config Change.

  3. Add the following to your config.yaml file:

    astronomer:
    houston:
    config:
    deployments:
    enableUpdateDeploymentImageEndpoint: true
    registry:
    protectedCustomRegistry:
    enabled: true
    updateRegistry:
    enabled: true
    host: <your-airflow-image-repo>
    secretName: <name-of-secret>
    info

    To use different registries for each Deployment, do not set astronomer.houston.config.deployments.registry.protectedCustomRegistry.updateRegistry.host.

  4. Push the configuration change. See Apply a config change.

  5. For any existing Deployments, run the following command to sync the registry credentials.

    kubectl create job -n <release-namespace> --from=cronjob/astronomer-config-syncer upgrade-config-synchronization
    info

    If you're using different registries for each Deployment, skip this step.

Push code to a custom registry

After pushing images for your Astro project to your private registry, you can run a GraphQL query to push these images from your registry to Astronomer Software. At a minimum, your query has to include the following:

mutation updateDeploymentImage {
updateDeploymentImage(
releaseName: "<deployment-release-name>", # for example "analytics-dev"
image: "<host>/<image-name>:<tag>", # for example docker.io/cmart123/ap-airflow:test4
airflowVersion: "<airflow-version-number>" # for example "2.2.5"
)
{
id
}
}

Alternatively, you can run this same query using curl:

curl 'https://houston.BASEDOMAIN/v1' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' -H 'DNT: 1' -H 'Origin: https://houston.BASEDOMAIN/v1' -H 'Authorization: <your-token>' --data-binary '{"query":"mutation updateDeploymentImage {updateDeploymentImage(releaseName: \"<deployment-release-name>\", image: \"<host>/<image-name>:<tag>\",airflowVersion: \"<airflow-version-number>\"){id}}"}' --compressed