This is where you'll find instructions for installing Astro on the Google Cloud Platform (GCP).
To complete the installation process, you'll:
- Create an account on Astro.
- Activate your Astro data plane by enabling Google Cloud APIs and adding service accounts to your project's IAM.
- Share information about your Google Cloud project with Astronomer.
When you've completed the installation process, Astronomer will create a cluster within your Google Cloud project to host the resources and Apache Airflow components necessary to deploy DAGs and execute tasks.
For more information about managing Google Cloud projects, see GCP documentation.
- A Google Cloud project with billing enabled. For security reasons, the install process is not currently supported on a Google Cloud project that has other tooling running in it.
- A user with Owner permissions in your project.
- Google Cloud Shell.
- A minimum CPU quota of 36. To adjust your project's quota limits up or down, see Managing your quota using the Cloud console.
- A minimum N2_CPU quota of 24. To adjust your project's quota limits up or down, see Managing your quota using the Cloud console.
- A subscription to the Astro Status Page. This ensures that you're alerted when an incident occurs or scheduled maintenance is required.
For more information about the resources required to run Astro on GCP, see GCP Resource Reference.
VPC Peering Prerequisites (Optional)
If any of your GCP resources are on a private network, you can access them using one of the following options:
- Private Services Connect
- A VPC Peering connection between Astronomer's VPC and the VPCs for your broader network
Astro uses 4 different CIDR blocks for creating the infrastructure for your Astronomer cluster. If you plan on peering with an existing VPC and want to use custom values for your CIDRs, then you must additionally provide your own CIDR ranges (RFC 1918 IP Space) of
/19 or better for the following services:
- Subnet CIDR: Used by nodes in your GKE cluster (Default:
- Pod CIDR: Used by GKE pods (Default:
- Service Address CIDR: Used by GKE services (Default:
- Service VPC Peering: Used by Private Service Connections (Default:
Step 1: Access Astro
To get started with Astro, create an account at https://cloud.astronomer.io/.
When you first authenticate to Astro, you can sign in with a Google account, a GitHub account, or an email and password.
If you're the first person from your team to authenticate, Astronomer adds you as a Workspace Admin to a new Workspace named after your Organization. From there, you'll be able to add other team members to that Workspace without the assistance of Astronomer.
After completing your initial installation, we recommend setting up an identity provider (IdP) so that users can log in to Astro through your IdP.
Step 2: Activate the data plane
The data plane is a collection of infrastructure components for Astro that run in your cloud and are fully managed by Astronomer. This includes a central database, storage for Airflow tasks logs, and the resources required for task execution.
To activate the data plane on your GCP project:
Run the following commands in your Google Cloud Shell:
gcloud services enable storage-component.googleapis.com
gcloud services enable storage-api.googleapis.com
gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable deploymentmanager.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable cloudkms.googleapis.com
gcloud services enable sqladmin.googleapis.com
gcloud services enable servicenetworking.googleapis.com
gcloud services enable dns.googleapis.com
--header "Authorization: Bearer `gcloud auth application-default print-access-token`" \
--header 'Accept: application/json' --compressed
Run the following commands in your Google Cloud Shell:
export MY_PROJECT_NUMBER=$(gcloud projects describe $GOOGLE_CLOUD_PROJECT --format="value(projectNumber)")
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member=serviceAccount:$MY_PROJECT_NUMBER@cloudservices.gserviceaccount.com --role=roles/owner
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member=serviceAccount:email@example.com --role=roles/owner
Step 3: Provide setup information to Astronomer
Once you've activated your data plane, provide Astronomer with:
- Your preferred Astro cluster name.
- The GCP region that you want to host your cluster in.
- Your preferred node instance type.
- Your preferred CloudSQL instance type.
- Your preferred maximum node count.
- (Optional) Your custom CIDR ranges for connecting to Astronomer's services.
If you don't specify your organization's preferred configurations, Astronomer creates a cluster in
us-central1 with default configurations for Astro on GCP. For more information, see GCP resource reference.
Astro supports Private Services Connect, which allows private consumption of services across VPC networks that belong to different projects or organizations. If you have created custom services that are not published using Private Services Connect, then you might want to peer with Astronomer. To set up peering, provide the following information to Astronomer:
- VPC Name/ID and region for peering with Astronomer.
- The IPs of your DNS servers.
You then need to accept a VPC peering request from Astronomer after Astro is installed. To accept the request, follow Using VPC network peering in GCP documentation.
Once VPC peered with Astronomer, configure and validate the following to ensure successful network communications between Astro and your resources:
Step 4: Let Astronomer complete the install
Once you've provided Astronomer with the information for your setup, Astronomer finishes creating your first cluster on GCP.
This process can take some time. Wait for confirmation that the installation is successful before proceeding to the next step.
Step 5: Create a Deployment
When Astronomer confirms that your Astro cluster has been created, you are ready to create a Deployment and start deploying DAGs. Log in to the Cloud UI again and create a new Deployment. If the installation is successful, your new Astro cluster is listed as an option below the Cluster menu:
Now that you have an Astro cluster up and running, take a look at the docs below for information on how to start working in Astro: