Skip to main content

Resources Required for Astro on GCP

Overview

Unless otherwise specified, new Clusters on Google Cloud Platform (GCP) are created with a set of default resources that our team has deemed appropriate for most use cases.

Read the following document for a reference of our default resources as well as supported Cluster configurations.

Default Cluster Values

ResourceDescriptionQuantity/Default Size
VPCVirtual private network for hosting GCP resources1x /19
SubnetA single subnet is provisioned in the VPC1, IP Range is 172.20.0.0/19
Service Network PeeringThe Astro VPC is peered to the Google Service Networking VPC1, IP Range is 172.23.0.0/19
NAT Router (External)Required for connectivity with the Astro Control Plane and other public services1
GKE ClusterA GKE cluster is required to run the Astro Data Plane. Workload Identity is enabled on this cluster.1, IP Ranges are 172.21.0.0/19 for Cluster IPs and 172.22.0.0/19 for Cluster Services
Workload Identity PoolAstro uses the fixed Workload Identity Pool for your project; one is created if it does not existThe default pool (PROJECT_ID.svc.id.goog) is used
Cloud SQL for PostgreSQLThe Cloud SQL instance is the primary database for the Astro Data Plane. It hosts the metadata database for each Airflow Deployment hosted on the GKE cluster1 Regional Instance with 4 vCPUs, 16GB Memory
Storage BucketGCS Bucket for storage of Airflow task logs1 bucket with name airflow-logs-<clusterid>
NodesNodes power the Data Plane and Airflow components. Nodes autoscale as deployments are added.3x n2-medium-4 for the system nodes; worker nodes default to e2-medium-4 and are provisioned as required, up to Max Node Count
Max Node CountThe maximum number of EC2 nodes that your Astro Cluster can support at any given time. Once this limit is reached, your Cluster cannot auto-scale and worker pods may fail to schedule.20

Supported Cluster Configurations

Depending on the needs of your team, you may be interested in modifying certain configurations of a new or existing Cluster on Astro. This section provides a reference for which configuration options are supported during the install process.

To create a new Cluster on Astro with a specified configuration, read Install on GCP or Create a Cluster. For instructions on how to make a change to an existing Cluster, read Modify a Cluster.

GCP Region

Astro supports the following GCP regions:

  • us-central1
  • us-west1
  • us-east1
  • europe-west4

Modifying the region of an existing Cluster on Astro is not supported. If you're interested in a GCP region that is not on this list, reach out to Astronomer Support.

Node Instance Type

Astro supports different GCP machine types. Machine types comprise of varying combinations of CPU, memory, storage, and networking capacity. All system and Airflow components within a single Cluster are powered by the nodes specified during the Cluster creation or modification process.

  • e2-standard-4
  • e2-standard-8

For detailed information on each instance type, see GCP documentation. If you're interested in a machine type that is not on this list, reach out to Astronomer Support. Not all machine types are supported in all GCP regions.