Skip to main content

GCP cluster settings

Unless otherwise specified, new Clusters on Google Cloud Platform (GCP) are created with a set of default resources that our team has deemed appropriate for most use cases.

Read the following document for a reference of our default resources as well as supported cluster configurations.

Default cluster values

ResourceDescriptionQuantity/ Default SizeConfigurable
GKE ClusterA GKE cluster is required to run the Astro data plane, which hosts the resources and data required to execute Airflow tasks. Workload Identity is enabled on this cluster.1x, IP Ranges are for cluster IPs and for cluster services
Worker node poolA node pool that hosts all workers with the default worker type for all Deployments in the cluster. The number of nodes in the pool auto-scales based on the demand for workers in your cluster. You can configure additional worker node pools to run tasks on different worker types.1x pool of e2-standard-4 nodes✔️
Airflow node poolA node pool that runs all core Airflow components, including the scheduler and webserver, for all Deployments in the cluster. This node pool is fully managed by Astronomer.1x pool of n2-standard-4 nodes
Astro system node poolA node pool that runs all other system components required in Astro. The availability zone determines how many nodes are created. This node pool is fully managed by Astronomer.1x pool of n2-standard-4 nodes
Cloud SQL for PostgreSQLThe Cloud SQL instance is the primary database for the Astro data plane. It hosts the metadata database for each Airflow Deployment hosted on the GKE cluster.1x regional instance with 4 vCPUs, 16GB memory
VPCVirtual private network for hosting GCP resources1x /19✔️
SubnetA single subnet is provisioned in the VPC.1, IP Range is
Service Network PeeringThe Astro VPC is peered to the Google Service Networking VPC.1, IP Range is
NAT Router (External)Required for connectivity with the Astro control plane and other public services1.
Workload Identity PoolAstro uses the fixed Workload Identity Pool for your cluster. One is created if it does not
Google Cloud Storage (GCS) BucketStores Airflow task logs.1 bucket with name airflow-logs-<clusterid>
Maximum Node CountThe maximum number of worker nodes that your Astro cluster can support. When this limit is reached, your Astro cluster can't auto-scale and worker Pods may fail to schedule.20✔️

Supported cluster configurations

You might need to modify configurations of a new or existing cluster on Astro. This section provides a reference for cluster configuration options.

To create a new cluster on Astro with a specified configuration, see Create a cluster. To request a change to an existing cluster, see Modify a cluster. Astronomer is currently responsible for completing all cluster configuration changes.

Cluster regions

Depending on how you installed Astro, you can host Astro clusters in the following GCP regions:

CodeNameAstro - Bring Your Own CloudAstro - Hosted
asia-east1Taiwan, Asia✔️
asia-northeast1Tokyo, Asia✔️✔️
asia-northeast2Osaka, Asia✔️
asia-northeast3Seoul, Asia✔️
asia-south1Mumbai, Asia✔️
asia-south2Delhi, Asia✔️
asia-southeast1Singapore, Asia✔️
asia-southeast2Jakarta, Asia✔️
australia-southeast1Sydney, Australia✔️
australia-southeast2Melbourne, Australia✔️
europe-central2Warsaw, Europe✔️
europe-north1Finland, Europe✔️
europe-southwest1Madrid, Europe✔️
europe-west1Belgium, Europe✔️✔️
europe-west2England, Europe✔️✔️
europe-west3Frankfurt, Europe✔️
europe-west4Netherlands, Europe✔️
europe-west6Zurich, Europe✔️
europe-west8Milan, Europe✔️
europe-west9Paris, Europe✔️
northamerica-northeast1Montreal, North America✔️
northamerica-northeast2Toronto, North America✔️
southamerica-east1Sau Paolo, South America✔️
southamerica-west1Santiago, South America✔️
us-central1Iowa, North America✔️✔️
us-east1South Carolina, North America✔️
us-east4Virginia, North America✔️✔️
us-east5Columbus, North America✔️
us-south1Dallas, North America✔️
us-west1Oregon, North America✔️
us-west2Los Angeles, North America✔️
us-west3Salt Lake City, North America✔️
us-west4Nevada, North America✔️

Modifying the region of an existing Astro cluster isn't supported. If you're interested in a GCP region that isn't listed, contact Astronomer support.

Worker node pools

A node pool is a group of nodes within a cluster that all have the same configuration. On Astro, worker nodes are responsible for running the Pods that execute Airflow tasks. Each worker node pool can be configured with a node instance type and a maximum node count. All Astro clusters have one worker node pool by default, but you can configure additional node pools to optimize resource usage.

If your cluster has multiple worker node pools with different worker node instance types, users in your organization can configure tasks to run on those worker types using worker queues. To enable a new worker type for your cluster, contact Astronomer support with a request to create a new node pool or modify an existing node pool.

Astronomer monitors your usage and the number of nodes deployed in your cluster. When your Airflow use increases, Astronomer support might contact you and provide recommendations for updating your node pools to optimize your infrastructure costs or increase the efficiency of your tasks.

Worker node resource reference

Each worker node in a pool runs a single worker Pod. A worker Pod's actual available size is equivalent to the total capacity of the instance type minus Astro’s system overhead.

The following table lists all available instance types for worker node pools, as well as the Pod size that is supported for each instance type. As the system requirements of Astro change, these values can increase or decrease.

Node Instance TypeCPUMemory
e2-standard-43 CPUs13 GiB MEM
e2-standard-87 CPUs29 GiB MEM
e2-standard-1615 CPUs61 GiB MEM
e2-highmem-43 CPUs29 GiB MEM
e2-highmem-87 CPUs61 GiB MEM
e2-highmem-1615 CPUs125 GiB MEM
e2-highcpu-43 CPUs1 GiB MEM
e2-highcpu-87 CPUs5 GiB MEM
e2-highcpu-1615 CPUs13 GiB MEM
n2-standard-43 CPUs13 GiB MEM
n2-standard-87 CPUs29 GiB MEM
n2-standard-1615 CPUs61 GiB MEM
n2-highmem-43 CPUs29 GiB MEM
n2-highmem-87 CPUs61 GiB MEM
n2-highmem-1615 CPUs125 GiB MEM
n2-highcpu-43 CPUs1 GiB MEM
n2-highcpu-87 CPUs5 GiB MEM
n2-highcpu-1615 CPUs13 GiB MEM
c2-standard-43 CPUs13 GiB MEM
c2-standard-87 CPUs29 GiB MEM

If your Organization is interested in using an instance type that supports a larger worker size, contact Astronomer support. For more information about configuring worker size on Astro, see Configure a Deployment.

Maximum node count

Each Astro cluster has a limit on how many nodes it can run at once. This maximum includes worker nodes as well as system nodes managed by Astronomer.

The default maximum node count for all nodes across your cluster is 20. A cluster's node count is most affected by the number of worker Pods that are executing Airflow tasks. See Worker autoscaling logic.

If the node count for your cluster reaches the maximum node count, new tasks might not run or get scheduled. Astronomer monitors maximum node count and is responsible for contacting your organization if it is reached. To check your cluster's current node count, contact Astronomer Support.