Skip to main content

Resources required for Astro on GCP

Unless otherwise specified, new Clusters on Google Cloud Platform (GCP) are created with a set of default resources that our team has deemed appropriate for most use cases.

Read the following document for a reference of our default resources as well as supported cluster configurations.

Default cluster values

ResourceDescriptionQuantity/Default Size
VPCVirtual private network for hosting GCP resources1x /19
SubnetA single subnet is provisioned in the VPC1, IP Range is 172.20.0.0/19
Service Network PeeringThe Astro VPC is peered to the Google Service Networking VPC1, IP Range is 172.23.0.0/19
NAT Router (External)Required for connectivity with the Astro control plane and other public services1
GKE ClusterA GKE cluster is required to run the Astro data plane, which hosts the resources and data required to execute Airflow tasks. Workload Identity is enabled on this cluster.1x, IP Ranges are 172.21.0.0/19 for cluster IPs and 172.22.0.0/19 for cluster services
Workload Identity PoolAstro uses the fixed Workload Identity Pool for your project. One is created if it does not existThe default pool (PROJECT_ID.svc.id.goog) is used
Cloud SQL for PostgreSQLThe Cloud SQL instance is the primary database for the Astro data plane. It hosts the metadata database for each Airflow Deployment hosted on the GKE cluster1x regional instance with 4 vCPUs, 16GB memory
Google Cloud Storage (GCS) BucketGCS bucket to store Airflow task logs1 bucket with name airflow-logs-<clusterid>
Worker node poolsNode pools run all Airflow workers. The number of nodes in the pool auto-scales based on the demand for workers in your cluster. You can configure multiple worker node pools to run tasks on different instance types1x pool of e2-standard-4 nodes
Astro node poolA node pool runs all proprietary Astronomer components. This node pool is fully managed by Astronomer1x pool of e2-standard-4 nodes
Airflow node poolA node pool runs all core Airflow components such as the scheduler and webserver. This node pool is fully managed by Astronomer1x pool of e2-standard-4 nodes
Maximum Node CountThe maximum number of nodes that your Astro cluster can support. When this limit is reached, your Astro cluster can't auto-scale and worker Pods may fail to schedule.20

Supported cluster configurations

Depending on the needs of your team, you may be interested in modifying certain configurations of a new or existing cluster on Astro. This section provides a reference for which configuration options are supported during the install process.

To create a new cluster on Astro with a specified configuration, read Install on GCP or Create a cluster. For instructions on how to make a change to an existing cluster, see Modify a cluster.

GCP region

Astro supports the following Google Cloud Platform (GCP) regions:

  • asia-east1 - Taiwan, Asia
  • asia-northeast1 - Tokyo, Asia
  • asia-northeast2 - Osaka, Asia
  • asia-northeast3 - Seoul, Asia
  • asia-south1 - Mumbai, Asia
  • asia-south2 - Delhi, Asia
  • asia-southeast1 - Singapore, Asia
  • asia-southeast2 - Jakarta, Asia
  • australia-southeast1 - Sydney, Australia
  • australia-southeast2 - Melbourne, Australia
  • europe-central2 - Warsaw, Europe
  • europe-north1 - Finalnd, Europe
  • europe-southwest1 - Madrid, Europe
  • europe-west1 - Belgium, Europe
  • europe-west2 - England, Europe
  • europe-west3 - Frankfurt, Europe
  • europe-west4 - Netherlands, Europe
  • europe-west6 - Zurich, Europe
  • europe-west8 - Milan, Europe
  • europe-west9 - Paris, Europe
  • northamerica-northeast1 - Montreal, North America
  • northamerica-northeast2 - Toronto, North America
  • southamerica-east1 - Sau Paolo, South America
  • southamerica-west1 - Santiago, South America
  • us-central1 - Iowa, North America
  • us-east1 - South Carolina, North America
  • us-east4 - Virginia, North America
  • us-east5 - Columbus, North America
  • us-south1 - Dallas, North America
  • us-west1 - Oregon, North America
  • us-west2 - Los Angeles, North America
  • us-west3 - Salt Lake City, North America
  • us-west4 - Nevada, North America

Modifying the region of an existing Astro cluster isn't supported. If you're interested in a GCP region that isn't on this list, contact Astronomer support.

Worker node pools

Node pools are a scalable collection of worker nodes with the same instance type. These nodes are responsible for running the Pods that execute Airflow tasks. If your cluster has a node pool for a specific instance type, you can configure tasks to run on those instance types using worker queues. To make an instance type available in a cluster, reach out to Astronomer support with a request to create a new node pool for the specific instance type. Note that not all machine types are supported in all GCP regions.

Astronomer monitors your usage and number of nodes deployed in your cluster. As your usage of Airflow increases, Astronomer might reach out with recommendations for updating your node pools to optimize your infrastructure spend or increase the efficiency of your tasks.

Worker node size resource reference

Each worker node in a pool runs a single worker Pod. A worker Pod's actual available size is equivalent to the total capacity of the instance type minus Astro’s system overhead.

The following table lists all available instance types for worker node pools, as well as the Pod size that is supported for each instance type. As the system requirements of Astro change, these values can increase or decrease.

Node Instance TypeCPUMemory
e2-standard-42 CPUs7.5 GiB MEM
e2-standard-86 CPUs22.5 GiB MEM

If your Organization is interested in using an instance type that supports a larger worker size, contact Astronomer support. For more information about configuring worker size on Astro, see Configure a Deployment.

Maximum node count

Each Astro cluster has a limit on how many nodes it can run at once. This maximum includes worker nodes as well as system nodes managed by Astronomer.

The default maximum node count for all nodes across your cluster is 20. A cluster's node count is most affected by the number of worker Pods that are executing Airflow tasks. See Worker autoscaling logic.

If the node count for your cluster reaches the maximum node count, new tasks might not run or get scheduled. Astronomer monitors maximum node count and is responsible for contacting your organization if it is reached. To check your cluster's current node count, contact Astronomer Support.