For all teams, incorporating an alerting framework is critical to the health of your data pipelines. In addition to the observability functionality provided on Astro, Airflow's alerting framework includes support for:
- Slack notifications
- Email alerts
Slack alerts and SLAs do not require additional configuration on Astro. For best practices, see the Astronomer guide on Airflow Alerts.
This guide focuses on setting up email alerts on Astro with an SMTP service.
Configure Airflow Email Alerts
On Astro, setting up email alerts requires configuring an SMTP service for delivering each alert.
If your team isn't already using an SMTP service, we recommend one of the following:
The following topics provide setup steps for integrating each of these external SMTP services on Astro, but any external SMTP service can be used.
Integrate with SendGrid
SendGrid is an email delivery service that's easy to set up for Airflow alerts. A free SendGrid account grants users 40,000 free emails within the first 30 days of an account opening and 100 emails per day after that. This should be more than enough emails for most alerting use cases.
To get started with SendGrid:
Create a SendGrid account. Be prepared to disclose some standard information about yourself and your organization.
Verify a Single Sender Identity. Because you're sending emails only for internal administrative purposes, a single sender identity is sufficient for integrating with Astro. The email address you verify here is used as the sender for your Airflow alert emails.
Create a Sendgrid API key. In SendGrid, go to Email API > Integration Guide. Follow the steps to generate a new API key using SendGrid's Web API and cURL.
For more information, see Sendgrid documentation.
Skip the step for exporting your API key to your development environment. Instead, execute the generated curl code directly in your command line, making sure to replace
--headerfield with your copied key.
Verify your integration in SendGrid to confirm that the key was activated. If you get an error indicating that SendGrid can't find the test email, try rerunning the cURL code in your terminal before retrying the verification.
In the Deployment view of the Cloud UI, create an environment variable with the following values:
For more information on this environment variable, see Airflow documentation.
In the Airflow UI, create an Airflow connection with the following values:
- Connection ID:
- Connection Type::
- Connection ID:
Click Save to finalize your configuration.
To begin receiving Airflow alerts via email for task failures within a given DAG, configure the following values in the DAG's
Integrate with Amazon SES
Use your existing Amazon SES instance to send Airflow alerts by email.
Sign in to the AWS Management Console and open the Amazon SES console at https://console.aws.amazon.com/ses/.
Click the menu icon and then click Verified Identities.
Optional. Complete one of the following tasks:
- To confirm the email account is working and the Amazon SES service can send emails to it, click an email address in the Identity column and then click Send test email. When the owner of the email address receives the test email, they need to select the validation link in the email.
- To add a new email address, click Create Identity, add the email address, and then click Create Identity.
For email alerts, Astronomer recommends using one email address as the sender and a second email address as the recipient. All email addresses must be verified.
Click Account dashboard.
In the Simple Mail Transfer Protocol (SMTP) settings pane, copy the value displayed below SMTP endpoint.
Obtain your Amazon SES SMTP credentials:
- Click Create SMTP credentials in the Simple Mail Transfer Protocol (SMTP) settings pane.
- Enter a name for your SMTP user in the IAM User Name field or accept the default value.
- Click Create and then Show User SMTP Security Credentials.
- Click Download Credentials or copy them and store them in a safe place.
- Click Close Window.
Log in to the Cloud UI, click Deployments, and then select an existing Deployment.
In the Environment Variables area, click Edit Variables and add these variables:
ENV AIRFLOW__SMTP__SMTP_HOST: Enter the value you copied in step 5
ENV AIRFLOW__SMTP__SMTP_STARTTLS: Enter
ENV AIRFLOW__SMTP__SMTP_SSL: Enter
ENV AIRFLOW__SMTP__SMTP_USER: Enter the value you copied in step 6.
ENV AIRFLOW__SMTP__SMTP_PASSWORD: Enter the value you copied in step 6.
ENV AIRFLOW__SMTP__SMTP_PORT: Enter
ENV AIRFLOW__SMTP__SMTP_MAIL_FROM: Enter your from email.
ENV AIRFLOW__EMAIL__EMAIL_BACKEND: Enter
To begin receiving Airflow alerts by email for task failures within a given DAG, configure the following values in the DAG's