Skip to main content
Version: 0.25

Configure Airflow Alerts on Astronomer Software

Overview

Whether you're just starting to use Apache Airflow or your team is running it at scale, incorporating an alerting framework is critical to the health of your data. On Astronomer Software, users have access to three types of alerting solutions:

  • Airflow task and DAG-level alerts, which notify you via email when an Airflow task or DAG fails, succeeds, or retries.
  • Deployment-level alerts, which notify you when the health of an Airflow Deployment is low or if any of Airflow's underlying components are underperforming, including the Airflow Scheduler.
  • Platform-level alerts, which notify you when a component of your Astronomer platform is unhealthy, such as Elasticsearch, Astronomer's Houston API, or your Docker Registry.

This guide focuses on configuring task and DAG-level alerts. For information on configuring platform and Deployment-level alerts, read Alerting on Astronomer Software.

Subscribe to Task-Level Alerts

As an Airflow user, you can configure event-based email alerts directly in your DAG code by leveraging Airflow's email util. Depending on your use case, you may choose to be notified if a particular task or DAG fails, succeeds, retries, etc. On Astronomer, setting up task-level alerts requires configuring an SMTP service to handle the delivery of these emails.

If your team isn't already using an SMTP service, we recommend the following:

Step-by-step instructions on how to integrate these two services with Astronomer are provided below, but you can use any SMTP service for this purpose.

Note: By default, email alerts for process failures are sent whenever individual tasks fail. To receive only 1 email per DAG failure, refer to the Limit Alerts to the DAG Level topic below. For more information and best practices on Airflow alerts, read Error Notifications in Airflow.

Integrate SendGrid with Astronomer

SendGrid is an email delivery service that's easy to set up for Airflow task-level alerts. A free SendGrid account grants users 40,000 free emails within the first 30 days of an account opening and 100 emails per day after that. This should be more than enough emails for most alerting use cases.

To get started with SendGrid:

  1. Create a SendGrid account. Be prepared to disclose some standard information about yourself and your organization.

  2. Verify a Single Sender Identity. Because you're sending emails only for internal administrative purposes, a single sender identity is sufficient for integrating with Astronomer. The email address you verify here is used as the sender for your Airflow alert emails.

  3. Create a key using SendGrid's web API. In SendGrid, go to Email API > Integration Guide. Follow the steps to generate a new API key using SendGrid's Web API and cURL.

  4. Skip the step for exporting your API Key to your development environment. Instead, execute the generated curl code directly in your command line, making sure to replace $SENDGRID_API_KEY in the --header field with your copied key.

  5. Verify your integration in SendGrid to confirm that the key was activated. If you get an error indicating that SendGrid can't find the test email, try rerunning the cURL code in your terminal before retrying the verification.

  6. In the Software UI, go to the Variables tab of your Deployment and add the following Environment Variables using the +Add button:

    AIRFLOW__SMTP__SMTP_HOST=smtp.sendgrid.net
    AIRFLOW__SMTP__SMTP_STARTTLS=True
    AIRFLOW__SMTP__SMTP_SSL=False
    AIRFLOW__SMTP__SMTP_USER=apikey
    AIRFLOW__SMTP__SMTP_PASSWORD=<your-api-key>
    AIRFLOW__SMTP__SMTP_PORT=587
    AIRFLOW__SMTP__SMTP_MAIL_FROM=<your-sendgrid-email>

    Your Environment Variables should look something like this:

    Astronomer Deployment Email Settings

    To prevent unauthorized users in your Workspace from seeing sensitive information, we recommend selecting the Secret? checkbox for your email and password profile variables.

  7. Click Deploy Changes to push your configuration to your Airflow Deployment.

Note: For more information on how to configure Environment Variables both locally and on Astronomer, refer to Environment Variables.

Integrate Amazon SES with Astronomer

This setup requires an AWS account and use of the AWS Management Console.

  1. In the AWS Management Console, go to AWS Console > Simple Email Service > Email Addresses to add and verify the email addresses you want to receive alerts.

  2. Open the inbox of each email address you specified and verify them through the emails sent by Amazon.

  3. In the AWS Console, go to Simple Email Service > SMTP Settings and use the Create My SMTP Credentials button to generate a username and password. This will look similar to an access and secret access key. Write down this username and password for step 5, as well as the Server Name and Port.

    Note: You won't be able to access these values again, so consider storing them in a password manager.

  4. Choose an Amazon EC2 region to use, then write down the code of this server for the next step. Refer to Amazon's list of available regions and servers to determine which server best fits your needs.

  5. Open your Airflow Deployment in the Software UI and go to the Variables tab. Then, add the following Environment Variables using the +Add button:

    AIRFLOW__SMTP__SMTP_HOST=<your-smtp-host>
    AIRFLOW__SMTP__SMTP_PORT=587
    AIRFLOW__SMTP__SMTP_STARTTLS=True
    AIRFLOW__SMTP__SMTP_SSL=False
    AIRFLOW__SMTP__SMTP_USER=<your-aws-username>
    AIRFLOW__SMTP__SMTP_PASSWORD=<your-aws-password>
    AIRFLOW__SMTP__SMTP_MAIL_FROM=<your-email-address>

    Your Environment Variables should look something like this:

    Astronomer Deployment Email Settings

    To prevent unauthorized users in your Workspace from seeing sensitive information, we recommend selecting the Secret? checkbox for your email and password profile variables.

    Note: For more information on how to configure Environment Variables both locally and on Astronomer, refer to our "Environment Variables" doc.

  6. Click Deploy Changes to push your configuration to your Airflow Deployment.

Limit Alerts to the DAG Level

By default, email alerts configured via the email_on_failure param (source) are handled at the task level. If some number of your tasks fail, you'll receive an individual email for each of those failures.

If you want to limit failure alerts to the DAG-run level, you can instead set up your alerts using the on_failure_callback param (source). When you pass on_failure_callback directly in your DAG file, it defines a Python function that sends you one email per DAG failure, rather than multiple emails for each task that fails:

 :param on_failure_callback: A function to be called when a DagRun of this dag fails.

The code in your DAG might look something like this (source):

     from airflow.models.email import send_email
def new_email_alert(self, **kwargs):
title = "TEST MESSAGE: THIS IS A MODIFIED TEST"
body = ("This is the text "
"That appears in the email body..<br>")
send_email('my_email@email.com', title, body)