Adapter for Apache Airflow

Integrate Airflow Activities Into Your Enterprise Scheduling Ecosystem

• Incorporate complex Airflow Directed Acyclic Graphs (DAGs) into business workflows
• Enable Developers to focus on higher value work instead of scheduling tasks
• Leverage a richer set of scheduling constructs than what is available in Airflow

Adapter Overview

Apache Airflow is an open-source platform for creating, running and monitoring workflows. Users develop Python language scripts that define Directed Acyclic Graphs (DAGs). These DAGs specify dependencies and a sequence of steps (or tasks) to run either on a defined schedule or based on external event triggers. Airflow is popular among developers that need to monitor and manage data pipelines.

Our adapter for Airflow provides seamless integration and automation of Airflow DAG executions into your Tidal-managed processes. With more robust scheduling functionality than the basic features in Airflow, Tidal Automation enables you to include complex DAGs as Airflow jobs within your enterprise workflows and schedule those jobs based on other dependencies and resources. Centralized management frees up developers from monitoring and scheduling tasks so they can focus on higher value activities.

The Tidal adapter for Airflow enables you to:

• Schedule Airflow jobs based on business activities, time-based triggers and/or events external to a DAG
• Create schedule dependencies based on the status of other Airflow jobs
• Verify the availability of external resources before starting an Airflow job
• Distribute Airflow jobs across multiple Airflow agents for improved performance

How It Works

• The Tidal adapter for Airflow creates a secure connection between the Tidal and Airflow servers.
• The adapter pulls selected DAGs into Tidal, allowing the Tidal Administrator to define Tidal job and Airflow DAG configuration parameters, as well as dependencies between the DAGs and other Tidal jobs.
• Airflow DAGs job runs can be monitored through the Tidal interface as part of schedule execution.

Airflow DAGs can be viewed directly in Tidal through a built-in DAG viewer.