Adapter for Apache Airflow

Integrate Airflow Activities Into Your Enterprise Scheduling Ecosystem

• Incorporate complex Airflow Directed Acyclic Graphs (DAGs) into business workflows
• Enable Developers to focus on higher value work instead of scheduling tasks
• Apply a richer set of scheduling constructs than what is available in Airflow

Download the Tidal Adapter for Airflow Overview.

Adapter Overview

Apache Airflow is an open-source platform for creating, running and monitoring workflows. Users develop Python language scripts that define Directed Acyclic Graphs (DAGs). These DAGs specify dependencies and a sequence of steps (or tasks) to run either on a defined schedule or based on external event triggers. Airflow is popular among developers that need to monitor and manage data pipelines.

Our adapter for Airflow provides seamless integration and automation of Airflow DAG executions into your Tidal-managed processes. With more robust scheduling functionality than the basic features in Airflow, Tidal Automation enables you to include complex DAGs as Airflow jobs within your enterprise workflows and schedule those jobs based on other dependencies and resources. Centralized management frees up developers from monitoring and scheduling tasks so they can focus on higher value activities.

This adapter offers compatibility for Airflow activities embedded in Google Cloud Composer.

The Tidal adapter for Airflow enables you to:

• Tie Airflow activities into cross-application workflows
• Schedule Airflow jobs based on time, events or dependencies external to A DAG
• Verify the availability of external resources – such as Kubernetes – before starting an Airflow job
• Distribute Airflow jobs across multiple Airflow clusters for improved performance

How It Works

• The Tidal adapter for Airflow creates a secure connection between the Tidal and Airflow servers.
• The adapter pulls selected DAGs into Tidal so you can define Tidal job and Airflow DAG configuration parameters, as well as dependencies between the DAGs and other Tidal jobs.
• The Airflow server runs the requested process.
• Airflow DAG job runs can be controlled and monitored through the Tidal interface as part of schedule execution.

Airflow DAGs can be seen directly in Tidal through a built-in DAG viewer.