Dec. 6-8 — Gartner IT Infrastructure, Operations & Cloud Strategies Conference Join Us

Big Data PipelineDeveloper

Adapter for Apache® Airflow

Automate Airflow activities with advanced scheduling functionality

Tidal Hero Grid Image

Integrate Airflow DAGs into business workflows

Tidal Hero Grid Image

Enable Developers to focus on higher value work instead of scheduling tasks

Tidal Hero Grid Image

Leverage more advanced scheduling functionality

Orchestrate Airflow DAGs

Airflow is popular among developers that need to manage and monitor data pipelines. Users develop Python language scripts that define Directed Acyclic Graphs (DAGs). These DAGs specify dependencies and a sequence of steps (or tasks) to run either on a defined schedule or based on external event triggers.

Our adapter for Airflow provides seamless integration and automation of Airflow DAG executions in your Tidal-managed processes. With more robust scheduling functionality than the basic features in Airflow, Tidal enables you to include complex DAGs as Airflow jobs within your enterprise workflows and schedule those jobs based on other dependencies and resources. Centralized management frees up developers from monitoring and scheduling tasks so they can focus on higher value activities.

Dag screenshot

Airflow Directed Acyclic Graph (DAG)

Adapter Capabilities

This adapter offers compatibility with Airflow v1 and v2 as well as Airflow activities embedded in Google Cloud Composer. With our integration, you can:

  • Tie Airflow activities into cross-application workflows
  • Schedule Airflow jobs based on time, events or dependencies external to a DAG
  • Verify the availability of external resources – such as Kubernetes – before starting an Airflow job
  • Distribute Airflow jobs across multiple Airflow clusters for improved performance

Tidal can orchestrate Airflow workflows whether they're running on-prem or in the cloud.

How It Works

Apache Airflow diagram

  • The Tidal adapter for Airflow creates a secure connection between the Tidal and Airflow servers.
  • The adapter pulls selected DAGs into Tidal so you can define Tidal job and Airflow DAG configuration parameters as well as dependencies between the DAGs and other Tidal jobs.
  • The Airflow server runs the requested process.
  • Airflow DAG job runs can be controlled and monitored through the Tidal interface as part of schedule execution.

 

Apache Airflow Adapter

Resource

Learn More About Our Adapter for Airflow

Overview & Demo Video

Tidal can simplify and optimize the scheduling of Airflow jobs and add advanced workflow management capabilities. Learn about the value of the adapter and how to integrate Airflow DAGs into enterprise processes in this short video.