Apache® Airflow
The Tidal by Redwood adapter for Apache Airflow automates and orchestrates activities with advanced scheduling functionality.
Apache Airflow is an open-source platform used to programmatically author, schedule and monitor workflows, especially for data pipelines, and gives data engineers the ability to define and manage complex data workflows as code.
Uplevel your automation with the Tidal adapter for Apache Airflow
Explore what’s possible with this powerful combination.
Extend your workflows
Integrate Airflow’s Directed Acyclic Graphs (DAGs) into business workflows, then use Tidal to orchestrate them.
Focus on strategic work
Enable your developers to engage in higher-value work instead of scheduling tasks.
Manage complexity
Leverage more advanced scheduling functionality to extend the impact of Apache Airflow.
Orchestrate Airflow DAGs
Airflow is popular among developers who need to manage and monitor data pipelines. Users develop Python language scripts that define Directed Acyclic Graphs (DAGs). These DAGs specify dependencies and a sequence of steps (or tasks) to run on a defined schedule or based on external event triggers.
Effortless integration with Tidal’s adapter allows you to build Airflow DAG execution automations into your Tidal-managed processes. With more robust scheduling functionality than the basic Airflow features, Tidal enables you to include complex DAGs as Airflow jobs within your enterprise workflows and schedule those jobs based on other dependencies and resources. Centralized management frees up your developers from mundane monitoring and scheduling tasks so they can put their efforts toward more valuable activities.
Airflow Directed Acyclic Graph (DAG)
What the adapter enables
This adapter offers compatibility with Airflow v1 and v2 and Airflow activities embedded in Google Cloud Composer. With this integration, you can:
- Distribute Airflow jobs across multiple Airflow clusters for improved performance
- Schedule Airflow jobs based on time, events or dependencies external to a DAG
- Tie Airflow activities into cross-application workflows
- Use Tidal to orchestrate Airflow workflows on-prem or in the cloud
- Verify the availability of external resources, such as Kubernetes, before starting an Airflow job
How it works
- The Tidal adapter for Airflow creates a secure connection between the Tidal and Airflow servers
- The adapter pulls selected DAGs into Tidal so you can define the Tidal job and Airflow DAG configuration parameters and dependencies between the DAGs and other Tidal jobs
- The Airflow server runs the requested process
- You can control Airflow DAG job runs via the Tidal interface as part of a schedule execution
See it in action
Tidal can simplify and optimize the scheduling of Airflow jobs and add advanced workflow management capabilities. Learn about the value of the adapter and how to integrate Airflow DAGs into enterprise processes in this short video.
More Tidal integrations
Automation drives innovation and efficiency and reduces risk
See what Tidal can help you achieve.
Tidal and Apache Airflow integration FAQs
-
What is Apache Airflow used for?
Apache Airflow is used to programmatically author, schedule and monitor workflows, particularly data pipelines.
-
What is an Airflow integration used for?
Airflow integrations connect and interact with various tools, services and platforms, allowing you to easily orchestrate and manage data pipelines and workflows by leveraging Airflow's open-source platform for authoring, scheduling and monitoring.
-
Is Apache Airflow an ETL tool?
Apache Airflow is an open-source platform designed to create, schedule and monitor workflows. Due to its flexibility and scalability, it is widely used for orchestrating ETL processes. The Tidal adapter for Airflow facilitates seamless integration between Tidal and Apache Airflow, enabling you to orchestrate Airflow jobs in enterprise workflows and leverage Tidal's advanced scheduling capabilities.