Adapters for Apache Hadoop

Adapter for Sqoop

Sqoop is a tool designed to transfer data between Hadoop and relational databases. This adapter enables you to automate the various tasks performed by Sqoop – import, transform and export data. The integration enables the following job definitions in Tidal:

• Code Generation – generate Java classes which interpret imported records
• Import – import structure data from a relational database management system (RDBMS) to the Hadoop Distributed File System (HDFS)
• Export – export files from HDFS back to an RDBMS
• Merge – combine two datasets where entries in one dataset will overwrite entries of an older dataset

Adapter for MapReduce

MapReduce is the programming model in the Hadoop framework that is used to access and process large amounts of data stored in the HDFS. The adapter uses the Hadoop API to submit and monitor MapReduce jobs with the full scheduling capabilities of Tidal. Through this integration, jobs in Tidal divide the input data into independent chunks that are processed by the map tasks in parallel. The adapter serves as the job client to automate the execution of MapReduce jobs as part of Tidal-managed processes.

Adapter for Hive

This adapter enables you to access and manage data stored in the HDFS using Hive’s querly language (HiveQL). The seamless integration with the Tidal platform enables you to define, launch, control and monitor HiveQL commands submitted to Hive via JDBC on a scheduled basis.

HDFS Data Mover Linux Agent

This data mover agent enables you to easily manage file transfers in and out of the Hadoop file system.