Airflow
Apache Airflow is an open-source workflow orchestration platform used to programmatically write, schedule, and monitor complex data workflows and pipelines. Airflow workflows are represented as DAGs — Directed Acyclic Graphs . Concept Meaning DAG Defines the workflow and dependencies Task A Task is the smallest unit of work in an Airflow DAG . Operator Template/class used to execute a task Scheduler Determines when tasks should run Executor Determines how/where tasks are executed Worker Executes tasks XCom Allows tasks to exchange small pieces of information Sensor Waits for an event/condition Connection Stores connection information for external systems Variable Stores configuration values Webserver/UI Used to monitor and manage workflows Metadata DB Stores Airflow's state/metadata Benefit 1. Workflow automation Automates ETL and data pipeline tasks instead of running them manually. Example: Automatically extract data from MongoDB and load it into Teradata every day. 2. Sc...