Posts

Airflow

Image
  Apache Airflow is an open-source workflow orchestration platform used to programmatically write, schedule, and monitor complex data workflows and pipelines. Airflow workflows are represented as DAGs — Directed Acyclic Graphs . Concept Meaning DAG Defines the workflow and dependencies Task A Task is the smallest unit of work in an Airflow DAG . Operator Template/class used to execute a task Scheduler Determines when tasks should run Executor Determines how/where tasks are executed Worker Executes tasks XCom Allows tasks to exchange small pieces of information Sensor Waits for an event/condition Connection Stores connection information for external systems Variable Stores configuration values Webserver/UI Used to monitor and manage workflows Metadata DB Stores Airflow's state/metadata Benefit 1. Workflow automation Automates ETL and data pipeline tasks instead of running them manually. Example: Automatically extract data from MongoDB and load it into Teradata every day. 2. Sc...
Post a Comment
Read more

Interview sql

Image
🚀 Find Duplicate Records SELECT col1, col2, COUNT ( * ) FROM table_name GROUP BY col1, col2 HAVING COUNT ( * ) > 1 ; 🚀 Second Highest Salary SELECT MAX(salary) FROM employees WHERE salary < ( SELECT MAX(salary) FROM employees); 🚀 Nth Highest Salary SELECT salary FROM ( SELECT salary, DENSE_RANK() OVER ( ORDER BY salary DESC ) rnk FROM employees ) t WHERE rnk = N; 🚀 Top 3 Sales per Region SELECT * FROM ( SELECT * , ROW_NUMBER() OVER (PARTITION BY region ORDER BY sales DESC ) rn FROM sales ) t WHERE rn <= 3 ; 🚀 Consecutive Purchases SELECT customer_id FROM ( SELECT customer_id, order_date, LAG(order_date) OVER (PARTITION BY customer_id ORDER BY order_date) prev_date FROM orders ) t WHERE order_date = prev_date + INTERVAL '1' DAY ; 🚀Remove Duplicates DELETE FROM table_name WHERE id NOT IN ( SELECT MIN(id) FROM table_name GROUP BY col1, col2 ); 🚀  Write a qu...
Post a Comment
Read more