Airflow Udemy

Posted By admin On 18/01/22

Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In the Ultimate Hands-On Course to Master Apache Airflow, you are going to learn everything you need in order to fully master this very powerful tool and take it to the next level. Every aspects of Airflow are addressed such as mastering your DAGs, monitoring, scalability, security and much more.

'Apache Airflow is a platform created by community to programmatically author, schedule and monitor workflows.' Airflow is going to change the way of scheduling data pipelines and that is why it has become the Top-level project of Apache. Using Helm to configure and set up Airflow on Kubernetes. Configuring the official Helm chart of Airflow to use the Kubernetes Executor and many different features. Deploying DAGs in Airflow with Git-Sync and AWS EFS. Deploying DAGs/Airflow through CI/CD pipelines with AWS CodePipeline. Testing your DAGs automatically.

You will find the materials under the title of the video Development Environment in Udemy as shown below:

Section 1: Course Introduction

  1. Important Prerequisites
  2. Course Structure
  3. Who I am
  4. Development Environment

Section 2: The basics of Apache Airflow

  1. Introduction
  2. Why Airflow?
  3. What is Airflow?
  4. How Airflow works?
  5. [Practice] Installing Airflow
  6. [Practice] Quick Tour of Airflow UI
  7. [Practice] Quick Tour of Airflow CLI
  8. [Practice] Controlling your DAGs with the CLI

Section 3: The Forex Data Pipeline

  1. Introduction
  2. Docker reminder
  3. Troubleshoot Docker performances on MacOS
  4. Project: The Forex Data Pipeline
  5. What is a DAG?
  6. Defining your first DAG
  7. What is an Operator?
  8. [Practice] Checking if the API is available – HttpSensor
  9. [Practice] Checking if the currency file is available – FileSensor
  10. [Practice] Downloading the forex rates from the API – PythonOperator
  11. [Practice] Saving the forex rates in the HDFS – BashOperator
  12. [Practice] Creating the Hive table forex_rates – HiveOperator
  13. [Practice] Processing the forex rates with Spark – SparkSubmitOperator
  14. [Practice] Sending an email notification – EmailOperator
  15. [Practice] Sending a Slack notification – SlackAPIPostOperator
  16. Operator Relationships and Bitshift Composition
  17. [Practice] Adding dependencies between tasks
  18. [Practice] The Forex Data Pipeline in action!

Section 4: Mastering your DAGs

  1. Introduction
  2. Start_date and schedule_interval parameters demystified
  3. [Practice] Manipulating the start_date with schedule_interval
  4. Backfill and Catchup
  5. [Practice] Catching up non triggered DAGRuns
  6. Dealing with timezones in Airflow
  7. [Practice] Making your DAGs timezone aware
  8. How to make your tasks dependent
  9. [Practice] Creating task dependencies between DagRuns
  10. How to structure your DAG folder
  11. [Practice] Organizing your DAGs folder
  12. [Practice] How the Web Server works
  13. How to deal with failures in your DAGs
  14. [Practice] Retry and Alerting
  15. How to test your DAGs
  16. [Practice] Unit testing your DAGs

Airflow Demo

Udemy

Section 5: Distributing Apache Airflow

  1. Introduction
  2. Sequential Executor with SQLite
  3. Local Executor with PostgreSQL
  4. [Practice] Executing tasks in parallel with the Local Executor
  5. [Practice] Ad Hoc Queries with the metadata database
  6. Scale out Apache Airflow with Celery Executors and Redis
  7. [Practice] Set up the Airflow cluster with Celery Executors and Docker
  8. [Practice] Distributing your tasks with the Celery Executor
  9. [Practice] Adding new worker nodes with the Celery Executor
  10. [Practice] Sending tasks to a specific worker with Queues
  11. [Practice] Pools and priority_weights: Limiting parallelism – prioritizing tasks
  12. Kubernetes Reminder
  13. Scaling Airflow with Kubernetes Executors
  14. [Practice] Set up a 3 nodes Kubernetes Cluster with Vagrant and Rancher
  15. [Practice] Installing Airflow with Rancher and the Kubernetes Executor
  16. [Practice] Running your DAGs with the Kubernetes Executor

Section 6: Improving your DAGs with advanced concepts

  1. Introduction
  2. Minimising Repetitive Patterns With SubDAGs
  3. [Practice] Grouping your tasks with SubDAGs and Deadlocks
  4. Making different paths in your DAGs with Branching
  5. [Practice] Make Your First Conditional Task Using Branching
  6. Trigger rules for your tasks
  7. [Practice] Changing how your tasks are triggered
  8. Avoid hard coding values with Variables, Macros and Templates
  9. [Practice] Templating your tasks
  10. How to share data between your tasks with XCOMs
  11. [Practice] Sharing (big?) data with XCOMs
  12. TriggerDagRunOperator or when your DAG controls another DAG
  13. [Practice] Trigger a DAG from another DAG
  14. Dependencies between your DAGs with the ExternalTaskSensor
  15. [Practice] Make your DAGs dependent with the ExternalTaskSensor

Section 7: Deploying Airflow on AWS EKS with Kubernetes Executors and Rancher

  1. Introduction
  2. Quick overview of AWS EKS
  3. [Practice] Set up an EC2 instance for Rancher
  4. [Practice] Create an IAM User with permissions
  5. [Practice] Create an ECR repository
  6. [Practice] Create an EKS cluster with Rancher
  7. How to access your applications from the outside
  8. [Practice] Deploy Nginx Ingress with Catalogs (Helm)
  9. [Practice] Deploy and run Airflow with the Kubernetes Executor on EKS
  10. [Practice] Cleaning your AWS services

Section 8: Monitoring Apache Airflow

  1. Introduction
  2. How the logging system works in Airflow
  3. [Practice] Setting up custom logging
  4. [Practice] Storing your logs in AWS S3
  5. Elasticsearch Reminder
  6. [Practice] Configuring Airflow with Elasticsearch
  7. [Practice] Monitoring your DAGs with Elasticsearch
  8. Introduction to metrics
  9. [Practice] Monitoring Airflow with TIG stack
  10. [Practice] Triggering alerts for Airflow with Grafana
  11. Airflow maintenance DAGs

Section 9: Security in Apache Airflow

  1. Introduction
  2. [Practice] Encrypting sensitive data with Fernet
  3. [Practice] Rotating the Fernet Key
  4. [Practice] Hiding variables
  5. [Practice] Password authentication and filter by owner
  6. [Practice] RBAC UI

Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

Airflow udemy course

Airflow Udemy App

In the Complete Hands-On Introduction to Apache Airflow, you are going to learn everything you need in order to start using Airflow and discover how powerful it is. Through theory and practical videos, you will see basic as well as advanced concepts.

Tell me where do you come from and enter you email to receive the link. Don’t worry, you will not get spammed. I don’t like spams either.

Air Flow Due To Differential Pressure

Airflow udemy app

Don’t forget to check your spams for the link.

Section 1: Course Introduction

  1. Important Prerequisites
  2. Course Objectives
  3. Course Structure
  4. Who I am
  5. Development Environment

Section 2: First Approach to Airflow

  1. Introduction
  2. Why use Airflow?
  3. What is Airflow?
  4. How Airflow works?
  5. [Practice] Installing Airflow
  6. [Practice] Quick Tour of Airflow UI
  7. [Practice] Quick Tour of Airflow CLI
  8. Quiz Time!
  9. Recap

Section 3: Coding Your First Data Pipeline with Airflow

  1. Introduction
  2. What is a DAG?
  3. [Practice] Time to code your first DAG
  4. What is an Operator?
  5. [Practice] Let’s use Operators
  6. Operator Relationships and Bitshift Composition
  7. [Practice] Adding dependencies
  8. How the Scheduler Works?
  9. [Practice] A Quick Play With Backfill and Catchup
  10. What is a Workflow?
  11. Quiz Time!
  12. Recap

Section 4: Databases and Executors

  1. Introduction
  2. Sequential Executor with SQLite
  3. Local Executor with PostgreSQL
  4. [Practice] Configure a DAG with Local Executor and PostgreSQL
  5. Celery Executor with PostgreSQL and RabbitMQ
  6. [Practice] Configure a DAG with Celery Executor, PostgreSQL and RabbitMQ
  7. Quiz Time!
  8. Recap

Section 5: Advanced Concepts

Airflow Udemy Download

  1. Introduction
  2. Minimising Repetitive Patterns With SubDAGs
  3. [Practice] Minimising a DAG with SubDAGs
  4. How to Interact With External Sources Using Hooks?
  5. [Practice] Get Results From PostgreSQL Using Hooks
  6. How to Share Data Between Your Tasks With XCOMs?
  7. [Practice] Share Your First Messages Using XCOMs
  8. How to Execute Tasks According To Criteria Using Branching
  9. [Practice] Make Your First Conditional Tasks
  10. Control Your Tasks With SLAs
  11. [Practice] Defining a SLA in a DAG
  12. Quiz Time!
  13. Recap

Section 6: Customising Your Apache Airflow Installation With Plugins

  1. Introduction
  2. Adding Functionalities to Apache Airflow
  3. [Practice] Creating a Hook to Interact With Elasticsearch
  4. [Practice] Creating a Transfer Operator PostgresqlToElasticsearch
  5. [Practice] Adding a View to Apache Airflow UI
  6. Quiz Time!
  7. Recap

Airflow Training Udemy

  1. Introduction
  2. Quick Reminder About Docker
  3. [Practice] Building And Running Your First Airflow Docker Image
  4. [Practice] Running Airflow Using Sequential Executor With Docker
  5. [Practice] Running Airflow Using Local Executor With Docker
  6. [Practice] Scaling Airflow Using Celery Executor With Docker
  7. Quiz Time!
  8. Recap