Upload
robert-sanders
View
137
Download
3
Embed Size (px)
Citation preview
Airflow Clustering and
High AvailabilityBy: Robert Sanders
2Page:
Agenda
• Airflow Daemons• Single Node Deployment• Cluster Deployment• Scaling
• Worker Nodes• Master Nodes
• Limitations• Airflow Scheduler Failover Controller• Failover Controller Procedure
3Page:
Airflow Daemons
• Web Server• Daemon that runs the Airflow Webserver• 1 to many gunicorn processes to accept and process requests in
parallel.• Allows you to track jobs progress, run jobs and more
• Scheduler• Periodically runs (every X seconds) to determine if a DAG or Task
needs to be ran based off the DAG schedule• Pushes messages to the Queuing Service to be executed
• Worker• Daemon runs if you’re using the CeleryExecutors (as opposed to
SequentialExecutor and LocalExecutor)• 1 to many dedicated celeryd processes which execute functions• Pulls messages from a Queuing services to determine what
functions to execute
4Page:
Single Node Deployment
5Page:
Cluster Deployment
6Page:
Why setup a Cluster Deployment?
• Distributes heavy processes onto many machines for better use of resources
• More Highly Available Airflow environment• If you have many Workflows with many Tasks your executors
would not be able to get to all the messages in the queue. Adding more executors would fix this issue.
7Page:
Scaling Workers
• Horizontally• Add more machines to the cluster• No need to register the machines with the master. You
just need to start up the Airflow Worker task on the new Machine.
• Vertically• Increase the number of executors (celeryd processes) per
node and restart the workers
8Page:
Scaling Master
9Page:
Limitations
• There can only be one scheduler running at a time• If you have multiple Scheduler processes running, there's
a possibility that multiple instances of a single task that will be scheduled to run.
• If the Scheduler Daemon or Machine with the process goes down then no jobs will get scheduled
10Page:
Airflow Scheduler Failover Controller
• Dedicated Daemon that runs with Airflow on the Master Nodes
• Ensures that there is always one and only one Scheduler running on the Master nodes at a time
• Developed Internally and Open Sourced• https://github.com/teamclairvoyant/airflow-scheduler-fail
over-controller
• High Level Steps• Polls (every x seconds) to check if the scheduler is
running• If scheduler isn’t running, restart the scheduler• If it still doesn’t start up, then try starting it up on the
other master nodes
11Page:
Failover Controller Diagram
12Page:
Start Up Scenario
13Page:
Failover Controller Process (Start Up)
Master Node 1
Failover Controller(standby)
Master Node 2
Failover Controller(standby)
On startup, the processes start out in STANDBY
14Page:
Failover Controller Process (Start Up)
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
The first one to enter data into the Metastore is elected as the active controller.
15Page:
Failover Controller Process (Start Up)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
The Failover controller checks to see if the Scheduler is running, but it isn’t.
16Page:
Failover Controller Process (Start Up)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Failover Controller starts up the Scheduler
17Page:
Scheduler Failure Scenario
18Page:
Failover Controller Process (Process Failure)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Scheduler process has died
19Page:
Failover Controller Process (Process Failure)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Failover Controller restarts the Scheduler
20Page:
Scheduler Failure and Failed Restart
Scenario
21Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Scheduler process has died
22Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Failover Controller tries to restart the Scheduler, but its still not running
23Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Failover Controller tries to restart the Scheduler on a different node
24Page:
Failover Controller Process (Process Failure 2)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Failover Controller succeeds to restart the scheduler and the cluster is back to normal
25Page:
Node Failure Scenario
26Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover Controller(active)
Master Node 2
Failover Controller(standby)
Everything is running as expected
27Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover Controller
(dead)
Master Node 2
Failover Controller(standby)
Master Node 1 dies and all the processes running on it are gone
28Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover Controller
(dead)
Master Node 2
Failover Controller(active)
Failover Controller on Master 2 becomes active because the one running on Master Node 1 has stopped sending a heart beat
29Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover Controller
(dead)
Master Node 2
Failover Controller(active)
The newly active Failover Controller tries to check-in with and restart the Scheduler on the daemon the Metadata says its running on and fails.
30Page:
Failover Controller Process (Node Failure)
Scheduler
Master Node 1
Failover Controller
(dead)
Master Node 2
Failover Controller(active)
The Failover Controller then starts it on another node and it succeeds
Scheduler
31Page:
Failover Controller Process (Node Failure)
Master Node 1
Failover Controller(standby)
Master Node 2
Failover Controller(active)
When Master Node 1 is brought back, the old Failover Controller goes into STANDBY state
Scheduler
32Page:
Q&A