Embed Size (px)
Citation preview
YYZ
Modeling Your Applications Based on Airport Operations
Yes, even the TSA Part
My day job…
But my hobby is…
❖ ATL
❖ AUS
❖ BOS
❖ CDG
❖ CLE
❖ CLT
❖ CMH
❖ CVG
❖ DAL
❖ DCA
❖ DEN
❖ DFW
❖ DTW
❖ DUB
❖ EWR
❖ FLL
❖ FPO
❖ HOU
❖ HPN
❖ IAD
❖ IAH
❖ JFK
❖ LAS
❖ LAX
❖ LGA
❖ LHR
❖ LUK
❖ MIA
❖ NAS
❖ NRT
❖ ORD
❖ ORL
❖ ONT
❖ PDX
❖ PHL
❖ PHX
❖ PSP
❖ PUR
❖ PVD
❖ SAN
❖ SFO
❖ SLC
❖ SMF
❖ SIN
❖ STL
❖ YHZ
❖ YYZ
Which gives me time to…
…and stand in line.
And of course complain.
Yet…
80%
On-Time
6.48% - Aviation System
6.87% - Late Aircraft
5.09% - Maintenance/Crew
0.34% - Weather
0.02% - Security
Waiting on me was never listed as an issue.
145,200 pieces per day
416 mishaps
0.287%
Mishandled Bag Rate
So while most of our experiences at airports contain stories of horror…
…they all work pretty much the same.
Airports must be doing something right.
Disperate, detached operations, all working together. Albeit loosely.
What lessons can I steal from airport operations and apply to system design?
7 Patterns
ORD
Airport Queue Patterns
Moving data through a system like people through an airport.
Typical Airport Day :)
❖ Get Your Boarding Pass
❖ Check Your Bag
❖ Get Routed to a Security Checkpoint
❖ Get ID Checked
❖ Get Scanned
❖ Get Food
❖ Jockey for Boarding Position
❖ Line Up for Take Off
❖ Line Up for Landing
❖ Jockey for Deplaning
❖ Retrieve Bag
Typical Airport Day :(
❖ Get Frisked/Searched
❖ Wait in Customs
❖ Retrieve Bag at Entry Point
❖ Get Scanned Again
❖ Re-Check Bag
❖ Repeat Boarding Process
Most of these are covered in the patterns we will discuss.
1 Queue/Queue Manager
aka, “The Tag Team”
Queue Worker
Capacity Queued Jobs
Typical Airport Scenario
Current Hell
Manager
Worker
Previous Hell
Next Hell
WorkerWorker
Typical Queue Scenario
Feeder
Queue Worker
Queue Manager
Queue Worker
Queue Worker
PreviousFeeder
NextFeeder
Capacity Queued Jobs
Queue Worker
2 Infinite Scalability
aka, “Open Up Another Damned Line!”
TSA
Normal Worker
TSA Manager
Pre-Check Worker
Frisk Worker
Gate
Physical Capacity
Resource Capacity
We can mimic the physical aspect of the airport model by adding more queues and corresponding workers…
Feeder
Queue Worker
Next Feeder
- Adding more queues with a corresponding worker can work.- Must be able to carve off work from the overloaded queue.- Work units need to be discrete.- QM needs to also balance the load.
Previous Feeder
Queue Worker
Queue Worker
Queue Manager
Personally though, I would rather do this…
Feeder
Queue Worker
Next Feeder
- Physical capacity is irrelevant (mostly).- Resource capacity is virtually infinite.- Manager determines how many workers to get to green. Resource
Capacity
Previous Feeder
Queue Manager
Simple Queue Manager
Scaling Formula
Increase scale
((Qd/Qw)/t) >= tmx
When the calculated time it takes for all the workers to complete all of the jobs is greater than or equal to the maximum allowable time for any job to complete this step - make more workers.
But only up to some maximum. Accidental $1500 cloud bills teach you a thing or two.
Decrease scale
((Qd/Qw)/t) < tmx
When the calculated time it takes for all the workers to complete all of the jobs is less than the maximum allowable time for any job to complete this step, you’re wasting resources. Time to trim the fat!
But we need to keep one lackey around to do the menial tasks.
3 Standard Prioritization #1
aka “The Haves”
1 1 11222334
You
Cool Kids
1 Queue Worker
Aside from Re-Insertions, which we will discuss later, I have not seen this in an airport environment.
Which is good, lest…
3a Standard Prioritization #2
aka “How the ‘Other Haves’ Live”
???
TSA
Normal Worker
TSA Manager
Pre-Check Worker
Frisk Worker
Ticketing
Gate
Physical Capacity
Resource Capacity
Courtesy Sorting
Random Inputs
Feeder
Queue Worker15
Previous Feeder
Next Feeder
10 3 1111214
7 6 2131621
14
4
2
5
1
11
21
3
8
15
17
16
910
12
18
1920
13
6
7
8 5 4917181920
Queue Manager
Queue Worker
Queue Worker
Queue Worker
Processes before anything else.
Processes only if nothing in queue above.
Processes only if nothing in queue above.
Processes only if nothing in queue above.
Depending on the timing of how jobs arrive in the queue,#15 may get processed, 1st, 2nd, etc, but no later than 15th.Where as #2 may not process before 8th.
4 Arbitrary Prioritization
aka, “Every Man for Himself”
Gate Area
AA Gate
10 Gate Worker12 151311
Gate Agent
141662
Security
Aircraft
21 13 184587191720
9
14
4
2
5
1
11
21
3
8
15
17
16
910
12
18
1920
13
6
71st Class
Diamond/Platinum
Windows
Aisles
- Due to the constrained nature of aircraft access, loading more passengers faster is not possible.- To a certain extent the passenger can control their access order (mob surfing).
Gate Area
SW Gate
2 Gate Worker5 7134
Gate Agent
61489
Security
Aircraft
10 15 1312111920161721
18
1
4
11
7
10
16
12
5
13
15
8 9
1821
17
14
1920
3
6
2
A
C
B
- Due to the constrained nature of aircraft access, loading more passengers faster is not possible.- To a certain extent the passenger can control their access order.
Random Inputs
Feeder
Queue Worker
15
Previous Feeder
Next Feeder
10 3 1111214
7 6 2131621
14
4
2
5
1
11
21
3
8
15
17
16
910
12
18
1920
13
6
7
8 5 4917181920
Queue Manager
Processes before anything else.
Processes only if nothing in queue above.
Processes only if nothing in queue above.
Processes only if nothing in queue above.
Depending on the timing of how jobs arrive in the queue,#15 may get processed, 1st, 2nd, etc, but no later than 15th.Where as #2 may not process before 8th.
5 Re-Insertion
aka, “Hey, can you let me cut? I had a thing.”
Ticketing
Ticket Manager
Ticket Worker
Baggage Worker
Security
Problem Worker
Kiosk Worker
The threshold of a green vs. yellow vs. red status is different for each queue type due to the average length of a transaction for that type.
Feeder
Queue Manager
Queue Worker
Problem Worker
Next Feeder
Notification Worker
In this scenario, when a Queue Worker encounters an error, it can:
- Attempt to fix the issue and re-queue;- Assign it to a problem queue to be handled via another process;- Notify the interested user.
The Problem Worker can fix the issue then complete processing or:
- Return it to the the main queue for normal processing;- Notify the interested user.
Previous Feeder
Feeder
Queue Worker1
Next Feeder
3
Queue Worker
Resolution Queue
Problem Queue
Instead of being moved to the back of the “Normal” line,problem jobs with resolutions get expedited treatment.
6 5 4789
Normal Queue
2143
Queue Manager
Queue Worker
6 Throughput Measurement
aka, “Metric Mule”
Measuring❖ Where to Measure
❖ At the Beginning of the Process
❖ At Important Junctions in the Process
❖ At the End of the Process
❖ What to Measure
❖ Time
❖ # of Transactions
❖ Size of Payload
❖ Frequency of Certain Transaction or Data Types
???
TSA
Normal Worker
TSA Manager
Pre-Check Worker
Frisk Worker
Ticketing
Gate
Physical Capacity
Resource Capacity
Courtesy Sorting
XX
XX
XX
- Mark time, distribute a card to passenger.- Retrieve card from passenger, mark card.- Profit!
X
Feeder
Queue Worker
Next Feeder
- Physical capacity is irrelevant (mostly).- There is no “unknown” or holding pen.- Resource capacity is virtually infinite.- Priority is set by business rules.- Manager determines how many workers to get to green.
Resource Capacity
Previous Feeder
Queue Worker
Queue Worker
QM applies formula to get to green.
Queue Manager
XXXXXXXXX
XXXXX
X- Mark time at queue entry.- Mark time at queue exit.- Adjust worker create/destroy thresholds based on latest data.
{
"qmsg":
{
"id":"log::97e3fa40-1aa6-49cb-8a50-d8b067b98898",
"company_id":"cmp::cc349da1-cf57-49b6-b1b3-4a8af9d52388",
"action":"log:api",
"objtype":"QAPILog",
"entered":"2014-09-12T06:18:19.169408997Z",
"processed":"0001-01-01T00:00:00Z"
},
"method":"PUT",
"remote_ip":"64.134.173.96",
"route":"/guest/graham.savage/location/74C9BFCF-4698-4D83-B867-77407C075C33/proximity/3",
"agent":"Grand%20Lucayan/1.0 CFNetwork/672.1.15 Darwin/14.0.0",
"status":404,
"size":96,
"ms":0.354969
}
Using the timing data…
((Qd/Qw)/t) < tmx
((Qd/Qw)/t) >= tmx
7 Begetting
aka, “You Had One Job!”
OPS
Passenger Worker
Baggage Worker
Security Worker
Flight Worker
Ticket Worker
The act of getting a boarding pass starts a process in multiple areas simultaneously.
Logging Worker
Error Worker
Metrics Worker
Notification Worker
Transaction Worker
Each transaction starts a chain reaction which could lead to sub jobs and a plethora of other messages.
So 1 transaction can spawn a dozen additional operations that will travel to their respective queues.
1. Queue/Queue Manager “The Tag Team”
2. Infinite Scalability “Open Up Another Damned Line!”
3. Standard Prioritization #1 “The Haves”
3a. Standard Prioritization #2 “How the ‘Other Haves’ Live”
4. Arbitrary Prioritization “Every Man for Himself”
5. Re-Insertion “Hey, can you let me cut? I had a thing.”
6. Throughput Measurement “Metric Mule”
7. Begetting “You Had One Job!”
