Upload
julia-barker
View
227
Download
0
Tags:
Embed Size (px)
Citation preview
P.C. Burkimsher IT-CO-BE July 2004
Scaling Up PVSS
Showstopper Tests
Paul Burkimsher IT-CO
Aim of the Scaling Up ProjectWYSIWYAF
Investigate functionality and performance of large PVSS systems
Reassure ourselves that PVSS scales to support large systems
Provide detail rather than bland reassurances
What has been achieved?
18 months PVSS gone through many pre-release versions– “2.13”– 3.0Alpha– 3.0Pre-Beta– 3.0Beta– 3.0RC1– 3.0RC1.5
Lots of feedback to ETM. ETM have incorporated
– Design fixes & Bug fixes
Progress of the project
Has closely followed the different versions. Some going over the same ground, repeating tests as bugs were fixed.
Good news: V3.0 Official Release is now here (even 3.0.1)
Aim of this talk: – Summarise where we’ve got to today.– Show that the list of potential
“showstoppers” has been addressed
What were the potential showstoppers?
Basic functionality– Synchronised types in V2 !
Sheer number of systems– Can the implementation cope?
Sheer number of displaysAlert Avalanches
– How does PVSS degrade?Is load of many Alerts reasonable?Is load of many Trends reasonable?
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches–How does PVSS degrade?
Is load of many Alerts reasonable?Is load of many Trends
reasonable?
}Skip
Sheer number of systems
130 systems simulated on 5 machines
40,000 DPEs~5 million DPEs
Interconnected successfully
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches–How does PVSS degrade?
Is load of many Alerts reasonable?Is load of many Trends
reasonable?
}Skip
Alert Avalanche Configuration
2 WXP machinesEach machine = 1 systemEach system has 5 crates declared x
256 channels x 2 alerts in each channel (“voltage” and “current”)
40,000 DPEs total in each systemEach system showed alerts from both
systems
9491UI
UI
Traffic & Alert Generation
Simple UI script
Repeat– Delay D mS– Change N DPEs
Traffic rate D \ N– Bursts.– Not changes/sec.
Option provoke alerts
Alert Avalanche Test Results - I
You can select which system’s alerts you wish to view
UI caches ALL alerts from ALL selected systems.
Needs sufficient RAM! (5,000 CAME + 5,000 WENT alerts needed 80Mb)
Screen update is CPU hungry and an avalanche takes time(!)– 30 sec for 10,000 lines.
Alert Avalanche Test Results - II
Too many alerts -> progressive degradation
1) Screen update suspended – Message shown
2) Evasive Action. Event Manager eventually cuts the connection to the UI; UI suicides.– EM correctly processed ALL alerts
and LOST NO DATA.
Alert Avalanche Test Results - III
Alert screen update is CPU intensive
Scattered alert screens behave the same as local ones. (TCP)
“Went” alerts that are acknowledged on one alert screen disappear from the other alert screens, as expected.– Bugs we reported have now been
fixed.
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches– How does PVSS degrade?
Is load of many Alerts reasonable?
Is load of many Trends reasonable?
Agreed Realistic Configuration
3 level hierarchy of machinesOnly ancestral connections, no peer
links. Only direct connections allowed.40,000 DPEs in each system, 1 sys per
machineMixed platform (W=Windows, L=Linux)
L
L L L L L L L L L L L L
W W
W
91
92 93 94
95 04 05 06 07 08 09 10 11 12 1303
Viewing Alerts coming from leaf systems
1,000 “came” alerts generated on PC94 took 15 sec to be absorbed by PC91. All 4(2) CPUs in PC91 shouldered the load.
Additional alerts then fed from PC93 to the top node.– Same graceful degradation and evasive action seen
as before. PC91’s EM killed PC91’s Alert ScreenDisplay is again the bottleneck.
Rate supportable from 2 systems
Set up a high, but supportable rate of traffic (10,000 \ 1,000) on each of PC93 and PC94, feeding PC91.
PC93 itself was almost saturated, but PC91 coped (~200 alerts/sec average, dual CPU)
91
92 93 94
95 04 05 06 07 08 09 10 11 12 1303
Surprise Overload (manual)
Manually stop PC93PC91 pops up a message Manually restart PC93Rush of traffic to PC91 caused PC93 to
overloadPC93’s EM killed PC93’s DistMPC91 pops up a message
91
92 93 94
95 04 05 06 07 08 09 10 11 12 1303
PVSS Self-healing property
PVSS self-healing algorithm– Pmon on PC93 restarts PC93’s DistM
Remarks
Evasive action taken by EM, cutting connection, is very good. Localises problems, keeping the overall system intact.
Self-healing action is very good. Automatic restart of dead managers
BUT…
Evasive action and Self-healing
Manually stop PC93PC91 pops up a messageManually restart PC93Rush of traffic to PC91 causes
PC93 to overloadPC93’s EM killed PC93’s DistMPC91 pops up a messagePmon restarts PC93’s DistM
91
92
93
94
Self-healing Improvement
To avoid the infinite loop, ETM’s Pmon eventually gives up.
Configurable how soon – Still not ideal!
ETM are currently considering my suggestion for improvement:– Pmon should issue the restart, but not
immediately.
(Old) Alert Screen
We fed back many problems with the Alert Screen during the pre-release trials. – E.g. leaves stale information on-
screen when systems leave and come back.
New Alert/Event Screen in V3.0
3.0Official release now has a completely new Alert/Event Screen which fixes most of the problems.
It’s new and still has some bugs, but the ones we have seen are neither design problems nor showstoppers.
More work for ETM:
When DistM is killed by EM taking evasive action, the only indication is in the log.
But Log viewer, like Alert viewer, is heavy on CPU and shouldn’t be left running when it’s not needed.
Reconnection Behaviour
No gaps in the Alert archive of the machine that isolated itself by taking evasive action. No data was lost.
It takes about 20 sec for 2 newly restarted Distribution Managers to get back in contact.
Existing (new-style!) alert screens are updated with the alerts of new systems that join (or re-join) the cluster.
Is load of many Alerts reasonable?
~200 alerts/sec average would be rather worrying in a production system. So I believe “Yes”.
The response to an overload is very good. Though can still be tweaked.
Data integrity is preserved throughout.
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches– How does PVSS degrade?
Is load of many Alerts reasonable?Is load of many Trends
reasonable?
Can you see the baby?
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches– How does PVSS degrade?
Is load of many Alerts reasonable?Is load of many Trends
reasonable?
Is the load of many Trends reasonable?
Same configuration:91
92 93 94
95 04 05 06 07 08 09 10 11 12 1303
Trend windows were opened on PC91 displaying data from more and more systems. Mixed platform.
Is Memory Usage Reasonable?RAM
(MB)
Steady state, no trends open on PC91 593
Open plot ctrl panel on 91 658
On PC91, open a 1 channel trend window from PC03 658
On PC91, open a 1 channel trend window from PC04 657
On PC91, open a 1 channel trend window from PC05 657
On PC91, open a 1 channel trend window from PC06 658
On PC91, open a 1 channel trend window from PC07 658
Yes
Is Memory Usage Reasonable?
RAM
Steady state, no trends open on PC91 602
On PC91, open 16 single channel trend windows from PC95Crate1Board1 604
On PC91, open 16 single channel trend windows from PC03Crate1Board1 607
On PC91, open 16 single channel trend windows from PC04Crate1Board1 610
Yes
Test 34: Looked at top node plotting data from leaf machines’ archives
Performed excellently.Test ceased when we ran out of
screen real estate to show even the iconised trends (48 of).
Bland result? No!
Did the tests go smoothly? No!– But there was good news at the end
Observed gaps in the trend!!
Investigation showed gap was correct – Remote Desktop start-up caused CPU load– Data changes were not generated at this time
Zzzzzzz
Proof with a Scattered Generator
Steady traffic generationNo gaps in the recorded archive
– Even when deliberately soak up CPU
Gaps were seen in the display– Need a “Trend Refresh” button (ETM)
Scattered UI on PC93
TrafficEM
Trend UI on PC94
Zzzzzzz
Would sustained overload give trend problems?High traffic (400mS delay\1000
changes) on PC93, as a scattered member of PC94’s system.
PC94’s own trend plot could not keep up.
PC91’s trend plot could not keep up.
“Not keep up” means…
Zzzzzzz
“Display can’t keep up” means…
Trend screen values updated to here
Timenow
Zzzzzzz
Evasive action
Trend screen values finally updated to here
Timenow
EM took evasive action, (disconnected the traffic generator) just here
Last 65sec queued in Traffic Generator. Lost when it suicided.
Zzzzzzz
Summary of Multiple Trending
PVSS can copePVSS is very resilient to overload
Successful tests.
Wakey!
Test 31 DP change rates
Measured saturation rates on different platform configurations.
No surprises. Faster machines with more memory are better. Linux is better than Windows.
Numbers on the Web.
Test 32 DP changes with alerts
Measured saturation rates; no surprises again.
Dual CPU can help in processing when there are a lot of alert screen (user interface) updates.
What were the potential showstoppers?
Basic functionality– Synchronised types in V2!
Sheer number of systems– Can the implementation cope?
Alert Avalanches– How does PVSS degrade?
Is load of many Alerts reasonable?
Is load of many Trends reasonable?Conclusions
Conclusions
No showstoppers.
We have seen nothing to suggest that PVSS cannot be used to build a very big system.
Further work - IFurther “informational” tests will be
conducted to assist in making configuration recommendations, eg understanding the configurability of the message queuing and evasive action mechanism.
Follow up issues such as “AES needed more CPU when scattered”.
Traffic overload from a SIM driver rather than a UI
Collaborate with Peter C. to perform network overload tests.
Further work – II
Request a Use Case from experiments for a non-stressed configuration:– Realistic sustained alert rates– Realistic peak alert rate + realistic duration
• i.e. not a sustained avalanche– How many users connected to control room
machine?– % viewing alerts; % viewing trends; %
viewing numbers (eg CAEN voltages)– Terminal Server UI connections– How many UIs can control room cope with?
What recommendations do you want?
In greater detail…
The numbers behind these slides will soon be available on the Web at http://itcobe.web.cern.ch/itcobe/Projects/ScalingUpPVSS/welcome.html
Any questions?
Can you see the baby?
Example Numbers
Name O/S GHz GB Rate@~70% CPU
PC92 Linux 2.2 x 2
2 1000\1000
PC93 W2000 1.8 0.5 1000\500
PC94 WXP 2.4 1 2000\1000
PC95 Linux 2.4 1 1000\1000
PC03 Linux 0.7 0.25 2000\1000
Table showing the Traffic Rates on different machine configurations, that gave rise to 70% CPU usage on those machines. See the Web links for the original table and details on how to interpret the figures.