Upload
devconfu
View
130
Download
2
Tags:
Embed Size (px)
Citation preview
Partners:
April 12, 2023 www.ExigenServices.com
Hints on complex system performance optimization
Ainars Galvans
Exigen Services Latvia
3 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
System Performance
Introduction: context, terms and basics
4 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Terminology
Testing
Load
Stress
Scalability
Endurance
…
Performance …
Targets
Thresholds
Objectives
Other
Stability
Workload
Response time
Scenario
5 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Performance is:
Requirement MeasurementNumber of users ThroughputHardware spec. CPU utilization“Response time” Response time
i.e. time user wait for page to load in the browser
6 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
A simple 3-tier Web Application
Prepare request
Parse, process
Select or update
Select or update
More Logic
Render HTML
Render UI
Client App. DB
Response tim
e
User E
xperience
Repeated
7 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
A simple 3-tier Web Application
Client1
App. DB
Client2
8 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
With 3rd party WebServices
Client1App. DB
Client2
9 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Other Apps share the same DB?
App. DB Other (Corp.)App.
10 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Performance is:
Requirement MeasurementNumber of users ThroughputHardware spec. CPU utilization“Response time” Response time
11 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Rules of thumb
Response time– >2 seconds: is it all CPU time?– 0.5-2.0 seconds: why?– <0.5 seconds : OK
Concurrent users– The term is misleading– What matters technically is throughput
Resource utilization:– DB CPU should be below 50%
12 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
My issue localization techniques
First step: where (what host) is the bottleneck?– Single user repeat the same action (i.e. with Selenium)– Monitor “CPU time” delta (secs.), not CPU (%)– Use excel to understand how it scales
If it’s the App Server: analyze HTML– FireFox “Save Page As” -> complete page – Look for large, unused data
If it’s DB or Web Services: analyze traffic, logs– Put milliseconds to log files, see what takes most of it– Look for repeated requests for single page rendering
If it’s client (browser): profile JavaScript– IE -> Dev. Tools (F12) -> Profiler– Are all the JavaScripts cached (or compiled each time)?
13 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
System Performance
Performance optimization stories
14 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Locating database call issues
Client1
App. DB
Client2
15 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Story #1: Just a few tables
Problem description– Web Page shows a number of tables– Page response time is a bit above 2 seconds– Under load DB becomes overloaded quickly– All DB queries takes <0.1 sec
Investigation results:– Single page load cause about 100 select executions– Each took 0.02 secs– There were two distinct statements
Solution– After adjusting DB caching – Response times reduced to 0.4– Almost no load to DB under load
16 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Locating WebService call issues
Client1App. DB
Client2
17 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Story #2: One call only takes 0.5 secs.
Problem description– Web Page shows user’s data overview– Typical customer page response time 5 seconds– Response time up to 30 seconds for specific customers– Under load response times don’t increase too much
Investigation results:– Data in tables is requested via web services– Separate call made on each business item– Call response time is 0.5 secs
Solution– Protocol adjusted to request all info with one call– Response times reduced to 0.9
18 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Application Logic (HTML processing)
Client1
App. DB
Client2
19 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Story #3: Tables, again: .NET
Problem description:– Page shows main page and a dynamic table with hot links– Response times >5 secs. – App server loaded significantly– Under load response times increase
Investigation– In page source (HTML) I’ve discovered huge callbackstate
Solution– http://www.devexpress.com/Support/Center/Question/Details/Q42
5597– Response times < 3secs. now [Browsable(false)]
public string UnboundLargeData { get; set; }
20 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Client Side (rare case though)
Client1
App. DB
Client2
21 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Story #4: Client side
Problem– Visually page loads 8-12 seconds and vary– Looks like response time depend on client workstation
Investigation– Server response time is only 3 seconds– It takes browser 1-2 seconds to send request– It takes browser 3-4 seconds to render response– It takes more if workstation CPU is slow (<3 GHz)
Solution– Optimize JavaScript and remove features from page– Browser choice recommendation– Unfortunately: optimal client CPU recommendation
22 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
… everything. The business itself
Client1
App. DB
Client2
23 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Story #5. The reports and queries
Production problem – Special feature “accounting reports” usually work 1-2 mins– Sometimes 10 times as much– Suspected reason a “morning lag” (all users)
Investigation– One report loads DB CPU to 100% for 1-2 minutes – Only technical solution: rewrite whole app, changing DB
structure and logic The actual solution
– Nightly batch jobs on demand– The default nightly batch jobs– Daily execution only with special privileges and only one at a
time
24 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Conclusions
Localizing the problem
25 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
My issue localization techniques
First step: where (what host) is the bottleneck?– Single user repeat the same action (i.e. with Selenium)– Monitor “CPU time” delta (secs.), not CPU (%)– Use excel to understand how it scales
If it’s the App Server: analyze HTML– FireFox “Save Page As” -> complete page – Look for large, unused data
If it’s DB or Web Services: analyze traffic, logs– Put milliseconds to log files, see what takes most of it– Look for repeated requests for single page rendering
If it’s client (browser): profile JavaScript– IE -> Dev. Tools (F12) -> Profiler– Are all the JavaScripts cached (or compiled each time)?
26 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com
Happy to share more details
Skype name: ainarsgalvans
Phone: +371 29432698
Facebook: Ainars Galvans
Email: [email protected]
Blog: find me at SoftwareTestingClub.com
Feedback
Partners: