8th Sakai Conference

4-7 December 2007Newport Beach

Collaborative Performance TestingCollaborative Performance TestingHealthy Medicine for Robust Sakai CodeHealthy Medicine for Robust Sakai Code

Collaborative Performance Collaborative Performance TestingTesting

• Using the example of Samigo, how performance testing can help developers find and fix performance defects before production.

Who are we?Who are we?

• Lydia Li – Sakai Developer, Stanford University

• Chris Franz – President, Unicon

• Drew Zhu – Oracle DBA, University of Michigan

• Chris Kretler – Performance Tester, University of Michigan

What is performance testing?What is performance testing?

• Replication of middleware calls• Tests have different goals:

– “Load”– “Stress”– “Benchmark”– “Capacity Planning”

• Response times, user capacity limits and infrastructure metrics

What performance testing is notWhat performance testing is not

• A Simulation– Actual traffic is generated

• GUI test

• High-volume functional test– Scope is narrower

When do we test?When do we test?

• All major releases – “Significant” patches

• Hardware Evaluations

• Configuration Changes

• Sakai performance testing

Michigan Test EnvironmentMichigan Test Environment

• Dedicated testing environment:– Load Balancers (Netscaler RS9000s)– 8 application servers (Dell 2650s running RedHat)– 1 Database server (Sun T2000 running 10g)

• System Administrators (Jeff, Adi, Rick, Ken, Jenny)

• Database Administrator (Drew)• LoadRunner software

Samigo Setup ProcessSamigo Setup Process

• Working Sakai build w/Samigo• Determined requirements and focus• Courses and students were created• Sample tests imported• Script creation

– Lydia’s help was invaluable here.

Sample Script PortionSample Script PortionUnderstanding expected result is crucial to script creation.

Load Test ResultsLoad Test Results

DBA Tasks in Load Tests DBA Tasks in Load Tests

• Monitoring Database and DB Server

• Produce AWR reports

• Analyze AWR Reports and Identify Issues

• Communicate with Developers, Testers, and Sysadmins

• Query Details and Analyzing Execution Plans

• Provide suggestions

Oracle OEM Screenshot -1Oracle OEM Screenshot -1

Oracle OEM Screenshot -2Oracle OEM Screenshot -2

Oracle OEM Screenshot -3Oracle OEM Screenshot -3

Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 3,184 3,296 68,972 0.05 43.8 38xh90r4gx5cz• select itemgradin0_.ITEMGRADINGID as ITEMGRAD1_139_, itemgradin0_.ASSESSMENTGRAD• INGID as ASSESSME2_139_, itemgradin0_.PUBLISHEDITEMID as PUBLISHE3_139_, itemgra• din0_.PUBLISHEDITEMTEXTID as PUBLISHE4_139_, itemgradin0_.AGENTID as AGENTID139_• , itemgradin0_.SUBMITTEDDATE as SUBMITTE6_139_, itemgradin0_.PUBLISHEDANSWERID a

• 858 858 254,115 0.00 11.4 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A

• 834 980 79,456 0.01 13.0 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_

• 203 203 43,210 0.00 2.7 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_

• 142 142 55,652 0.00 1.9 5n3rvywwp9y7u• select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1• _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE• MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_• from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1

Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 10,105 10,659 2,125,189 0.00 15.2 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A

• 8,184 8,459 99,794 0.08 12.1 8wcadnu8rwd5b• select count(assessment0_.ASSESSMENTGRADINGID) as col_0_0_ from SAM_ASSESSMENTGR• ADING_T assessment0_ where assessment0_.FORGRADE=1 and assessment0_.AGENTID=:1 a• nd assessment0_.PUBLISHEDASSESSMENTID=:2

• 7,290 8,974 665,235 0.01 12.8 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_

• 5,979 6,189 77,580 0.08 8.8 3cswn2dndzcm0• select assessment0_.ASSESSMENTGRADINGID as ASSESSME1_140_, assessment0_.PUBLISHE• DASSESSMENTID as PUBLISHE2_140_, assessment0_.AGENTID as AGENTID140_, assessment• 0_.SUBMITTEDDATE as SUBMITTE4_140_, assessment0_.ISLATE as ISLATE140_, assessmen• t0_.FORGRADE as FORGRADE140_, assessment0_.TOTALAUTOSCORE as TOTALAUT7_140_, ass

• 2,174 2,232 317,661 0.01 3.2 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_

• 1,950 2,026 22,183 0.09 2.9 8su2pcsy7fbq5• select assessment0_.ASSESSMENTGRADINGID as col_0_0_, publisheda1_.ID as col_1_0_• , publisheda1_.TITLE as col_2_0_, assessment0_.AGENTID as col_3_0_, assessment0_• .SUBMITTEDDATE as col_4_0_, assessment0_.ISLATE as col_5_0_, assessment0_.FORGRA• DE as col_6_0_, assessment0_.TOTALAUTOSCORE as col_7_0_, assessment0_.TOTALOVERR

New Indexes for SamigoNew Indexes for Samigo• create index idx_ASSGRAD_AID_PUBASSEID on

SAM_ASSESSMENTGRADING_T(AGENTID,PUBLISHEDASSESSMENTID) tablespace ctools_indexes; create index IDX_ITEMGRADING_A_GRADINGID on SAM_ITEMGRADING_T(ASSESSMENTGRADINGID) tablespace ctools_indexes; create index IDX_PUBANS_ITEMTEXTID on SAM_PUBLISHEDANSWER_T(ITEMTEXTID) tablespace ctools_indexes; create index IDX_PUBMETDATA_ASSESSMENTID on SAM_PUBLISHEDMETADATA_T(ASSESSMENTID) tablespace ctools_indexes;

Samigo Load Test AWRSamigo Load Test AWR• CPU Elapsed CPU per % Total• Time (s) Time (s) Executions Exec (s) DB Time SQL Id• ---------- ---------- ------------ ----------- ------- -------------• 4,840 5,046 1,007,039 0.00 21.0 0byxm85k79suc• select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS• WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE• RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as• TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A

• 3,489 4,245 315,353 0.01 17.7 3px7qs284b3xk• select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7• _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1• 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe• rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_

• 1,072 1,096 152,652 0.01 4.6 9xmw3p1px195k• select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from• SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i• n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and (• ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_

• 817 839 220,813 0.00 3.5 5n3rvywwp9y7u• select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1• _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE• MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_• from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1

• 623 643 15,350 0.04 2.7 c55kf9dj75jy4• select count(1) from SAKAI_REALM_RL_FN MAINTABLE LEFT JOIN SAKAI_REALM_RL• _GR GRANTED_ROLES ON (MAINTABLE.REALM_KEY = GRANTED_ROLES.REALM_KEY AND• MAINTABLE.ROLE_KEY = GRANTED_ROLES.ROLE_KEY), SAKAI_REALM REALMS, SAKAI_REAL• M_ROLE ROLES, SAKAI_REALM_FUNCTION FUNCTIONS where ( ROLES.ROLE_NAME in(

Server Side MonitoringServer Side Monitoring• Ran Perl script to capture CPU utilization on

one of eight app servers as well as the DB server

• The amount of free space in the JVM on one of the application servers was also captured

JAVA_OPTSJAVA_OPTSJAVA_OPTS="-server -Xms1500m -Xmx1500m"

JAVA_OPTS="$JAVA_OPTS -XX:NewSize=400m XX:MaxNewSize=400m"

JAVA_OPTS="$JAVA_OPTS -XX:PermSize=192m -XX:MaxPermSize=192m"

JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

JAVA_OPTS="$JAVA_OPTS -Dsakai.home=/usr/local/sakai/home -Dsakai.components.root=/usr/local/sakai/components"

JAVA_OPTS="$JAVA_OPTS -Dsakai.security=/usr/local/sakai/home"

JAVA_OPTS="$JAVA_OPTS -Dnetworkaddress.cache.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"

JAVA_OPTS="$JAVA_OPTS -Dsun.net.inetaddr.ttl=0 -Dsun.net.inetaddr.negative.ttl=0"

JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

CPU UtilizationCPU Utilization

• Both application server and database server utilization was spiky

• Application server was in the 20-60% range

• Database server was in the 60-100% range

CPU Utilization GraphsCPU Utilization Graphs

App Server Heap UsageApp Server Heap Usage

Application ProfilingApplication Profiling

• If bottleneck was determined to be on application server side, we were prepared to use JProfiler to capture internal JVM statistics

• Due to its overhead, a profiler can generally

only capture statistics under lighter loads

Purpose for the tripPurpose for the trip• Identify bottlenecks under load.

– no local performance test environment and expertise

• Establish a baseline for future regression testing.

• Gain confidence in our local deployment.– Samigo is an important tool for our users.

• Publish performance test report to the Sakai community.

Prepare for Performance TestPrepare for Performance Test• What test case scenarios are we testing?

– Focus on Samigo’s delivery feature.

• What is the expected usage pattern– Our user support team provided the expected # of

concurrent users at Stanford– Emailed and collected other schools’ usage information

• Prepare for the test– Created sample assessments with question types. – Send Chris XMLs to be pre-loaded to the test

environment. – Assisted Chris with load test scripts.

During the testDuring the test

• Monitored Oracle DB

• Monitored Application Servers,

• Monitored LoadRunner’s test results/graphs.

• Iterative performance tuning based on test results.

What we learnedWhat we learned

• Database:– Added indexes upon Drew’s recommendation.

• Application:– Optimized application code to reduce excessive

DB calls (SAK-10153)

• No memory leaks found.

Things we didn’t test in deliveryThings we didn’t test in delivery

• Focused on Samigo’s Delivery only– Did not test Audio and File Upload questions due

to limited time– Audio/File upload questions are heavily used at

Stanford. Please let us know if you have questions.

SummarySummary

• Performance test helps to identify bottlenecks under load.

• Establishes a baseline for future testing. – Identify performance improvement/degradation in future

releases.

• Load test should be an integral part of release.

• Performance test report can be found at http://bugs.sakaiproject.org/confluence/display/SAM/Home