© 2012 Whamcloud, Inc.
Whamcloud and Quality
• Chris Gearing & Mike StokSoftware EngineersWhamcloud, Inc
Lustre User GroupAustin TXApril 2012
Version 1.0
© 2012 Whamcloud, Inc.2
• Whamcloud’s View Of Quality
• A Year’s Progress
• The Year Ahead
• Maloo ‘Your Window onto Test’ – Mike Stok
Agenda
© 2012 Whamcloud, Inc.3
Whamcloud’s View Of Quality
© 2012 Whamcloud, Inc.4
Eric vs. Non Delivery
‟ The only thing I’m non-Linear about is
Whamcloud not delivering what it says
it will deliver ”
Whamcloud’s View Of Quality
Eric BartonCTO Whamcloud
© 2012 Whamcloud, Inc.5
• Whamcloud’s view of quality development
Whamcloud’s View Of Quality
StabilityQualityFeat
ures
Perf
orm
ance
Time
© 2012 Whamcloud, Inc.6
• Whamcloud is investing money, time and expertise in Quality
• …is continuing to develop tools and invest in infrastructure to enhance the Lustre community
• The whole of its engineering team is orientated around quality principles
Whamcloud’s View Of Quality
© 2012 Whamcloud, Inc.7
A Year’s Progress
© 2012 Whamcloud, Inc.8
• For landing testing we have carried out;– 687 days of landing testing– 2381 landing sessions– 2092758 individual tests– 2087607 passed
• Statistically– 1425 passes – 956 failures– 59.85% Session passed
Landing Test Performance
© 2012 Whamcloud, Inc.9
• This chart shows the percentage of landings that passed all tests since LUG 2011
Landing Test Performance
05/2
011
06/2
011
07/2
011
08/2
011
09/2
011
10/2
011
11/2
011
12/2
011
01/2
012
02/2
012
03/2
012
30.00%
35.00%
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
Pass RateLinear (Pass Rate)
© 2012 Whamcloud, Inc.10
Distributed Test
Source code
repository
PublicSector
ISVs
Resellers
© 2012 Whamcloud, Inc.11
Distributed Test
Source code
repository
PublicSector
ISVs
PublicSectorResellers
Resellers
But!
© 2012 Whamcloud, Inc.12
• Juelich financed a fully equipped cluster• Used for testing all head releases• Specialises in failover testing
– 36TB of multi attached storage
• Good for performance regression tests– Private network with no contention so results are repeatable
• Also used for manual large Lun testing for 2.2 release
• I’d like to thank Juelich and in particular Frank Heckes for making this happen
Juelich Supercomputer Centre
© 2012 Whamcloud, Inc.13
• Indiana provided a 36 node cluster for the development and rollout of a backup test system
• Used for secondary 2.2 release testing• Enabled the transfer of the primary test cluster
from the West Coast to Colorado without any break in the Lustre test and landing processes
• Again I’d like to thank Indiana and Steve Simms for enabling this
Indiana University
© 2012 Whamcloud, Inc.14
The Year Ahead
© 2012 Whamcloud, Inc.15
Development vs. Landing Test• Which is landing test?• Which is development test?
05/2
011
06/2
011
07/2
011
08/2
011
09/2
011
10/2
011
11/2
011
12/2
011
01/2
012
02/2
012
03/2
012
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Pass RateLinear (Pass Rate)
© 2012 Whamcloud, Inc.16
• Distinct paths for development vs. landing– Similar process but distinct purpose
• Provide for maximum flexibility in development testing– Encourage systematic test to be part of the development process– Test during development does improve product quality
• Encourage peer review as the code is written– Earlier review leads to better code and more opportunity for
education
• Be auditable– Improvement requires knowledge of the past– Development test part of the landing collateral
• 100% pass rate for landing test– Developers should push tried and tested code for landing
Development vs. Landing Test
© 2012 Whamcloud, Inc.17
Development Test Cloud
© 2012 Whamcloud, Inc.18
The Goal
0%
20%
40%
60%
80%
100%
Development Test
FailPass
0%
20%
40%
60%
80%
100%
Landing Results
FailPass
© 2012 Whamcloud, Inc.19
• Whamcloud’s Quality Approach
• A Review Of The Last Year
• Our Plans For The Coming Year
Summary
© 2012 Whamcloud, Inc.20
Maloo ‘Your Window onto Test’
© 2012 Whamcloud, Inc.21 LUG Austin, TX - April 2012
• What is Maloo?• Recent changes to Maloo
– High points since last LUG
• Development priorities– Tool quality– Easy access to timely, accurate data
• Some planned features– User preferences– Automated scanning of incoming log files for “interesting” data
Maloo Agenda
© 2012 Whamcloud, Inc.22 LUG Austin, TX - April 2012
• A repository for lustre test result data– Collects the test results and the logs generated– Allows users to query the database– Contains about 1TB of log files
• https://maloo.whamcloud.com
What is Maloo?
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201223
Recent changes to Maloo
© 2012 Whamcloud, Inc.24
Release report
LUG Austin, TX - April 2012
https://maloo.whamcloud.com/reports
© 2012 Whamcloud, Inc.25
Node utilization report
LUG Austin, TX - April 2012
https://maloo.whamcloud.com/reports/show_node_utilization_report
© 2012 Whamcloud, Inc.26
Internal changes
• Invisible work– Development practices– Testing– Packaging– Deployment– …
LUG Austin, TX - April 2012
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201227
Development priorities
© 2012 Whamcloud, Inc.28 LUG Austin, TX - April 2012
• Tool quality– Mechanics of Maloo and its development– Visibility into the tool– Usability
Development priorities
© 2012 Whamcloud, Inc.29 LUG Austin, TX - April 2012
• Data quality– Accurate– Timely– Accessible
Development priorities
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201230
New features
© 2012 Whamcloud, Inc.31 LUG Austin, TX - April 2012
Log file scanning
• Automates a tedious task
New features
© 2012 Whamcloud, Inc.
Maloo Footprints
© 2012 Whamcloud, Inc.33
Maloo Footprints
© 2012 Whamcloud, Inc.34
Maloo Footprints
© 2012 Whamcloud, Inc.35
Maloo Footprints
© 2012 Whamcloud, Inc.36 LUG Austin, TX - April 2012
User profiles
• One size needn’t fit all
New features
© 2012 Whamcloud, Inc.37 LUG Austin, TX - April 2012
What was covered• Changes in Maloo since last year• Our development priorities• A couple of the planned changes
Wrap up and questions
© 2012 Whamcloud, Inc.
Thank you very much
38