Upload
hugh-mccamphill
View
212
Download
0
Embed Size (px)
Citation preview
@hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis: Towards robust, reliable and timely tests
Hugh McCamphill
@hughleo01 Liberty Information Technology2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centersLIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us
3Liberty IT
Hugh McCamphill
Principal Software Engineer in Test
Writing Automation since 2004
@hughleo01
@hughleo01 Liberty Information Technology4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
@hughleo01 Liberty Information Technology5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, NorthumberlandImage Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests
@hughleo01 Liberty Information Technology6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto" by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?
@hughleo01 Liberty Information Technology7Liberty IT @hughleo01
@hughleo01 Liberty Information Technology8Liberty IT @hughleo01
@hughleo01 Liberty Information Technology9Liberty IT @hughleo01
~ Henry David Thoreau
@hughleo01 Liberty Information Technology10Liberty IT @hughleo01
@hughleo01 Liberty Information Technology11Liberty IT @hughleo01
Example dashboard
@hughleo01 Liberty Information Technology12Liberty IT @hughleo01
RunTests ResultsPost
Results
A Simple Process Flow
@hughleo01 Liberty Information Technology13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data
@hughleo01 Liberty Information Technology14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json
@hughleo01 Liberty Information Technology18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject","environment":"QA","classname":“TestProject.Foo.Bar","name":"ShouldBeAbleToAddItem","duration":"47.12","created":"2016-09-14 21:35:53","status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results
@hughleo01 Liberty Information Technology19Liberty IT @hughleo01
A Simple Process Flow
RunTests ResultsPost
Results Visualize
@hughleo01 Liberty Information Technology20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch
@hughleo01 Liberty Information Technology21Liberty IT @hughleo01
@hughleo01 Liberty Information Technology22Liberty IT @hughleo01
New visualization results in a single bucket
@hughleo01 Liberty Information Technology23Liberty IT @hughleo01
Splitting error messages into buckets of top occurring error messages
@hughleo01 Liberty Information Technology24Liberty IT @hughleo01
Showing top ten most occurring error messages
@hughleo01 Liberty Information Technology25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology27Liberty IT @hughleo01
Top Failure Reasons By Test
@hughleo01 Liberty Information Technology28Liberty IT @hughleo01
Top Slowest Tests
@hughleo01 Liberty Information Technology29Liberty IT @hughleo01
Top Failing Tests
@hughleo01 Liberty Information Technology30Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology31Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology32Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology33Liberty IT @hughleo01
Example Dashboard With Time Filter
@hughleo01 Liberty Information Technology34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/
@hughleo01 Liberty Information Technology35Liberty IT @hughleo01
Thank you
Hugh McCamphill@hughleo01
Run Tests ResultsPost
Results Visualize
In Summary