of 34 /34
FRAMEWORK FOR AUTOMATION TESTING – PRACTICE SHARING June 2013 KMS Technology: http :// kms-technology.com

Framework For Automation Testing Practice Sharing

Embed Size (px)


This is the presentation that Mr. Duong Phan and Mr. Vu Duong shared at KMS's public seminar on Saturday, June 15, 2013.

Text of Framework For Automation Testing Practice Sharing

  • 1. FRAMEWORK FOR AUTOMATION TESTING PRACTICE SHARING June 2013 KMS Technology: http://kms-technology.com

2. AGENDA Duong Phan QA Architect Introduction Challenges and Solutions KMS Test Automation Framework Advanced Features Vu Duong Engineering Manager ROI and Metrics 2 3. KMS TECHNOLOGY OVERVIEW 3 US Company/Passionate Global Workforce 400 Resources & Growing in Vietnam and the US 160 Testers ~ 50% Workforce Proven Leadership Team World Class Infrastructure Built for ISO 27001, Planned Certification in 2013 Best-in-Class SDLC Practices CMMI and Agile focus QASymphony - Commercial Agile Testing Solutions recognized by Forrester with over 4,500 users Best Clients Raving Fans 100% Referenceable and Ecstatic 100% in long-term dedicated team 4. A CASE STUDY 4 About the Client is a leader in eCommerce software and solutions, Provide multifaceted eCommerce platform, reliable 24/7/365 service, scalable IT infrastructure, integration with third party, and custom- built features, Serve over 160 retail sites Technology: Platform: Windows XP, Windows 7, Linux, OSX, iOS and Android. Technology: Java, .NET/C#, Oracle Testing Tools: JIRA, Testlink, KMS Automation Framework Achievements KMS is responsible for automation of client Core Platform Product and also for distribution platforms Reusable Automation Framework can stand independent with KMS run-time plug-in (support web testing) although can adapt other tools (Selenium) Framework support multiple OS: Windows, Linux and OSX, iOS and Android Execute test daily, Automation script is able to find a lot of defects Confidential Automation Goals Reduce cost of development and quality assurance Continuous platform upgrade to provide new features and patch releases Testing is performed on core platform and more than 200 of distributions Testing on many browsers (IE, Firefox, Chrome, Safari) 5. CHALLENGES AND OUTCOMES Project Specification Multiple Instance Application Types Flexible Components Regular Sanity Test Automation must find bugs Customer Satisfaction Cost Saving Product Quality Improvement Bug Escape Reduction 5 6. KEY FEATURES Framework using Excel as Scripting tool Easy to use and learn Easy to apply Data Driven and Keyword Driven ideas Prototype plug-in Able to develop wrapper plug-in with several tools One script can run under any plug-ins Log System General report and log types History report 6 7. KMS TEST AUTOMATION FRAMEWORK 7 8. EXECUTION MODEL AND PLUG-INS 8 9. SCRIPTING 9 10. 2013 KMS Technology ADVANCED FEATURES 11. SMART OBJECT RECOGNITION Multiple GUI instances 1 GUI for many kinds of instance objects, apply for multi sites product Example: 1 gui object txt_Email may be detect for 4 sites (gmail, yahoo mail, facebook, Linkedin) 11 12. KEYWORD DRIVEN - FLEXIBLE REUSABILITY 12 A library can use other libraries Test case can use any library Test case can call other test cases Data-driven on test case calls test cases 13. DATA DRIVEN - FLEXIBLE CONFIGURATION 13 Multiple Datasets 14. DATA DRIVEN - FLEXIBLE CONFIGURATION 14 Able to switch runtime engines Sample case Configure steps in IE Execute some steps in Firefox Verify result steps on Chrome 15. LOG AND REPORT 15 16. LOG AND REPORT History report (excel, html and email) Helpful to follow bug fixing HTML format can be opened anywhere Email (HTML format) is portable Various type of reports/logs Statistic of result Detail report/log HTML / Excel format report Reminders Send email notification Define groups to send notification 16 17. LOG AND REPORT Follow Drill-down idea History Report Step report Detail report 17 18. 2013 KMS Technology ROI & METRICS VU DUONG 19. ROI WHAT? 19 20. ROI WHAT? 20 21. ROI WHY? 21 22. ROI WHY? 22 23. ROI HOW? 23 24. ROI ANALYSIS OVERVIEW 24 To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a Percentage or a Ratio. 25. ROI BEST PRACTICE 25 ROI (Effort) ROI (Cost) 26. ROI - EFFORT 26 ESV (Effort saved from using automation) ROI (effort) = ------- = (------------------------------------------------- ) ESP (Effort involved in automation) ESV = Effort saved as a result of test execution using automation (in hours) ESP = Effort involved in automation like: Framework Development, Automation Scripts Development, Automated Script Execution, Automation Scripts Maintenance, Automation Training 27. ROI ANALYSIS EFFORT 27 Parameters: 28. ROI ANALYSIS EFFORT 28 Pay-Back Period (Effort) 29. ROI ANALYSIS COST 29 Parameters: 30. ROI ANALYSIS COST 30 Pay-Back Period (Cost) 31. METRICS 31 32. METRICS EFFORT/COST 32 Metrics for Effort/Cost Running Time Per Month Critical/High Defect Per Component Critical/High Defect Per Month 33. METRICS QUALITY 33 Metrics for Quality Defect Escape % Automation Coverage Per Sites/Features Defect Found by Automation Defect Rate (Compare with Manual) . 34. 2013 KMS Technology THANK YOU 34