Embed Size (px)
DESCRIPTION
Extend the automated Firefox UI testing with Mozmill to allow contributors around the world to participate.
Citation preview
Crowd-sourced AutomatedFirefox UI Testing
Henrik Skupin <[email protected]>Lead of QA Test Automation
quality.mozilla.org qa automation
Thursday, November 4, 2010
Who is Mozilla QA?
Thursday, November 4, 2010
WHO IS MOZILLA QA
Group of about 30 employees / contractors
Are you serious?
No!
Thursday, November 4, 2010
WHO IS MOZILLA QA
1000s of community members across the globe
who are partners to the Mozilla QA core team
Thursday, November 4, 2010
Mozilla Scale
400+Developers
10,000+Nightly Testers
2,500,000Beta Testers
400,000,000Users
400,000,000
350,000,000
300,000,000
250,000,000
200,000,000
150,000,000
100,000,000
50,000,0000Thursday, November 4, 2010
What are our Areas of Work?
Thursday, November 4, 2010
• Testing new features in beta versions and nightly builds
• i.e. Switch to Tab, Add-ons Manager, ...
• Testing release builds before they get shipped
• Smoketests across all platforms
• ~ 300 basic functional tests (BFT)
• Software update tests
WHAT ARE OUR AREAS OF WORK?
Thursday, November 4, 2010
A Brief History of Automation in Mozilla QA
Thursday, November 4, 2010
A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA
•No automated functional tests existent by early 2009
• 100% manual testing of new Firefox features
• 100% manual testing of release builds (limited to a single platform)
Huge amount of manual resources necessary!
Thursday, November 4, 2010
A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA
So what has been changed so far?
Thursday, November 4, 2010
A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA
• Integration of MozMill (based on WindMill) for functional tests
• Automation of basic tests (30% done - ~100 tests)
• Automation of update tests (nearly done - Only 1single person needed to run 192 different update paths across 5 platforms and 4 P1 locales in under 2 hours)
Perfect time to open up testing for everyone
Thursday, November 4, 2010
Crowd Sourced Testing
Thursday, November 4, 2010
CROWD SOURCED TESTING
Jeff Howe: "It’s not outsourcing; it’s crowdsourcing."
Thursday, November 4, 2010
CROWD SOURCED TESTING
Crowdsourcing relies on people, and thus works when participants are treated as people
Thursday, November 4, 2010
CROWD SOURCED TESTING
What can be achieved?
• Support testers, localizers, and add-on authors to allow them to focus on their main tasks
• Constantly testing builds across platforms and locales
• Assembling test results from a large number of different environments (settings and performance)
How can this be done?
Thursday, November 4, 2010
MozMill Crowd Extension
Thursday, November 4, 2010
MOZMILL CROWD EXTENSION
Pre-requisites:
• Trivial setup (“One click install”)
• Uses MozMill to run tests in any version of Firefox
• Support for different test-runs (basic tests, add-ons, ...)
• Reporting test results to a central database
Under Construction
Thursday, November 4, 2010
MOZMILL CROWD EXTENSION
What remains:
• How reliable are our tests in the crowd
• Fix focus issues to allow background tests (webdriver)
• Analyzing results based on the environment
• ...
Thursday, November 4, 2010
DEMO
What will be shown?
• Setup of the test environment by using MozMill-Crowd
• Test-run for a subset of general Firefox tests
• Analyzing test results
• Test-run for the Google Toolbar extension
Thursday, November 4, 2010
REFERENCES
QA Test Automation team page:http://quality.mozilla.org/teams/automation/
Mozmill Tests Repository:https://developer.mozilla.org/en/Mozmill_Tests
Mozmill Code:https://github.com/mozautomation/mozmill
Mozmill Crowd Extension Project:https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Crowd_Testing
Thursday, November 4, 2010
QUESTIONS & ANSWERS
Questions?
Thursday, November 4, 2010
Thanks
Thursday, November 4, 2010
