Text of Testing in the Web â€Spaceâ€™ Vonnie FaiumuOct 2001
Testing in the Web Space Vonnie FaiumuOct 2001
The business community in general is more aware of the benefits of effective and efficient test practices throughout the Software Delivery Life Cycle . deliver a quality product in the shorter time frames expected by the business..
Agenda Differences and Similarities Some extra considerations for web development Experiences from Project # 1 Some things I learned Experiences from Project # 2 Some thoughts to take to the next project.
Whats different ? Customer expectation of rapid delivery High visibility throughout lifecycle Broad user base (the world !) Broad range of User interface (Operating Systems and Browsers) Large variation in user experience High user expectations (everyones an expert) Greater reliability needed (24 x 7)
Whats the same? Still need firm business requirements and reusable test scripts Interface with new and existing systems Non functional testing is needed - eg Availability and Reliability Need to transition to Business as Usual
Whats in the Test Component? Strategy/Planning Agree the terminology Agree environments and responsibilities Evaluate use of Test Tools Plan approach Test phases - entry and exit criteria Detailed Test Plans What and who Test analysis, script creation, set up test data Code migration process - release management Test execution Analysis and reporting Handover to next phase
Project # 1 Project Structure Internet Online Banking application Mixed development team on customer premises Front end and back end development Back end development - in house Content management and web functionality Formal and structured test processes Existing test tools Joined team near the start of development
Initial challenges Detailed Test Plans Multiple Requirements Documents at varying levels of detail- which is the master? Navigation - where does the back button go? (The wall !!!!) Requirements documents cant keep up with the prototype Error handling and messaging not confirmed
Items to be tested Functional Testing of Use Case scenarios Validation of web pages navigation and links page content and layout (as per design) Access via supported web browsers Performance and Load Testing Availability and reliability (connectivity and system stability) Exception handling - messages and alerts
Planned approach for test execution Use back end emulator used by developers Test bench different O/S browser combinations per PC Test different functional areas on different combinations to get full coverage early Testers to enter defects directly into SQA manager Test Scripts to be reusable and used for phases through to UAT
The plan goes awry. Test bench desktop setup Test entry criteria not met Not all development completed No documented unit test plans or results Testing resources not 100% available Unable to use emulator needed to write stubs Error messages not finalised Page design not finalised All links not working
Way Forward Page content and layout excluded Staggered testing as code and stubs available Accept more frequent code release Reduced OS/browser coverage Paper based defect sheet with centralised entry into SQA Manager Daily reporting and review of progress
Participated in End to End Testing Managed test execution resourced with back end developers No visibility of back end system test results Still tweaking requirements Error messages still being defined Now in main test window - less control over environment and data - wider usage Dynamic update, therefore not easily repeated
Things that worked well Co-location with developers Early involvement with UAT pers Test Document Repository and approval process Using SQA Manager from outset Test script format - excel Daily Reporting during test execution QA process - encouraged stocktake and review
What caused the most pain? Incomplete business requirements Test bench set up Test environment Identification and set up of test data Recovery and Reliability Changes in available testers That old chestnut - too many tests and never enough time
Project # 2 Project Structure Internet application Development team on customer premises Front end and back end development Back end development - third party Content management and web functionality Less formal existing test processes No existing test tools Joined team during design phase
I wanted to achieve . Earlier review of business requirements Comprehensive plan earlier Clearly defined responsibilities for environments, test phases and tasks Early selection of test data Schedule and enforce proper reviews of Requirements Design Test cases etc Better use of resources OS / Browser combos More accurate schedule assessment for design and execution
Processes Requirements captured in Use cases Complex business rules recorded in separate documents Visio diagrams for page flow and menu items Word docs with page layout and link destinations defined Central repository for clarification of reqts Early agreement to keep documentation current
Planned Test Phases Code Verification and Unit Test to agreed standard checklist - developers Function and Testing Front End including end to end test team Back end existing IM support team Navigation and Boundary Testing Non-Functional Testing test team in conjunction with the operational support team. Usability Test User experience analyst Process Test process owners. Production Test Comfort Test on production environment in lieu of acceptance test - BAU
Approach for Function Test Test cases and scripts grouped by Use Case VMWare for OS/Browsers Test all functionality on one OS/Browser combination first Test major functionality over a range of other OS/Browsers Test data input pages and some navigation on alternate browsers as time allowed
O/S Browser Combinations Fully tested Win 95 with IE 4.0 and Netscape 4.0 Part Coverage major functionality Win 98 with IE 5.0 and Netscape 4.5 Sample Coverage Win 95 with IE 3.0 and Netscape 3.0 Win 98 with IE 4.0, IE 5.5, Netscape 4.5 Win 2000 IE 5.5 and Netscape 4.5 Extra coverage NT Development environment MAC because lot of people use it
Some Challenges Development delayed - test functional areas as they become available Changes identified from Usability testing required change in design re-development rework in test scripts Metrics and reporting requirements - not defined Performance requirements - what does acceptable mean? Meaningful Volume testing without tools.
Tests by developers and test team Developers Code Verification and Unit Test - including 128 bit encryption Secure session / public pages Test Team functional scripts (front and back end) navigation and links boundary testing on all input pages final test of major functionality on the production environment configuration and back end test
Tests by developers and test team Non-Functional Testing oApplication Availability testing was carried out by the test team, assisted by the development and operational support teams. oVolume testing - conducted by the development team simulating incremental users across the production environment using Perl scripts. Results were monitored using the BCM monitoring tools set up in the IM team for Business as Usual monitoring and reporting.
Tests done by project team Usability Test ease of use designed and conducted by User Experience Analyst Process Test Processes were documented by the business, verified by peer review, and were validated in training. Production Test Comfort Test On go live, a limited number of tests were run against the production environment.
Tests done by other groups High Availability and Load Balancing external to the Firewalls. Back Up and Recovery Network Availability and Security testing Ethical hack New input and enquiry screens for back end Reporting and Statistics not defined passed to BAU Green screen maintenance on back end system for new data fields
Things That Worked Well Co-location with rest of team - one room Early identification of test responsibilities Central repository for clarification of reqts Work co-operatively with developers (not over the fence) VM Ware - flexible - some overheads Make early contact with operational group - non functional testing Use of same Defect Management Tool from Testing through to Business as Usual
Lessons Learned Inspection of Use Cases and navigation enforces standards Encourage walk throughs at