Integrated Development and QA Team with Scrum

Ethan HuangPerficient GDC, China

Challenges We Were Facing

At the beginning of the project, team was suffering from having: No testers

– Our resource plan could not support a fully structured team which contains dedicated QA roles

A fixed schedule– We only had 90 calendar days in total as a fixed deliver timeline– So we didn’t have much buffer time for ‘stabilization’ in terms of bug fixing

Unclear requirement– We had no ‘Finalized’ requirement even in the late development phase– And we could not afford the documentation/communication effort for writing

both requirement and test case documents No enough communication with the client

The Way We Go

Team decided to resolve the problems and secure high quality by: Forming an Integrated Dev/QA team

– Developers agreed that we should secure high quality by ourselves– Which means the developers were to test for themselves

Running short period Sprints (one week) – Short Sprints helped to reduce risks– And secure high quality code by frequent inspections

Running Test Case Driven Requirement– We were using test cases to represent/confirm requirement directly

Detailed Actions

Ramping up in the first a couple of Sprints:– Designing the team model and made clear of the Developers responsibilities– Designing our Test Case template– We defined and posted a white board diagram to highlight our process

In the later Sprints we continuously improved our approach:– We had one member to act as the “QA Goalkeeper”– And for each User Story we had a volunteer to be the “Story Owner”– Built several quality principles– Setup 4 Quality Gates

Ramp up – Forming the Team

Manager

Developer Developer Developer

Ramp up – Defining the Sprint Approach

Practice – Test Case Driven Requirement

Developers were developing test cases before coding And use the test case as the tool to confirm the requirement

– It made significant difference on helping the team understand the business– Also scaled down the effort on confirming the requirement back and forth

One possible format for the test cases:

Practice – Conducting tests by developers

Developers develop code following the test cases developed by us own Everyone takes both development and testing tasks

– could be testing on others code – buddy test

No bugs to be checked in into repository – run enough test on local

Do daily verification after the daily build, and resolve today’s issues today

Two IMPORTANT principles that all team members came to agreement on:

Practice – The 4 Q-gates before we deliver

Q-gate # 1 – Local verification + Continuous Code Review (CCR) before code check in

– The Story Owner runs the test step by step on local dev environment– The identified Reviewer to review the local code– Exit Criteria: Resolve all the issues

Q-gate # 2 – Daily Build and Daily Verification on Dev Server– At least one daily build, sometimes multiple builds– Run UT on servers, track the Sonar report for variations – The QA Goalkeeper and Story Owners test the new feature after the build– Exit Criteria: Resolve all the issues found on that day

Q-gate # 3 – Team Verification on Test Server before Sprint closure– The whole team gather together in one room and verify all the integrated

feature one by one– Exit Criteria: Resolve all the ‘Major’ above issues and log the rest into Jira

Q-gate # 4 – Formal Code Review (FCR) every one or two sprints– Exit Criteria: All findings logged into Jira have been fixed

Results - 12 weeks development vs. 1 week stabilization

We ran near 13 one-week sprints (89 calendar days) 2 RED Sprints for we failed to deliver high enough quality But we finished integrating all the planned feature in sprint # 11 In sprint # 12 we spent one whole week for regression and bug fixing In sprint # 13 we finished the clean-up

Development + TestingDevelopment + TestingFinal Final

TestingTesting

CleaningCleaning

UpUp

Quality Statistics – UT Coverage/Test Success

From Sonar reportFrom Sonar report

Quality in Statistics – Code Summarization

From Sonar reportFrom Sonar report

Quality Statistics – Code Complexity

From Sonar reportFrom Sonar report

Quality Statistics – Violations

From Sonar reportFrom Sonar report

Quality Statistics – Time Machine

From Sonar reportFrom Sonar report

Best Practices

Local Verification/Buddy Test before code check in– It’s the first step of Continuous Code Review– The Story Owner does the verification together with the developer– Developer walks the Story Owner through the test case step by step– All the findings should be fixed and then do the verification again

This activity was happening daily and could find most of the functional defects or even potential defects as early as possible

It takes about 30 minutes for both the author and the reviewer for one verification – the effort is much less than the regular testing activities which happens after the integration

Best Practices

Daily Verification on Dev Server for the new build – Our daily build happens at 12:00 PM everyday– The QA Goalkeeper will do the manual verification for all the new features

newly integrated (sometimes the author will prefer to walk the whole team through the new feature)

– It takes about 30 – 60 minutes, and the findings will be logged into a spreadsheet and be posted on the whiteboard

– The issue owners will fix them before the end of the day– Do another manual build if necessary to verify the fixed bugs– In cases of some issues could not be resolved in that day, they’ll be treated

as the most important tasks which we should not start new implementation before we fix them

This activity was happening daily and could find most of the integration issues as early as possible

The more important, team members committed to resolve the issues before they leave so that for every build we will have solid quality, no debts

Best Practices

Team Verification before the Sprint Review– As the exit criteria of the entire sprint, team will be sitting in one room and

do the verification/manual functional test for all the feature developed within this Sprint

– Authors run the test following the test cases– All findings will be logged into Jira as the highest priority tasks for the next

Sprint– Sprint exit criteria: having no major above issues found in the

session – the key fact to decide whether the sprint status is Green or Red

This activity helps the whole team to have a chance of inspecting the whole set of Sprint deliverables.

Since most of the defects have been inspected/fixed during the Sprint day-to-day work, therefore, Team Verification helps more on finding those highest priority/severity issues, and getting a quality overview/summarization for that Sprint

Best Practices

Simplify the way we log/manage defects– Tools we were using to log/manage defects: Yellow Stickies,

Spreadsheet/txt files + White Board, and Jira– Yellow Stickies: To log Local Verification findings– Spreadsheet/txt files: To log Daily Verification findings– White Board: The place we post all Yellow Stickies and Spreadsheets– Jira: To log Team Verification findings/Regression Test findings

# Issue Found Date Fixed Date Status1 Categories pages haven't configured the 3 eSpots on the left. closed2 Product Listing: 'Sale !' for gallery/list view closed3 Product Detail: 'Sale !' closed4 Product Listing/Search: 'Add to Cart' and 'Add to Wish List' don't work closed5 Shopping cart detail: delete cart confirmation displays incorrect closed6 Contact us: content spot content is empty 2-Sep 3-Sep closed7 Footer: shipping, pricing & return policies, content is emtpy 2-Sep 3-Sep closed8 Footer: privacy policy, content is emtpy 2-Sep 3-Sep closed9 Footer: terms of service, content is emtpy 2-Sep 3-Sep closed

10 Product Listing: Price break is broken: Flyer and layout issue 2-Sep 2-Sep closed

11Product Listing: When user's not logged in: buttons 'add to cart' and 'add to wish list' should be in-actived (right now they were invisable)

2-Sep 4-Sep closed

12Product Listing: Display 'base contract price range' together with 'your contract price range' - rule is having 'your contract price range + base contract price range' then only display 'your...'; having 'base contract price range' then display 'base...', having no then display base price.

2-Sep 4-Sep closed

13 Product Detail: price break (flyer and layout issue) 2-Sep 3-Sep closed14 Product Detail: 'Descriptive Attributes' -> 'Attributes' 2-Sep 2-Sep closed15 Product Detail: 'Fishy Rattle' has no large img 2-Sep 2-Sep closed

16Product Detail: 'base contract price range' together with 'your contract price range' - rule is having 'your contract price range + base contract price range' then only display 'your...'; having 'base contract price range' then display 'base...', having no then display base price.

2-Sep 3-Sep closed

17 Data Issue: Account '80112', Toys Online, Account Representative: Johnny Yao 2-Sep 3-Sep closed18 Data Issue: Hangzhou, FL US 2-Sep 3-Sep closed19 Accounts: There should be 3 eSpots on the bottom, now only have 2 2-Sep 3-Sep closed20 Accounts: Default tab should be highlighted 2-Sep 3-Sep closed

21Search: If there's only one result, after displaying the search result page, it will redirect to the product detail page directly. But the breadcrumb is not correct: 'Home > ' ---> 'Home >Top Level Category > Second Level Category > Product '

2-Sep 2-Sep closed

22Shopping Cart Detail/Wish List Detail: Copy and Save as doesn't support '#' and '%' as the cart/list name 'cut the string if contains '#''?

2-Sep 2-Sep closed

23Wish List Detail: Input '#' or '%' as the wish list name, a system error msg displayed: 'A system error has occured. Please try again later. '

2-Sep 2-Sep closed

24Shopping Cart Detail: Price break: 1-5, $ 12.49; 6-10, $ 12. Added 4-1-1, when item # was 5, total amount is correct (=12.49*5); but when item # was 6, sub total = 12*6; while total = 12.49*6

2-Sep 2-Sep closed

25 Checkout - view and submit order: repeat this order section missed; cost center missed if log in as admin 2-Sep 3-Sep closed26 Checkout - view and submit order: shipping address selection - length of the drop down is not limited 2-Sep 3-Sep closed27 Left Navigation: Layout issue for second level categories - not aligned to left 2-Sep 3-Sep closed28 Product Listing: Quantity input box should be marked as required 2-Sep 3-Sep closed29 Product Listing: Display both 'base contract price range' and 'your contract price range' at the price section - priority of 2-Sep 3-Sep closed30 Left Navigation: Layout issue for second level categories - not aligned to left 2-Sep 3-Sep closed31 Data issues: Account '80112', Toys Online, Account Representative: Johnny Yao - Johnny Yao is not a valid user 2-Sep 3-Sep closed32 Data issues: Hangzhou, FL US - wrong city name 2-Sep 3-Sep closed33 left navigation: Done Shipping Address missing in Account 3-Sep 3-Sep closed34 either Reg price or Your price is enabled: your contract price is disabled, while the your contract price range is enabled 3-Sep 3-Sep closed35 Speera - 120, your price is not displayed 3-Sep 3-Sep closed36 product detail - price section, layout issue 3-Sep 3-Sep closed

37Account - both the buttons 'View Cart' and 'Cehckout' should be disabled if there's no cart at all Order History Detail page - price should NOT be visible when the dealer switch doesn't allow the user to see them(Product/Cart/Order/Header Mini Cart)

8-Sep 10-Sep closed

38 1. View Cart - remove item, shipping charge displays wrong 7-Sep 7-Sep closed39 2. My Account - account drop down displays in 2 rows 7-Sep 7-Sep closed40 3. Contact, shipping address missing 7-Sep 7-Sep closed41 4. Cost Center missing for crector 7-Sep 7-Sep closed42 1. search - product detail page - left eSpot img issue - covered the confirmation dialogue 7-Sep 7-Sep closed43 2. Dealer Switch - switch User Price to NO while Reg Price is YES, neither price is displayed 7-Sep 7-Sep closed44 3. Change the Dealer Switch test case to "speera 80110 yyny 80120 nnny"? 7-Sep 7-Sep closed45 4. JS issue - all 'Add to Wish List' buttons don't work 7-Sep 7-Sep closed46 5. Upcoming Repeat Order - Order Detail - Date Submitted : 'N/A' while in Order list page displays '07/09/2009' 7-Sep 7-Sep closed47 6. Upcoming Repeat Order - Order Detail -Return to all orders - returned to 'Order History' 7-Sep 7-Sep closed48 7. All "contact" are missing 7-Sep 7-Sep closed49 8. Some "Shipping Address" are missing 7-Sep 7-Sep closed

Further Improvements

FCR should happen more frequently Test Automation – reduce the regression test effort significantly

Several key-words to be highlighted

Integrated Dev/QA team Story Owner QA Goalkeeper Local Verification Daily Verification Team Verification

Thanks!谢谢观赏！