HP QuickTest Professional : ": 061001368AgendaShort Introduction ArenaTier 3 support processDescription and analysis of the processIntroduction of main problemsDefine key performance indicators for successful improvements of the processSuggest improvements to processWhat is Functional Testing?Testing software based on its functional requirements. It ensures that the program physically works the way it was intended and all required menu options are present. It also ensures that the program conforms to the industry standards relevant to that environment; for example, in a Windows program, pressing F1 brings up help.How Does QuickTest Help?When performing functional testing by automatic scripting you get:Scenarios are easily reproduced.More coverage since it can run 24 hours.Regressions are easily detected.More accurate verifications.

LegendHorizontal IntegrationVertical IntegrationCompanyProductsCustomerArenaSAPOracleOracle and SAP sells their products with Tests suite for their products that can be run under QuickTest Professional.

Service Test is another testing tool that is developed in HP Software which Perform functional testing on SOA systems and is sold with QTP as unified functional license testing tool.QuickTest Professional

Service Test

ITOrgnizationMarket Analysis

Tier 3 Support ProcessTier 2 Creates an escalation R&D AnalysisR&D Creates DLL with SolutionR&D creates patchQA performs testing on verificationCustomer Verification~3 Days

~1 Day

~0.5 Day

~0.5 Day

~2 Days

ISO 9000ISO 9000 is a standard that says that the process needs to be defined and enforced.Currently HP SW Tier 3 support process, that was described, meets ISO 9000 standard.The maturity of the process is 4, since it is currently the 7th year that it is enforced and fine tuning is done at the beginning of a new QuickTest Professional version release.6It is hard to talk about variety in the process since each customer has its special case and each case is resolved by different R&D engineer. We can pin point a few problems with the current:a) Cases that are forgotten to be handled by TIER 2 or TIER 3Suggestion: Monitor engineers by their response time when handling cases.b) Cases that are reassigned to other R&D engineer since the first analysis where the problem resides was wrong.Suggestion: Add a logging system to the product and teach the engineers how to pin point the problem from the log.PerformanceWhen talking about the performance of our support process we take into consideration the following parameters:

Productivity: each customer pays about 3,000$ for support license. Assuming that each customer has approximately 4 support calls that handled by Tier 3 (R&D) when each support cycle is about 5 working days, the cost is about 300$ per day. It comes that our profit is about 1500$ per support license.

Improvement: according to the process, reducing the maximum time spent on analysis by half can increase the profit by 30%. This can be done by the following:a) creating a simple diagnostic program that generates a report. b) fixing our logging system. c) teaching our TIER 2 support to analyze the logs.

Performance cont.Overhead Since there is no direct communication between customer and R&D engineer, most of the time is spent on cycles between R&D engineer and Tier 2 support person.Improvement: Performing WebEx with customer to demonstrate his problem. First analysis can reduce cycles and help with analysis stage.

Response Time When tier 3 escalation is created it takes about 7 working days to close it. But the cycles between R&D engineer and customer increase a time to close the tier 3 escalation to about 20 days.

Improvement: With the above suggested improvements it is possible to reduce the response time by 50%. Performance cont 2.SLA about 84.5% of the TIER 3 escalations have resolved in the given SLA.Improvement: With the above suggestions we can increase the above number to 92%.

Quantity Each product version has about 320 Tier 3 escalations and in every moment there are about 70 cases which are opened. Each R&D engineer has about 2 escalations that he is currently working on.Improvement: more regression testing in QA and working with more real world application, when performing the QA, will reduce the number of total escalations that needs to be handled by tier 3. This reduction will be by 25%.

Performance cont 3.Quality in this indicator we speak about to aspects:Robustness the quality of the robustness of the process is measured by how many Tier 3 escalations are not met by the SLA, this number is currently about 15.7% of all Tier 3 escalations.Improvement: with the above suggestions we can reduce the number of errors in the process to less than 10%. Agility there are small number of cases where a customer needs a special treatment to solve his problem either by working directly with R&D or by increasing man power. This number is rather smaller (less than 5%).Improvement: no improvement needed here.Problems in current processSince the number of cycles between R&D and TIER2 is the main cost of the process reducing that time can provide much more profit.The following problems in the process cause many cycles:Reassignments to between R&D engineersProblematic logging system.Escalation was not justified.

Pareto Analysis on ProblemsIshikawa AnalysisLack of knowledge in Analyzing LogsLack of DB to store Customers LogsProblematic Logging SystemNot all Scenarios are loggedNo R&D enforcement for Writing to LogsActions to be takenAfter a short discussion we have concluded that the following steps should be taken:

ResourceTaskTimeTechnical Architect Research for new logging system2 weeksIT managerCreate DB for logs1 weekR&D Team leaderInstruct R&D engineers with to consider logs in their code review1 weekL&D Team leaderCreate a new course for analysis of logs4 weeksChanges to the arenaWith the introduction of the WEB 2.0 the support process became complex because of the following:Variety of toolkits.Toolkits are open source and it is hard to keep track with changes in them.That is why HP has decided to create a taskforce that needs to provide methodologies and patents to solve the problems introduced by Web 2.0.Referenceshttp://www.pcmag.com/encyclopedia_term/0,2542,t=functional+test&i=43588,00.aspAll the information provided in this presentation is taken from HP Software and is used for the academic purpose only and should not be forwarded or used in any other way.