Upload
jerrykprague
View
791
Download
7
Embed Size (px)
DESCRIPTION
Citation preview
PRACTICAL SOTWARE QUALITY AND TESTING 2008Zpráva z konferenceIng. Jaroslav Kalvoda(prezentace na pracovní snídani, čtvrtek 19.6.2008)
Představení firmyHlavní kompetence
Vývoj informačních systémů na zakázkuData Management
Řada dalších kompetencíQuality Assurance
Přibližně 70 konzultantůPřibližně 270 milionů tržeb ročněOd 08/07 Oracle partner pro Českou spořitelnuOd 11/07 součást rakouského New Frontier Holding
1. Quality Assurance• Česká spořitelna
2. Studie automatizované testy• Česká spořitelna
• Česká pojišťovna
3.Testovací týmy• Česká spořitelna
• Česká pojišťovna• Vodafone• Sitronics
• O2
• IBMKonference PSQT
Shrnutí PSQT 2008 Las Vegas, květen 2008 Praktická a nekomerční konference Šest souběžných tracků
Test Process Performance Testing Test Management Quality Management Quality Management Test Automation
Zdroje informací z konference Popis nejzajímavějších myšlenek z
nejzajímavějších přednášek na následujících slidech
Popis navštívených přednášek v souboru PSQT 2008.doc (6 stran)
Podrobné materiály k celékonferenci v souboruWelcome.jpg (1021 stran)
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Magdy Hanna, PhD
Conference Chair
Luis Carlos da SilvaGRID: Practical Experience of the Independent
Software Test GroupCESAR vývojové centrum při univerzitěDříve 200 test engineers rozptýlených po výzkumném
centru nahrazeno 13 členy Independent Test GroupVyvinuli metodologii pro odhad počtu test engineers na
každý projektV týmu nemají rozdělené role (analyst, designer,
manager…), nýbrž univerzální test engineers (senior)Během nešpiček doporučuje využít tým na konzultaci
business requirements a stimulaci developers
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Greg StoneFrom Ad Hoc Testing to Quality Assurance using
Process as the GuideIowa Foundation for Medical CareAsi 400 IT profesionálů, DWH asi 10 terabytes, kvartálně
vyřizují 120 milionů claims, jejich SQA se za dva roky rozrostlo ze 45 na 100 lidí
Zpočátku testování prováděli Product CoordinatorsPozději zahrnuli SQA do každé části software life cycle4 fáze SQA (Planning, Preparation, Execution, Wrap Up)Netestují vše, klient ví co testují a co netestují
a souhlasí se scopem
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Marina Gil-SantamariaHow to Keep and Motivate Your QA Team without
Increasing Your BudgetTester jako „second class citizen“ (vs development)Software bugs stojí US economy $59.5 billion ročněGlobal software testing market byl $13 Billion v roce 2006
= 30% z total software development market (Gartner)Uvádí 5 mýtů o testování a proč neplatíPodrobněji viz dále…
Solving the Software Quality Puzzle Page 11 www.psqtconference.com
Myth #1: “Anybody can do QA”
• Testing is a skilled activity that– Requires the ability to think,
explore and follow logic while questioning and reasoning
– Performed under time-constraints, with little or no documentation
– Uses various types of infrastructure and experimentation, logic, models, mathematical probabilities and supporting tools
• Sorry, you just can’t take anybody off the street to do QA!
Solving the Software Quality Puzzle Page 12 www.psqtconference.com
Myth #2: “Any out of school kid can test our applications”
• You systems/applications are critical company assets. If they go down or perform poorly– Direct profit & revenue will suffer– Employee productivity will be
affected– Support costs will increase– A company’s reputation can also
be seriously damaged
• Would you hire inexperienced financial advisors to take care of your critical investments and financials assets? I didn’t think so!
Solving the Software Quality Puzzle Page 13 www.psqtconference.com
Myth #3: “A QA Engineer is really a Developer Wanna Be”
• Some QA engineers DO enjoy coding BUT high performing QA teams need a variety of skills– Domain expertise,
communication, customer-facing, logical thinking to build end-to-end test cases , automation, etc
• QA teams develop the plan that will ensure the success of a company’s product in the marketplace!
Solving the Software Quality Puzzle Page 14 www.psqtconference.com
Myth #4: “QA is a boring repetitive task with no creativity involved”
• This one could be true, it is up to you. How are your– Internal processes and QA
cycles that you have in place
– On-going automation and new initiatives
• How involved is your QA team with customer interactions and product requirements?
Solving the Software Quality Puzzle Page 15 www.psqtconference.com
Myth #5: “QA doesn’t provide much value to the organization”
• QA represents the heterogeneous users of the products that your company produces to– Improve end-user experience,
quality and reduce support costs
– Think about all your DAILY responsibilities and tasks!
• Everybody “shoots” the messenger….BUT…How well are you communicating the information that is important to your stakeholders?
Marina Gil-SantamariaHow to Keep and Motivate Your QA Team without
Increasing Your BudgetTester jako „second class citizen“ (vs development)Software bugs stojí US economy $59.5 billion ročněGlobal software testing market byl $13 Billion v roce 2006
= 30% z total software development market (Gartner)Uvádí 5 mýtů o testování a proč neplatíUvádí 10 způsobů motivace testovacího týmu, zejména
Vylepšit atmosféru v týmuZlepšit vztah testovacího a vývojového týmuPodrobněji viz dále…
Solving the Software Quality Puzzle Page 17 www.psqtconference.com
How to motivate your team• Evaluate internal atmosphere –do you see any of the 5
“myths” there? Take the time, act as “new” or a consultant on the job, and identify
1 or 2 areas for improvement For you: More knowledge of what are the team roadblocks! For the team: Increase sense of appreciation and boost morale
• Communicate, communicate and communicate with your stakeholders– Understand what your stakeholders are measured on so you can
report back what is valuable and important to them– Don’t be afraid of the business side! Report back on business
metrics as well• For you & the team: Demonstrate more value
Solving the Software Quality Puzzle Page 18 www.psqtconference.com
How to motivate your team cont
“Brag” about your team – Come up with a list of periodic achievements from your team,
and talk about them whenever you are interacting with your peers (status meetings, department Cross Functional meetings, etc)
• For you & the team: Demonstrate more value
• Challenge your team: Explore automation and/or new testing initiatives– Investigate first your own processes and past automation
initiatives• For you: Team will be more efficient, you will show results, you will
look great with upper mgmt!• For the team: Learn & gain valuable skills, avoid repetitive tasks
Solving the Software Quality Puzzle Page 19 www.psqtconference.com
How to motivate your team cont
• Improve QA-development relations – Examine your own relations and reporting structure as well– Come up with very specific goals (new cross-functional meetings,
more team building exercises, etc)• For you: More productive team, better collaboration, reflect very
positively on you with upper management• For the team: More sense of appreciation & respect
• Be a “good” manager – Employees usually leave a manager not a company!– Show appreciation, coach & mentor, consider allowing
telecommuting and/or flexible schedules• For you: Employees will stick with your for the longer term• For the team: Fun, enjoyable working environment
Solving the Software Quality Puzzle Page 20 www.psqtconference.com
How to motivate your team cont
• Enhance your QA job descriptions – Take the time to put together a more enticing job description
• For you: Simple• For the team: Shows that the company values their QA resources.
• Involve QA in your dev cycles from the beginning– Explore the Agile methodology
• For you: More efficient teams (higher collaboration, better product understanding and more view points towards a particular task), you will sleep better at night.
• For the team: More job satisfaction due to peer to peer type of relationship with their dev counterparts
Solving the Software Quality Puzzle Page 21 www.psqtconference.com
How to motivate your team cont
• Consider rotating projects and tasks – Be also respectful of product attachments within the team…
For you: Implement best practices (“more eyes” on the project) For the team: Challenging, fun and creative environment
• Involve your team members in customer interactions – If there is an interest within the team…
• For you: More comprehensive test plans and more realistic testing scenarios
• For the team: It’s fun, more empowerment and appreciation because they can contribute more during the development cycle (requirements, test plans, etc)
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Eric NewcomerMeeting New Challenges in Testing SOA
Flat World Market TrendsBusiness TransformationIT TransformationSOA is the way
to achievetransformation
Testing SOAPodrobněji viz dále…
Solving the Software Quality Puzzle Page 24 www.psqtconference.com
Market Trends
• Companies have become virtual…
• Blurred lines between company and supplier…
• Outsourcing is location independent
• Key asset is information• Frequency of change is
accelerating!– Innovation– Product Cycles
Solving the Software Quality Puzzle Page 25 www.psqtconference.com
IT Transformation Challenges
• Applications are built not assembled
• High-cost specialists
• Delivery cycles are costly and often too slow
Revenue
Costs
New Product Delivery
$
Opportunity for business transformation
Mass Production is needed
Standardize
Reuse
Average developers
Solving the Software Quality Puzzle Page 26 www.psqtconference.com
Move From Monolithic Applications in StepsIntermediate stage:
Break out individual services
Application
Application
service
service
Application
Application
service
Goal: Service-oriented
architecture
service
service
service
service
Application
Monolithic applications
Application
Solving the Software Quality Puzzle Page 27 www.psqtconference.com
Service Contracts Are the Key
• The key principles of SOA– Services should be business-oriented
– Services should have well-defined interfaces (aka service contracts)
– Service contracts should separate interface from implementation
• Service contracts are critical to achieving reuse and abstraction
Solving the Software Quality Puzzle Page 28 www.psqtconference.com
WSDL: Typical Service Contract
• WSDL is very flexible– Import existing WSDL contracts– Create new WSDL contracts using
XML Schema– Create new WSDL contracts from
an external metadata source such as CORBA IDL
– Annotate with policy metadata
• Benefits– Abstraction– Encapsulation– Loose-Coupling– Separation of Concerns
Service
Port
Binding
XML Data Type
Part
Message
Operation
PortType
WSDL
Logical
Contract
Physical
Contract
• Logical Contract is what other applications care about
• Physical Contract is extensible to support any middleware binding
• XML Schema provides independent type system
Solving the Software Quality Puzzle Page 29 www.psqtconference.com
Pre-Certification Process Helps
Early Collaboration and Validation
Extends unit testing to include systems integration level testing
Moves defects discovered prior to Test deployment
Unit Test 1
Unit Test 2
Unit Test 2
Unit Test 1
Advance QABasic QADeploy $$
Basic QA
Basic QA
TestDevelopment
Solving the Software Quality Puzzle Page 30 www.psqtconference.com
SOA: Interface Simulation and Testing• SOA is based on interface contracts
• Rapid Simulation Solution
• Designed for distributed people/projects
– SOA, Outsourced, Inter-Departmental, B2B, etc.
• Enable parallel development of components
• More efficient use of testing resources
• Enable pre-certification between peer teams
Sim
ula
te Middleware
Web
Back-End
Sim
ula
teS
imu
late
Sim
ula
te
© 2008 IONA Technologies
Solving the Software Quality Puzzle Page 31 www.psqtconference.com
Building a Validation Framework
SystemTest
E2E Test
Unit Test
QA TestingDevelopmentRequirements
IAIA
RequirementsRequirements
Code1010101010101010101010101010101010101010
Unit TestIAIA
Code1010101010101010101010101010101010101010
IAIA
Standard Interface Agreements (IAs)Standard Interface Agreements (IAs)
Augment current processes to populate and utilize frameworkAugment current processes to populate and utilize framework
Interface AgreementManagement SystemInterface AgreementManagement System
Meta data about code droplets across SDLCMeta data about code droplets across SDLC
Interface Simulator and Testing
Interface Simulator and Testing
Validates functional code against design (IAs)
Validates functional code against design (IAs)
Validation Framework
SystemTest
Solving the Software Quality Puzzle Page 32 www.psqtconference.com
Example: Time to Market Compression
PR
OD
UC
TIO
N
Req
uir
em
en
ts
CO
NS
ULTA
TIO
N
DefDef DevDev TestTest
Backend Billing Systems (eCare/Telegence)
Middleware Tier (CSI)
Middleware Tier
WOW/BAS Tier
DefDef DevDev TestTest
DefDef DevDev TestTest
DefDef DevDev TestTest
IONA Framework
IONA Framework
DefDef DevDev TestTest
IONA Framework
IONA Framework
DefDef DevDev TestTest
IONA Framework
IONA Framework
DefDef DevDev TestTest
7-14 Weeks TTM
Improvement
7-14 Weeks TTM
Improvement
Solving the Software Quality Puzzle Page 33 www.psqtconference.com
Example: Rapid Website Update• .CO.UK business needed high-quality software
delivered at “Internet Speed”– Up to 30 “releases” per year
– Delays are unacceptable
– Only “Priority 1 Defects” can derail releases
– Development teams “must address” P1 defects prior to launch
Solving the Software Quality Puzzle Page 34 www.psqtconference.com
• Reduce Cost of Correcting Defects– Discovering defects one phase earlier saves $6,600 on
average
– Interface framework users have found up to 25 defects per interface: an average of 18 defects
• Reduce Duration of Projects– Reducing defects in earlier phases allow subsequent
phases to complete more rapidly
• Increase Organizational Capacity– Eliminating defects reduces workloads across the
organization
– Resources can be re-invested into improving functionality
Some Quantified Benefits Net Results
Save $118K per Service InterfaceSave $118K per Service Interface
Eliminate ten (10) business days per project
Eliminate ten (10) business days per project
Increase Overall Capacity 6%Increase Overall Capacity 6%
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Lew PrivenSoftware Inspection
Identifikují chyby ve vývoj software již v době sběru požadavků
Čím dříve v software life cycle je chyba objevena, tím nižší jsou náklady na její odstranění
Za 20 let nasbírali zkušenosti a stanovili kde je nejlepší v software life cycle dělat inspekci
Inspekcí snižují náklady na testování
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Tony SimmsTips and Templates for Test Managers, Tony Simms
Vytvořil a posbíral templates pro podporu testováníTemplates nejsou Brain TransplantsPokrytí celého testovacího procesu
Test StrategyEstimating Test Effort & ResourceReporting & Tracking Test MilestonesDaily (weekly) Test Progress ReportingEnd of Testing Report
Rozdělení zdrojů (40% Requirements,20% Development, 40% Testing)
Podrobněji viz dále…
Solving the Software Quality Puzzle Page 39 www.psqtconference.com
Test Strategy
Solving the Software Quality Puzzle Page 40 www.psqtconference.com
Reporting & Tracking Test Milestones
Solving the Software Quality Puzzle Page 41 www.psqtconference.com
Reporting & Tracking Test Milestones
Solving the Software Quality Puzzle Page 42 www.psqtconference.com
Reporting & Tracking Test Milestones
Solving the Software Quality Puzzle Page 43 www.psqtconference.com
Reporting & Tracking Test Milestones
Solving the Software Quality Puzzle Page 44 www.psqtconference.com
Reporting & Tracking Test Milestones
Solving the Software Quality Puzzle Page 45 www.psqtconference.com
Daily (weekly) Test Progress Reporting
Click image abovefor template
Daily/Weekly Progress Report
0
5
10
15
20
25
30
Total Scriptsto Run:
Total Run Todate:
Total ScriptsPassed:
Total ScriptsFailed:
Test Run Statistics
0
500
1000
1500
2000
2500
1 2
Test Steps run todate
Total Steps to Run
Total Steps:
Issues Raised & Closed
0
2
4
6
8
10
12
14
Sev 1 Sev 2 Sev 3 Sev 4 Sev 5
Raised
Closed
Simple, easy to complete daily or weekly report forms make it easy to show project managers the progress and trends in testing. The attached template reports on:
• Test run rate for scripts
• Script pass and fail rates
• % of steps executed
• Issues raised and closed
Tony SimmsAnatomy of a Test Strategy, Tony Simms
Detailně popsal fáze testovacího projektu u uvedl související templates
Různí stakeholders mají různé požadavky a je třeba je uspokojit všechnyIT Requirements Governance RequirementsBusiness Requirements
Podrobněji viz dále…
Solving the Software Quality Puzzle Page 47 www.psqtconference.com
Test Phases [template section 6]
Detail the relevant Test Phases for the project, a diagram helps visualise the sequence and a phase definition helps all involved ‘speak the same language’
Component Testing
Component Testing
IntegrationTesting
IntegrationTesting
SystemTesting
SystemTesting
SystemsIntegrationTesting
SystemsIntegrationTesting
Performance Testing
Performance Testing
OperationsTesting
OperationsTesting
UserAcceptance Testing
UserAcceptance Testing
Solving the Software Quality Puzzle Page 48 www.psqtconference.com
Work Products
Test Planning Test Preparation Test Execution
Test Strategy
MasterTest Plan
Test Environment Requirements
AcceptanceCriteria
High Level Test Specifications
DetailedTest Plan
Test Data
Acceptance Criteria/Test Spec. Matrix
Detailed Test Specifications
Test Execution Schedule
Environment Utilisation Plan
Test Results Report
End of Phase Test Reports
Incident Reports
TestProcess
This diagram shows which Work Products are to be generated for each phase and activity. Red bordered items are produced once, for the whole project, blue bordered items will be produced for each test phase.
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Bill MillerTest Driven Quality
Vyjmenovává cíle testováníUkazuje kdo všechno jsou zákazníci testovacího týmuUvádí kvality dobrého testovacího týmuUkazuje testovací cyklusDefinuje release criteria a závažnost chybPodrobněji viz příklad dále…
Solving the Software Quality Puzzle Page 51 www.psqtconference.com
Defect Severity
• Critical– Loss of data, instability, and hazardous outcomes..– Blocks testing.– Would harm reputation of company.– Would harm sales.– Would lose money.– Would risk company being liable in a product liability case.– Would be a danger to people.
• High– Key features do not work regardless of workaround.– Leaves the customer with a bad impression.
• Medium– Minor feature does not work with no work around.– A visual error.
• Low– Minor feature does not work, but there is a work around.
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Patrick FreemanSoftware Quality Metrics for Critical Decision Making
Uvádí sedm Software Quality Metrics a typické počty chyb pro každou metriku
Uvádí seznam metrik, na základě kterých nedoporučuje rozhodovat
Jako Capture Deffect Tool doporučuje Quality Center, Excel nebo Bugzilla (zejména tu doporučuje)
Nejzajímavější přednášky Luis Carlos da Silva, Case Study Greg Stone, Case Study Marina Gil-Santamaria, Testing Team Eric Newcomer, Testing SOA Lew Priven, Software Inspections Tony Simms, Testing Templates Bill Miller, Test Driven Quality Patrick Freeman, Software Quality Metrics
Shrnutí Headlines PSQT 2008
1. Testing Team Motivations by Marina Gil-Santamaria
2. Software Inspections by Lew Priven
3. Testing SOAby Eric Newcomer
4. Testing Templatesby Tony Simms
Poučení z PSQT 2008Testovací tým
Vylepšit atmosféru, prostředí - odbourání stresu, vyšší výkon
Důležité, aby testeři věděli, co testují, jaké je použití, znali historii produktu, mohou pak nejen testovat, ale i test připravovat
Regresní testy dělat až v okamžiku, kdy je aplikace stabilní
Zaznamenávat výsledky testů a prezentovat je testerům pro zvýšení motivace
Poučení z PSQT 2008Vztah TEST a TECH týmů
Nutno popsat SLA mezi TECH a TESTZaznamenávat výsledky testů a reportovat je TECHReportovat podíl chyb ve skupináchVšechny chyby reportovat konsistentním způsobemZavlečené chyby re-open a reportovat TECHUrčit zodpovědnost za provádění integračního testuZástupce TECH v TEST týmu při plánováníZapojení businessu do testování (alespoň na konci UAT)
• pro TECH zavlečené chyby• pro správu testovacího prostředí
chyby prostředí• pro navazující systémy chyby v
navazujících systémech• pro TEST tým chyby testů• pro BUS tým chyby vzniklé z
nepochopení dokumentace
Poučení z PSQT 2008Automatizace testů
Vhodná pouze u regresního testuVhodná pouze je-li aplikace stabilníVhodná pouze jsou-li testovací data stabilníNepodařilo se najít srovnatelnou formu pro posouzení
vhodnosti automatizace
Podrobněji o PSQT 2008Popis nejzajímavějších přednášek v souboru
PSQT 2008.doc (6 stran)Podrobné materiály k celé konferenci v souboru
Welcome.jpg (1021 stran)Na emailu: [email protected]
Odpovědi na další otázkyPomoc při kontaktování přednášejícíchUspořádání seminářů nebo konzultací s Tony SimmsUspořádání konzultací s Lew Priven