Upload
techwellpresentations
View
183
Download
1
Embed Size (px)
DESCRIPTION
The classic software testing team is becoming increasingly obsolete. Traditional processes and tools just don’t meet today’s testing challenges. With the introduction of methodologies such as agile, testing processes with a "test last" philosophy cannot succeed in a rapid deployment environment. To exacerbate our testing difficulties, we now have to deal with "big data" which introduces an entirely new set of problems. In the past, we have relied on tools such as test automation to solve these problems; however, classic test automation simply will not suffice on its own and must be integrated with the right testing activities while being supported by correct procedures. When you combine these problems with inadequately defined requirements and limited resources, you have a recipe for testing disaster. Regg Struyk shares real-world examples and offers constructive ways to move away from traditional testing methods to a more integrated process using concepts such as test-driven development and TestOps.
Citation preview
nt Session
Presented by:
Regg Po re
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
W5 Concurre4/9/2014 12:45 PM
“Why Classic Software Testing Doesn’t Work Anymore”
Struyk
larion Softwa
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Regg Struyk Polarion Software
With twenty years of commercial software development and testing experience, Regg Struyk has held many different positions ranging from the head of technical product management for Agfa HealthCare to, most recently, product evangelist for Polarion QA. Regg has developed for several software testing tools including test integrity, iTest, and Polarion QA. Dedicated to the domain of test management, Regg is continually analyzing testing trends and their potential impact on the discipline of software testing.
2/11/2014
1
QA Evolves
20 plus years software dev experience
Who am I?
Senior Product Manager @ MKS (PTC)
Group Product Manager @ Agfa Healthcare
Product Manager @ Polarion for QA Software
Polarion Software® Polarion® QA™ - www.polarion.com
g @ Q
2
2/11/2014
2
•Introduction•Company Profile•Speed of Light•Big Data•Mobile•Tools•Tools•Automation•Resources
“We produce all significant documents for specifications, QA and test plans easily and rapidly with Polarion QA”
Andreas Deuter , Phoenix Contact Electronics
Waterfall– Test Specifications
Traditional QA
– Test Cases – Test Execution
Polarion Software® Polarion® QA™ - www.polarion.com 4
2/11/2014
3
Agile– Get ready to test
The Speed of Light
– More time constraints– Originated from development
Hybrids/Spin-Offs– Scrum, Agile-Fall– Does not means getting rid of documentation
Polarion Software® Polarion® QA™ - www.polarion.com
Market Pressures– Regulatory– Quick check-in
5
The Speed of Light
“Testing is not a phase on Agile teams, testing is a way of life. Agile teams should test continuously. It’s the only way to ensure that the feautures implemented during a given iteration or sprint are actually done”
Polarion Software® Polarion® QA™ - www.polarion.com 6
2/11/2014
4
Time Constraints - Deliver Faster
Pressure to test more with Limited Resources
Focus becomes task completion vs coverageQuality may suffer (may miss critical bugs)
Examples: Madden Football 2008 – prematurely releasedXbox – hardware crashes = LAWSUIT
Lack of Planning – Still need to plan not a free for all
What‘s going on QA – Pitfalls
– More time constraintsLack of Documentation
– Key component of Agile is removing clunky documentation– Does not means getting rid
Daily Meeting Hassle– Waste of time?????
Polarion Software® Polarion® QA™ - www.polarion.com
– Quick check-in
“I’m surprised at the name Agile —it should be called couch potato. There are too many meetings. It’s ironic that there are all these books explaining how simple it is.”
8
2/11/2014
5
The real goal of testing should be to improve the software– A continuous search for information
What‘s going on QA
– Instead of pass or fail think in terms of whether or not tests provide valuable information
Understanding project and goals– Define coverage requirements– Understand/establish goals of a sprint
O f %
Polarion Software® Polarion® QA™ - www.polarion.com
“Out of 200 participants, 64 % said that switching to Agile Development was harder than it initially seemed.”
9
•Risk-based testing accommodates change (focus on main areas)– Reduce the scope of testing based on risk
What‘s going on QA -Solutions
•Careful automated testing can accomodate change*
•Good unit testing is essential (limited bug-finding 30%)
•System Testing will cover more on average
Polarion Software® Polarion® QA™ - www.polarion.com
Combine risk-based testing with maintainable automated regression testing at the system test level
10
2/11/2014
6
The big data construct
What is Big Data?
2/11/2014
7
“Data sets whose size is beyond the ability of commonly used tools to process it within acceptable time”.
Big Data – What is it?
Transformation in how society processes information.
Explosion of big data is just as transformative as the internet was.
According to IBM, “Every day, we create 2.5 quintillion bytes of data —so much that 90% of the data in the world today has been created in
Polarion Software® Polarion® QA™ - www.polarion.com
so much that 90% of the data in the world today has been created in the last two years alone”.
Gartner believes Big Data will drive a total IT spend of $34billion in 2013 rising to $232billion by 2016!
13
Big Data – Sources
Polarion Software® Polarion® QA™ - www.polarion.com 14
2/11/2014
8
What is Big Data?
The big data construct theory
Source: Gartner
To gain the right insights, big data is typically broken down by three characteristics:
Big Data – 3Vs?
•Volume: How much data•Velocity: How fast data is processed•Variety: The various types of data
Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
Polarion Software® Polarion® QA™ - www.polarion.com
processing for enhanced insight and decision making.
16
2/11/2014
9
Data Volume
"I have too much data to analyze" “We live in a world that is getting more digitized and more connected
Big Data Volume - Challenges
every day.”Data is now created by employees, customers, 3rd partyExtracting information from large data sets Massive, complex and semi-structuredRegulatory demands increase pressure on testingConsolidation of data from systems, apps, internetMi i d t & d t i t it
Polarion Software® Polarion® QA™ - www.polarion.com
Missing data & data integrity
18
2/11/2014
10
Data Velocity
Speed at which new data is being created
Big Data – Velocity
The need for real-time analyticsacquisition of data vs. actionable insight
Polarion Software® Polarion® QA™ - www.polarion.com
“Every business is now a digital business”
20
2/11/2014
11
The ability to implement a data driven testing process
Big Data – Velocity Challenges
Improving response rates for decision making
Expectations – when fast isn’t fast enough
Polarion Software® Polarion® QA™ - www.polarion.com 21
Data Variety
2/11/2014
12
structured/unstructured datatext
Big Data – Variety
textsensor dataaudiovideoclick streamslog files
Polarion Software® Polarion® QA™ - www.polarion.com
g
23
80% data growth in images & videoNumber of data types Unstructured/semi
Big Data – Variety Challenges
Number of data types Unstructured/semiVolatility of changing dataSources of data
Mobile devicesInternetEmail
Polarion Software® Polarion® QA™ - www.polarion.com 24
2/11/2014
13
Traditional methods not enough
Big Data – Challenges
Traditional tools are inadequate
Lack of knowledge as to what to test and how much data to test
Polarion Software® Polarion® QA™ - www.polarion.com
Test strategies for structured vs unstructured data
25
Test early – Test fastData driven testing
Big Data – Solutions
Data driven testing Develop a strong testing environmentMultiple testing methodologies
functional/non-functionalData Sampling
Identify suitable sampling technique
Polarion Software® Polarion® QA™ - www.polarion.com
y p g qInclude correct uses casesRight test data sets
26
2/11/2014
14
Mobile Testing
Mobility application testers, face many critical challenges compared to conventional software testingchallenges compared to conventional software testing
Number of mobile phones will exceed world population by 2014 (7.3 billion)
Mobile – Landscape
y ( )
Polarion Software® Polarion® QA™ - www.polarion.com 28
2/11/2014
15
BYODComplexity added due to combination of multiple
Mobile– Challenges
Complexity added due to combination of multiple platforms each having numerous devices and various OS versions
Various versions of each platform and deviceMobile user: range from tech savvy to newbieTesting on various networks and network vendors
Polarion Software® Polarion® QA™ - www.polarion.com
CDMA and GSM with their 2G, 3G and 4G versions
29
Rapid Roll-out ScheduleExpectations for apps
Mobile– Challenges
Expectations for appsUltra competitive to release new apps
Complexity added due to combination of multiple platforms each having numerous devices and various OS versions
Polarion Software® Polarion® QA™ - www.polarion.com 30
2/11/2014
16
Emulation Software (Zap, Keynote)
Mobile– Solutions
Cloud Computing (Perfecto, Soasta)Crowd sourcing
Performance Testing (Loadster, Neotys)
Polarion Software® Polarion® QA™ - www.polarion.com 31
Tools
2/11/2014
17
Tools
Unavailability of right tools– Budget, availability and funding– Type of Tools:
• DEFFECT TRACKING• TEST CASE CREATION & Execution• Test Automation
Integrations: lack there of– Integrations: lack there of – MS Excel & MS Word equation
• Time consuming to manage and maintain• Metrics • Integrations
Tools - Solution
Test Management Software– Combines Test Cases, Test Executions,
Defects, Test Automation– Replace or leverage MS Excel/Word– Customizable workflow– Automation– Automation– Integrations
2/11/2014
18
Tools - Solution
Network with other testers to find information about l k llesser-known test tools. Online QA/Testing forumsInteroperable tools
Automation
2/11/2014
19
Test Automation
1. Test Automation Will Eliminate Human Testers1. Test Automation Will Eliminate Human TestersFALSEFALSE
2. What tests should be automated first?2. What tests should be automated first?Automate your testing procedure when you have lot of regression work.
Test Automation - Implementation
– Commitmentcan take time to set p & maintain• can take time to set-up & maintain
• If not implemented correctly = $$$ Desire to automate tests across apps & platformsMost automated test tools do not tell you what to testInitial cost can be high
• purchase and set-up
100% Automation does not exist!
2/11/2014
20
Test Automation - Solutions
The right automation – Right skill sets (programing scripts)– Ensure we create reusable tests
• can take time to set-up & maintain– Unit test will provide highest ROI
There is a large gap between the people who ownautomated test tools and the people who actually realize the benefits from test automation.
5 Lessons Learned
2/11/2014
21
Lessons
To handle the volume and ensure live data i t ti Bi D t t ti d t integration, Big Data testing needs to empower developers, testers and enterprises with freedom to experiment and innovate.
Lessons
Successful software test automation is possible if f d t l i dd d d d fundamental issues are addressed and managed. Automated software testing is truly a different way of testing and requires adjustments to current test methods and organizational structures. However, thepayback from test automation can far outweigh the costs.
2/11/2014
22
Lessons
With nearly a third of all page views now coming f bil d t bl t t t d t b from mobile and tablets, testers need to embrace mobile. Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers.
Lessons
Development cycles will continue to accelerate b d i f t ( k t based on various factors (market, process, technology), and quality expectations will need to meet the challenge through increased visibility, new ways of thinking combined with advances in software.
2/11/2014
23
Lessons
What did you learn????
Resources“Game Testing Evolves” whitepaper @ Polarion
“Big Data Testing” whitepaper @ Polarion Big Data Testing whitepaper @ Polarion
http://www.polarion.com/products/qa/index.php
T itt R St k1Twitter: ReggStruyk1
LinkedIn: Software Test Case Management