Upload
parasoft
View
208
Download
0
Embed Size (px)
Citation preview
Parasoft Copyright © 2016 1
2016-11-23
Evolving from Automated to Continuous Testing
Better Software 2016 - Orlando
Parasoft Copyright © 2016 22
Intro
Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame
Follow him @codecurmudgeon
Blog: http://codecurmudgeon.com
Web: http://parasoft.com
Parasoft Copyright © 2016 33
Agenda
Today’s Challenges
What IS Continuous?
Barriers
Best practices
Role of automation
Role of static analysis!?
Parasoft Copyright © 2016 44
Normal Situation
Pressure to deliver software quickly
Pressure to reduce defect rate
Late stage “bug-hunting”
No time for error prevention
Quality can be rushed
Security is forgotten
Parasoft Copyright © 2016 66
How to you know a build is successful?
When it compiles?
When all unit-tests have run?
When the right coverage goal is achieved?
When the right % of test failures occurs?
When it’s deployed?
Parasoft Copyright © 2016 77
Continuous
Build, Integration, Testing, Release, Delivery, Deployment
Builds on and enables DevOps
High dependency on automation:
processes
assessment
decisions
Parasoft Copyright © 2016 88
Feature Release
Business Stakeholder Developer Customer Support
Continuous Testing is Beyond Automation
SoftwareFeature
Continuous measurements mean continuous refinement of the process
Real-time feedback from
Objective assessment and go/no go Defects are eliminated at the point
that they are easiest to fix
Business Stakeholder Developer Customer Support
Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met
Parasoft Copyright © 2016 99
Challenges of Continuous
Tests must produce binary decision go/no-go
Reuse unit test and functional tests from devto QA
High level of automation
Requires disciplined mature process
Testing must automatically answer
Is it stable
Will it do what it’s supposed to do
Parasoft Copyright © 2016 1010
Importance of Testing
How do you know when you're done?
How do you know if a fix for a minor bug broke a major function of the system?
How can the system evolve into something more than is currently envisioned?
Testing, both unit and functional, needs to be an integrated part of the development process.
Parasoft Copyright © 2016 1111
Continuous Delivery
Continuous Integration
Delivery Team
Version Control
Build & Unit Test
Automated Acceptance
Tests
User Acceptance
TestsRelease
Check In Trigger
Trigger Trigger
TriggerTriggerApproval Approval
Check In
Check In
Feedback
Feedback
Continuous Deployment
Production
Business
Continuous *
Parasoft Copyright © 2016 1212
Elements of Continuous Testing
Continuous Testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?”
ContinuousTesting
RiskAssessment
PolicyAnalysis
RequirementsTraceability
TestEnvironment
Access
TestOptimization
AdvancedAnalysis
Ensure access to complete test environments
Automate defect prevention andPolicy measurement
Expand test coverage and measure test effectiveness
Define actionablepractices
Connect functional with non-functional requirements
Process improvementopportunities
Parasoft Copyright © 2016 1313
Prerequisites
Tools
• Version control
• Build server
• Deployment server
• CI tools
• Automation tools (test, etc)
Process
• Commit/update often (each change)
• Always create tests (pass and fail)
• Test regularly
• Run regularly
Parasoft Copyright © 2016 1414
Continuous Testing
Tests are logically componentized
Tests correlated with business requirements
Tests are incremental
Tests are repeatable
Tests are deterministic
Tests are maintainable in a process
A process that isprescriptive based on test results
Parasoft Copyright © 2016 1515
Penetration of Automated Testing Practices
Aggregated from analyst research: Gartner, Forrester, voke, IDC, Ovum for enterprise IT (embedded market eliminated)
Practice Penetration Opportunities
Static Analysis 16 –38% • Business/Risk driven models• Process enabled
Unit Testing 15 – 32% • Workflow • Test optimization• Deterministic tests
Peer Review ~ 26% • Integrated results• Contextual review• Compliance triggers
API Testing 12 – 21% • 2nd lowest penetration• Regression models
Load Testing 32 – 49% • Smoke testing • Continuous
Service Virtualization 9 - 16% • Green field• Process enabler• ROI is a NO-Brainer!
Parasoft Copyright © 2016 1616
Continuous needs
Proper infrastructure
A binary definition of “done”
Reliable quality gates
Extreme automation
Parasoft Copyright © 2016 1717
Infrastructure
Real requirements system
Flexible developer level project/scrum management system
Automated build/testing system
Data collection that can answer
Is it done?
When will it BE done?
Is it good enough to release
Parasoft Copyright © 2016 1818
Defining Done
Are requirements coded?
Do tests exist for the requirements and code?
Are the tests passing
To your satisfaction (often not 100%)
Will the tests be enough?
Parasoft Copyright © 2016 1919
Delivery is Part of Software
Delivery is an important part of software too
Development and Test need access to production-like systems
Deployment has to be repeatable and reliable
Quality must be validated (measured)
Parasoft Copyright © 2016 2020
Defining Quality
• Consistent
• Repeatable
• Meaningful
Automated objective measure
• Unit tests
• Functional tests
• Regressions tests
• Static analysis
Made up of
Parasoft Copyright © 2016 2121
Effective Continuous Testing
Avoid manual end-to-end testing
Invest more into automated tests at component
Follow testing pyramid rules
Measure: test results
test effectiveness
Determine: Risk
Parasoft Copyright © 2016 2222
How to do it better ?
Give absolute priority to automated tests
Invest time into designing interfaces (API)
Use API (service) testing tools to cover interfaces
Measure the quality of the test
Parasoft Copyright © 2016 2323
How Can Static Analysis Help?
Coding standards
Quality gate (bug detection)
Prevention
Parasoft Copyright © 2016 2424
Bug Gates
Binary Decisions
No “bugs”
No “critical” static analysis findings
Parasoft Copyright © 2016 2525
Quotable Quotes
An ounce of prevention is worth a pound of cure. – Benjamin Franklin
Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place. – Deming
Simply finding bugs with static analysis isn’t enough.
Avoid risky behavior
Parasoft Copyright © 2016 2727
Static Analysis Continuous Feedback Loop
Identify Error
Isolate Root Cause
FixNew Rule to Prevent
Monitor
Code review
Regression
QA
Field bugs
Parasoft Copyright © 2016 2828
Choosing rules
Things happening in
the field
Things you worry will
happen
Things happening in
the news
Standards you must comply
with
Parasoft Copyright © 2016 2929
Main Points for Today
We must define “all” quality expectations upfront
We must change the primary goal of testing
We must accept that testing is only a task in the
quality process
We must be able to measure trust of the process
We must automate breaking the application
Parasoft Copyright © 2016 3030
Re-Inventing the SDLC
The evolution of the software development lifecycle over the past 5 years is forcing the transformation of software testing
Transformation Business Impact
• “Digital Industrial Revolution” makes all a software company
• Trend toward Dev/Test insourcing as competitive advantage is via software
• Strong adoption of “agile” development practices
• Forces more technical testing, early. Agile is forcing “shift-left”
• DevOps is applying lean practices to SDLC
• DevOps is exposing 20 years of processbarriers protected by silos and culture
• Cloud has disrupted traditional SDLC and is driving down costs
• Switching costs associated with applications at an all time low
• Business risk associated with software failure at an all time high
• Public companies lose an average of $2.5 Bn in market cap on the day of the announcement
Parasoft Copyright © 2016 3131
DevOps is Continuous Improvement
Speeding up the conveyor belt does not yield better results…
Modern DevOps must embrace systematic process improvement—Focused on testing
Parasoft Copyright © 2016 3232
How to Achieve Quality @ Speed
Business risks drive quality activities
Teams collaborate on risk definition
All team members trained on risks
Acceptance criteria visible and measured
DevTest team activities prioritized per risk definition
Results of quality activities are visible and translated for all levels of business
Parasoft Copyright © 2016 3333
Survey on Non-Functional Requirements
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Waterfall
Hybrid
Iterative
Agile
Agile-ish
“Yes” My Team Monitors Compliance to NON-Functional Requirements (NFRs)
34%
58%
Parasoft Copyright © 2016 3535
Continuous Testing Mitigates Business Risk
Why ContinuousTesting?
Systematic, centralized decision making to factor
business risk into SDLC
Safety net to allow developers to bring innovations to market faster
…and automation to break the application
A feedback system for better trade-off decisions between release scope,
time and quality
Parasoft Copyright © 2016 3636
Enabling Technologies
Stubs
Service Virtualization
API testing
Test data management
Environment management
Self-service test environments
Parasoft Copyright © 2016 3737
How to do it better ?
Use service virtualization to improve testing automation
Isolate at the message layer
Simulate functional scenarios and performance conditions
Parasoft Copyright © 2016 3838
The Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the process
IT OperationsParallel development delays… Need simple, realistic access to dependent components…
Too much time waiting for access…Need reliable test data
Need a realistic testenvironment easy to maintain
Scheduling
Configuration
Access Limits
DependentApplications
Staged Assets
3rd Party Assets Virtual Environments
Hyper Visor
App App App
Parasoft Copyright © 2016 3939
The Test Environment Challenge
Test environment access is outside the control of development and test leaving gaps in the process
IT Operations
Scheduling
Configuration
Access Limits
DependentApplications
Staged Assets
3rd Party Assets Virtual Environments
Hyper Visor
App App App
Create, Manage, Provision
Parasoft Copyright © 2016 4040
The Challenge Multiple teams using the same test database
Teams not respecting data integrity & others test data records
Regression tests consistently failing. Takes >1 hour to determine that it was due to “data changes”.
“Real problems” were getting lost in the noise
SharedDatabase
Eliminated 83% of configuration time for a
major telecom company
Test data management for complex transactions
✔
✖
✖
Parasoft Copyright © 2016 4141
The Solution Setup Virtual Assets to model the SQL queries and use API testing tool
to manage automated nightly regressions against both virtual assets and live systems
The Business Benefit Test teams able to focus on ‘real regressions’ and separate out data
integrity issues from functional test failure
Virtual Asset
Virtual Asset
Virtual Asset
✔
✔
✔
Eliminated 83% of configuration time for a
major telecom company
Parasoft Copyright © 2016 4242
Reduced wait time for test team by 60% for a major media conglomerate
The Challenge
Large agile development effort to adopt Service Oriented Architecture (SOA)
High risk project but the Test team “stuck waiting for the first build”
Development of functionality was not easy to coordinate as different teams had different schedules; not all finished at the same time
Agile/Parallel development limited by system dependencies
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
Current Development/Testing
dependencies
Parasoft Copyright © 2016 4343
The Solution
Use descriptions of the new services (WSDL, XSD, example JSON payloads) to build Virtual assets prototyping the new functionality.
Test team builds tests with against the prototypes with API testing tool and the independent development tests use the prototypes to perform early stage Integration Testing
The Business Benefits
Met business goals and timelines, were able to test functionality “as soon as” it was available. Practiced TDD against prototypes to get a head-start on ‘full system testing’
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
✔✖✔
✖✔✖
Reduced wait time for test team by 60% for a major media conglomerate
Parasoft Copyright © 2016 4444
Summary
Continuous * requires mature infrastructure and process
Quality gates must produce a binary answer
Automation is everything
Policy is critical, because automation is critical
Static analysis gets you to the gate
Parasoft Copyright © 2016 4545
Web http://www.parasoft.com
IoT Hall-of-Shame http://bit.ly/iotshame
Blog http://alm.parasoft.com
http://codecurmudgeon.com
Book: http://alm.parasoft.com/continuoustestingbook