Shift Left - Approach and practices with IBM

© 2013 IBM Corporation

Shifting Left Approach and Practices

Paul Bahrs Chief Architect, Emerging Technologies, IBM Software, Rational

Nov 6, 2014

Who is this guy?

• Paul is the Chief Architect for emerging technologies, focused on new or acquired solutions and accelerating early client successes

• Today he works with IBM clients with their adoption of DevOps, Cloud, and Quality solutions to meet market and business demands

• Paul is the published on DevWorks, ibm.com, a presenter at industry/IBM conferences, patent author and supported open standards

Paul Bahrs

Market shifts are fundamentally changing what and how we do IT

3

Software Development Customers

Development Testing Deployment

Macro Business Environment Volatile economic and changing regulatory

environments require businesses to adapt

quickly to changing market conditions

Empowered Users The exponential increase

in empowered users drives

expectations for higher quality

customer experiences and the

need for continuously updated

software

Technology Trends Mobile, cloud, big data, social,

agile and delivery analytics

increase application complexity

while promising better

accessibility, flexibility

and agility

Business Owners

Develop / Test

Deploy Steer Operate

IBM DevOps – most comprehensive capabilities Addresses bottlenecks and waste across the delivery lifecycle

Continuously plan, measure and

bring business strategy and

customer feedback into the

development lifecycle.

Enable collaboration between

business, development, and QA

to deliver innovative, quality

software continuously.

Reduce the cost of testing while helping

development teams balance quality and

speed.

Deliver software to customers

and internal users faster and

more frequently with better

quality, lower cost, and reduced

risk.

Understand and accommodate the

user perspective to achieve service

levels with better visibility and

continuous feedback across the

entire software lifecycle.

Continuous

Business Planning Collaborative

Development

Continuous

Testing

Continuous Release

and Deployment Continuous

Monitoring

Continuous

Customer Feedback

& Optimization

Provide the visual evidence and full

context for analyzing customer behavior

and pinpointing pain points.


Some Observations

6-12 Month Delivery Cycles Are Still the Norm

Where do Testers want to spend their time?

Creating Automated Tests

Performing Exploratory Tests

Executing Automated Tests

Designing Tests

Planning Tests

Reviewing Test Results

Running Pre-defined Tests

Maintaining Automated Tests

I want to spend MORE time…

I want to spend LESS time…

Creating Reports

Maintaining Automated Test Tools

Configuring Test Tools

Creating Test Data

Re-running tests

Investigating, Documenting and Submitting Defects

Setting Up Test Labs

What are deployment engineers doing?

*Data based on UrbanCode customer survey

Bottlenecks affecting Test

Delays for

Business Scope

Provision,

Deploy, Test,

Delivery,

Integration, Build

Production

Releases

Continuous Changes

from Business

Water Fall Projects

Biz-Dev-Test-Ops are organizational aligned and interact formally

Integration test phase immediately follows Development to “enable testing”

Testing follows INT and is usually impacted by delays or bottlenecks

– Time delays to scope project

– Time to move changes through test to production

Change from business is continuous throughout the project

Feature deliveries

weekly or bi-weekly

Integrations are resource

intensive and manual

QA and beyond require

formal builds

Build and Deploy process are

governed but usually not

automated nor continuous.

Managed or virtual environments

are not necessarily impacted

Deployments are performed by

developers in Dev/INT and more

formally (by organizations) in other

environments.

Feedback to features / stories

may still be delayed by weeks

or provided by the developer

The need for INT is relative to

the testing performed during

iterations. QA is still formal

test within interface context

Water-Agile-Fall Projects

Biz-Dev-Test-Ops are organizational aligned and interact formally except for construction

Integration test phase may still follow Development to “enable testing”

Iterations achieve planning flexibility, but most testing still occurs after development

– Some departments may use periodic iterations to integrate, build, deploy, test

– Developers are usually burdened with doing their own testing during iterations


Feature / story

deliveries during

iterations

Integrations may still be

resource intensive but

performed more often

Iteration testing requires

periodic builds to verify

but may not test


either distinct or heavily

interdependent

Managed environments require 2-3

weeks to 2-3 months to prepare

Deployments are performed by

multiple organizations with different

processes/technologies

Feedback to features may be

delayed by weeks or months

or not at all due to time

INT and QA are first time to

perform test within interface

context

Agile Projects

Biz-Dev-Test are usually collaborative for Dev purposes. Ops may still be organizational

aligned and interact formally

Integration and QA test phase needs are reduced if testing performed during construction

Bottlenecks begin at

– Business planning to support iterative or accelerate release cycles

– Ability of environments, application deployments to keep up with agile teams

– Test team to support iteration testing – early and expand testing to exploratory/performance/loading


Stories delivered

during iterations

Integrations are ongoing

between team members,

components

Builds services may be

provided for individual,

team and applications


run often to verify and to run

test/scanning during process

Environment availability is

dependent on complexity of

changes and capacity of ops

Time through the “post construction”

pipeline is determined by quality of

code, risk and velocity of pipeline.

Feedback to developers to

dependent on periodicity, type

and quality of testing

Testing periodicity is

dependent on the provision,

deploy process capacity

On to Shifting Left

What is Shift Left Testing?

"Shift Left" is a DevOps practice that provides an effective means to perform testing with

or in parallel to development activities.

– Development, test and operations work together to plan, manage and execute automated and

continuous testing to accelerate feedback to developers

– The rate of the feedback is determined by an organization’s desired velocity.

– Technology is used to automate processes and virtualize components.

Testing may be performed as a part of the development process or as a service running in

parallel to development activities. In either case, shifting left accelerates feedback to

developers and improves the quality of code delivered to QA.

Shift left is an essential capability for Agile teams but may be successfully leveraged by all

development teams.

What Shifts Left?

Perspectives

– Testing: Begin test execution earlier

– Development: Act on feedback to achieve value

– PM: Resource allocation to address feedback

– Infrastructure: Shift left environments available

early, capacity drives availability

– Deployment: Accelerated capacity to meet test

periodicity

Activities:

– plan, create and automate integration test

cases in advance

– support a periodic cycle of integration testing

for the components/applications under

change

– prioritize, process, and resolve feedback by

development teams within the feedback

period

– plan, automate and deploy components,

applications and virtual services

– plan, create and virtualize component

services relevant to the development

activities

– execute delivery, integration, build, deploy,

testing, feedback and correction during the

defined periodicity.

Whatever testing you can perform shifts left - Integration testing could be

your first step as a means to provide a means to prepare for a formal QA

and reduce bottlenecks associated with Integration testing post

development.

Why Some Clients Shift Test Left

Move Defect Curve to left – Reduce costs because defects are cheaper earlier (Dev/Test teams cannot test

adequately)

– Change enough to find more defects early (Dev/Test teams have a long wait time to

get the right environment)

– No change to project management processes (We have too many configurations to

be tested. There is no consistent way to spin the right environments in time without

errors)

Reduce Issues in Production – Dramatic quality improvements earlier in pipeline to reduce risk to production

– Continue quality maturing through to production

– Support across technology

– Promotion criteria should ensure quality is met

Prepare for Continuous Delivery – Reduce bottleneck in QA and set stage for continuous delivery

– Start QA on delivery of first feature

– Ensure automation supports usage for Developer, Team, Application, Project

– Ensure the proper SLA’s and Reuse are supported

Support for Agile teams – Ability for Agile teams to test daily

– Ability to provide and act on feedback daily

– Ability to scale as Agile adoption scales

Where should you change?

Environment

The “What” The “How” The “Where” The “Verb”

• Application Development

• Application Deployment

• Environment Provision • All types of Testing

Steer Develop / Test Deploy Operate

Sc

ale

d

Reli

ab

le

Rep

ea

tab

le

Pra

cti

ced

Practice improvements

Define release with business objectives

Measure to customer value

Optimize applications

Use enterprise issue resolution procedures

Standardize and automate cross-enterprise

Automate patterns-based provision and deploy

Manage data and virtual services for test

Deliver and integrate and build continuously

Link objectives to releases

Centralize Requirements Management

Measure to project metrics

Link lifecycle information

Deliver and build with test

Automate testing

Embed Quality Reporting

Document objectives locally

Manage department resources

Manage Lifecycle artifacts

Schedule SCM integrations and automated builds

Test following construction

Plan and manage releases

Standardize deployments

Monitor resources consistently

Collaborate Dev/Ops informally

Plan and source strategically

Dashboard portfolio measures

Automate problem isolation and issue resolution

Optimize to customer KPIs continuously

Improve continuously with development intelligence

Test Continuously

Leverage Quality Tends

Manage environments through automation

Provide self-service build, provision and deploy

Adopting IBM’s approach: https://ibm.biz/BdDaMe

Plan departmental releases and automate status

Automated deployment with standard topologies

Monitor using business and end user context

Centralize event notification and incident resolution

https://ibm.biz/BdDaMe

Considerations

Overall

– Identify feedback velocity and means to

measure it

– Scope of automation pipeline (build, deploy,

test)

– Decide which changes will improve team’s

success (versus introduce functionality)

– Continuously improve and plan for next steps

(they will be needed)

Software Configuration Management

– Delivery and integration processes velocity and

measures

– Complexity that causes bottlenecks and delays

Build

– Time to build – number of application level

builds

– Accessibility for developer, team, application

– Periodicity of builds and disposition of results

Deploy

– Time to deploy

– Rate of deployment

– Scalability to extend to post deployment

actions

– Cross platform, approved versions and virtual

services

Test

– Provide complete or virtual environments

each event

– Only start testing early only then improve

– Test automation should follow immediately

Lifecycle Measurements 2008 2010 2012 – 2014 Total

Improvement

Project Initiation 30 days 10 days 2 days 28 days

Groomed Backlog 90 days 45 days On-going 89 days

Overall Time To Development 120 days 55 days 3 days 117 days

Composite Build Time 36 hours 12 hours 5 hours 700 %

BVT Availability N / A 18 hours < 1hour 17 hours

Iteration Test Time 5 days 2 days 14 hours 4 days

Total Deployment Time 2 days 8 hours 4 hours -> 20

minutes

2 days

Overall Time To Production 9 days 3 days 2 days 7 days

Time Between Releases 12 Months 12 Months 3 Months 9 Months

Innovation / Maintenance 58% / 42% 64% / 36% 78% / 22% +20% / -20%

Double-digit revenue growth, increased client adoption, improved client satisfaction

How IBM Rational Cloud Hosted Products have improved!

A Waterfall Client’s Experiences with Shift Left

Focus was on shifting integration test left, but not boiling the ocean on change. The

practices that would be affected included code changes, delivery and integration, build

automation, deployment automation. Test practices would merely execute earlier

Outcomes

Rework reduction: First time in 3 years to impact defect rate going into QA – 90%

reduction.

Practice impact: Demos of UI and retrospective improved requirements understanding and

business acceptance. No pilot project ever reported as medium or high risk during pilot.

Work reduction: Over 50% of testing was executed first week in QA almost 100% success

(normally done in weeks with almost 50% defects)

Collaboration: Qualitative improvement of end to end testing. Faster issue resolution and

cross team coordination. Automated build, deploy, unit test/scan and virtual service

activation processes.

First Steps – Integration Test Feedback

Environment

The “What” The “How” The “Where” The “Verb”

• Application Dev

• Code coverage

• Static Scans

• Code Reviews

• Security Scans

• Two/Screen

• Code deliveries

• Change Integration

• Binary management

• Documentation

• Technology maintenance

• Process execution

• Extensibility

• Speed, repeatability

• Scalability

• Environment consistency

• Levels of Delays

• Resource requirements

• Error prone processes

• Documentation

• Agility

• Availability

• Manual processes

• Automated processes

• Unit testing

• Regression testing

• Integration testing

• Requirements coverage

• Code coverage

• Test Data

• Virtual services/stubs

Shift Left Planning Workshop

Goals and

Assessment

•Client Capabilities: Current Status

•Business Goals for Improvement

•Solution Capability Oriented

•People, Practices, Technology, information

Capability

Priorities

•IBM Best Practices for Solution Capabilities

•Discussions to refine Objective Capability

•Prioritize Incremental Capabilities

•Vision, User Value, Pain and Complexity

Executive

Sponsor

Review

•Review Outcomes

•Assumptions & Risks

•Gain concurrence

•Identify next steps

Solution

Improvement

Roadmap

•IBM Best Practices for Adopting Solution

•Identify Key Milestones

•Roadmap activity to define actionable plan

•Define Quick Win Pilot

Da

y 1

D

ay 2

D

ay 3

D

ay 4

Current State

Assessment

Objective & Prioritized

Capabilities

Adoption Roadmap

Draft Results

Detailed 1-3 month roadmap

for initial win. High level

roadmap for remainder of

initiative. Executable week

following the workshop

Theme Activities Objective

Yes, we have tools that help

Rational Team Concert –Continuous Integration

UrbanCode Deploy –Application Deployment Automation

Rational Test Virtualization Server –Service Virtualization

Now what?

Check out our Practices Self Assessment Tool to assess your current

practices in context to Shift Left

– Ibm.biz/devops-practices-assessment

Review our Shift Left Practices @ Jazz.net

–Jazz.net/shift_left_practices

Contact us to discuss our workshop to plan your Quality improvements

–[email protected]

Reach out to me for more discussion

–@paulbahrs

–[email protected]

25

mailto:[email protected]

mailto:[email protected]

Next step - Shift Monitoring Left

26


www.ibm.com/devops

http://www.ibm.com/software/rational


© Copyright IBM Corporation 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/devops

http://www.ibm.com/software/rational