Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Scrum Embedded Systems

Orlando Scrum Gathering, March 16, 2009

Myllerup Consult, Hasseltoften 11, 8361 Hasselager, DENMARK +45 2834 9084, info@myllerup.dk

Bent MyllerupScrum Coach, Agile Trainer and Software Development Manager

- An experience report from

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Presentation outline

• The background for implementing Scrum at TC Electronic (TCE)

• The challenges in combining embedded software development and Scrum

• How we responded to the challenges

• Challenges we see today and our next steps

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

My background

Myllerup Consult

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

• Established in 1976 by John and Kim Rishøj, making quality guitar effects pedals

• Market leader in professional signal processing

– approximately 80% of all broadcasted music has at some point been processed by TC equipment

• Customers include performers/musicians, broadcast, studios, live and installations

• Key technologies include in particular digital effects and other signal processing

• Part of the TC Group with total of 420 people having a turnover of approx USD 100 mill

– Each company within the Group focus on a specific market

segment

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

World Locations

TCG Americas

TC Helicon

TCG Americas

TC Applied Technologies

Tannoy TC Electronic

Lab.gruppen

TC China TC Japan

Tannoy IndiaTannoy Dubai

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

How TC Group discovered Scrum

• We had a burning platform in non-performing device drivers for our product line of audio interfaces– Engineers were distributed across the world (India, Canada and

Denmark) with lack of prioritization, direction and communication

• Temporary collocation and Scrum made the difference and showed by example that Scrum actually works

• Executives now see Scrum as a major key to success and “surviving” the financial crisis

• Engineers are still distributed across the world, but the use of scrum makes the difference– We have actually added an additional location: California

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

The organizational environment at TCE

• The product development department in TCE is divided into three business areas, who are responsible for their own product portfolio

• The monthly Steering Group Meeting the whole team meet with management to discuss “state of the nation” and decide on approaches and actions

Steering Group = Management

Guitar Fx Instrument Amp

HD Recording

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Looking at the team in a Business Area

BM

SW eng

HW engTester

PO SM

ProcurementMechanicsICT eng MarketingSourcing

Beta Test

Writer

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Why did we split the PO role?

• The coordinating with procurement calls for traditional project management skills for the PO

• The market calls for musician like attitude of the PO

• We have found it hard find persons who possesses both skills

BM

PO

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

The Product Lifecycle at TC Electronic

Scrum

Vision Backlog Sprints

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Hardware development cycles

Schematic,Mechanical drawings,Mockup tests and95% BOM

Sourcing critical components

Layout and100% BOM

Build prototypes, verification,EMC and safety tests

Sourcing PCB,production and transport

Pre-production and transport

In-circuit test equipment

Beta test

Mass production and transport

Launch

Additional print spin

Verification of pre-production

Possible corrective actions after beta test

Team

External

Legend

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Benefits of having the hardware developers on the team as well

• Mutual commitment towards a common goal – Delivery of the embedded product

• Synergy between hardware and software in design and problem solving– Making the right solution at the right place and at the right price

• Improved quality– No “them and us” mentality

– Quality assurance and test on the whole product

• Scalability– Hardware and software engineers help each other (e.g. building

prototypes and do diverse tests)

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Challenges in applying Scrum in embedded projects at TC Electronic

• When the cross functional team includes hardware engineers, the whole team experiences constraints to external resources:

– Producers of prototypes (printed circuit boards and mechanical prototypes)

– The production plant (in circuit test equipment, sourcing of components, preproduction)

– Safety and Electromagnetic Compatibility (EMC) certified test centers

• You can use nightly builds and tests, but have to test on the target in order to be certain that things actually work!

• Automatically verification of digital processed audio is a science in itself

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

How we responded to the challenges

• Intensive coaching and teambuilding

• The idle time when awaiting prototypes to be produced makes especially the hardware developers players in several projects in parallel– Instead of having team members defocused on several projects, we

made the team responsible for the portfolio plan – one prioritized sprint backlog for several projects

• Product backlog– We love user stories, but found a need for adding technical,

production and approval related backlog items as well

– For each backlog item we specifies an accept or DONE criteria

• We implemented a setup of several build servers connected to our code repository– Mac Based

– Windows Based

– Embedded (AVR)

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Handling requirements and monitoring progress in a multi-product BA

Pro

duct

A B

acklo

g

Pro

duct

B B

acklo

g

Pro

duct

C B

acklo

g

Com

mon S

pri

nt

Backlo

g

Product A Increment

Product BIncrement

Product CIncrement

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Business Manager

The Scrum Processat TC Electronic

• Cross-functional, seven plus/minus two members

• Selects the iteration goal and specifies work results

• Has the right to do everything within the boundaries of the project guidelines to reach the iteration goal

• Organizes itself and its work

• Demos work results to the Business Manager and Program Manager

Team

• Ensures that the team is fully functional and productive

• Enables close cooperation across all roles and functions and removes barriers

• Shields the team from external interferences

• Ensures that the process is followed.

• Invites to daily scrum, sprint review and sprint planning

ScrumMaster

• Is responsible for portfolio and project planning

• Is responsible for the project budget (expenses, deliveries and time schedule)

• Initiates and maintains Product Backlog, and prioritizes features according to business values

• Facilitates making an absolute prioritization between features from various projects

Program Manager

• Is responsible for initiating the product development and the project/product profitability

• Defines user related features and their business value in the Product Backlog

• Can change user related features and their business values at every sprint planning meeting

• Accepts or rejects work results with respect to user related features

24 hours

2 to 4 weeks

Daily Scrum:• What have you done since last meeting?• What do you intend to do until next

meeting?• What is in your way?

Prior to the meeting:• Estimate remaining work

After the meeting:• Re-planning if off schedule

Sprint Review:• Demonstration of DONE functionality• Acceptance• Figuring out what is best to do nextSprint Backlog:

• Desired functionality selected by Business Manager and Program Manager

• Remember the bucket of water approach: ∑Size and complexity <= Team velocity

Sprint Planning:• Business Manager, Program Manager and Team discuss desired functionality• Team defines necessary tasks in order to develop functionality• Team defines sprint goals and commit to deliver• User Stories are discussed (first half) and broken into tasks (second half)

Sprint Retrospective:• Team takes pride in what it has

accomplished• What went well?• What can be improved?• Define concrete actions for next sprint• Assess team velocity

Product delivery• Potentially ready for use

Sprint Plan:• Defined by the Team• Task size <= 2 days

Product Backlog:• Described in User Stories with acceptance criteria’s. Story size <= half a sprint• Prioritized by the Program Manager and Business Manager• Maintained by the Program Manager• Estimated by the team (Planning Poker)• Anyone can contribute

Prod

uct

Ow

ner

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Testing…

• Unit test– Hard to implement on legacy code

– We start new projects with a TDD approach

• System test– We are incorporating SeqZap from Sequanto in order to perform

automated test on target

• Beta test– We have a corps of beta testers who test products in action before

product launch.

• Most of TCE employees are musicians and users them selves – enforcing high responsibility and awareness of quality

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Automated test on embedded platforms

• We are using SeqZap from Sequanto for script based automated test on target

• We are planning to incorporate it into our nightly build and test environment

• SeqZap is easy to use and expand with customized modules and interfaces

Verdict Report

MIDIControl/Logic verification

AESDigital audio verification

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Changing the mindset

From: “This simply isn’t possible in a hardware environment”

To: “We couldn’t have made this great product with out it”

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Breaking down boundaries with coaching and teambuilding

• Scrum is more than just roles and rituals– So in order to be successful, you must foster high performance

teams

• We did (and are still doing) a lot of intensive coaching and teambuilding. Some examples are:– Team awareness exercises

– Ability spotting

– Team coaching after principles taken from Appreciative Inquiry

– Conflict resolution

– Individual coaching of key players

• The above is based on my experience as a Systemic Coach

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

The theory behind it

StormingForming Norming Performing

Tuckman combined with Katzenbach and Smith:

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Another theory behind it

Relations

Direction

Latitude

Scope

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Ability spotting

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Appreciative team coaching

Phase 5: DeliverPlan how to

implement and maintain the

changes

Phase 4: DesignDefine concrete

goals

Phase 3: DreamingCreate a shared image of how it should be in

the future – what would we like to

achieve?

Phase 2: DiscoverFocus on best practice. How and when does it

work in the team.

Phase 1: Definition Define what needs to

be explored and developed on

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

What we have gained through coaching and teambuilding

• A shorter path to performing teams

• Hardware developers have bought into using Scrum as the development process– And they actually like it

• Individual minded team members have broken down the mental walls they have been hiding behind

• Improved communication in the team– It’s okay to ask for and offer help

– It’s okay to say “NO!”

– There is a positive and appreciative attitude

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Also: Use the Scrum rituals to foster team identity

• Planning

• Daily Scrum

• Sprint Review

• Sprint retrospective

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Sprint reviews

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Next steps

• Implement Test Driven Development

• Enlarge the team with Procurement and ICT Engineer

• Think Lean in the whole organization

• ...

Ag

ile C

oach

ing

-te

am

em

po

werm

en

t…

Thank you for your attention

www.agilecoaching.dk