How we sped up delivery using feature branchesagiliaconference.com/engine/wp-content/uploads/2018/02/2018-AC... · How we sped up delivery using feature branches Agilia Conference

How we sped up delivery

using feature branches

Agilia Conference 2018

Jan Vilímek, Platform Architect

Oriflame Software

About Oriflame

Complexity we face

Challenges we had when we

were growing

How to get effectivity and

flexibility back?

Take away

AGENDA

2Copyright ©2018 by Oriflame Cosmetics Global SA

3

BUT FIRST.. A FEW QUESTIONS FOR YOU..

Copyright ©2018 by Oriflame Cosmetics Global SA

Do you use agile for

developing software?


Which source control do

you use?


Are you already using

feature branches?


About Oriflame

Complexity we face


were growing


flexibility back?

Take away

WHERE ARE WE NOW?


8

Whole globe spread

250+ million online visits per

year

Direct selling company

3 millions active

consultants


~300 people across Europe

Provides IT solutions exclusively to Oriflame

Dynamic team with a family spirit in a wonderfull town of Olomouc, branch in Brno

Partners in Ukraine, Poland

Multinational

Center of technology innovations

ORIFLAME SOFTWARE


Enterprise Resource Planning / Invoicing / Warehouse

systems

Data Warehouse / Business Intelligence

Product catalogues DTP, pricing

Digital e-commerce platform

See next slides

DEVELOPING & SUPPORTING


E-COMMERCE CONSTANT TRANSFORMATION

11

1990 - 2001 2002 - 2013 2014-2015 2016-2019

Orico Orisales Online V4 Cumulus

1-2 Servers Farm of servers Elastic cloud computing

WebDigital

MarketingDigital Business

Autonomous

Business

+Smart

Machines

2000 2005 2017 2025

2020-2025

Combination of people,

connected things, analytics and

algorithms, providing a platform

for business operations

Operators only

(~100s users)

Online orders

growing…99% orders done online (3M users)


E-COMMERCE

Copyright ©2018 by Oriflame Cosmetics Global SA 12

SEVERAL MOBILE APPLICATIONS


14 cross-functional distributed development teams

2 weeks sprint

Every team:

Front-End & Back-End developers

Test Analyst and Engineer

Product Owner, Scrum Master, Architect (guilds)

Owns specific business domains

E-COMMERCE TEAMS


About Oriflame

Complexity we face


were growing


flexibility back?

Take away

WHERE ARE WE NOW?


60 countries (legal / functional requirements) in one monolith

Central driven content

Many 3rd party systems (sms / payment providers)

Legacy sales system integration

24/7 site availability, many parts (x6 regions) => complex deployment

Cloud (specific development)

OUR SITUATION


SALES PEAK HANDLING

17

One month

One day

Page views per hour in one country

One of the reasons

to use cloud..


QA ENVIRONMENT

18

Cluster in the South

Central US

Cluster in East Asia


About Oriflame

Complexity we face


were growing


flexibility back?

Take away

WHERE ARE WE NOW?


20

LIVEDEVTEST

User Story 2

User Story 1

MAIN

Smoke and functionaltesting

“Regression" testingLive deploy + post deploy

support

PRE-LIVE

Release

Once per 2 weeks full merge

HotfixMerge based

on label

What was in MAIN

branch in time of the

merge was eventually

pushed to Live

Is it OK to

check-in now or

should I wait?

WHEN WE HAD ONLY SEVERAL TEAMS, DEVELOPMENT WAS EASY…DEVELOPMENT LIFECYCLE OVERVIEW


More developers, but also testers, analysts

More teams

Wheel re-inventing

How to coordinate?

WE HAVE BEEN GROWING…

21

Architect guild was

established


Once per week Architect Synchro

Dual track agile

Architecture track

Delivery track

SOFTWARE ARCHITECTS GUILD

22

LEAD DEV

ARCHITECT

LEAD DEV

ARCHITECT

LEAD DEV

ARCHITECT

ARCH LEAD

DEV MANAGER

Architect Lead DEV


No stable code in Main

Needs for formal Quality Assurance

Quality gate:

User Story can’t be deployed to production unless it’s tested

Part of Definition of Done

WE HAVE BEEN GROWING…


Team Foundation Server

No environment for testing

Environments too complex (Tamagotchi)

SHOULD WE USE FEATURE BRANCHES?

24

So… why not do

some branching?


DEPLOYMENT WITH QA GATE

25

QA LIVEDEVTEST

User Story 2

User Story 1

DEV

Smoke testingFunctional testing +

acceptanceRegression testing

Live deploy + post deploy support

QAcherry

picking

Release/HF X.YMain

cherry

picking

new

branch

PRE-LIVE


Developer

• Implement and test locally

Developer

•Check-in to Dev branch

Developer

•Ask for CODE REVIEW

•Test on DEVTEST environment

• (daily deploys)

Developer

•Code Review OK?

•Task [Implementation] ⇒Done

Quality Assurance

•Smoke test on DEVTEST environment

• If OK, assign MERGE task to Cherry picker

Cherry picker

•Merge to QA

•Task [Merge QA] ⇒Done

• (daily deploys)

Quality Assurance

•Full test on QAenvironment

• If OK, assign MERGE task to Cherry picker

Cherry picker

•Merge to Main

•Task [Merge Main]

⇒ Done

ACTIVITY FLOW: REGULAR DEVELOPMENT


IT SOUNDS GREAT

SO WHERE IS THE

ISSUE?


TOO MANY CONFLICTS..

28

MAIN MAINQADEV

Former: Now:


MERGE HELL

29

DEV

Ch01 (US01)

Ch02 (US02)

Ch03 (US01)

Ch04 (US02)

Ch05 (US01)

QA

Ch02 (US02)

Ch01 (US01)

Ch03 (US01)

Ch05 (US01)

Ch04 (US02)

MAIN

Ch02 (US02)

Ch01 (US01)

Ch03 (US01)

Ch04 (US02)

Ch05 (US01)

ChXX: changeset XX

USXX: User Story XX

Code in Main not stable..


Development is not able to quickly deliver new increment

respectively get customer feedback

TIME MATTERS…

30

1st day

• Commit to DEV

• Code review with peer, no changes

2nd day

•Deployed to DEVTEST

•Smoke tests (of the feature + code integration)

•Merge to QA

3rd day

• Deployed to QA Env

• Functional + integration tests

• Merge to Main

If issue found in DEV => +1 day (+possibly broken environment)

If issue found in QA => +2 days (+possibly broken environment)


3 days to deliver …… but usually 6 days or more

… in 10 days sprint

©Oriflame Cosmetics AG, 2018 31

About Oriflame

Complexity we face


were growing


flexibility back?

Take away

WHERE ARE WE NOW?

©Oriflame Cosmetics AG, 2016 32

HOW TO GET EFFECTIVITY AND FLEXIBILITY BACK?

33

Change starts from the people

Retro, Yammer, Slack, E-mails…

Software Architects Guild

Technical solution

Scrum Masters Guild

VSTS Epic & Feature boards on

Global level


FEATURE BRANCHES DEPLOYMENT MODEL

34

QA LIVEDEVTEST

Master

Feature full scope testing

Code integrationRegression testing

+ acceptanceManual + auto

regression testingLive deploy + post

deploy support

Release X.Ynew branch

Feature 2

Branches

Environment

mergingFeature 1

Feature 2

PRE-LIVE

Feature 1

Branches

Environment

Separated

environments per FB

Just one Master

branch


35

MAINQADEV

Do you remember?


MAINQADEV

NO CONFLICT IN MASTER BRANCH

36

Feature01

Master

Feature01 Feature01 Feature01


You remember this…

TIME MATTERS… WITH PREVIOUS SETUP

37

1st day

• Commit to DEV

• Code review with peer, no changes

2nd day

• Deployed to DEVTEST

• Smoke tests (of the feature + code integration)

• Merge to QA

3rd day

• Deployed to QA Env

• Functional + integration tests

• Merge to Main


FROM 3-6 DAYS TO 3 HOURS

38

1st hour

• Create branch

• First commit

• Code review with peer

• Start build + new environment

2nd hour

• Deploy changes

• Smoke test

• Run integration (auto) tests

3rd hour

• Functional tests (Quality Assurance )

• Merge to Master


39

THE BENEFITS


Deployment from 3-6 days to 3 hours

Faster feedback from Product Owners / customers

=> Less feature redevelopment

Higher team responsibility

End of the merging hell, more stable environments

WHAT HAS THE SOLUTION BROUGHT US?


Isolated Continuous Integration environment per feature branch

Branch master

Process change

Teams involvement

Mindset change

WHAT DO WE NEEDED FOR FEATURE BRANCHING?


42

CHALLENGES


How big should a feature be?

Environment costs money + time

Let’s limit environments per team (2-3)

FEATURE SIZE MATTERS...


We needed to mock 3rd parties

Legacy sales systems

Sms/email/payment providers

It’s good:

you develop independently

Ultimate responsibility for environment => team

it’s testable

But: the mock needs to be developed and kept up to date

ISOLATION OF DEVELOPMENT (MOCKS)


Environment (VM, storage…) creation with one click

No image, but install scripts

Database/storage snapshot

Build

Deployment

Testing

AUTOMATION (SELF SERVICE)


About Oriflame

Complexity we face


were growing


flexibility back?

Take away

WHERE ARE WE NOW?


47

Every change come with people

Communicate

Automate everything

Separate concerns, independent

development

Use feature branches ☺


Jan Vilímek

[email protected]@VilimekJan

48

Thank you for your attention!

Questions?

49

Documents

How we sped up delivery using feature branchesagiliaconference.com/engine/wp-content/uploads/2018/02/2018-AC... · How we sped up delivery using feature branches Agilia Conference