Scaling R&D Organization While Maintaining Quality

Aviran MordoHead Of Back-End Engineering @ Wix

@aviranmhttp://www.linkedin.com/in/aviran

http://www.aviransplace.com

Wix In Numbers

• 44M users from 190 countries

• Static storage is >800TB of data

• 3 Data centers + 2 Clouds (Google, Amazon)

• 700M HTTP requests/day

• ~600 people work at Wix

• Of which ~ 200 in R&D

Double The People Every Year

From 15 to 200 in 3 yearsPlanned growth 400 -> 800 in the next 2 years

Jan 2014

1250Deployments (production changes) per month

Double the velocity from last year

Structures for Scalability

• There are 2 key common structures in the industry

• Functional unit model• Business unit model

Functional Model

• Disadvantages• Lack of product ownership • Lack of product level expertise • Hard to predict and plan product

roadmap• Cross-function communication is

hard• Less focus on delivery and TTM

Business Unit Model

• Disadvantages• Product integration is hard• Resource and work duplication• Architecture alignment is hard• Technology knowledge sharing is

hard• Limited opportunity for

professional development

• There is no perfect model

• It depends on the company’s current challenges, life cycle phase and culture

• Every model should be tuned constantly and evolve with the company

Our Assumptions

• Guild – a group of people that share expertise, knowledge, tools, code, and practices

• Gang – a group of people that work in a related products• Compose of all required

resources from the different disciplines (Guild)

Wix’s Gangs & Guilds Model – How?

Guild

Gang

Guild

Gang Gang

Wix’s Gangs & Guilds Model – Why?• Technical power of the Guild• Independence of the Gang• Healthy balance between product and tech• Product features and technical equal in priority

Guild LeaderWhat

How

Guild Leader

How

Team member

Team member

How Does It Work

Dev-Centric Culture

Traditional Dev Pipeline

Product Dev QA Operations

23:50

Product Dev QA Operations

23:50

What Is The Common Denominator?

• Product manager• Project manager• QA• Operations• DBA

Developers can do these jobs

Dev Centric Culture – Involve The Developer – How?

• Product definition (with product) • Development (with architect)• Testing (with QA developers)• Deployment / Rollback(with operations) • Monitoring / BI (with BI team)• DevOps – to enable deployment and

rollback, fully automated

Developer

Product

QA

ManagementOperation

BI

Because It Makes You ROQ

•Responsibility•Ownership•Quality

Dev Centric Culture – Why?

Project Rotation – How?

• Team usually has more than one project• Team members rotate tasks between projects• Developers can switch teams and Gangs

Project Rotation – Why?

• There is no single person with a knowledge on a specific component• Ongoing review• Keep it interesting• It is everyone's responsibility• Everybody owns everything

Quality Thursday – How?• A software engineer works 4 days with his Gang• Thursday is a Guild day.• Developers stop working on their designated

projects and conduct quality enhancing activities with the Guild.

Quality Thursday – Why?• Assimilates people into the Guild• Builds cross-team relationships• Shares knowledge • Enhances the culture• Promotes innovation

Bug Hunt – How?

• Service owner picks a service running on production• Explains the service to all Guild members• What the service does• Every exception thrown in production• Performance matrix

• Get a list of AI from group feedback (sometimes for dependent services)

Bug Hunt - Why?

• Teach developers about services they don’t own• Understand how your application behaves on

production• Open discussion and ideas from group members• Discover unexpected use patterns• Find bugs on production

Retrospective – How ?

• Whenever developers encounter issues, they are encouraged to post them on a board for public discussion• Anything can be discussed• Topics posted on the board during the week

constitute the agenda of the retrospective

Retrospective – Why ?

• Solve problems • Share lesson learned.• Suggestions on how to improve:• Our team• Quality of products• Process • Effectiveness of the R&D organization.

Tech Talks – How?

• Every developer can give a talk about anything• People from other departments in the company• Guest speakers• Open to anyone from Wix• Using Meetup http://www.meetup.com/at-wix/ to

invite outsiders to our internal talks (if appropriate)• Talks videos http://goo.gl/IDqXTi

Tech Talks – Why ?

• Training new and old employees• Educating about other activities in the R&D• Educating about other departments activities and

concerns in the company

Thursday Tasks – How ?

• Pay technical debt (preferably not your own)• Work on a task / feature NOT in your project• Read and learn something new• Open source a project• 1 on 1 mentoring

Thursday Tasks – Why?

• Learn other projects (easier to step in if necessary)• Unbiased review of other projects• Learn about problems and solutions other teams

faced and solved that you may also encounter.• Find repeating patterns across projects and

generalize a solution• Find interesting projects to switch to.• Leave the code in a better shape than they got it

Test Driven Development – How?

• No new code is pushed to Git without being fully tested• We currently have around 10,000 automated tests

• Before fixing a bug first write a test to reproduce the bug

• Cover legacy (untested) systems with Integration tests

23:50

Test Driven Development – Why?

• Developers are responsible for their own code

• Less bugs

• Easier to enter someone else’s project

It Is All About The People

Do Not Compromise On Hiring

• Hire only good people• Fit the culture• Excellent technically

• Candidates can be dropped• By anyone• At any time

• If there is any doubt, then there is no doubt

Quality Trust Independence Growth

The Path To Growth

Independence

Dedicated Resources = Organization Structure

• Have dedicated teams for each product group• Avoid product priority collisions• A team should have all the resources it needs

Minimize Architectural Dependencies

• Service oriented architecture (SOA)• Separation of client (UI) and server projects• Dedicated data stores for each service• Stateless services

Lesson learned: System architecture that decouples concerns

Communication Channels• To company wide activities• To knowledge centers• To key personnel

Quality Trust Independence Growth

Lessons Learned: Growing Teams

Seeds Of Ambassadors

• Train a group of ambassadors that practice dev-centric culture from the Guild• Build new teams with at least one dev-centric

ambassador to assimilate new teams into the culture.• Beware of hiring more people than you can train /

assimilate successfully

Quality

TrustIndependence

Growth

It’s Not a Path, It’s a Cycle

Aviran MordoHead of Back-End Engineering @ Wix

@aviranm

linkedin.com/in/aviran

aviransplace.com

Q&A

Read some more about maintaining quality R&D: http://goo.gl/c3WLsz