ConversionSummit 2015 - Testing culture: making testing look easy!?

Testing culture: making testing look easy!?Martijn Scheybeler / Lead Growth, SEO & Analytics / [email protected] / @MartijnSch

EASY, WHAT?

• Leads Marketing: Growth, SEO & Analytics

• Marketing team The Next Web: 12 (60+)

• Into testing for the last 3 years

• Weekly running 10 new tests, we can do 15-20 :(

• Scaling, coding and automating

ConversionSUMMIT 2015 - @MartijnSch

Who is this guy?

What is The Next Web?


7 million unique users, 13 million pageviews

TNW Conference: Amsterdam & New York: 20.000 attendees

TNW Deals: The online marketplace for digital tastemakers

TNW Academy: Hand-picked online courses to improve your professional life


• From no testing to …? in 6 months

• Setting up a testing culture from scratch

• Scaled from: 1 test a month > 10 a week • 10 NEW tests as week, the ability to do: 15

What qualifies me to speak here?


• How to create a culture of testing?

• How were / are we able to scale so fast?

• How it influences our (company / team) culture?

• Summary: How do YOU scale your testing program?

What do I like to talk about…


• Team structure & dynamics

• Creating experiment ideas + planning

• Creating & coding experiments

• Analysing experiments

• What’s next?

… and it goes something like this


• Design, Development, QA, Project Management, Legal > CEO approval: you’re doing it wrong!

• Can you design, code & analyse? You don’t need a team.

• Cross-functional or not?

Team structure


TNW Style: Team structure


• 1 Web Analyst: Designing, Coding & Analysis

• 2 Interns: Analysis + Coding

• Me: Strategy, Coding, Analysis

• Support: 1 designer + 1 front-end developer

Not cross functional (at the moment)

TNW Style: Team structure


• “Our developers won’t let us change the code”• “I need to run our tests against the legal and *BS* team”• “We’re still waiting for the design reviews”• “We don’t think A/B testing can improve our results”• “I asked my girlfriend to test the site, she didn’t have issues”

Team dynamics

What does our culture looks like?


• “This is how you access our live servers, need GTM as well?”

• “Legal, who?”

• “We still have 5 design reviews laying around, DAMMIT!?”

• “10 new tests this week! YOLO!”

TNW Style: Team dynamics


• A lot of ideas, but no REAL ideas: button colours?

• =AVG(TIME*0,272+EFFORT*0.123+HAIRCOLOUR*0,605)

• A planning heavily discussed with the …

Experiment ideas + planning

Make sure you don’t have a HIPPO in the room

Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”


• Ok, we do have a planning

• “It doesn’t need to be rocket science”

• We don’t have a huge backlog

TNW Style: Planning, Planwhat?


• Design by committee, according to style guides

• Won’t get you a lot further

• Outsourced testing? In-house teams (with no time)?

Creating & coding experiments


• Designer: “I created a new design, we should test it!” • Developer: “I looked at your A/B testing script: refactored it” • Analytics: “We have a backlog but we could use new ideas” • Me: “Can we do 20 tests a week?”

TNW Style: Creating experiments

Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs


• Designed by our CRO team

• Working with our design team to test very big things

• Responsible: no way of hiding behind bad code

TNW Style: Creating experiments


• Google Tag Manager

• A/B Testing & Multivariate Testing: http://tnw.to/testingJS

• Custom JavaScript / jQuery for changes

• Supported by the dataLayer for targeting / segmenting

• You could do better: back-end testing!

TNW Style: Coding experiments

http://bit.ly/testingJS


• Google Analytics? Adobe Analytics?

• You do make sure to check for power + significance, right?

• You’re also tracking secondary KPIs?

• You decide to ‘re-test’?

Analysing experiments

Automated reporting via TNW Analytics: -20%


• Google Analytics API

• Custom Dimensions / Event Tracking

• Automated reporting

• Ultimate pleasure and fun + no annoyed team members

TNW Style: Analysing experiments

TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%


• Take it easy… do you know what to do?

• First: make friends!

• Second: make sure you know what you’re doing!

• Third: map out your strategy, do you have the resources?

• Worst case? Read back the slides!

What’s next for you?


• Grow our team: 15 people?

• Cross functional, a mistake?

• Automate & Innovate even more

• Drink more …

TNW Style: What’s next?


• Create / Embrace a culture of testing: don’t create silos• Make the whole team + organisation come up with ideas• Design ideas yourself or ask your designer(s)• Learn to code so you can create experiments yourself• Want to scale? Make sure you have the resources to automate

Take aways


Martijn Scheybeler Lead Growth, SEO & Analytics - TNW

Email: [email protected]: @MartijnSch

Have fun! Thank you!Slides: http://tnw.to/csummit Code: http://tnw.to/testingJS

mailto:[email protected]

http://tnw.to/csummit

http://tnw.to/testing


Data & Analytics

ConversionSummit 2015 - Testing culture: making testing look easy!?