Upload
optimizely
View
181
Download
0
Embed Size (px)
Citation preview
Introducing OptimizelyProgram Management
Claire Vo, VP of Product, OptimizelyHeather Miller, Senior Experimentation Analyst, BBC
SCALE YOUR PROGRAM AND 5X YOUR EXPERIMENT VELOCITY
“Our success is a function of how many experiments
we do per year, per month, per week, per day.”
“Instead of saying ‘I have an idea,’ what if you said
‘I have a new hypothesis, let’s go test it.’
“Our company culture encourages
experimentation and free flow of ideas.”
3
“One of the things I’m most proud of, and I think
what is the key to our success, is this testing
framework we’ve built.”
Experimentation is the Next Great Business Transformation
Jeff Bezos Larry Page
Mark Zuckerberg Satya Nadella
The Surprising Power of Online Experiments
Digital Experience Optimization:Digital Products, Commerce & Campaigns
Up to 5X Increase in Yield:Revenue, Share of wallet, Funnel conversion, Risk mitigation, Ops efficiency
Partner of Choice:Work with leading global enterprises & “digital disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:Next gen “Test and Learn” System
Replaces Digital Guesswork: with Evidence-based optimization
Speeds Innovation & Optimization:Single platform for marketing & product teamsBest in class stats & machine learningConsumer grade usability Enterprise program management & prof. services
2
OUR SOLUTION
PrioritizationIdeation can be siloed leading to inefficient prioritization and execution
Oversight & knowledge sharingLack of executive oversight can lead to serious problems in a global organization
CoordinationAs you scale, coordinating across additional teams and stakeholders becomes more and more difficult
ResourcesTime, people, and effort is wasted duplicating and reinventing different processes
Challenges of Building a Program
IDEATION & PRIORITIZATION
TESTING & IMPLEMENTATION
REQUIREMENTS GATHERING & DEVELOPMENT
REPORTING & ANALYSIS
DOCUMENTATION & IP PRESERVATION
Does this reporting match internal data?
What are we going to test next?
Can someone code this test for me?
Manage your program across the entire experiment lifecycle
Teams Web & Full Stack Experimentation
Program Reporting
OPTIMIZELY PROGRAM MANAGEMENT
IDEATION REQUIREMENTS CREATIVE DEVELOPMENT TESTING ANALYSIS SHARING
Democratize ideation and empower your entire team to participate
Make the experimentation process more accessible and transparent
Improve your program workflow with new tools for collaboration
TeamsCollaborate more effectively with an integrated hub for your team
Track the velocity of your program
Gain more oversight across projects and teams
Uncover new insights and iterate on your best ideas
Program ReportingGain powerful insights into your program with a new executive dashboard
Program Management at the BBC – with Optimizely
Heather MillerSenior Experimentation and Optimisation Analyst17 January 2018
•The BBC is the oldest and largest broadcaster in the world
•Founded in 1922, we employ 35,402 staff in total across the world
•Our primary audience is the UK - but some of our work reaches over 200 countries
•We deliver a wide range of services, from news, radio, video, music and online content
•The BBC is considered to be of ‘significant cultural importance’ to the UK
“To enrich people’s lives with programmes and services that inform, educate and entertain.”
- John Reith, Founder of the BBC
Experimentation at the BBC: our team•The Experimentation and Optimisation team sits within Data
Insights
•Our team operates as a Centre of Excellence
•There are five us in the Experimentation team, who leverage our expertise to advise and consult with teams across the BBC
•Our team assists with the entire end-to-end process - from strategy, to test design, to analysis, to final reports
•We want to upskill teams and make them self-sufficient in experimentation best practice
•Right now, 13 teams use Optimizely across the whole of the BBC
•Our ultimate goal is to have 40 teams using the tool
•On any given day dozens of people use Optimizely
•We strive to deliver continuous experimentation across web, mobile and connected devices
•Our team maintains a close working relationship with Optimizely to maximise productivity and deliver the best results
● The BBC reaches millions
● We strive to deliver digital excellence
● Experimentation plays a vital role
● Ultimately saves time, money and mitigates risk
● We have run around 200 experiments with
Optimizely
Why does testing matter to the BBC?
How we generate ideas and prioritise tests
● Test ideas are generated and prioritised at product level
● Qualitative & Quantitative: data insights, usability work, internal ideas and previous tests
● Tests are prioritised based on wider BBC Objectives, product roadmaps and the PIE Model
Why knowledge sharing is important to us● Sharing insights plays a key role in upscaling our
testing programme to new teams
● We deliver a monthly newsletter to 300 subscribers and counting
● Confluence knowledge hub with historical result reports helping teams make informed decisions
● Upskilling teams via our bespoke training supports our adoption of a consistent approach to experimenting
● Host regular internal showcases, engineering meet ups and workshops
The challenges we face
As we grow our programme across more teams and touch points we face some challenges
• Multiple tools are used to document ideas and information
• Different teams, different preferences leading to multiple places with similar information
• Ability to measure maturity, and track experiment end-to-end velocity as a BBC overview and product level
• A need for splitting out BBC tests by type to identify expansion into other forms of testing
• Demonstrating and visualising test scale across the BBC
How Program Management
will help the BBC
As we continue to increase our pace of innovation, we are excited about what Program Management has to offer.
• Consolidate multiple tools into a single slick environment
• One stop shop for past results which are easily found
• Improves collaboration and encourages transparency, allowing anyone within a product to add ideas
• Drives efficiency of our end-to-end workflow
• Offers progress reporting at BBC level and product level
• Supports our goal to scale Optimizely to more teams