TEST ROADMAPSHow to win sales and influence success
Presenter
Presentation Notes
1. MVT AND AB TESTING: ROADMAP FOR SUCCESS
Hi I am Tim Stewart from trsdigital
You might know starring roles such as: Commercial Manager for the largest English language motorcycle site in the worldConsultant with the highest success rate, longest emails AND undefeated go-kart champion during Maxymiser’s highest growth yearsWeb Optimization Consultant for SiteSpect with client’s such as well, er Connor you just met, Confused.com, Hotels.com and Guess Jeans EUSome may simply know me as that bloke who talked for 5 hours about Excel on the #MeasureFest training course yesterdayThe internet mostly knows me as @pjeedai on most social networksThe p stands for Padawan as I am always learning
INTRODUCTION
@pjeedai
Presenter
Presentation Notes
2. INTRODUCTION� Hi I am Tim Stewart from trsdigital��You might know starring roles such as: Commercial Manager for the largest English language motorcycle site in the world Consultant with the highest success rate, longest emails AND undefeated go-kart champion during Maxymiser’s highest growth years Web Optimization Consultant for SiteSpect with client’s such as well, er Connor you just met, Confused.com, Hotels.com and Guess Jeans EU�Some may simply know me as that bloke who talked for 5 hours about Excel on the #MeasureFest training course yesterday The internet mostly knows me as @pjeedai on most social networks The p stands for Padawan as I am always learning
Effective testing starts with the boarding processVendor or Agency bringing on a client, an Internal team rolling out a new initiative or flying soloIts not as simple as adding the tool and taking offYou can’t just go up the ramp, turn left to First Class and wait for the complementary peanuts
BOARDING
Presenter
Presentation Notes
3. BOARDING Effective testing starts with the boarding process Vendor or Agency bringing on a client, an Internal team rolling out a new initiative or flying solo Its not as simple as adding the tool and taking off�You can’t just go up the ramp, turn left to First Class and wait for the complementary peanuts
The energy when people talk about CRO can be breathtakingIt may feel like Boarding is simply a question of hanging on and letting the wave push youThere’s nothing more to do except enjoy the ride.
HIGH ENERGY BOARDING
Presenter
Presentation Notes
4. HIGH ENERGY BOARDING� The energy when people talk about CRO can be breathtaking It may feel like Boarding is simply a question of hanging on and letting the wave push you There’s nothing more to do except enjoy the ride.
But some stakeholders don’t have as much energy as othersYou need to understand the level of buy-in and competence of all the stakeholdersIt might mean your plans to roll out your testing program have to be adapted so they don’t overwhelm some parties.
LOWER ENERGY BOARDING
Presenter
Presentation Notes
5. LOW ENERGY BOARDING� But some stakeholders don’t have as much energy as others You need to understand the level of buy-in and competence of all the stakeholders�It might mean your plans to roll out your testing program have to be adapted so they don’t overwhelm some parties.
And some are just along for the ride
MINIMAL ENGAGEMENT BOARDING
Presenter
Presentation Notes
6. MINIMAL ENGAGEMENT BOARDING� And some are just along for the ride
BOARDING
For initial boarding you should establish:• Who needs to be involved in the process• What responsibilities each person has• How you will roll out your testing plan
Otherwise your life-changing trip might end with a bang… and even worse a Celine Dion song
Presenter
Presentation Notes
7. AGREE A PROCESS OR FAIL BEFORE YOU START� Having an agreed process for Boarding is recommended – even a simple one Identifying responsibilities and requirements is important
DISCOVERY
Arguably the most important part of any roadmap processYou find out what you are working withAnd what you need to work around.
Presenter
Presentation Notes
8. DISCOVERY PHASE Arguably the most important part of any roadmap process You find out what you are working with And what you need to work around.
DISCOVERY: UNDERSTAND YOUR LAUNCH CONDITIONTry understand the inertia you have to overcome
What is currently weighing your business down?• How much weight do you need to shift?
What is the shape of the business?• Where is money being made?• Which objectives have highest importance• What outcomes have the most volumes?• Which generate the most revenue?
Can you quantify this?• How do web metrics report these values?• How closely do web metrics relate to outcomes?
Presenter
Presentation Notes
9. DISCOVERY IS ABOUT THE BUSINESS
DISCOVERY: NOT JUST NUMBERS, ALSO RESOURCE
Before looking at any website issues
Be clear on the following:• What the business needs for take-off• What needs to be handled to achieve this
Who looks after this?
Who needs to be involved?
What is required in terms of: • Development resources• Design resources• Sign-offs
Presenter
Presentation Notes
10. THE BUSINESS IS NOT JUST THE NUMBERS BUT THE RESOURCE What is the holistic environment you need to consider? It will vary for every company, there is no single answer You need to know: The congestion Road conditions If the vehicle is suitable If you are fully fuelled for the journey All your passengers are on board Are you ready to launch?
DISCOVERY: TRAJECTORY IS IMPORTANTTesting is often pitched as a way to• Bypass IT • Fast track past release schedules. It can be used for this. But care is needed.
Too much pushToo little direction Will mean a short, sharp trip A spectacular end to your plans
Presenter
Presentation Notes
11. POOR DISCOVERY LEADS TO DISASTER� How fast your plan can take off depends very much on What you have to achieve that On what velocity you need to make a difference If you have enough push to overcome resistance Establish the people, the potential, the limitations�This is the point where these things can be investigated and mitigated Making every following step easier to achieve.
Binary choices: • Leave a page or move forward• Add or Not Add• Checkout Now or Continue Shopping• Click to Read More• Sign up for our mailing list
Where do the best opportunities lie?• What makes up the total?• Where are the biggest holes? • Where is the most money spent on traffic?
Where is the signal loudest?• Like - Will continue• Don’t Like - Will go no further?
Are there channel or device differences?
DATA
Presenter
Presentation Notes
12. DATA DRIVEN DECISIONS Don’t just start with the headline site conversion rate and pick a desired improvement Look at how that rate changes over a year, over a week, over channels, over devices. Your tests will be trying to move the needle enough to be measureable You need to understand what a typical variation in conversion looks like You need to be clear on what buying patterns and buying cycles to test against You can’t affect the whole site conversion in one test. You can…(it is possible, I’ve done it) but, really you can’t (rely on that) What you can do is shift the needle enough on one step of the buying process in one lucky test that it makes a big enough difference to the whole But those opportunities are rare. Usually those holes were big enough to be spotted and if not fixed then improved already. ��So you need Data. Analytics on what makes up those steps towards a sale or macro conversion You need to know how much you have to measure against. Understand what you think is the current benchmark How much signal? How much noise? How much money is being spent in areas that underperform? How much money is being made in areas that perform OK but you would like more? What numbers do you have to show where the worst friction happens for a user Is it the same for all users? Return, new, frequent flyers, researchers, serial purchasers, one-off interest? Do you see different choke points for users on different devices?
Data about how those choices happen• Click patterns• Heat maps• User Journey• Detailed step conversion tracking
Data about users and their motivation• Never will Buys• Might Buys• Could be Persuadeds• Plan to Buy• Will Buy no matter what
Do you have customer feedback? • User testing• Customer survey• Live Chat sessions• Personas
SERIOUS DATA
Presenter
Presentation Notes
13. DATA ABOUT HOW THOSE NUMBERS COME TO HAPPEN� What is the click pattern, heat map, user journey for Never will Buys, Might Buys, Could be Persuadeds, Plan to Buy, Will Buy no matter what Do you have analytics on micro events? Heatmaps, user testing, customer survey, Live Chat sessions, Personas? You might have none, some or all of this and more But move away from only money or goal focused KPIs Look at the behaviours that correlate with Buying and Nearly Buying and Not Buying Anything that will help understand how those holes in the conversion process are created You look at buyers to see what they like But also look at the users who didn’t buy, how is their behaviour different? Can you correct that without annoying the buyers?
Quantitative and qualitative• Cohort and behaviour profiles• NPS• Customer Buying Patterns• Typical Basket composition• CRM and userID• BI: Margin per product, AOV, LTV, RPU
Can you stitch it together?• Get an idea of themes• Areas to explore• Key templates• Decision points?• Customer Persona• Non-Customer Persona
It is a term that is often misusedIts not Big Data in terms of machine learning
REALLY SERIOUS DATA
Presenter
Presentation Notes
14. QUANTITATIVE AND QUALITATIVE – HOW DOES QUALITY RELATE TO ACTIONS? Can you stitch it together to get an idea of themes, areas to explore, key templates or decision points? �Its not really Big Data in terms of machine learning. Its several sets of small to medium data�But it’s a lot more data to look at than just Headline site conversion rate A typically optimistic forecast of 20% YoY, regardless of circumstance� Because you don’t just want to look at pages you and the board target and assume a quick set of best practice tests will work You need to look at what causes the user to not take the actions you want. And you need to look at all the potential friction points that show the user is not reacting the way you intend
But it is a hopefully large collection of data sets and useful intelligenceAnd importantly it is not a fixed set of Data
It will be added to over time• Through tests results• Through additional research• Through improved analyticsOnce your testing shows more detail around what drivers user choices you SHOULD be tweaking your measurement on site to get a clearer picture
And obviously this new data should be fed back into the process and used to create and refine testing ideas and roadmap areas.
NOT REALLY “BIG DATA” (BUT IT MIGHT BE LOTS)
(This is actually Lore)
Presenter
Presentation Notes
15. IT IS NOT BIG DATA But it is a hopefully large collection of data sets and useful intelligence And importantly it is not a fixed set of Data It will be added to over time Through tests results Through additional research Through improved analytics Once your testing shows more detail around what drivers user choices you SHOULD be tweaking your measurement on site to get a clearer picture And obviously this new data should be fed back into the process and used to create and refine testing ideas and roadmap areas.
It can be overwhelming
But this is why Boarding and Discovery play such an important role. You aren’t on your own in this. You brought in the Business Intelligence team, the Department stakeholders, the Board, the HiPPOs
When faced with a mountain of Data to look at it’s generally best to look for what makes the biggest difference to the companyAnd look at the customer and analytics data to see what makes the biggest difference for the users.
LOTS OF DATA CAN OVERWHELMING
Presenter
Presentation Notes
16. LOTS OF DATA; WHERE DO YOU EVEN START? It can be overwhelming But this is why Boarding and Discovery play such an important role. You aren’t on your own in this. You brought in the Business Intelligence team, the Department stakeholders, the Board, the HiPPOs When faced with a mountain of Data to look at it’s generally best to look for what makes the biggest difference to the company And look at the customer and analytics data to see what makes the biggest difference for the users.
So if your stakeholders and customers have given you some clues… Start there.
Be methodical. Drill down and work outwards to eventually the periphery of “other interesting stuff” when you come to focus on an area. Initially at least you are looking for a few places to start and some holes which need investigating
INVESTIGATE YOUR DATA TO FIND SOME CLUES
Presenter
Presentation Notes
17. START WITH CLUES FROM THE BUSINESS AND LOUDEST SIGNALS FROM THE USERS So if your stakeholders and customers have given you some clues… Start there. Be methodical. Drill down and work outwards to eventually the periphery of “other interesting stuff” when you come to focus on an area. Initially at least you are looking for a few places to start and some holes which need investigating
At this point you haven’t looked in any depth at any specific pages or processesYou’ve looked at the business, the people in the business, the audience you can work with.
IDEAS
So you’ve established
Presenter
Presentation Notes
18. TIME TO CHECK SEE WHAT YOU HAVE At this point you haven’t looked in any depth at any specific pages or processes You’ve looked at the business, the people in the business, the audience you can work with.
Who you need involved from the businessWhat level they need to be involvedThe best way to communicate the strategy and the results
IDEAS: WHO IS BEING INCLUDED?
Stakeholders
Presenter
Presentation Notes
19. STAKEHOLDERS Who you need involved from the business What level they need to be involved The best way to communicate the strategy and the results
What they need for the businessWhat they consider to be the critical results to meet the business strategyDo they have a timeline in mind?Is it reasonable?
Fundamentally what you need to achieve from testing to ensure its continued support and growth. HiPPOs are not stupid. They are looking out for the best for the business.
If you can show value and align your testing program with what the business needs to grow and succeed then HiPPOs will back youWhich is how they became the Highest Paid People. And delivering a testing roadmap that delivers success for the business will make you more Highly Paid too
IDEAS: THE HIPPO’S IDEAS DO MATTER
The HiPPO
Presenter
Presentation Notes
20. THE HIPPO What they need for the business What they consider to be the critical results to meet the business strategy Do they have a timeline in mind? Is it reasonable? Fundamentally what you need to achieve from testing to ensure its continued support and growth. HiPPOs are not stupid. They are looking out for the best for the business. If you can show value and align your testing program with what the business needs to grow and succeed then HiPPOs will back you Which is how they became the Highest Paid People. And delivering a testing roadmap that delivers success for the business will make you more Highly Paid too
The Audience is your sample. It is what you have to test againstIt is their behaviour that you want to influence. • How much you can test?• How it can be segmented?What is the composition?New, Returning, Regular Purchasers, Irregular, Distress Purchase, Subscription.They all have different motivations and will react differentlyWhat is the shape?Frequency, Recency, Device preference Are users doing the same thing on every visit?Do different devices get used for different purposes?Are there obvious places where mobile outperforms or underperforms Desktop? WHY?
IDEAS: WHO AND HOW MANY TO TEST AGAINST
Audience
Presenter
Presentation Notes
21. AUDIENCE The Audience is your sample. It is what you have to test against It is their behaviour that you want to influence. How much you can test? How it can be segmented? What is the composition? New, Returning, Regular Purchasers, Irregular, Distress Purchase, Subscription. They all have different motivations and will react differently What is the shape? Frequency, Recency, Device preference Are users doing the same thing on every visit? Do different devices get used for different purposes? Are there obvious places where mobile outperforms or underperforms Desktop? WHY?
How many of these have audience?Sometimes area of great internal debate have much lower engagement or user volumes to test against.
Sometimes you want to measure increase to Final SaleBut there simply aren’t enough volumes Or the Final Sale is 4 visits, 2 devices and 18 pages after the area you want to test. So is Final Sale the most closely related metric to what you are changing?
You will, eventually, influence it. You may want to track it every test for a benchmarkBut unless you are testing the page directly before and have large volumes of commonly distributed purchases every day.. It will limit your testing abilityIt should not be your sole focus or your primary decision metric on every test. If you focus on just the cash register but don’t stock the shelves, promote the offers, price match competitors, see how users navigate the aisles…Your Cash Register is going to remain too quiet and will provide you very little intelligence on why.
IDEAS: WHERE AND WHAT NEEDS FOCUS
Places
Presenter
Presentation Notes
22. PROBLEM PAGES, PROBLEM PROCESSES How many of these have audience? Sometimes area of great internal debate have much lower engagement or user volumes to test against. Sometimes you want to measure increase to Final Sale But there simply aren’t enough volumes Or the Final Sale is 4 visits, 2 devices and 18 pages after the area you want to test. So is Final Sale the most closely related metric to what you are changing? You will, eventually, influence it. You may want to track it every test for a benchmark But unless you are testing the page directly before and have large volumes of commonly distributed purchases every day.. It will limit your testing ability It should not be your sole focus or your primary decision metric on every test. If you focus on just the cash register but don’t stock the shelves, promote the offers, price match competitors, see how users navigate the aisles… Your Cash Register is going to remain too quiet and will provide you very little intelligence on why.
The potential for Win in those places and with the Audience identified means you can start to think on prioritiesYou also start to understand the size of the challenge you have to convince the Audience to change their behaviourYou will NOT go from 0.3% conversion rate to 3% easily
An idea of the value? It won’t be perfect but you need an idea of the value, in revenue, happy customers, business objectives (acquisition, AoV, RPU etc)Possibly from the data alone you will already have a clear idea on some of the questions you need to answer and options you can test
Which of these are traffic wins? (more people into a key area)Which of these are business wins? (more sales, greater value, the “right” product being seen/chosen) Do these align or are they always separate tracks?
IDEAS: WHICH ARE BIG ENOUGH TO MATTER
Scale of Opportunity & Challenges
Presenter
Presentation Notes
23. THE SIZE OF THE HOLES, THE VALUE OF THE POTENTIAL WINS. The potential for Win in those places and with the Audience identified means you can start to think on priorities You also start to understand the size of the challenge you have to convince the Audience to change their behaviour You will NOT go from 0.3% conversion rate to 3% easily An idea of the value? It won’t be perfect but you need an idea of the value, in revenue, happy customers, business objectives (acquisition, AoV, RPU etc) Possibly from the data alone you will already have a clear idea on some of the questions you need to answer and options you can test Which of these are traffic wins? (more people into a key area) Which of these are business wins? (more sales, greater value, the “right” product being seen/chosen) Do these align or are they always separate tracks?
What are the targets you have?What budgets? What current actuals?
Want != NeedI want more sales now, for less spendTranslates toI need to find a way to increase conversion from the same audienceI need to identify where this is being impacted and reduce thisThis NEEDS time, experimentation and commitment.
If one test gave all the answers, halved the CPA .. Then there wouldn’t be a need for a Roadmap.
IDEAS: WHAT TARGETS NEED TO BE MET
What the Business wantsHopefully what the business actually needs (these two rarely align)
Presenter
Presentation Notes
24. WHAT THE BUSINESS WANTS, WHAT THE BUSINESS NEEDS What are the targets you have? What budgets? What current actuals? Want != Need I want more sales now, for less spend Translates to I need to find a way to increase conversion from the same audience I need to identify where this is being impacted and reduce this This NEEDS time, experimentation and commitment. If one test gave all the answers, halved the CPA .. Then there wouldn’t be a need for a Roadmap.
Look at the pages with fresh eyes.
Think – What do WE want here? Think – What does the CUSTOMER want here? Look – What does the Customer ACTUALLY do?
Why is this different to what we want/intend?(it will be, customers are usually a LOT less keen to give you all their money immediately)
IDEAS: THINK LIKE A CUSTOMER
What the Customer DOESN’T want
Presenter
Presentation Notes
25. UNDERSTAND WHAT USERS DON’T WANT Look at the pages with fresh eyes. Think – What do WE want here? Think – What does the CUSTOMER want here? Look – What does the Customer ACTUALLY do? �Why is this different to what we want/intend? (it will be, customers are usually a LOT less keen to give you all their money immediately)
Go to the places your data has shown need attentionGo to the places the business and stakeholders want to improve
TRY to break the user journey. Screen grab every error, every misstep, every button/link/option that is NOT move forwardLook for friction, miscommunication, ways where Where Next is not clearly answered
Blur your eyes, see if you can still see a CTATry different screen sizes, is that important information above fold (spoiler: it still matters, even if you can encourage people to scroll)The UX book is called Don’t Make Me Think. Do you? Do you make the customer have to *shudder* think? And Read?Can they do it in < 3 seconds from landing?
Then think about how you can change that, some alternative ways to handle those choice points
IDEAS: GET CREATIVE
Presenter
Presentation Notes
26. BE CREATIVE: LOOK AT THE SITE, GET IDEAS Go to the places your data has shown need attention Go to the places the business and stakeholders want to improve TRY to break the user journey. Screen grab every error, every misstep, every button/link/option that is NOT move forward�Look for friction, miscommunication, ways where Where Next is not clearly answered Blur your eyes, see if you can still see a CTA Try different screen sizes, is that important information above fold (spoiler: it still matters, even if you can encourage people to scroll) The UX book is called Don’t Make Me Think. Do you? Do you make the customer have to *shudder* think? And Read? Can they do it in < 3 seconds from landing?��Then think about how you can change that, some alternative ways to handle those choice points
And you will end up with ideas, lots and lots of ideas. It’s probably a good idea to write them down somewhere.
LOTS AND LOTS AND LOTS OF IDEAS
Presenter
Presentation Notes
27. THERE WILL BE LOTS OF IDEAS And you will end up with ideas, lots and lots of ideas. It’s probably a good idea to write them down somewhere.
You’ve sat down• Got the business objective(s)• Got the web stats which relate to that objective• Got a budget• Got a timeline• Got lots and lots and lots of ideasJust a question of pointing it in the right direction and heading to Success-ville?
A ROADMAP TO SUCCESS
Presenter
Presentation Notes
28. SO THAT IS ALL YOU NEED, RIGHT? You’ve sat down Got the business objective(s) Got the web stats which relate to that objective Got a budget Got a timeline Got lots and lots and lots of ideas Just a question of pointing it in the right direction and heading to Success-ville?
If that is the conclusion of your boarding and discovery – Get Ideas, Test Ideas, Epic Win!
I have news for you. You Shipment of Fail has arrived.
Because you may have created the biggest shopping list of issues ever and “testing will fix that”But its at this point a lot of businesses choke. And I bet at this stage some of your boarders have run out of energy and you won’t have the traffic or resource to test everything.
EXPECTATION VS. DELIVERY
Presenter
Presentation Notes
29. BAD NEWS – EXPECTATIONS MAY NOT MATCH REALITY� If that is the conclusion of your boarding and discovery – Get Ideas, Test Ideas, Epic Win! I have news for you. You Shipment of Fail has arrived. Because you may have created the biggest shopping list of issues ever and “testing will fix that” But its at this point a lot of businesses choke. And I bet at this stage some of your boarders have run out of energy and you won’t have the traffic or resource to test everything.
Too many, dare I say it, Ideas
Because we can’t test Ideas. Ideas are formed into hypotheses. Hypotheses are formed into tests.
Because the road is never that smooth. The choke points of resource and traffic are always too small. If your boarding and discovery showed that you have everything you need, no impedance, no need for alternatives or expectation that you will deviateYou are going to have a bad time
EXPECTATION VS. DELIVERY
Presenter
Presentation Notes
30. TOO MUCH DATA, NOT ENOUGH FOCUS ON DELIVERY Too many, dare I say it, Ideas Because we can’t test Ideas. Ideas are formed into hypotheses. Hypotheses are formed into tests. Because the road is never that smooth. The choke points of resource and traffic are always too small. If your boarding and discovery showed that you have everything you need, no impedance, no need for alternatives or expectation that you will deviate You are going to have a bad time
Set a destination that looks more achievable for your circumstancesPlan a more realistic way to get there with the resources you have
PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS?
Presenter
Presentation Notes
31. SUCCESS REMAINS THE GOAL BUT CONSIDER A MORE REALISTIC APPROACH Set a destination that looks more achievable for your circumstances Plan a more realistic way to get there with the resources you have
PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS?
Now we all work in digital. This isn’t anything new, or at least it shouldn’t be. The Board & hungry HiPPOs may think this is what they will get – a direct line to Success
The reality is that you won’t fly first class to success even with the best Boarding & Discovery process
Presenter
Presentation Notes
32. A STRAIGHT LINE IS NOT A ROADMAP It summarises your objective, it shows direction of your strategy But it does not account for the tactics needed on the way to adjust for the conditions
PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS?The reality of release schedules, sign off delays, legal & brand policy departments getting involved will be specific to your situation and will always be evolving.
Your tests won’t always work out the way you expect; you might need to test again and check results, stop testing in one area to work in another, account for changes in market or tactical need etc.
It is very likely there will be congestion on the ideal route, diversions, unplanned stop-overs & lost luggage
If you only have a single focus plan, only test when conditions are perfect then you will step your way there. But slowly, through adverse conditions.
You will back step regularly, roadblocks can & will stop you completely unless you have something prepared.
Presenter
Presentation Notes
33. REALITY CHECK A test plan without planning is a very inefficient way to get to your destination
PROCESS TO RELIABLY REACH SUCCESS
Now obviously you don’t know, what you don’t know.
You can’t draw a plan to account for every possible outcome upfront, no matter what Data you have
But you can and should have a few alternative paths to explore that will also get you to your destination
A successful Roadmap will combine multiple areas of investigation, several paths forward to your objectives
The idea is to have a Process that keeps you testing somewhere & keeps momentum in your Roadmap
Presenter
Presentation Notes
34. DEVELOP A PROCESS FOR DEALING WITH ADVERSITY Plan alternative approaches Consider choke points and what tactics could be used
What we need are tests. What a test needs is a reason to exist; a HypothesisYou need to have a think, cogitate on your next move. Think a few moves ahead
COGITATE: PLAN YOUR MOVES
So I said earlier that we don’t really want Ideas. But initially we do need them, we just need to craft them into hypothesesWe need to put them together. Chew them over. Have a think. Make a list
Presenter
Presentation Notes
35. A BIG PILE OF IDEAS WILL CHOKE YOUR ROADMAP What we need are tests. What a test needs is a reason to exist; a Hypothesis You need to have a think, cogitate on your next move. Think a few moves ahead
IDEAS WILL COME IN MANY FORMS, SOME EASIER TO COMPILE THAN OTHERS
Ideas will come in from all stakeholders and Persons with Opinions (Highly Paid or not)They will come in many different formats and need to be consolidated into a big pile of “stuff we could look at”And you will welcome these (because understanding who and what was part of the process and involvement helps you get traction)
COLLECT & COLLATE
An unordered list of business problems and design wishlists
A set of ideas, grouped into tracksSome simple time & sequence planning
A fully quantified spreadsheet, with tracks matching business areas and priorities accounting for release schedules
An email with only basic ideas
Presenter
Presentation Notes
36. IDEAS WILL COME IN MANY FORMS, SOME EASIER TO COMPILE THAN OTHERS Ideas will come in from all stakeholders and Persons with Opinions (Highly Paid or not) They will come in many different formats and need to be consolidated into a big pile of “stuff we could look at” And you will welcome these (because understanding who and what was part of the process and involvement helps you get traction)
These ideas need to be refined into something you can actually test with: HypothesesStructured Test Concepts not just “make it better and more punchy”
What parts do you have?What parts are missing?
CONSOLIDATE: ORGANISE THE PUZZLE PIECES
These ideas are likely to be pieces of the puzzle, parts of a hypothesis, not a complete testThe next phase therefore is to pull these together into a pile of parts you can work with
Presenter
Presentation Notes
37. CHECK OVER WHAT YOU HAVE COLLATED These ideas need to be refined into something you can actually test with: Hypotheses Structured Test Concepts not just “make it better and more punchy” What parts do you have? What parts are missing?
You won’t get this right on the first pass. Or the second. Like every part of testing this is an ongoing optimisation. Permanently re-adjusting and tweaking until you have something that worksBut initially at least… good enough will do, even if there are some bits that just don’t look quite right.
Split Ideas into groups; themes, site sections, areas of business interestDivide sometimes (often?) conflated and unviable ideas into discrete achievable tests
Match available hours, ability and traffic to achieve the most appropriate mix
CONSOLIDATE: SEE WHAT FITS
Presenter
Presentation Notes
38. COMBINE THE PIECES: THIS IS ALSO AN ITERATIVE PROCESS� You won’t get this right on the first pass. Or the second. Like every part of testing this is an ongoing optimisation. Permanently re-adjusting and tweaking until you have something that works But initially at least… good enough will do, even if there are some bits that just don’t look quite right.
Just because you can, doesn’t mean you shouldEXPERIMENTAL DESIGN
TIME FOR A QUICK DETOUR INTO EXPERIMENT DESIGN
Seeing as we are talking about turning ideas into tests. So a quick refresher on Experimental design
Presenter
Presentation Notes
39. TIME FOR A QUICK DETOUR INTO EXPERIMENT DESIGN Seeing as we are talking about turning ideas into tests. So a quick refresher on Experimental design
THE PROBLEM WITH MOST EXPERIMENT DESIGNS
When given a tool that allows you to change almost anything
And a list of business problems which, if fixed, could make LOTS more money
The typical reaction is to dive in and try to do it all in one big test…
The typical result is #epicfail
Presenter
Presentation Notes
40. YOU CAN CHANGE ALL THE THINGS. YOU SHOULDN’T The problem with changing all the things is there is no clear hypothesis No Focus for the test. Therefore no Answer that you can get Or, if you do sit down and work out a hypothesis and counter hypothesis for every problem on the site then the test is too large to run or analyse.
Hypothesis Kit 3
Craig Sullivan @OptimiseOrDieColin Mcfarland, Lukas Vermeer, Rik Higham, Doug Hall, Michael Aagard and many others who helped shape this
Simple Kit1. Because we saw (data/feedback)2. We expect that (change) will cause (impact)3. We’ll measure this using (data metric)
Advanced Kit1. Because we saw (qual & quant data)2. We expect that (change) for (population) will cause (impact(s))3. We expect to see (data metric(s) change) over a period of (x business cycles)
“We don’t like this page” is NOT a valid HypothesisThere are several ways to frame this but here is one I like for it’s clarity and simplicity
Presenter
Presentation Notes
41. USE A HYPOTHESIS FRAMEWORK A couple of great ways to frame what an idea needs to have before it can become the basis for a test.
A lot of stakeholders don’t think in terms of hypothesesThey’ve focused on the business problems and who actually has to do thisSo a Scope, Brief, Qualifying Criteria for a test may be better explained if you can answer the 5W1H questions
QUALIFYING CRITERIAWhat• Do we want to achieve?
Where• Are the changes to be made?
Why• Based on what data?• Based on what need?
Who• Is this aimed at?• Is responsible for sign off?
When• Can we do this?• Is this running?
How• Is that measured? How will that show in the metrics we have defined• Much difference could be made?
Presenter
Presentation Notes
42. DEFINE CRITERIA FOR WHAT MAKES A “TEST” IN YOUR BUSINESS A lot of stakeholders don’t think in terms of hypotheses They’ve focused on the business problems and who actually has to do this So a Scope, Brief, Qualifying Criteria for a test may be better explained if you can answer the 5W1H questions
DECISION METRICS: WHAT DOES SUCCESS LOOK LIKE?Decision Metrics• Appropriate to the question you are asking?• Directly relate to Test Objective?
What are you changing?• Expected effect• What should users do more of, more often?
Unexpected effect• What if they do less?• Or worse… No Significance?• What if it is substitution?
Primary Call To Action or Outcome• Ancillary Outcomes• Value or Meta Metrics
Presenter
Presentation Notes
43. MEASURE THE RIGHT THING The most effective metric is the closest Next Action for a user who has seen your change What are you changing? Why? What should the user do if it works? What could a user do if it doesn’t? Measure that as your main signal Sometimes users do less of your main metric but substitute another If you can, measure that too Sometime the main metric increases but an alternate path to Next Action decreases This can mean no NET gain, what you have is substitution not incremental increase So measure that too.
The most effective metric is the closest Next Action for a user who has seen your change• What are you changing?• Why?• What should the user do if it works?• What could a user do if it doesn’t?Measure that as your main signal
Sometimes users do less of your main metric but substitute anotherIf you can, measure that tooSometime the main metric increases but an alternate path to Next Action decreasesThis can mean no NET gain, what you have is substitution not incremental increaseSo measure that too.
You want a clear resultIt doesn’t need to be Up, sometimes quantifying the Risk of Down is criticalThe only failed test is one where you don’t learn something
UNCLEAR METRIC DEFINITIONS: UNCLEAR RESULTS
Presenter
Presentation Notes
44. POOR TEST STRUCTURE & POOR METRIC SELECTION = UNCLEAR RESULTS You want a clear result It doesn’t need to be Up, sometimes quantifying the Risk of Down is critical The only failed test is one where you don’t learn something
• Weighting• Priority
• Potential• Importance• Ease• Reusability
• Dependencies• Restrictions
TRIAGE
Many methods to prioritise YOUR tests to suit YOUR situation, limitations and objectivesYou know what they are, right?
After all we established this in Boarding and Discovery… didn’t we?
Presenter
Presentation Notes
45. SORT IT ALL OUT http://www.processexcellencenetwork.com/lean-six-sigma-business-transformation/articles/process-excellence-methodologies-using-prioritizat/ http://conversionxl.com/3-frameworks-help-prioritize-conduct-conversion-testing/ PIE or PIER is a simple option; also see Six Sigma, FUWDM, Plan/Measure/Improve frameworks or roll your own. Pick one that works for you, adapt it as needed. Explain to Stakeholders how priority will be judged Adjust the weight of each factor as needed, update regularly Or be Agile, have a regular stand-up meeting and argue a lot debate Sprints & minimum deliverables However you do it you want a Working List of 3 to 10 tests which are being planned, scoped, mocked up, developed, QA’ed, QA’ed again and Analysed If you are working in a team you don’t need to wait for a test to complete, go to another area and start planning the next one whilst the first two are in Build and QA If you are on your own, rather than pointlessly checking results of a live test 5 times a day (we all do) get on with planning a new test and QA the next one.
Emotion and tactical need will enter into itBut if your whole roadmap is led by whim and gut feel it is easy to go off course
PRIORITISATION
Many people will think Priority is decided by which HiPPO is the most loudy shoutyWhilst this can be accurate, it is not the most reliable way to plan an effective test roadmap
Presenter
Presentation Notes
46. PRIORITISATION SHOULD BE METHODICAL Emotion and tactical need will enter into it But if your whole roadmap is led by whim and gut feel it is easy to go off course
PRIORITISATIONIt is the HiPPOs job to focus on• The end result• Big difference for the whole company• Technically difficult site-wide changes
They will want BIG RESULTS quickly
They need to quantify risk or reward
But if you wait for only those tests…
Your roadmap will stallTest for intelligence on what winsYou can’t test to only get wins
Explain, carefully, sometimes slowly:• Some tests are small• Some are far away
Presenter
Presentation Notes
47. HAVE A MIX OF SIZE AND COMPLEXITY We talked about a number of threads of enquiry, alternative routes. You need to plan a mix of all types of test Tests you can do now Tests you can plan ahead for Tests which will only be added later IF preceding results show those later tests are worth investigating. Keep refreshing that list Killed something as it turned out to be lower priority or too hard right now? Bring forward 2 or 3 potential candidates and try them on for fit in the hole you made in the roadmap
• Keep a central record of what is agreed• Keep a revision history of that over timeCheck it from time to time to remind yourself how you have had to re-planRemind yourself and stakeholders that changing plans is expected and required
PRIORITISATION
This is where a framework and Discovery are very useful. Don’t get into emotional debateQuantify what happens when, why and what is needed to make later tests happen sooner.
Presenter
Presentation Notes
48. QUANTIFY AND RECORD YOUR CHOICES Keep a central record of what is agreed Keep a revision history of that over time Check it from time to time to remind yourself how you have had to re-plan Remind yourself and stakeholders that changing plans is expected and required
CRO IS UNDERSOLD. IT CAN DO MORE THAN “JUST” GET WINS
But the ease with which consistent victories are achieved is massively oversoldTechnology will not solve all the issuesWhat makes most difference is the people and the teamwork
SUPERHEROES OF CRO
We like to portray CRO as heroes of the business, fixing bad UX and saving the worldBut, a confession.
In reality it is not the company-saving Alpha-male magic beans we were sold
Presenter
Presentation Notes
49. CRO IS UNDERSOLD. IT CAN DO MORE THAN “JUST” GET WINS But the ease with which consistent victories are achieved is massively oversold Technology will not solve all the issues What makes most difference is the people and the teamwork
DIFFERENT BREED OF HERO
In fact it needs a different kind of hero. One that is much more … office-based & organised
And wears a lot less lycra
Presenter
Presentation Notes
50. PLANNING, ORGANISATION, DOCUMENTATION. IS THAT SUPERHERO STUFF? This is not a world of glamour and excitement In fact it sounds a lot like …. Project Management And … ick… Common Sense
Working through what is needed, not just what looks good requires a methodical approachYou can run around smashing heads and breaking hearts but it is easy to make more noise than value
The biggest gains are made from long-term cultural shift in behaviourIt doesn’t mean you have to go slow. With a good repeatable process you can go fasterBecause you already have a list of targets that add up to a clear goalYou waste less time than looking around for only the headline grabbing big battles after you recover from the last
DIFFERENT BREED OF HERO
That’s not to say that this different breed of heroes can’t kick assWith the right approach and the right tools they are very much in the world-saving business
They just do so with a little more of a measured approach and less collateral damage.
Presenter
Presentation Notes
51. DON’T UNDERESTIMATE THE QUIET MAN Working through what is needed, not just what looks good requires a methodical approach You can run around smashing heads and breaking hearts but it is easy to make more noise than value The biggest gains are made from long-term cultural shift in behaviour It doesn’t mean you have to go slow. With a good repeatable process you can go faster Because you already have a list of targets that add up to a clear goal You waste less time than looking around for only the headline grabbing big battles after you recover from the last
PROJECT MANAGEMENT
In fact we aren’t talking about a revolutionary approachWhen it comes to Roadmapping for success an essential tool is… Project Management
I recommend and use www.liquidplanner.com but there are many options out there
Presenter
Presentation Notes
52. PLAN TO TURN IDEAS AND OBJECTIVES INTO PROJECTS AND DELIVERY So we aren’t talking about the glamourous super hero company saving Alpha male activity we were sold This sounds a lot like …. Project Management And … ick… Common Sense
Excel, Smartsheet, Whiteboard, Piece of Paper. Ultimately its what fits and what you can use.But something you can share with the team and easily update is recommended
PROJECT MANAGEMENT
A good basic (and cheap) option is www.smartsheet.com which offers several templatesProTip: You can also link sheets so the Projects in the Gantt links to items from your Priority/Ideas sheet
It also has good Google integration and allows creation of Forms for adding new items from Stakeholders
Presenter
Presentation Notes
53. EVEN A SIMPLE GANTT WILL HELP Excel, Smartsheet, Whiteboard, Piece of Paper. Ultimately its what fits and what you can use. But something you can share with the team and easily update is recommended
Task lists don’t give you an oversight of how all tests fit togetherBasic lists of Projects won’t show you what Task is needed next from each personSimple Gantts or Card Views don’t let you move Tasks independently of ProjectsIt is entirely likely (and recommended) to be in planning phase of Test 4 and design phase of Test 3 when Test 1 is live and Test 2 is in Final QA
PROJECT MANAGEMENT
Task based tools like Asana or Basecamp can also work, as can card style tools like Trello.These type of tools are simple to use but lack oversight on scheduling resource over multiple tests
(The image is actually Card view from Liquid Planner – simple view, underpinned by a proper scheduling system)
Presenter
Presentation Notes
54. PRIORITY NEEDS TO BE FOR PROJECTS BUT ALSO TASK LEVEL Task lists don’t give you an oversight of how all tests fit together Basic lists of Projects won’t show you what Task is needed next from each person Simple Gantts or Card Views don’t let you move Tasks independently of Projects It is entirely likely (and recommended) to be in planning phase of Test 4 and design phase of Test 3 when Test 1 is live and Test 2 is in Final QA
PROJECT MANAGEMENT
Liquid Planner works well for CRO because it is flexible & any plan needs to be flexible• Several ways to look at, reprioritise and reschedule multiple overlapping projects• Move tasks up the priority list & work on them out of sequence when you need to work ahead• Every time you update priority it re-calculates the likely delivery for each task & project
Presenter
Presentation Notes
55. LIQUID PLANNER IS THIS POWER USER’S CHOICE Ranged estimates for delivery and roll-up, drill down from Client to Project to Task level Each task can have a checklist, deadline, comments, notes, attachments, tags and much more There is also time tracking; a full Resource, Project, Client time analytics suite and an API Zoom out to Year, Quarter and Month, Zoom in to Week and Day views Users see their own task list, but the order they work on them fits the most efficient way for the whole show to move forward.
COMMON SENSE: SO RARE IT’S A SUPERPOWERPrioritise and plan for delivery of Tests, not just ideas.How MUCH you do depends on your setup & situation:
Resource Capacity• Traffic and sample size for metrics• Developer availability• Analyst & Planning availability• Design availabilityResource Capability• Technical ability within the team• Technical function of the Testing tool• Experiment planning ability• Analytics and Statistical ability
Want to go fast and run small tests. Plan for thatWant to plan more and test for impact? Do thatWant BIG TESTS and quickly. Be big. Pay big (still fail lots)
Ultimately use Common Sense.
It’s still so rare it’s a super power
Presenter
Presentation Notes
56. PLAN TO GET BUSIER – INCREASE CAPACITY AND CAPABILITY The more capable the team is the more (good) work they can do in the same time So increasingly ability increases capacity. But you might need to plan on reducing capacity to create the overhead needed to improve people’s skills If you are short on capacity or capability there are highly experienced professionals out there who can help *cough* www.trsdigital.com *cough*
Simply planning Tests will breed new test ideasResults will create more need to explore in some areas than originally expected.Results will kill whole planned sets of tests if they are shown to have less value than anticipated.
EVOLVE
Test 1
Mobile 1
Mobile 1a Mobile 2
Mobile 3
Mobile LP
Landing Pages
LP 1
LP 2
LP 3
LP 4
LP 2a
LP 2b
Checkout
Checkout 1
Checkout 1a
Checkout 1b
Checkout 2
Checkout 3
Checkout 4
Checkout 5
Test 1a
Presenter
Presentation Notes
57. TESTS WILL BREED TESTS, WINS WILL BREED MORE TESTS Each test should teach you something, even if it is to not test like that again Every metric should point to other areas to explore, user reactions you can investigate Some tests will show great value and prompt many follow ups
Nothing is set in Stone• Revise those plans and documents after each update• Document each revision, track what takes time• The aim is to improve the process as well as the site
It takes an investment – a commitment on timeYou can’t just cut out fat to make efficiencies and optimizeYou have to measure and invest where your system is lacking
Anyone says otherwise is telling Porkies
EVOLVE
Presenter
Presentation Notes
58. ACCEPT CHANGE. MAINTAIN INVESTMENT DON’T CUT CORNERS This is often where successful testing initiatives fall over Initial wins “prove” the original plan was right and the plan becomes fixed Not deviating from the original plan can miss opportunities The other side to that is Costs and Time Initially it may be proof of concept, with a limited budget and resource. But once your Process has shown results cutting back on budget and resource (or leaving it static as your ambition grows) will choke growth Expecting to grow from process efficiency alone is foolish Expecting the same results when you remove support after the initial excitement is unrealistic Investment in time, resource, training and increasing the power given to the roadmap will accelerate success Keeping investment the same but expecting more, will not work.
For every test on the Working ListCreate Test Documentation• Document Your Hypotheses• Confirm decision metrics• Record what you intended to happen• Scope how you want that to be built• Specify URLs to include/exclude• Specify Segments to include/exclude
If you came back to this in a year could you explain to a new starter what was intended? Memory is fallible.
If you had to come back to your developer in a week and explain why it failed QA.. Is it in here?
If you’re “just trying something” without data, objectives, a Hypothesis and Metrics…It’s not a test.
You can build it and run it.
But it’s not a test
EXECUTION: DOCUMENT EVERY TEST
Buy Today
Buy Today
Buy Today
click
Why is it bigger?How much bigger?What should users
do more of?
Is this on all pages?All devices?
Image or CSS?What does this do to
nearby elements?
Presenter
Presentation Notes
59. EXECUTE ON YOUR PLANS BUT WRITE STUFF DOWN! One of the areas many initiatives fail is when the plans change but people don’t remember why New blood in the team, scope creep, unexpected use cases Document every step Use version control Monitor what was the original concept and how it changed to the delivered product Look at commonly occurring issues. Make changes in your process to remove issues and double down on good practice
Wireframes• Blueprint your changes• Show what it should look likeTake a closer look at the page• Look at what else could happen• Is the change clear?• Could something else be clicked on?• Are you measuring that too?
This is your blueprint, the concept to get approved before you spend time and money on design.
This is the idea your stakeholders must buy into, your designer must understand, your developer must build.
Have you shown all the states the page can be in for the test? Does it work in all cases?
EXECUTION: BLUEPRINT THE CHANGES
click clickclick
click clickclick
click clickclick
Presenter
Presentation Notes
60. RECORD YOUR DESIGN CONCEPTS Again looking back in a year your site may well have changed again Again you need to know what “right” looks like when it’s built Changes that come in during the build process are untested, you might run a test that does not fit your hypothesis or metrics Looking at wireframes, mockups and screengrabs of the final QA in your documentation can help explain where you deviated Keep a record of before and after A snapshot of the environment where the change you made was found Don’t assume a “Win” from this year will work the same next year if everything else around it has changed
Issue Resolution• Plan for Use cases• Try to break the page• Try it logged in, not logged in• Checkout with nothing in basket• Submit forms with fields empty
Build and try to find omissions• QA• QA again
Hold a pre-mortemBlame no-one, open mind and think:• If in n weeks you are reporting a failure• Why did it fail? What broke? What is the cause of death you have to pronounce?• Could it have been avoided?
Produce a conclusion report for EVERY test. What worked, what didn’t
EXECUTION: PRE AND POST MORTEM EVERY TEST
Presenter
Presentation Notes
61. RECORD YOUR WINS, LOSSES, MISTAKES Again looking back in a year your memory will have faded You are learning lessons in test structure, metric selection, how your site works, how well your process works Document this so you can learn from these Document so your fallible memory is not the only record Document so new team members have a knowledge base they can work from
Test Results should change your plans• Regular reviews
• Is this still in line with strategy?• Could we go faster? Should we go slower?
• Test Results• Did the result show further potential?• Did the result show greater risk?
• Test non-results• Why is it not clear?• Did the user not care enough?• Did one segment “outvote” another?
Don’t underestimate Tactical Issues• Releases• Team Changes• Market conditions• Seasonality
ADAPT
Review
Results
Non-Results
Releases
Team Changes
Tactical issues
Presenter
Presentation Notes
62. ADAPT TO CIRCUMSTANCES: SURVIVAL OF THE FITTEST Each test, each planning session, each delayed release, each change to plans should be recorded If you are being a new kind of Hero then you are tracking time and seeing how changes affect the overall plan If you are being smart you regularly look at what burns time, what causes issues in planning, what causes tests to fail QA, what makes for unclear results And each time you learn you revise your plan, look at what tests you have coming that would repeat those mistakes Eliminate those issues, Evolve your plan and your approach Optimise your process, not just your website. If you want to go faster you can add more resource If you want to go faster and get better you need to improve your process You reach a point where just adding more bodies will start to cause problems If you have treated the incremental optimisation of your program as seriously as the optimisation of your site Then more fuel and more power will make you go even faster If you haven’t and you add more fuel and power you can spin out or crash NB I hate the cogs on this Smart Art. That’s not how cogs work, the arrows need to be bi-directional on the big cog or some variable switching pulley on the counter-turning cog for this to even start to make sense
CONCLUSION
Bet you thought this would never end, nearly time for the coffee breakWe’ll finish with a quick recap
Presenter
Presentation Notes
63. NEARLY DONE Bet you thought this would never end, nearly time for the coffee break We’ll finish with a quick recap
BUILD FROM GOOD BOARDING
Good Boarding process No Boarding process
GOOD BOARDING = UNDERSTAND THE PEOPLE
Presenter
Presentation Notes
64. GOOD BOARDING = UNDERSTAND THE PEOPLE
DISCOVERY PHASE ENSURES SUCCESSFUL TAKE-OFF
DISCOVERY = UNDERSTAND THE ENVIRONMENT AND EQUIPMENT
Presenter
Presentation Notes
65. DISCOVERY = UNDERSTAND THE ENVIRONMENT AND EQUIPMENT
BE SERIOUS ABOUT YOUR DATA
DATA DRIVEN DECISIONS,OTHERWISE IT’S GUESSWORK
Presenter
Presentation Notes
66. DATA DRIVEN DECISIONS, OTHERWISE IT’S GUESSWORK
All too easy to think data and a plan has to be perfect to startAll you need is Good EnoughThe process is to refine all of this as you go along
TOO MUCH CAN BE INTOXICATING
DECIDE – DON’T JUST COLLECT DATA AND TAKE NO ACTION
Presenter
Presentation Notes
67. DECIDE – DON’T JUST COLLECT DATA AND TAKE NO ACTION All too easy to think data and a plan has to be perfect to start All you need is Good Enough The process is to refine all of this as you go along
BUT GET IT RIGHT, AND YOU’RE LAUGHING
STRIKE A PRAGMATIC BALANCE THAT SUITS YOUR SITUATION
Presenter
Presentation Notes
68. STRIKE A PRAGMATIC BALANCE THAT SUITS YOUR SITUATION
IDEAS FROM BUSINESS, USERS AND DATA
BE CREATIVETRY TO BREAK YOUR USER JOURNEYSIT IS NEVER THE FLOW YOU INTENDED
Presenter
Presentation Notes
69. BE CREATIVE
BALANCE YOUR PRIORITIES
BALANCE TO SUIT YOUR CAPACITY
Presenter
Presentation Notes
70. BALANCE TO SUIT YOUR CAPACITY
QUALIFY WHAT MAKES AN IDEA A TEST CANDIDATE
MAKE EVERY IDEA EARN ITS PLACE ON THE LIST
Presenter
Presentation Notes
71. MAKE EVERY IDEA EARN ITS PLACE ON THE LIST
QUANTIFY YOUR PRIORITIES TO THE HIPPOS
Calmly explain; facts and data back up strong argumentsAnger, fear, aggression lead to the dark side(turning off the lights has a similar effect)
OPEN COMMUNICATION WITH STAKEHOLDERSUSE DATA TO BACK UP YOUR POSITION
Presenter
Presentation Notes
72. EXPLAIN TO STAKEHOLDERS, USE DATA TO BACK UP YOUR POSITION
EXPLAIN HIGH PRIORITY DOESN’T MEAN “NEXT TEST”
BY DEFINITION EVERYTHING IS ASAPBecause Soon and Possible are subjective variables. Use Common Sense
Presenter
Presentation Notes
73. BY DEFINITION EVERYTHING IS ASAP Because Soon and Possible are subjective variables
BE A NEW BREED OF CREDIBLE HERO
DON’T GET CAUGHT UP IN YOUR OWN LEGEND
Be “Get things done” rather than “Look at me”
Presenter
Presentation Notes
74. DON’T GET CAUGHT UP IN YOUR OWN LEGEND Be “Get things done” rather than “Look at me”
PROJECT MANAGEMENT IS NOT A DIRTY WORD
A TESTING PLAN REQUIRES PLANNING
Don’t forget Common SenseCommon Sense and pragmatism go a long way
Presenter
Presentation Notes
75. A TESTING PLAN REQUIRES PLANNING Don’t forget Common Sense Common Sense and pragmatism go a long way
RECORD ALL PLANS (BUT NOTHING IS FROZEN)
WRITE IT DOWN, BUT BE PREPARED AND HAPPY TO CHANGE IT
Presenter
Presentation Notes
76. WRITE IT DOWN, BE PREPARED TO CHANGE IT
ADAPT: LEARN FROM BAD RESULTS
BAD RESULTS ARE STILL RESULTS, LEARN FROM THEM
Presenter
Presentation Notes
77. BAD RESULTS ARE STILL RESULTS, LEARN FROM THEM
EVOLVE: TEST FOR INTELLIGENCE NOT JUST RESULTS
BE AGILE: SEE AN UNEXPECTED OPPORTUNITY THEN EXPLOIT NEW AREAS
Presenter
Presentation Notes
78. BE AGILE: SEE AN UNEXPECTED OPPORTUNITY THEN EXPLOIT NEW AREAS
EFFECTIVE ROADMAPS
@pjeedai
www.trsdigital.com
Presenter
Presentation Notes
79. SEX PANTHER, 60% OF THE TIME IT WORKS EVERY TIME Thank you for your time That name again is Mr Plow Tim Stewart from trsdigital
