Embed Size (px)
DESCRIPTION
On the pre-conference day of PBIRG's Annual General Meeting 2012 in Chicago, SKIM presented about how to better deal with uncertainty in forecasts. Gerard Loosschilder, Jemma Lampkin and Eelke Roos explored Monte Carlo simulations for scenario planning and addressed 'what if' questions, building a Monte Carlo simulation from the ground up. Participants left with better ideas of how to deal with the certainty of uncertainty in forecasting, understanding how to just deal with it - turning uncertainty into a useful, and even playful, approach.
Citation preview
expect great answers
Uncertainty? Just deal with it!
Jemma Lampkin | Eelke Roos | Gerard Loosschilder
For PBIRG | Chicago | May 2012
… even more accepting of uncertainty in your forecasts, actually turning it into an integrated part of your scenario thinking.
At the end of this workshop, we hope you are …
The purpose of a forecast is to support business planning
Determine …
How much you are going to
sell.
If you will have a positive
return on your investment.
Your forecast … not a point estimate
4
Time
Pe
rfo
rma
nce
Your annual
peak sales
is $1 Billion
Time
Pe
rfo
rma
nce
Your forecast … a range estimate
5
At moment tx
Your annual peak
sales are
100% sure to
be $800 million
80% sure to be
$1 billion
10% sure to be
$1.5 billion At a likelihood of x%
The output is a range estimate of likely outcomes
0
200
400
600
800
1000
1200
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
Cum
ula
tive
reve
nu
e tre
atm
en
t (m
illio
n U
SD
)
Cumulative revenue test treatment
Probable revenue
range: 90% chance
of revenue falling
within this range
based on Monte
Carlo simulation
90% likelihood range
Minimum cumulative revenue
Maximum cumulative
revenue
Average cumulative
revenue
Sources of uncertainty can be categorized in two clusters:
The accuracy of metrics
• Metrics collected in our
studies
• Metrics available in the
public domain,
syndicated data and
with the client
The likelihood of events
• Market conditions that
may change
• Competitive actions
and reactions,
preempting and trailing
7
That is why we prefer to talk about scenario thinking
instead of forecasting, to properly focus the attention on
the question “what if?”.
Monte Carlo Simulation An alternative way to support scenario thinking
To deal with uncertainty and risk, we suggest using …
8
Monte Carlo Simulation is an extension of your modeling practice
9
Δ Input Δ Output Stochastic
Not deterministic
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10
Uniform if
uncertainty
is high
Inputs and
outputs follow a
distribution
Normal if
uncertainty
is low
A normal distribution if uncertainty is low
0
100
200
300
400
500
600
0% 20% 40% 60% 80% 100%
Nu
mb
er
of
sim
ula
tio
ns a
t th
is v
alu
e (
#)
Compliance value (%)
Compliance
10
The input variable
of “compliance”
assumes a
normal
distribution with a
mean of 50% and
a standard
deviation of 8%.
Input
A uniform distribution if uncertainty is high
0
100
200
300
400
500
600
1% 11% 21% 31% 41% 51% 61% 71% 81% 91%
Nu
mb
er
of
sim
ula
tio
ns a
t th
is v
alu
e (
#)
Uptake value (% of peak share)
Uptake after the 1st year
11
The input variable
of “uptake”
assumes a
uniform
distribution with
an equal
likelihood of all
values between
40% and 60% to
happen.
Input
The likelihood of events are inserted as discrete variables
25%
50%
25%
0%
20%
40%
60%
80%
100%
Launch scenario
20%
60%
20%
0%
20%
40%
60%
80%
100%
Worst Base Best
Efficacy scenario
12
These events have discrete
probabilities of happening
Input
The likelihood of events are inserted as discrete variables
10%
40% 50%
0%
20%
40%
60%
80%
100%
Launch scenario
50%
30% 20%
0%
20%
40%
60%
80%
100%
Worst Base Best
Efficacy scenario
13
These events have discrete
probabilities of happening
Input
Probability distribution of sales forecast if uncertainties in continuous inputs are high
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Pro
ba
bil
ity o
f m
ak
ing
th
e s
ale
s (
%)
Sales in billion USD
Probability distribution of sales
14
The distribution of
forecasted sales
values shows a
gradual decline
as a result of
higher
uncertainties in
continuous input
variables.
Output
Probability distribution of sales forecast if uncertainties in continuous inputs are low
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Pro
ba
bil
ity o
f m
ak
ing
th
e s
ale
s (
%)
Sales in billion USD
Probability distribution of sales
15
The distribution of
forecasted sales
values shows a
steep decline as
a result of lower
uncertainties in
continuous input
variables.
Output
Probability distribution of sales forecast if critical input variables have higher values
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Pro
ba
bil
ity o
f m
ak
ing
th
e s
ale
s (
%)
Sales in billion USD
Probability distribution of sales
16
The distribution of
forecasted sales
values shifts to
the right as a
result of higher
values for the
input variables.
Output
Probability distribution of sales forecast if strongly impacted by discrete input variables
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Pro
ba
bil
ity o
f m
ak
ing
th
e s
ale
s (
%)
Sales in billion USD
Probability distribution of sales
17
The distribution of
forecasted sales
values assumes a
step-wise shape
as a result of a
higher impact of
discrete input
variables.
Output
Working with uncertainties works best if we also manage our expectations
That is why we work with
action standards.
An action standard is a
threshold value that a key
performance indicator needs
to exceed at an acceptable
risk, before we to decide to
pursue the initiative.
I.e., we want to be 80% sure
to make $1 billion or more.
18
We met the action standard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5
0.8
1.1
1.4
1.7
2.0
2.3
2.6
2.9
3.2
3.5
3.8
4.1
4.4
4.7
5.0
5.3
5.6
5.9P
rob
ab
ilit
y o
f m
ak
ing
th
e s
ale
s (
%)
Sales in billion USD
Probability distribution of sales
19
Action standard
We want to be 80%
sure to make $1
billion or more.
Result
The probability of
making $1 billion worth
of sales is 84%, so we
have exceeded the
action standard.
How Ducendi Inc. wants to build a business case for its in-licensing agreement with Novus pharmaceuticals
Introduction into the business case
Novus is developing an oral type 2 diabetes drug with a novel mode of action
Novus pharmaceuticals is a
biotechnology company on the rise.
In order to raise new funding, Novus
has offered the new treatment in an
in-license agreement to Ducendi, a
big pharmaceutical corporation.
Ducendi wants to know how likely it
is for Periculum to get a positive
ROI.
21
Novus claims a high likelihood of success Ducendi is not so sure
Upside
• Survey sponsored by
Novus: 80% of physicians
are positive; 60% are likely
to prescribe it
• Advantages: safety and
tolerability profile,
risk/benefit profile and
Mode of Action
Downside
• A likelihood that efficacy is
only moderate
• Competitive treatments in
clinical development are
expected to have
interaction with Periculum
• Competitive treatments may
be launched sooner than
Periculum
22
How well do you deal with the uncertainty?
Your 5-year revenue will surely be $1.5 billion
Your 5-year revenue will have a 80% likelihood of being
$1.5 billion
Your 5-year revenue will have a 80% likelihood of being
$1.5 billion
It also has a 99% likelihood of being $300 million
and 30% of being $2 billion
23
Setting the action standard for this case
• What would be the accepted
amount of risk you are
willing to take?
• How would you set the
action standard?
• Would setting an action
standard like this fit with your
business practice and
resonate with your team?
24
Exercise – set the action standard for Ducendi’s $1.5 billion investment in Novus’ Periculum
• Senior management has asked you to assess the likelihood of a
positive ROI 5 years post-launch
• Ducendi has calculated a positive ROI to equal $1.5 billion in 5 years
• This investment includes the development, production, launch and
maintenance of Periculum
We will use these numbers in the business case.
25
5 year revenue How certain do we need to be of
reaching this revenue point?
$ 1.5 billion At %
$ 2.0 billion At %
The return on investment of Periculum launched in two major markets
Introduction into the Monte Carlo Simulator
for scenario thinking
Ducendi wants to forecast the potential in two crucial markets, the United States and Elbonia
United States of America
Strategically important
established market
• Largest T2D market in the world
in terms of revenue
• Health insurance provided by the
both public and private entities
• Complex payer dynamics
• T2D data available from many
sources at high precision, quality
and certainty levels
High risk, low uncertainty
Accounts for ~70% of revenue
Elbonia
Strategically important
emerging market
Big opportunity but …
• Market characterized by high out-
of-pocket expenses
• High use of branded generics
• Aggressive low cost local
competitors
• Not many data available. High
uncertainty and low quality.
Based on qualitative impressions
High risk, high uncertainty
Accounts for ~30% of revenue
27
With Periculum being launched in 2016, Ducendi wishes to break even in 5 years
0
5
10
15
20
25
30
2016 2017 2018 2019 2020
Mil
lio
n T
2D
pati
en
ts
Year
US, Minimum US, Maximum
Elbonia, Minimum Elbonia, Maximum
28
Let us assume the
size of the patient
population is a
given at a lower
and upper bound.
Ducendi uses conjoint methodology to measure demand for Periculum under various scenarios
Ducendi’s conjoint study replicates the following launch scenarios:
Efficacy (phase III) of Periculum
• Higher than phase II data (best case)
• Similar to phase II data (base case)
• Lower than phase II data (worst case)
Competitive launch
• Before Periculum
• At the same time as Periculum
• After Periculum
29
Competition is expected to launch a similar drug. However, who goes first?
First-mover advantage: the first mover preempts the follower, and
gets a lasting advantage throughout this 5 year period.
The first mover advantage is modeled as a likelihood in the scenarios:
what’s the likelihood of:
Periculum first,
competitor second
Periculum and competitor
at the same time
Competitor first,
Periculum second
30
2015 2016 2017 2018 2019 2020
P C
C P
P C
2015 2016 2017 2018 2019 2020
2015 2016 2017 2018 2019 2020
E.g.,
30%
E.g.,
40%
E.g.,
30%
What are the ranges we put in, and what level of uncertainty do we assume?
Now we need your input!
First, we look at the accuracy of market data: compliance/persistence and uptake
Uptake
What do you expect the uptake of the new drug to be
by the physician population?
Uptake is influenced by satisfaction with current
products, awareness/”buzz,” access/price, opportunity,
competition and the quality of the product
United States Elbonia Shape (uncertainty)
2016
Uniform (high)
Normal (low)
2017
2018
2019
2020
Compliance x Persistence
What do you expect the patient compliance
and persistence with the new drug to be?
Compliance is the patient’s adherence to the
prescribed dose per day
Persistence is the proportion of patients
persisting with the prescribed therapy
United
States Elbonia Shape (uncertainty)
Lower
Bound Uniform (high)
Normal (low)
Upper
Bound
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
% Min:
% Max:
%
%
%
%
35
40
65
75
95
100
100
100
100
100
75
80
Second, we look at the likelihood of events: efficacy and a competitive launch
Efficacy
Coming out of phase III, what is
the likelihood of Periculum to be
less, equally, or more efficacious
than measured in phase II?
Higher
(best case) _ _ _ %
Similar
(base case) _ _ _ %
Lower
(worst case)
_ _ _ %
Competitive launch
What is the likelihood of the competitor
drug to be launched before or after
Periculum, or at the same time?
United States Elbonia
Before _ _ _ % _ _ _ %
Same time _ _ _ % _ _ _ %
After _ _ _ % _ _ _ %
33
20
50
30
60
30
10
See what happens in the business case
Now let us plug in the numbers and …
34
So, did we make it?
Target Actual
Revenue % of risk Revenue % of risk
Total $ 1.5 billion At __ % $ 1.5 billion At %
Total $ 2.0 billion At % $ 2.0 billion At %
Do you want to go back and change a few parameters
to see what happens?
Set action standard Set market data Set launch data
So, how can we help the business make a decision while dealing with uncertainty?
That is all nice,
but my business cannot deal with uncertainty.
My business needs to make a decision!
Eventually, the business needs to make a few decisions to overcome the uncertainty
37
Did we
meet or
exceed the
action
standard?
Yes
No
First, the business needs to decide if it finds enough reason to continue
38
Did we
meet or
exceed the
action
standard?
Yes
No Now what?
Continue with
the initiative
Not meeting the action
standard usually
results in more
questions and
uncertainty. The
business needs to
decide what to do next.
If not, the business needs to decide if it is due to the quality and accuracy of the data
39
Did we
meet or
exceed the
action
standard?
Yes
No
Did we
have the
best data
we could
have had?
Yes
No
Continue with
the initiative
If not, the business needs to decide if it is due to the quality and accuracy of the data
40
Did we
meet or
exceed the
action
standard?
Yes
No
Did we
have the
best data
we could
have had?
Yes
No
Continue with
the initiative
Invest in more
accurate data
Now what?
Deciding that the
data were not
accurate is the
easiest way out.
But what if the
data were the best
we could have?
Last, the business needs to decide what is in its power to meet the action standard
41
Did we
meet or
exceed the
action
standard?
Yes
No
Did we
have the
best data
we could
have had?
Yes
No
Can the
business
invest to
have a
higher
probability
of meeting
the action
standard?
Yes
No
Continue with
the initiative
Invest in more
accurate data
Some parameters
can be in control
of the business,
like investments in
compliance or time
to market.
Last, the business needs to decide what is in its power to meet the action standard
42
Did we
meet or
exceed the
action
standard?
Yes
No
Did we
have the
best data
we could
have had?
Yes
No
Can the
business
invest to
have a
higher
probability
of meeting
the action
standard?
Yes
No
Continue with
the initiative
Revise the
business case
Stop the
initiative
Invest in more
accurate data
We hope that by now, you’re even more accepting of uncertainty in your forecasts
Turning it into an integrated part of scenario thinking
• Working with a Monte Carlo based simulator, thinking
in terms of ranges instead of point estimates
• Setting action standards in consultation with the
business, representative of their appetite to risk
43
Any great questions?
Jemma Lampkin | Senior Project Manager
[email protected] | +1 201 963 8430
Eelke Roos | Project Manager
[email protected] | +1 201 963 8430
Gerard Loosschilder | Chief Methodology Officer
[email protected] | +31 10 282 3535
