Uses of performance information Benedict Wauters

Uses of performance information

Benedict Wauters

Use of performance information

1. Next to “motivate”…

2. To budget/plan

3. To control

4. To promote

5. To learn

6. To evaluate

7. …ultimately, to improve

2

Bob Simons is the Charles M. Williams Professor of Business Administration at Harvard Business School. Over the last 30 years, Simons has taught accounting, management control, and strategy implementation courses in both the Harvard MBA and Executive Education Programs.

Robert D. Behn, Lecturer in Public Policy, focuses his research, teaching, and thinking on the leadership challenge of improving the performance of public agencies. He is the faculty chair of the School's executive program, Driving Government Performance: Leadership Strategies that Produce Results and conducts custom-designed executive education programs for government jurisdictions and public agencies.

3

This is where the use for budgeting/ planning happens

Here we use it for control, evaluation, learning, promotion

• The Netherlands and wider in the OECD: “When looking at…the Netherlands, it seems fair to

say that performance information has been used by line ministries extensively for compliance and legitimisation…”

“The 2011 OECD Performance Budgeting Survey shows that line ministries most commonly use PI to increase spending in their negotiations with central budget authorities”

“Experience in the Netherlands and other OECD demonstrates that the effects of PBB on budgetary deliberations by parliaments* have been nearly absent”

5M. De Jong et al., OECD Journal on Budgeting, 2013

*after finance/budget minister sends it to Parliament

For info

6

Budgeting/planning -1• From budget/planning to improvement? Ensure

readyness! we need to estimate (predictable) demand AND we need to

estimate our required capacity in terms of people and resources (in diverse parts of an organisation that need to coordinate) to meet this demand

Then add reserve capacity for the uncertain demand How? Based on how we have been doing before, given system

conditions (stock-taking AFTER the fact) Otherwise, we will NOT have the required capacities in place

and will fail to satisfactorily meet demand Budgeting therefore is an exercise in ensuring the organisation

is as ready as possible as, naturally, performance will be lower for unprepared organisations than for prepared ones

However, this clearly has nothing to do with setting targets

7

Budgeting / planning -2

At the macro level in government?• usually political priorities determine envelopes at macro level

(should we invest in roads or in education?) • performance budgeting created the expectation that overall

government performance can be improved by reducing/increasing budgets if poor/good performance in terms of “output”

• but how would government know if past output for the money is sufficient or could be improved?

better to know if service provision relative to its purpose (in the eyes of the customer) is under control and if there is waste

money is not always the answer e.g. lack of leadership in rethinking the system comes first

• only evaluation can determine which services deliver more/less outcomes and why

• government at macro level should engage its public service providers on this!

R. Behn, Public Administration Review, 2003M. De Jong et al., OECD Journal on Budgeting, 2013

For info

Budgeting/planning -3

Demand plan: planned (in/decrease of predictable demand)

Capacity requirements + free capacity maintained (in % used of available people and facilities)

Factoring in expected process improvements

Direct operational, maintenance AND improvement costs

= OPEXCapital expenditure

(replacement/expansion of existing facilities for current business)

=CAPEXIndirect operational cost=OPEX

Strategic initiatives

STRATEX(can be

bothexpenses

orcapitalisa

tion)

PLANS

BUDGET

Operational plans (with a one year time horizon) specify how much demand is expected to beaddressed by which parts of the entity.

9

Structural Funds programming• Also for the programmes in Structural Funds, it makes sense to

budget/plan outputs: How much output do we think the budget can buy?

• Otherwise, project promotors / beneficiaries could be very disappointed if they find out there is no money for them

• Or expectations of how much the money would deliver could be seriously wrong For known, well-understood, repeat services:

• We should know how much it cost in the past, which is a good estimate if there is no reason to think it will be more/less in the future

For complex projects (innovation, solutions development):• Output = project • Budget = maximum budget a project can have

But it is a mistake to turn this into YES/NO targets, with reward and punishment attached!

• As to “results” (outcomes): only evaluation can tell us what we really achieved (after the fact) hence it makes no sense to set “targets” and incentivise them on results

For info

10

• “…ensure that measures used for planning and budgeting purposes are not confused with measures used for improvement and development”.

Andy Neely et al. How to avoid the problems of target setting, 2010, PMO symposium

For info

Boundary systems

Strategy(plan)

Core values,mission, vision

Criticalperformance

variables

Strategicuncertainties

Risks to be avoided

Interactive control systems Diagnostic control systems

Belief systems

SEARCH CREATIVELY AND EXPAND OPPORTUNITY SPACE. LEARNING dominates over control.

Learning and evaluation-1

12


• From learning and evaluation to improving? What is (not) working AND why* To get better understanding in general (enlightenment) for those

involved (new insights, new ideas, changed perceptions)* Not so easy:

• we observe only what we measure, which may not matter or show random correlations

• different people understand the same data differently and draw different lessons from it

• measurement systems (incl. as used in evaluation) reflect what decision-makers expect to see

• however, real learning often is triggered by the unexpected How can you design a measurement system to detect what you

don’t expect?

R. Behn, Public Administration Review, 2003*”programme learning” as inM. De Jong et al., OECD Journal on Budgeting, 2013

13


• to detect the unexpected formal measurement systems must: cast a wide net (variety of measurement covering the entire system from end-to

end, internal and external) avoid excessive aggregation

• formal systems are not enough. There also needs to be informal “measurement”

this is the idea behind “management/data collection by walking around” where every story that people tell managers are the ultimate in disaggregation

• deviance is not always obvious as in the case of a clear failure: to detect deviances and understand if they are worth further investigation, service

providers must understand the internal and external environment (be sensitised) furthermore, to learn from failure, there cannot be an environment that focuses

primarily on assigning blame as people will then try to hide the deviant data

• after a significant deviance is detected, a learning strategy has to be deployed that probes for causes and implications:

will require expert knowledge and other sources of information beyond the measurement system itself

measurement itself is more likely to suggest topics for investigation than to directly impart key operational lessons

R. Behn, Public Administration Review, 2003*M. De Jong et al., OECD Journal on Budgeting, 2013

14


The implications are:

• Targets offer little scope for detecting interesting deviations (that helps us

probe further):

Hit them / Did not hit them

• “experience shows that most operational performance indicators deliver little,

if any value in terms of actionable insights unless they are charted… in time-

series format”*

*Andy Neely et al. How to avoid the problems of target setting, 2010, PMO symposium

15

SPC

No point in acting when values fall within limits of SPC. To improve you need to work on the system.

Seddon, Freedom from C&C

Learning from SPC charts

16

SPC


Trend data

Does not allow toseparate noise frominteresting events

time

Average

Week 1 Week 2 Week 3…



17

SPC

Assumption that worker is responsible for variation leads to shame for worker, followed by cheating and more shame


Target reporting

Tells you very littleOn one side sits success, on the other failure (you hit the target or you don’t)*

Y % good

X% bad

Trend data

Does not allow toseparate noise frominteresting events

time

Average

Week 1 Week 2 Week 3…



18


Evaluation is a specific way to “measure”:• Can be used to answer the question how well are we doing (‘narrow’

accountability) Typically compare with others, or with the past Danger of simplistic comparisons (ignoring context) with others, unclear

objectives and/or inadequate “programmatic” structure (multiple constraints, inadequate resources, unreasonable timetables)

Requires looking at outcomes (relative to input), taking account of context (what is due to other factors than our action)

• Can also relate to a “needs” assessment (what is happening/changing at citizen level) irrespective of some particular public action (key for relevance), sometimes part of “ex ante” evaluation

• As with any other measurement, in evaluation, “deviance” from what we expect is only a starting point to probe further so we can learn to help us improve

R. Behn, Public Administration Review, 2003M. De Jong et al., OECD Journal on Budgeting, 2013

Aggregate common indicators cannot catch much

X visits of teachers to parents

X kids put more effort into homework

+X kids show higher attendance at school

X kids have better

start in the labour market

Actual intervention

For info

Aggregate common indicators cannot catch much

EC states: Participants refer to persons benefiting directly from an ESF investment and who can be identified and asked for their characteristics, and for whom specific expenditure is earmarked. Other beneficiaries should not be counted as participants.

X participants(parents)

X participants in employment 6 months after

leaving

�participants gaining a qualification upon leaving+ �participants in employment

upon leaving

X visits of teachers to parents

X kids put more effort into homework

+X kids show higher attendance at school

X kids have better

start in the labour market

Actual intervention

Common indicators

For info

• »Managing by results is like looking in the rear view mirror. » E. Deming

21

• Visible figures reflect performance of the past

• Managing by (monthly) numbers is like driving a car by watching the road line in the rear-view mirror

• Management by numerical goal is an attempt to manage without knowledge of what to do and is in fact management by fear.

“To lead, one must understand the work that (s)he and his/her people are responsible for. An incoming manager must first learn from his/her people what they are doing.”

For info

Strategic measurement-1

• This measurement should reflect key performance questions (KPQs) that require data for informed discussion regarding possible answers: open questions (what, why, how) rather than closed questions (yes, no,

how much): e.g. Not: “was the budget spent? Rather, “what has happened with the budget?”, “why?”, “how will we act?”

relate to the strategy map formal “measurement” does not necessarily equate to an indicator but may

require qualitative/quantitative research

• Strategy testing questions relate to the map in its entirety and to the underlying strengths, weaknesses, opportunities and threats e.g. does it really matter to invest in networks to get our delivery partner to

be more innovative… We were not sure about this. How will we find something out?

e.g. we thought our programme would have 30% less financial resources? What happened? What will we do?

22

Interactivequestions

23

Strategic measurement-2• Strategy review questions related to objectives on the strategy map

and associated initiatives e.g. (among others) How are strategic initiatives developing (what has changed, why)? How

are they contributing to realising strategic objectives as hoped (what has changed / why)? What will we do?

• …e.g. we wanted to make sure staff were able to deal with requests one stop so we trained them for predictable demand. What have we learned?

How is the programme deployed by the PMO having an impact on Roma? To what extent are innovative actions of DPs for Roma developing properly? Why? What will we do?

• Frequency of strategic measurement depends on the nature of the KPQs: minimum quarterly for questions looking at progress of strategic

initiatives and possibly their impact on the strategic objectives minimum yearly basis for questions relating to the overall logic of

the strategy as visualised by the strategy map

Strategic measurement-3

24

E.g. Action research (probe, sense, act / plan, do , check ,act) responding to KPQs such as “What is getting developed, what is not getting developed, why and what are the implications?”

Traditional arsenal of social research (incl. theory based and counterfactual impact evaluation)

Interactivequestions

Programmme evaluation is one specific kind of measurement activity!

25

Strategic measurement and cascading• If relevant, questions and associated measurement ideas are

cascaded at the same time as the objectives they measure This means that cascaded objectives also have cascaded measurement

at unit level: staff think about KPQs at their level, just as they thought about objectives and engage in discussion with higher level

• A baseline is established when measurement is conducted for the first time discuss with relevant staff whether and how improvement can be

achieved and how the PMO will know if things are moving into the right direction

when using quantitative research, it entails ensuring that the direction of the desired trend is discussed

when using more qualitative research, the key characteristics of the desired situation should be discussed

in both cases we will want to answer “why” change has occured which is not something we can derive from descriptive info

26

/ quarterly

Does not question strategy as such

= strategy reviews are mainly interactive control systems! They

focus on learning!-what comes up from strategic

initiatives?-what comes up from operations

that they cannot handle?

Cascading to be performed again

27

/ quarterly


(could also be prompted by new threat or opportunity emerging from regular environmental scanning)

= strategy testing and adaptation meetings are also interactive control

systems, now questioningthe strategy as such

28

Control-1

• Control to improve?

Establish standards, in principle on the process however, there are

some considerations:

• sometimes its is not easy/possible to observe (complicated) processes (e.g.

a manager) and we would rather measure output

• or, monitoring the process , however tightly, does not mean the output will

necessary be satisfactory (e.g. we can observe a researcher, reading,

writing, discussing etc. but whether /how this leads to a good piece of

research in the end is not so certain) so we want to measure the output

anyway

• or it costs a great deal less to monitor the output rather than the process

R. Behn, Public Administration Review, 2003

R. Simons, Performance measurement and control systems, ch. 4,2000

29

Control-2• if one wants to constrain experimentation/innovation in the process

as much as possible by tightly controlling the process (e.g. where safety is an issue or where tinkering in only one part of the process can have serious consequences for downstream parts etc.), one cannot wait for the output to control

• when neither process nor output can easily be controlled, we need to resort to “input control”, usually in the form of people through the recuitment process (e.g. highly capable and integer individuals

but...*• “standards” are derived from production environments where they

are needed for interoperability / integration E.g. tolerances in terms of how much front-window opening of a car can

deviate from norm, before fitting the glass becomes a problem!

• in most services, the “standard” is continuously set by the users, with variation to be absorbed by staff in a co-creating process

if not, it may be best to automate with ICT (eg license plates)

*John Seddon

30

Operational measurement

• Operational reviews are:Diagnostic:

• By now it should be clear that we want to use systemic measures and statistical process charts

Interactive • Look for “signals” in diagnostic system and then look into

reasons behind these (interactive) To what extent are we achieving what we expect (what has

changed)? Why not? What to do?

• Inline experimentation initiatives to improve service How are process initiatives (not) improving KPIs? Why?

Tools in the hands of staff that do the work to self-regulate (decide what to do about it), NOT for managers to control them

31

/ quarterly


(could also be prompted by new threat or opportunity emerging from regular environmental scanning)

Operational reviews are mainly a diagnostic control system –focusing on

“control”- but, as it concerns continuous improvement, is also used interactively

32

Promote-1• To promote:

Convince others (politicians, journalists, citizens, ….) we are doing good job

• Performance info can be used to “legitimise”*: rationalise, justify, validate courses of (past/present/future) actions and decisions (incl. on budgets)

• It can be used to “reassure”* that government is doing what it is supposed to with taxpayers’s money (also called “transparency”)

• It can be used to “show compliance”* with regulations regarding performance management

• It can, by the above, unlock extra resources (e.g. earned autonomy, extra budgets, attract dedicated people,…) and hence improvement

Requires easily understood measures that the particular stakeholders care about

• Eg for DMV (dpt. for motor vehicles) more important to show how much you have to wait to get a check (compared to waiting perhaps for other services), than to show accident rates due to malfunctioning cars

R. Behn, Public Administration Review, 2003*M. De Jong et al., OECD Journal on Budgeting, 2013

Promote-2

33

Using performance information for “promotion”?

Trouble in measurement paradise?

34

Purpose Bias in terms of use

Budgeting/ planning

None: we want the information to be as accurate as possible to be as well-prepared as possible.

Learning None: we want the information to be as accurate as possible to learn

Motivation Managers want to “stretch” their expectations for staff, which leads staff to understate current performance. Which detracts from accuracy.

Control We want to know if standards are being met. This conflicts with the desire to motivate people to “stretch” as well as the desire to have an accurate picture of real (not within tolerance) performance levels.

Evaluation People have an interest to meet targets, if they are salient, in any way possible (also undesired ways). This detracts from the “accuracy” requirement. On the other hand, many factors beyond staff’s control contribute to / detract from the observed performance and should be adjusted for to accurately evaluate them.

Promotion The organisation has an interest in setting low expectations, that can surely be met, to maintain credibility. Conflicts again with accuracy.

R. Simons, Performance measurement and control systems, ch. 4,2000

Documents

Uses of performance information Benedict Wauters