Results Based Management WorkshopSetting context for Leadership Performance Measurement

Ging WongGing WongAndrew GrahamAndrew Graham

School of Policy Studies

ObjectiveSet the Project in context of Results Based Management developments in Canada and elsewhereProvide an overview of the linkages of results, accountability, performance measurement and riskSuggest ways that the Project can uniquely support the effective transfer of relevant and useful elements of RBM

School of Policy Studies

3

ACCOUNTABILITYACCOUNTABILITY

RISKRISKPERFORMANCEPERFORMANCEMEASUREMENTMEASUREMENT

School of Policy Studies

4

Flexibility and ControlFlexibility and Control

Tension in management thinking and practice between structure/accountability and initiative/flexibility.Challenge is how to create a balance that recognizes public sector accountabilities that create very high levels of expectation with the need to get the job done, i.e. meet the objectives of the program.

School of Policy Studies

5

Flexibility and ControlFlexibility and Control

“While flexibility is becoming essential, the public is not willing to forsake accountability to achieve it. Confronted with antiquated systems of governance, managers must exercise leadership, not to in an ‘anything

goes’ manner. In short mangers face the challenge of developing an alternative form of accountability that allows for

greater flexibility of action.”

Feldman and Khademian

School of Policy Studies

6

Flexibility and ControlFlexibility and Control

Public Sector accountability is complex and variedIt is changing with the role of public servants changed, becoming more involvedUnderstanding accountability in the public sector means understanding that there is a continuum of accountabilities: from operational to policy to political

School of Policy Studies

7

Flexibility and ControlFlexibility and Control

Reality that there are significant ‘gear slippages’ in public sector accountability in which the day-to-day at times takes on a much greater significance than so-called higher level accountabilities for policy outcomesIn the public sector it is more likely that a failed program can run really well and a successful one can founder on details

School of Policy Studies

8

The basic notion of accountability is that those acting on behalf of another

person or group report back to the person or group or are responsible to them in some

way. - Owen E. Hughes, Public Management and Administration

Defining the Concept of Defining the Concept of Accountability….Accountability….

School of Policy Studies

9

Defining the Concept of Defining the Concept of Accountability….Accountability….

“Accountability is about evaluating performance, meeting legitimate standards, fulfilling legitimate commitments, and

holding responsible those who fail to meet the standards. The right to judge government performance flows naturally from the role of citizen, as does the right to sanction those

who fail to meet the standards.”

Janice Gross Stein, The Cult of Efficiency

School of Policy Studies

10

Defining the Concept of Defining the Concept of Accountability….Accountability….•From a management perspective, accountability runs the full gambit of activities, from the specific to the general

•In the public sector, this means from detailed administration to accountability for program outcomes

•At no point is this either easy or simple.

•There are no fire walls between the detailed and the general.

School of Policy Studies

11

The Accountability ContinuumThe Accountability Continuum

“ Public mangers do not have the luxury of separating out expectations for flexible leadership from demands for

strict accountability in the form of structures or guarantees that

check and limit management action. They must grapple with

both.”

- Feldman and Khadamian

School of Policy Studies

12

The Accountability ContinuumThe Accountability ContinuumINTERNAL ACCOUNTABILITY EXTERNAL ACCOUNTABILITY

OPERATIONS

MANAGEM

ENT

COMPLIANCE

OUTPUTS

OUTCOMES

MANAGEMENT PROGRAMCOEFFICENTS COEFFICIENTS

DATA INFORMATION KNOWLEDGE

School of Policy Studies

13

Sustained accountability Sustained accountability through results based through results based managementmanagement

Key trend in governments in CanadaFederal government and provinces have adopted various methodologies

School of Policy Studies

Based on simple model of organizational learning and adaptation

14

School of Policy Studies

RBM leads to consideration of key elements of a management framework

Effective statement of goals and outcomesPerformance measurement systems

15

School of Policy Studies

RBM leads to consideration of key elements of a management framework

The role of monitoring and control in execution

16

School of Policy Studies

RBM leads to consideration of key elements of a management framework

The role of evaluation in determining outcomes and the validity of the of the outcome expectationsMeans to develop organizational learning across complex systems in both implementing and adapting policies (case studies, etc)

17

School of Policy Studies

18

desirable properties of performance measuresPerformance measures have a number of desirable properties. Most of these can be dispensed with fairly quickly to focus on three key problems:relevance and balancePerformance measures such as those embodied in the IDTs are undoubtedly relevant. There are, however, two problems of balance. First is that the measures may not span an agency’s portfolio. For example, the health targets in DFID’s PSA exclude health activities outside of the top ten recipients. And the targets as a whole do not apply to activities in middle-income countries. The second problem of balance is that, for reasons discussed below, it is better to have a set of indicators spanning the range of the log-frame from inputs to outcomes.measures known, understood and trustedPerformance measures used in development are usually clearly defined and well understood, at least by those who work directly with them. There has been some tension between top-down setting of targets and a more participatory approach. This has also been a factor behind the promotion of the Millennium Development Goals (MDGs) as rivals to the IDTs. Data quality can be more problematic. The IDTs include maternal mortality, for which data are notoriously unreliable. The measure was included in DFID’s first PSA, but when it came to reporting DFID told the Treasury that data were unavailable. Country coverage for most indicators is patchy. But the more serious problem is the timeliness of the data, discussed below.affected and attributableChanges in performance measures should be affected by the activities of the organisation and the extent of the effect measurable (attributable). However, attribution becomes harder to establish as we move through the log-frame from inputs to outcomes. As we saw above, USAID has abandoned claiming that changes in economic outcomes can be attributed to its work.achievableTargets should be achievable, but not too easily. Most analyses suggest that the IDTs will not be met, although there is regional variation in performance. But this is not the same as saying that they could not be met if aid and government resources were better focused toward meeting them.linked to existing management systemsOrganisations will already have in place management information systems. The rise of results-based measures was in part a response to the fact that existing systems focused on inputs and internal activities, such as spending and staffing, rather than achieving outcomes. But new results-oriented systems should not be separate or parallel to these existing systems. There are two further problems here: aggregation (can the various measures be added up to summarise performance for regions, sectors and the agency as a whole?) and alignment (do the data tell us about impact on the outcomes of interest?)

School of Policy Studies

19

three key problemsThe above discussion identifies three problem areas. These problems are less severe (though present) when using RBM at the project level, becoming more intense as we attempt to measure results at country level or for the whole portfolio.data availabilityData become available only with a time lag, this being particularly so for developmental outcome measures like infant mortality and income-poverty. DFID’s PSA covering 2001-2004 includes amongst its targets 'a decrease in the average under-5 mortality rate from 132 per 1,000 live births in 1997 to 103 on the basis of data available in 2004'. Implicit here is the recognition that data available in 2004 will probably be for 2001, namely the start of the period of the current PSA. How can results of current strategies be judged on the basis of results achieved before the strategy was put in place?attributionIt is very difficult to attribute changes in developmental outcomes to the activities of an individual agency. This problem is encountered where agencies work together, as in sector programmes or providing budget support, or for agency performance at the country level. One agency’s support to public sector reform in an African country (providing the funds to pay for retrenchment packages) had as the outcome indicator: maternal mortality. It is difficult to imagine what sensible use could have been made of maternal mortality data (even with no time lag) to judge the performance of this programme.integration with existing management information systemsAll donor agencies have some sort of monitoring and evaluation system at the project and programme level which should provide a basis for both feedback at the project level and 'feed-up' to management. But how do these systems tackle the problems of aggregation and alignment