Reforming Accountability Systems to Enhance Innovation

Professor Donald P. MoynihanRUC Sunrise Conference

A basic question…

How do we know performance systems make a difference or not?

Basic tension between practitioner needs and social science standards

Focus on one particular reform as example:◦Program Assessment Rating Tool

Tensions in Reform Research

Practitioner needs Social science constraints

We want new ideas, and we want them now

What are innovators doing?

Forward looking

It takes time to evaluate if reforms made a difference

What were the effects of reforms?

Backward looking

Implications

Practitioners rely on best practices that are promising but unproven

We accept claims about success too easily and uncritically

We do not study administrative changes with the same rigor as policy interventions

Implications

Value of reform is the eye of the beholder: we evaluate reform through personal, theoretical or political biases

There is no basis to defend the reform against the next wave of ideas

We continually flit back and forth between reforms that represent different values – are we making progress?

Taking social science seriously

Much of public management research has trappings of social science research, e.g. hypotheses, quantitative analysis

But what are big lessons about effects of reforms?

This is hard: ◦Reforms implemented governmmentwide rather

than in experimental fashion◦Lack clear measures of success

Need to be willing to look back

The Program Assessment Rating Tool

2005: An impressive innovation - winner Harvard Innovations in Government Award

2009: A political failure, ended with Bush administration

What was PART?

• Bush-era questionnaire used by Office of Management and Budget to rank programs from ineffective to effective

Four sections: program purpose and design, strategic planning, program management, and program results/accountability

Burden of proof on agenciesAlmost all federal programs evaluated

Goals of PART

Foster performance improvement, performance budgeting, accountability Bush, 2001: “After eight years of experience

[since the passage of GPRA], progress toward the use of performance information for program management has been discouraging…Performance measures are insufficiently used to monitor and reward staff, or to hold program managers accountable”

Why was the study of PART different

What were its actual effects?Lots of social science research on this – why?Better data:

◦Agency ideology measures (Clinton and Lewis 2008)◦PART scores – available online◦GAO surveys/Lewis survey

More diligent about asserting and testing causal claims (and acknowledging gaps/limitations)

Drew better from existing social science theory

Costs of PART

Mostly qualitative studies:◦Time and effort of agency and budget staff

(Moynihan 2008; Redburn and Newcomer 2008; Lavertu, Lewis & Moynihan)

◦Opportunity cost of this time and effort – policymaking attention of budget office (White 2012)

Did PART increase performance information use of managers?

Not really (Moynihan and Lavertu 2012)Survey asked if managers were involved in

PART – instrument for measuring its effectOn aggregate, those who said they were

involved were◦No more likely to use data for most purposeful

means (resource use, setting priorities to manage employees)

◦Much more likely to use data for passive means (setting goals and refining measures)

Did PART alter budgets?

Marginal effects within executive branch (Gilmour and Lewis 2006; Moynihan 2008)

No systematic effect in legislative branch (Heinrich 2012; Frisco and Stalebrink 2008)

Was PART political?

Designed to be good government, politically neutral reform (Dull 2006), and qualitative studies do not report overt partisanship, but…

More liberal agencies and programs get lower scores (Gallo and Lewis 2012; Gilmour and Lewis 2006)

PART scores only related to President’s budget proposals for liberal programs (Gilmour and Lewis 2006)

Conservative members more supportive of PART (Stalebrink and Frisco 2011)

Did politics affect response to PART?

Liberal agencies, though smaller, had significantly higher PARTs completed (Lavertu, Lewis and Moynihan 2012)

Two types of effort:◦Observable: self-reported effort in completing

PART – higher for managers in liberal agencies (Lavertu, Lewis and Moynihan 2012),

◦Discretionary: performance information use – lower for managers in liberal agencies (Lavertu and Moynihan 2012)

How did different types of programs do under PART?

The claim: some program types are at a disadvantage because unsuited for PART

Programs associated with resource redistribution (Greitens and Joaquin 2010), and environmental protection (Thomas and Fumia 2011) receive lower scores

Block grant programs, which tend to have diverse goals, score lower (Gallo and Lewis 2012), even controlling for agency ideology

Regulatory programs demanded greater effort (Lavertu, Lewis and Moynihan 2012)

Research & development programs score higher (Gallo and Lewis 2012) and required less effort (Lavertu, Lewis and Moynihan 2012), contrary to claims

Results may be related to political ideology if controls for ideology are not included

What else did we learn in the process?

Predictors of performance information use (Moynihan and Lavertu 2012):◦Leadership commitment ◦Goal clarity ◦Ability to link data to action◦Routine discussions of performance data with

supervisors/peers

What else did we learn in the process?

PART data provided measures of goal ambiguity

Goal ambiguity associated with lower scores (Thomas and Fumia 2011; Rainey & Jung 2011)

Ambiguity associated with lower agency discretion (Clinton et al. 2012)

Ambiguity associated with lower budgets (Heinrich 2012)

General lessons from PART

Governmentwide routines that prioritize data collection and dissemination do little for use or to spur innovation

Created dialogue centered on measures and performance, not on improvement

Political ideology matters to the implementation of reform, and how it is received

Credible commitment matters

Post-PART world

Obama administration abandoned PARTGPRA Modernization Act of 2010

◦Legislatively led◦Creating leadership commitment at multiple

levels (high priority goals, COOs, PIOs, goal leaders)

◦Focus on routines of useReforms make sense – but too early to tell

Continuing challenge: how to make use of performance data

Create learning routines:◦One liner: If performance management is to lead to

innovation, we must invest at least the same resources on creating routines of performance information use as we do on routines of performance information creation and dissemination.

Create learning cultures, rather than accountability cultures

Find actionable data – e.g. benchmarks, outliersIncorporate variety of analytical skills (e.g.

evaluators)

Conclusion

Had we known then what we know now, is PART worth implementing?

How can social science provide more timely relevance to actual public servants?

References

Clinton, Joshua D. and David E. Lewis. 2008. Expert Opinion, Agency Characteristics, and Agency Preferences Political Analysis 16:3-20.

Clinton, Joshua D., et al.  "Separated Powers in the United States: The Ideology of Agencies, Presidents and Congress." American Journal of Political Science 56(2)(2012):341-354.

Frisco, Velda and Odd J. Stalebrink. 2008. Congressional Use of the Program Assessment Rating Tool. Public Budgeting and Finance 28: 1-19.

 Gallo, Nick and David E. Lewis. 2012. The Consequences of Presidential Patronage for Federal Agency Performance Journal of Public Administration Research and Theory. 22(2): 195-217.

  Gilmour, John B., and David E. Lewis. 2006a. Assessing performance budgeting at OMB: The influence of politics, performance, and program size. Journal of Public Administration Research and Theory 16:169-86.

 --------. 2006b. Does Performance Budgeting Work? An Examination of the Office of Management and Budget’s PART Scores. Public Administration Review 66: 742-52

 --------. 2006c. Political appointees and the competence of Federal Program Management. American Politics Research 34: 22-50.

Greitens, Thomas J., and M. Ernita Joaquin. 2010. Policy Typology and Performance Measurement: Results from the Program Assessment Rating Tool (PART). Public Performance & Management Review 33: 555–70

Heinrich, Carolyn J. 2012. How Credible is the Evidence, and Does It Matter? An Analysis of the Program Assessment Rating Tool. Public Administration Review 72(1): 123-134.

Lavertu, Stéphane and Donald P. Moynihan. 2012. Agency Political Ideology and Reform Implementation: Performance Management in the Bush Administration. Forthcoming at Journal of Public Administration Research and Theory

 Moynihan, Donald P. and Stéphane Lavertu. 2012. “Does Involvement in Performance Reforms Encourage Performance Information Use? Evaluating GPRA and PART.” Public Administration Review 7(4): 592-602

Lavertu, Stephane, David Lewis and Donald Moynihan. 2012. Administrative Reform, Ideology, and Bureaucratic Effort: Performance Management in the Bush Era. Paper presented at APPAM Annual meeting.

Rainey, Hal and Chan Su Jung. 2011. Extending goal ambiguity research in government: from organizational goal ambiguity to program goal ambiguity in Richard M. Walker, George A. Boyne and Gene A. Brewer (eds) Public Management and Performance. Cambridge University Press. 

Stalebrink, Odd and Velda Frisco. 2011. PART in Retrospect: An Examination of Legislator’s Attitudes toward PART. Public Budgeting and Finance 31: 1-21.

 Thomas, Craig and Danielle Fumia. 2011. The Effect of Program Size and Goal Ambiguity on Performance: An Analysis of PART Assessments for 165 Environmental Programs. Paper presented at the National Public Management Research Conference, Syracuse University, June 2-4, 2011.

For more research on performance information use: http://www.lafollette.wisc.edu/publicservice/performance/index.html