Evaluation and Monitoring Methodologies Strengthening the Legislature – Challenges and Techniques...

Evaluation and Monitoring Methodologies

Strengthening the Legislature – Challenges and Techniques

K. Scott Hubli, NDI

Overview

General Comments on Monitoring andEvaluation

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Practical Tips and Considerations

General Comments onMonitoring and Evaluation

Evaluation (and Baseline Assessments) --Use to develop program design; use for major

course corrections --More costly and less frequent than monitoring

(every two to three years) --Typically done at the beginning and the end of a

program, but often also done after a major change in the political landscape (e.g., regime change, ethnic conflict settlement, etc.)

--Used for accountability to partners, donors, stakeholders, not for ongoing project management

General Comments onMonitoring and Evaluation

Performance Monitoring--Ongoing monitoring; used to manage performance

of implementation--Track changes (but less analysis)-- Informed by baseline assessment and, if well

designed, it can reduce future evaluation costs -- May indicate a need for a evaluation or updated

baseline-- Focus on low cost, regular data collection

(workshop evaluations, information available from parliament, regular focus groups, etc.)

General Comments onMonitoring and Evaluation

Always distinguish among: --Inputs (e.g., consultants, computers, etc.)

--Outputs (e.g., 40 people trained in a workshop onoversight techniques)

--Outcomes (e.g., increased knowledge of oversight investigation techniques)

--Objectives (e.g., increased oversight hearings)

--Goals (e.g., increased government accountability)

How are legislative strengthening programs different from other programs with respect to monitoring and evaluation?

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Legislatures are highly complex institutions --They involve multiple actors seeking to achieve multiple goals simultaneously

--Where possible, disaggregate data (by gender, party, region, etc.)

--Identify clear goals and targeted groups; watch for unintended consequences

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Long-term goals, short-term programs -- Resist the tendency to monitor outputs rather than progress in achieving desired outcomes, objectives and goals

-- Find ways to measure small changes in large goals; or outcomes that can be affected with the

project time frame

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Programs focus on process, not outputs--Example: number of laws passed--Emphasize qualitative over quantitative information--Use detailed process descriptions in establishing

baselines--Use monitoring and evaluation to help strengthen this process and to teach results-based management, where possible (“Monitoring and evaluation should be managed as joint exercises with development partners.”)

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Monitoring and evaluation is often highly political

--Involving partners can sometimes further politicize evaluation and monitoring; use caution and judgment--Can be hard to get necessary information--Politics may cause people to be less than fully honest--Results can be used as a political weapon

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Legislatures have natural cycles --Elections, post-election learning curves, legislative floor periods, recesses, budget processes, etc.--Example: constituency relations --Expect uneven development in performance monitoring, but try to attribute fluctuations in data--Time evaluations carefully – look for “normal” periods

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Many intervening variables --Economic conditions, geopolitical developments, ethnic conflict, death of a key politician, etc.

--No substitute for nuanced political analysis

--Measure outcomes, objectives, goals – not just outcomes; this can help identify these intervening variables

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Perceptions matter--Importance of qualitative over quantitative indicators

--Use of focus groups, opinion polls, etc.

--Even anecdotal evidence is useful if it captures a political mood or issue

Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs

Difficulty of comparative benchmarking--Only one national legislature; cross-country

comparisons are of limited utility

--Comparisons across time more important; use ofthorough baselines

--Implications on setting goals and targets – use ofreasonable/consensus expectations

What are some practical strategies for dealing withthese unique aspects of monitoring and evaluating legislative strengtheningprograms?

Practical Tips & Considerations

General Issues--Be pragmatic in designing an evaluation or monitoring plan; tie

evaluation and monitoring to the purpose or objectives. Avoid evaluation for evaluation’s sake. Consider:

--resource availability for evaluation--novelty of the program--confidence in program design or

implementation--needs of funder

--Budget sufficient resources – (Costs for legislative strengthening evaluation may exceed those for other program types – soft assistance, new field, etc.)

Practical Tips & Considerations

Issues in Doing a Baseline--Limit scope to allow for detailed coverage of program areas

--Protect against biases of person(s) doing the baseline by: --Using teams

--Using clear, detailed terms of reference

--Incorporating documentary evidence

--Seeking consistency in future assessments

Practical Tips & Considerations

Issues in Doing a Baseline (cont.) --Pick timing carefully; describe any special circumstances--Prepare carefully for baseline assessment team

--Cover the range of stakeholders

--Get out of the capital

--Consider focus groups or creative methods for documenting perceptions and processes (e.g., a sample of 10 legislators to track periodically every 3 years)--Pay attention to protocol; build good will.

Practical Tips & Considerations

Using outside evaluators --Outside evaluators can not only provide objectivity but also insulation from the political consequences

of an evaluation--Combine multiple backgrounds (academic or legislative strengthening specialists and MPs or staff from similar systems)--Recognize value of “time in the trenches”--Designate a lead person with responsibility for producing the document--Get a sufficient time commitment

Practical Tips & Considerations

Issues in Performance Monitoring--Draw on baseline and prior evaluations

--Design performance monitoring plan up front; adjust it as project evolves:

--Imposes discipline; keeps program on track

--Provides clarity of expectations to partners

--Keep it current, modify as needed

--Make these changes explicit

Practical Tips & Considerations

Issues in Performance Monitoring (cont.)--Tie to likely performance issues--Draw on low-cost existing information sources; may be more quantitative, with less analysis--May focus on outcome level, rather than objective

or goal level--Consider quarterly or semi-annual monitoring--Expect, but explain, fluctuations--When you can’t explain repeated fluctuations, consider updating a baseline to try to identify issues--Often done, in part, by those implementing program

Final Thoughts

Be creative: legislative strengthening is anart, not a science

Be willing to accept criticism; fight structural bias for “spinning” results

Share lessons learned, both internally and externally