The use of evidence in developing countries:

(South African) experiences and challenges

Thilde Stevens October 2010

Key message of this presentation The South African context and its approach to evaluation Experiences: Policy making in SA Experiences: Supply of and demand for evidence Experiences: Accountability and learning Experiences: Evaluation practice Experiences: The shift to an outcomes-based approach Challenges: Human and Financial Resources Challenges: Decision makers susceptibility to “weaker” forms

of evidence Challenges: Data collection and access Challenges: The relationship between planning and

evaluation Conclusion

Contents of this presentation

The use of evidence in dynamic political environments is unpredictable and contingent on a wide range of context-specific factors.

It is not a linear process where a set of research findings will necessarily translate into policy impact and feed into policymaker’s decisions and practical programmes

Taking these factors into consideration is important and needs a strategic and subtle understanding of each policymaking environment.

This presentation is based on the presenter’s own experiences, particularly in the Social Protection context.

It is NOT an official view.

Key message of this presentation

The rainbow nation is now 16 years into democracy and has emerged as a stable an exciting place.

Our sense in SA of being exceptional has passed and we now realise that we are one of a pack of developing countries that face similar problems.

Inequality, poverty, low-economic growth, HIV/AIDS and unemployment are the most pressing.

SA retains the ability to capture the global imagination and to unite around specific issues, sometimes surprising itself in the process (e.g. World Cup)

The South African context and its approach to evaluation 1

Important to remember that SA has had to re-invent itself since 1994 from a governance and public administration perspective, so in many ways it is a young country with immature systems that are still evolving and developing.

Since the early days of democracy, SA has recognised the importance of “M&E”. Mostly donor driven.

In practice this has really meant “M” with “E” very neglected and under-addressed.

Government Departments developed and implemented M&E on and ad hoc basis with no guidelines.

The GWM&E policy has attracted a lot of favourable coverage (and is a useful framework) but it has not been systematically implemented.

The South African context and its approach to evaluation 2

Policy making in SA is a complex process, highly politicised and very opaque.

Decision making on occasion is contrary to all evidence and is often driven by:◦ Electoral considerations based on the views of the supporters(e.g.

the extension of the CSG to children up to the age of 17 which should have focused on pregnant mothers rather than older children)

◦ External influences such as socio-economic and cultural beliefs (e.g. Aids policies based on the believe the Aids is caused by poverty)

◦ Financial Resources (e.g. If there is no money available or other priorities are more pressing, National Treasury will reject the best policy proposals)

◦ Ideological or personal factors(e.g. The ideology that formal housing was promised to people therefore the government keeps on building poor quality houses rather that improving shacks)

Experiences: Policy making in SA 1

Political leaders in SA are the decision makers on new policies or policy changes, but they rely heavily on a number of advisors who are the opinion leaders and influence the decision-making process.

Despite all of the above complexities, evidence does matter to a certain extent in the SA policy making process.

Evidence will only be influential in the SA policy making process if:◦ The demand comes from the policy makers and they are

involved in the process◦ The evidence is of high quality and the topic relevant to the

problem◦ The evidence provides solutions to the problem ◦ It can be advocated in simple and convincing terms through

the appropriate channels necessary to influence policy.◦ The timing of the evidence is right.

Experiences: Policy making in SA 2

Researches have to ensure that they use a specific set of appropriate lobbying and advocacy skills that is necessary if evidence in the South African context is to be persuasive.

The evidence needs clear interpretation in simple terms, translation and advocacy.

All of the above confirms what Edward Clay said in 1984 on the policy process: “ The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies”

Experiences: Policy making in SA 3

There are strong differences between and tensions around the supply and demand of evidence through public service systems in SA

This reflects the complexity of the governance system which distinguishes between three spheres of government: national (policy and M&E), provincial ( implementing social services incl. health; education and housing), local (providing basic services) and implementing partners (NGOs)

Capacity differs, being strongest nationally and weakest locally.

Demand for evidence is strongest furthest from the point of delivery (e.g. at NT) and supply is weakest at the site of delivery (local sites).

Experiences: Supply of and demand for evidence 1

These systemic inconsistencies cause many problems.◦Supply of evidence from local and provincial

levels is often done due to meet compliance regulations and not because they see any value in gathering the evidence

◦This results often in evidence that is unreliable, scattered and of poor quality

◦ Evidence gathered from Local and Provincial levels is rarely used by them to measure their own performance.

Experiences: Supply of and demand for evidence 2

M&E is generally accepted as having two intentions: supporting accountability and facilitating learning, which should lead to improved performance and better service delivery.

In SA the unfortunate reality is that the accountability driver, shown in the focus on monitoring, is much stronger than the learning driver.

Learning is not properly integrated into institutional cultures and is widely recognised as an area of weakness.

Extensive effort is put into gathering and formatting evidence – in the form of performance monitoring reports – but this evidence is barely used, except by the institutions responsible for financial accountability.

Experiences: accountability and learning

As noted above, the focus is on monitoring rather than evaluation – there are highly developed policy and practice guidelines on the former and none on the latter.

As a result, evaluation is not a requirement and is not widely practiced in the public service, except in an ad hoc and sporadic way.

Where evaluations are undertaken, these are not always rigorous and little use is made of theory-based approaches to evaluation.

For example, program logics and theories of change are rarely used when evaluations are planned and implemented.

The reliance on indicator-based performance assessment is problematic, especially when the indicators used are themselves so inadequate and do not extend meaningfully beyond outputs.

Experiences: evaluation practice

Despite its neglect in government circles, evaluation is thriving in the academic and non-state sectors.

The country has a number of prominent researchers working in evaluation but from a theoretical and fundamental perspective.

Much of the evaluation work being done in SA is not designed to be policy friendly and doesn’t explicitly seek to address policy issues.

Experiences: evaluation practice

Since the Zuma administration took office, a new component has been created in the Presidency which has recently taken shape as the Department of Performance Monitoring and Evaluation.

This Department is now driving a shift to an outcomes-based approach to M&E. 12 Outcomes have been identified for SA.

In response to its requirements that the intended outcomes of interventions (i.e. Programmes) be explicitly addressed, the DSD is seeking to develop its own O-B M&E model.

There appears to be little appreciation for the complexity of the task and the enormous challenges this poses.

The limitations of our development and redistributive policies (many of which are loud on intended impact, processes and outputs but silent on outcomes) are not being acknowledged.

Experiences: The shift to an outcomes-based approach

SA has been hard hit by the global recession and public finances are under serious pressure.

Rigorous impact evaluations and other types of evaluations, are expensive, especially if it is going to include the necessary stakeholders and reflect the concerns of various interest groups.

Evaluation is also very demanding from a human resource perspective, with technical skills hard to access and other strategic coordinating and integrating skills scarce and unavailable.

Where the value and importance of evidence generated from rigorous evaluation practice is not always appreciated, it is hard to motivate for the allocation of the resources required.

Challenges: Human and Financial Resources

In a dynamic and relatively inexperienced democracy such as SA’s, decision makers are easily influenced by less strong evidence and are often immune to or not convinced by rigour.

Images, personal representations and emotional forms of evidence can be more persuasive.

Some politicians and senior members of government (policy makers) have limited knowledge and understanding about rigorous evaluations, the processes, time and resources they require and the value that rigorous evidence can provide them with when making policy decisions.

Therefore it is easier sometimes for policymakers to revert to less complex and weaker forms of evidence

Challenges: Decision makers’ susceptibility to “weaker” forms of evidence

Data needs to be collected at the points at which interventions are delivered.

However, these are often remote and underserviced, making the collection of accurate and reliable data difficult with a particularly serious set of challenges that are expensive, time-consuming and technically difficult to overcome.

Data collection for evaluation is often outsourced and is usually of a high enough standard but is consequently inconsistent with patchy administrative data sets.

This leads to serious comparability troubles and “un-usability”.

Challenges: Data collection and access

There is little or no links between the strategic planning processes and M&E in Government at a Provincial and National level.

Little of the planning that is done is undertaken using reliable baseline data and targets are therefore often not credible (when they are set at all).

• There is no linkage between what gets measured at local and Provincial levels (policy implementation) and National level(policy making)

• This makes the gathering and use of evidence for policy making very difficult

Challenges: The relationship between planning and evaluation

South Africa has still a long way to go the recognise the value of using rigorous evidence in the policy making process

SA is moving in the right direction with the outcomes based approach recently introduced by the President.

There are a number of individuals in government that are already convinced and are the champions for promoting the collection and use of good and rigorous evidence in Government.

Training, training and training.

Conclusion

Thank you.