Evaluating system change in inclusive innovation systems

Evaluating system change in inclusive

innovation systems

Objectives of the Seminar (as per brief)

• Deepen our knowledge and understanding of the various impacts resulting from innovation practices on the nation’s inclusive development efforts

• Strategies and how we may be able to measure these contributions.

• Criteria to be considered, the pros and cons of each methodology as well as the ethical, logistical and policy considerations.

• Does innovation sufficiently promote and support South Africa’s developmental aspirations or does it act as a barrier in some areas? How? Why?

• Gain fresh policy-relevant insights on how we can better manage innovation in order to ensure that it is both relevant to the society and effectively addresses the citizens’ needs.

Agenda

• Introduction and problem statement• Evaluation and the logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?

The big questions…..

The research problem addressed in this article is the development of an analytical framework for the identification and analysis of inclusive innovation opportunities, the practical implementation and evaluation of that

The research questions that we aim to answer in the article are:

• Inclusive growth is key for the development and political stability of our country and our continent - how can inclusive innovation support help us achieve this?

• What will an analytical framework look like to analyse / explore opportunities for inclusive innovation?

• How can we discover and find ways to operationalise inclusive innovation opportunities

• How can we best support learning between inclusive innovation programmes and universities?

• How can we measure and evaluate impact of inclusive innovation projects?

Overarching framework

Mechanisms to identify and locate systems weaknesses / evaluate system

interventions

Frameworks to design policy mechanisms and facilitate policy

learning

PUB

HESPRIV

1

2

3

Analysis and operationalising

support and innovation

Identifying imperfections,

Measuring, Evaluating

Learning, planning and

designing

4 Theories of policy change

Com

Defining various levels of focus for a programme to support, explain and facilitate effective implementation of policy

6

• The research problem addressed on this level is the development of an analytical framework for the identification and analysis of innovation opportunities and its practical implementation

• How can we develop a more effective system to ensure the three levels between implementation, to identify imperfections and to support policy-learning is well integrated?

• Understanding: Analysis for understanding systems weaknesses and failures: cost drivers, risk, opportunities, sustainability, resilience, competitive advantage, localisation issues, strength of the supply base, routes to market

• Measuring: What are the indicators used to track the evaluation of systems change?

• Building capacity to learn, plan, coordinate, manage, implement the innovation strategy

1

2

3

4

Agenda

• Introduction and problem statement• The logic of interventions• Systems thinking in evaluation of innovation• Progress made in analysing innovation systems• Challenges in evaluating systems change• A practical proposal• The way forward?

Reason to evaluate initiatives

1. Provide accountability to funder for expected benefits 2. Gain a deeper understanding of the change theory to

increase practical utility; 3. Assist in making adjustments in funding strategy to better

align evolving theory and conditions; 4. Support projects as they adjust their activities; and/or 5. Determine how to sustain the beneficial aspects beyond

period of funding. 6. Help project leaders put their work in perspective within the

bigger picture

Source: Kellogg foundation

Evaluation purposes

A. Programme monitoring andProcess evaluation for the sake ofimproving programmes(Programme staff/director)

LEARNING

ACCOUNTABILITY

FORMATIVE SUMMATIVE

C. Evaluation studies for the sake of knowledge creationas part of contributing to scholarship(Scholars)

B. Monitoring and EvaluationStudies for the sake ofquality enhancements(External regulatory agencies/Compliance requirements)

D. Evaluations (ROI’s/ value forMoney/ cost-benefit) studies toInform decision-making - Continued funding/ upscaling)(Funders/ Trustees)

Source: Mouton

Evaluation model

Intervention Life Cycle

Evaluation Studies

Intervention outcomes/

effects

Outcome evaluation/

Impactassessment/(Summative)

Conceptualisationand design of

programme

ClarificatoryEvaluation/ EvaluabilityAssessment

First (pilot) version(s)

of programmeimplemented

Processevaluation

(Interactive/Formative)

Matureversion of

programmeimplemented

ProgrammeMonitoring(Formative/Summative)

Source: Mouton

Units (levels) of evaluation

Systems and sectors

Institutions

Corporate level Divisional level (departments)

Programmes and projects

Individuals

SYSTEMIC REVIEWS(Curriculum 2005 review)

INSTITUTIONAL AUDITS

DEPARTMENTAL REVIEWS

PROGRAMME EVALUATIONS

PERFORMANCE APPRAISALS

Level of evaluation Type of study

Initiatives (i.e. JIPSA) (PROGRAMME) EVALUATIONS

Source: Mouton

Separate issues

• Learning to do it better

• Systems performance

• Evaluation of policy

• Understanding of systems

Agenda


Kellogg foundation’s view

• “We have found that traditional evaluation approaches have not been as useful as we would like”

• Program staff and grantees have told us that evaluation misses both “the big picture” and the subtle changes in relationships, resource allocations, and roles that lead to the bigger changes

• In short, evaluation has not taken into account the complexity of social change efforts and the dynamic nature of the communities and systems in which we work

• Initiative leaders and evaluators are concerned not only with the components of an initiative but also with the patterns of change of the systems within or affected by the initiative

• It is the change over time and across locations and context that is of special importance in an initiative evaluation

Nature of human systems

• Living organism, has identify, memory, patterns of behaviour, feedback and complex behaviour, life stages

• Complex systems have non-linear behaviour – chaos theory - convergence, oscillation, chaos and then return to equilibrium

• Evolving all the time – works to connect, configure, reconfigure and, transform – elements added all the time

• Entropy, human system, struggle to survive but beyond a certain tipping point – it collapses if starved

• System dynamics a useful tool – vicious and virtuous circles

Morgan, P., (2005). The idea and practice of systems thinking and their relevance for capacity development

Nature of human systems

• Emergence– Not nature of parts but as a whole – the outcome of

overall systems behaviours and synergies– Comes from dynamism of interrelationships

• Learning as a way to improve capacity – learning is necessary for human systems to survive

• Interrelationships – systems are patterns of interrelationships– Process – focus on the patterns– Legitimacy – needs some sort of value structure– The role of values, rules - institutions


How systems thinking deals with complexity

• By focusing attention on a higher level of aggregation than the individual components or parts of the system

• By searching for patterns or order within complex situations that would allow people to understand the systems dynamics better.

• This understanding would hopefully lead, in turn, to better choices regarding the type and nature of any interventions undertaken to improve system capacity and performance

• By encouraging higher levels of self-organization within the system

• By changing the structure of the system in order to increase its performance


Systems thinking in evaluation – not a new proposition

• This particular approach which derives from earlier engineering systems models looks narrowly at the transformation of inputs into outputs in support of specific goals and objectives

• The most famous application of the ‘hard’ systems approach is that of the logical framework analysis approach introduced to development cooperation in the early 1970s and still used by most international development agencies

• The focus here is on things or parts rather than the interrelationships amongst these elements.

Logic models: Focus more on parts than interrelationships

Agenda


The systems of innovation approach

22

Defining “innovation” - Schumpeter

The systems of innovation approach provides a useful analytical framework for strategy development

• The systems of innovation approach is not a theory, it provides an analytical framework for the development of a regional innovation strategy

• Therefore to analyse regional innovation systems it is also important to integrate substantive theoretical elements such as: – evolutionary economic theory– regional science– the industrial district concept– the theorizing of rationalization strategies – and the governance concept

A useful analytical

framework

Substantive theoretical elements

The goal of an innovation system

24

Defining a “system”A system could be defined as: “a set of interrelated components working together towards some common goal or purpose” (Blanchard and Fabrycky, 1998).

Understanding the term “system of innovation”Considering the definitions of innovation and systems, it can be concluded that innovation systems consist of interrelated components that work together towards a common goal or purpose. The goal of the innovation system could be defined as:

The goal of an innovation system is to develop and diffuse innovations (Edquist, 1997)

Unpacking the component, attributes and relationships

25

Elements Generic systems definition (Blanchard and Fabrycky, 1998)

Innovation system (Carlsson, 2002)

Components Operating parts of the system and consists of input, process and output

Organisations e.g. firms, government departments, research councils, universities

Attributes Properties of the components which characterise the system

Institutional environment - the character and properties of the organisations e.g. absorptive capacity, R&D capacity

Relationships The links between components and attributes Linkages between organisations

• Interaction is central to the process of innovation, i.e. interaction between actors such as firms, universities, intermediaries. Central to this concept are cooperation and interactive learning (Lundvall, 1992).

• A second assumption is that institutions are crucial to economic behaviour and performance (Smith, 1997). - Legal (e.g. regulation and law) and customarily institutions

• Third, evolutionary processes, play an important role. They generate variety, select across that variety, and produce feedback from the selection process to variation creation (Hauknes and Nordgren, 1999).

• In all these basic elements, systemic imperfections can occur if the combination of mechanisms is not functioning efficiently

Copied from Woolthuis, R.K. et al, 2005. A system failure framework for innovation policy design, Technovation, Volume 25, Issue 6

Categorisations of system failures

Malerba (1998) Keith Smith (1998) Carlsson and Jacobs (1997)

Woolthuis et al (2005)

Learning failure Infrastructure provision and investment

Infrastructural

Exploration-exploitation variety selection trade-offs

Institutional failures Institutional failures

Institutional (Hard and Soft)

Appropriability traps Lock-in failures Network failures Interaction failures (Weak network, Strong network)

Dynamic complementarities failure

Transition failures Capabilities failure

Haukness J., Nordgren L., Economic rationales of government involvement in innovation and the supply of innovation-related services, STEP report, Oslo, NorwayWoolthuis, R.K. et al, 2005. A system failure framework for innovation policy design, Technovation, Volume 25, Issue 6

Combined approaches to consider innovation systems

27

Structure and process components approach for HE

Component Category Systemic instrument goals

Actors Presence Stimulate and organise the participation of various actors (NGOs, companies, government etc.).

Quality Create space for actors' capability development (e.g. through learning and experimenting).

Interaction Presence Stimulate the occurrence of interaction among heterogeneous actors (e.g. by managing interfaces and building a consensus).

Quality Prevent ties that are either too strong or too weakInstitutions Presence Secure the presence of (hard and soft) institutions.

Quality Prevent institutions being too weak or too stringent.Infrastructure Presence Stimulate the physical, financial and knowledge infrastructure

Quality Ensure that the quality of the infrastructure is adequate (strategic intelligence serving as a good example of specific knowledge infrastructure).

(Wieczorek & Hekkert, 2012) – structure and link to systemic instruments and goals

Inclusive functions from lit applied to HE

Function Considerations

F1: Entrepreneurial activity Involvement with BoP, BoP entry, Business planning, Business models for inclusive innovation

F2: Knowledge development and learning

Sources of knowledge, focus of knowledge development, research capacity, research collaborations, IP protection

F3: Knowledge dissemination / diffusion

Focus of dissemination, capacity for dissemination, absorptive capacity, dissemination methods

F4: Guidance of Search Focus of considering opportunities e.g. inclusion of development concern in policies, recognition of constraints in planning

F5: Market Formation How is university creating spaces for innovations to become market-ready, access to markets

F6: Mobilization of resources Access to capital, access to human resources

F7: Creation of legitimacy Commitment, engagement with community – legitimacy and sustainability of relationships

Van der Hilst (2012) - INCLUSIVE INNOVATION SYSTEMS HOW INNOVATION INTERMEDIARIES CAN STRENGTHEN THE INNOVATION SYSTEM

Two resources that are very useful!

IS analysis – basis for establishing a system of indicators?

Steps Components of the analysis1 Structural analysis • Technology: What are the technology trajectories?

• Actors: Who are the actors?• Research: The state of the knowledge system?• Education: Are the education needs met?• Market: What does the market look like?• Politics and policy: What re the policy goals?• Intermediaries: Which parties try to engage collaborations?• Networks: What does the network look like?

2 Determining the phase of development

• For the system to develop it needs to fulfil systems functions – this also develops on the stage of development of the various technologies

• Technology needs to be develop to a certain level of maturity and development needs of the system depends on that

3 System functions • How is the innovation system performing? Usually by experts

4 Structural causes of functional barriers

• Determine what systems functions are forming a barrier?• For each system function – what structural functions form barriers?• Relationships between causes and barriers

5 Obstacles for policy goals

• Choice of policy mechanisms depend on the structural causes and barriers identified

• Prioritisation needs to take place

31Wieczorek, A.J. & Hekkert, M.P., 2012. Systemic instruments for systemic innovation problems: A framework for policy makers and innovation scholars. Science and Public Policy, 39, pp.74–87.

Van der Hilst – some suggestions for indicators??

van der Hilst, B. J. G. (2012). Inclusive Innovation Systems: How Innovation Intermediaries can strengthen the Innovation System.

Van der Hilst – some suggestions for indicators??

van der Hilst, B. J. G. (2012). Inclusive Innovation Systems: How Innovation Intermediaries can strengthen the Innovation System.

Agenda


Challenges for systems thinking in M&E

• Using reductionist mental models and especially linear pursuit is undermining development and effectiveness

• Reductionist models not suitable to describe complexity across programmes

• Perception of systems thinking as fad / complicated / abstract• Practitioners remain unconvinced of operational use• Disempowering – used to structure , targets and plans – emergence not

attractive– Issue of attribution complicated, increasing complexity not reducing it

• Not clear how systems thinking can be used to analyse situations - Possibly supplementary?

• SI needs to adopt some issues – budgeting, results-based management, work programming

• Demanding intellectually – multi-disciplinary approaches: New staff, New systems, New values

Approaches

• In what ways are inclusive innovations different? – Type and scale of inclusive innovations – Consumer needs – Access to expertise, knowledge and finance– Market conditions for firms

• What is the impact of inclusive innovations? – Scaling up – Success factors for scaling – Microcredit: A successful inclusive innovation

Traditional versus Inclusive innovation systems

Component Traditional innovation systems Inclusive innovation system

Goal Development as economic growth Macro level analysis

Development as socio-economicInclusion - Micro-level analysis of livelihoods

Actors • Main focus on: • Higher income

markets/consumers • Formal supply- side organisations

in industrial sectors • Innovation intermediaries as R&D

brokers

Main focus on:• Low-income consumers• Non-traditional, less formal, demand-side

innovators• Chain of innofusion intermediaries• linking ‘distant’ supply and demand

Innovation • Located innovation pre, durante and post production

• Growth oriented innovation • Supply driven innovation • Technical innovation

Incremental innovation with a focus on diffusion processes• Local needs-oriented innovation as

appropriation, configuration, use variation, domestication

• Demand-driven and context-driven innovation• Non-technical innovation with focus on social

systems of sales and support• Reverse innovation


Traditional versus Inclusive innovation systems

Component Traditional innovation systems Inclusive innovation system

Learning • Learning by doing plus using and interacting: Learning about production and implementation Learning about technology Coherence and profit maximisation as guides

• Contextualised (supply, demand, other) learning by interacting and using and doing

• Learning about diffusion (sales and support) and use

• Learning about wider social processes including non-instrumental processes

• Survival and utility-maximisation a guidesInteraction • Formal, close relations

preference• Necessity (but also limitations) of• informal, loose but socialised relations

Institutions • Formalised, relatively static, direct impact overarching institutions

• Complex institutional terrain of informal and formal

• Indirect impact of core, formal institutional forces

• Importance (including potential negative impact) of informal institutions at local level


Agenda


Operationalising systems change for M&E

• Elements of a system– Pathways– Institutional structures

• Elements of positive systems change– More effective pathways

• Increased pathway capacity• Improved pathway connections

– More conduce institutional structures

• How system change influences system change results– Systems change initiatives managed by collaboratives create

intentional systems change– Three factors contribute to collaborative effectiveness: member

engagement, governance structure and process, and an accountability framework

A systems change framework

Agenda


Why is this useful??

• Includes connections between certain aspect of collaboration and the ability to lower structural barriers

• Developing leading indicators: Operationalising and assessing systems improvements – pathways and declining structural barriers

• Pathway approach allows choosing a pathway to evaluate – systems boundary

• Pathways also allow for concrete ways to bring the system analysis down to the meso-level – allows to trace the causal pathway of systems change

Way forward

• Toolkits for inclusive innovation system analysis?• Identifying opportunities• Integrating complex systems theory principles in

M&E?• Pathways approaches to inclusive innovation?• Big data – what can be done??• Composite indicators?? NOT good idea though –

process to understand what underlying contributions are good idea though

Documents

Evaluating system change in inclusive innovation systems