Evaluation of DFID Online Portals and Repositories

Eldis 20th Anniversary Workshop

Institute of Development Studies

Brighton

15th September, 2016Rachel PhillipsonMott MacDonald Senior Economist & Evaluation Specialist

5 evaluation objectives1. To describe user populations and examine how they interact with

online research portals and repositories;

2. To assess the quality and accessibility of online research portals and repositories and to collate and analyse the available evidence on their use;

3. To draw out and illustrate ‘plausible pathways’ between portal use and uptake of evidence in policy and practice;

4. To assess whether the DFID-funded portals and repositories present value for money, in their own right and in relation to sites not funded by DFID;

5. To provide recommendations for how the DFID-funded websites might be improved and better monitored.

R4D – DFID Repository (£300,000 p.a)

Eldis – A portal for aid information (£400,000 p.a)

SciDev.Net – Science & Tech news (£2.2M p.a)

A simplified Theory of Change

Evidence mapping for the ToC  Theory of Change

hypothesisSource Summary of conclusions from the

literature1 Portals should be mobile friendly Sylla et al. (2012)

Starkey (2013)Global Internet Report (2014) Debeljak, K. (2010)Euforic Services (undated)J.Adams & T. Loach (2015)Batchelor (2013)

Rapidly increasing use of smart phones to access internet services and social media; rapid increase in sharing citations and alerts about research via mobiles in Europe and USA; rapid catching up in mobile use in lower income countries, though little evidence of this yet extending to social media use by them to access or promote research. Strength of evidence: Medium

2 Content should be prioritised on search engines

Pew Research (2014)De Satge (2011) Intermedia (2010a)Prakash (2013).

Users of online research, particularly non-academics, often prefer to search portals using general search engines (eg Google, Yahoo) rather than dedicated site search functions. Strength of evidence: Strong

3 Portals should meet region specific cultural requirements

Chavan et al (2009)Chavan, (2004) Bidwell & Winschiers-Theophilus (2010) Moalosi et al (2007)Faiola & Matei (2005)

Perceptions of a system’s usefulness are culture-based; websites that are tailored for local audiences are demonstrably more effective in reaching that audience.Strength of evidence: Strong

4 Portals should be specialised Intermedia (2010b) Bayliss et al. (2012)  

Development policy makers often start a web search with a known ‘expert’ website (eg World Bank, the Lancet).Strength of evidence: Weak

5 Portals should be linked to reputed authors

McKenzie & Özler (2014)AidDATA (2015) 

References, mentions, citations by known sources, ‘influentials’ and ‘elite bloggers’ have demonstrable impact on attention given to the referenced item. These results are from Northern users only. Strength of evidence: Medium

Summary of research tools used Online survey sent to 10,000+ addressees: 40 questions,

half of which were free text.

3x Country case studies: Ghana, Tanzania, Nigeria using (i) 2-part ‘contextual inquiry’ and (ii) research diaries.

Heuristic evaluation ( independent ‘expert review’) of the websites’ usability.

VFM ‘3Es’ review: Eldis and SciDev site visits, web metrics and financial analysis. Light-touch comparisons with Pambazuka, Zunia and GSDRC (helpdesk) portals.

Plus...

‘Plausible Pathways’ drew on case studies, market research and portals’ personae and impact studies.

Our market research respondents

N=734 from market research including new multilateral/donor (South) category

Respondents by organisation type

Based on full sample of 945 completed questionnaires

Our case study participants

Male75%

Female25%

Gender overall

Network (e.g Proff. Assoc.)

Parliament/Politcal Party

Private charitable foundation (e.g. Bill and Melinda Gates)

Media (e.g. National news paper)

National Civil Society Org (Charity faith-based org.)

Gov't Donor agency (USAID, DFID)

National Gov't(excluding overseas Aid Dept)

Uni Dept/ Res Inst/ Think tanks

Consultancy

0 2 4 6 8 10 12

Organisation Type

Findings

Research evidence-seeking online is frequent, rapid and impatient

Half of all MR respondents said they search online for research evidence for work a few or many times a day

Case study participants frequently referred to lack of time to devote to searching

Searching often limited to 1st page of Google

Failures, eg of downloading, searching, broken links, lead to immediate abandonment

Although there are some sceptics, ‘Google is King’…

0.0%

5.0%

10.0%

Eldis SciDev Zunia GSDRC

Percentage of sessions with internal search - All traffic data 2015

All SessionNew VisitorReturning Visitor

- More efficient - will cover all portals in one search- More effective - Case study instances of Google being

better at searching the site than own search function- Returning users tend to use internal search more frequently

World Bank& UN are the ‘go-to’ sites for international data & guides

Governments’ own websites are popular• Own government portals frequently used for local stats

• But country variation in perceived reliability of Govt websites

• Stats are often sought independently of ‘research’

A wide range of formats & services are used when searching for evidence online

Credibility of evidence found online is assessed initially – and quickly - via the

source

Nearly two-thirds of respondents use the source of research evidence to help judge if they should trust it; 20% listed no other criteria

Sources can be author, affiliation, host site, respected referee (personal or journal based)

DFID, WB, UN are key trusted host sites

There is a probably a two-stage process: credibility of source is assessed online; validity of the evidence is assessed in longer time. But the first is key.

DFID is a trusted source and portals’content is perceived to be high quality - but not always accessible

“I used them because I trusted the information posted in these website” TZ5 diary (Scidev and r4d)

“That’s very useful … relevant to my work” TZ7

“I will come back to this website [R4D].... for more information on DFID projects”. NIG10

“I’ve never heard of these anywhere. I feel shameful... especially if you are working on a DFID-related project, you want to be efficient” GH11

Becomes surprised that the specific article is not the first title. Also realises that the specific article automatically opens on a different tab NIG02

Research evidence-seeking is going mobile

-50.00%

0.00%

50.00%

100.00%

150.00%

200.00%

Eldis SciDev.Net

DesktopMobileTablet

Year on year % change in sessions by device used

- 40% of MR respondents use a computer, phone and a tablet to access the internet

- Case study participants moved effortlessly between devices in search tasks [GH5, TZ5, TZ1]

Southern users per se are not different in their information behaviour

Biggest differences are between (global) academics/researchers and Southern civil servants

This suggests differences may be driven by training, time-constraints and research needs

Women (especially Northern) are more likely than men (especially Southern) to use email newsletters/alerts to keep up to date. This may also be linked to time constraints.

Webmetrics suggest Southern users generally spend somewhat longer on site - but could be due to internet availability problems

But other barriers can still be a problem Three-quarters of respondents (n=534) reported commonly experiencing one or more problems when using the internet to obtain research evidence for work. Of these:

1 in 3 reported problems relating to paywalls and firewalls

1 in 4 reported problems with searching the internet

1 in 5 reported problems with internet access

“Despite having Hinari access, some journals are still not freely available online. Internet speeds still poor, unreliable and expensive in Malawi. Power often down when speeds are faster!” [Southern Academic/Researcher in Health]

“For this office I don’t get much journal articles because we don’t subscribe to them ....I am able to get access to them when I go to the University” …“I don’t get internet in my office regularly, and even when it is available, it is not always fast”. GH12

Uses phone and hotspots for internet connection outside Accra GH1

Strong evidence of individual and inter-personal level behaviour change

occurring Increased supply of evidence is the main reason for perceptions of own more frequent and easier use of internet research:

particularly improved availability (better and cheaper internet access, more free journals)

Increased ‘discoverability’ (eg Google and Google Scholar facility) is also commonly quoted

Increased demand from colleagues and organisations is noted: ‘There is more demand for researched-based quality reports and discussion with colleagues is a necessary activity to improve quality.’‘[...]now we discuss the references and sometimes can triangulate data we have found on the web.’

And indications of both organisational and institutional behaviour changes emerging

Most evidently driven by the demands of (aid) funders:

‘Stronger emphasis from clients on research evidence’‘[B]ecause there is greater need now to understand the quality of the evidence, particularly as donors are pushing harder for proof that the evidence is robust.’‘More focus on evidence from politicians and universities’‘It's become more of a topic across the NGO sector, with more awareness about it and tools for trying to assess quality (i.e. BOND evidence principles)’

‘Plausible Pathways’ between online evidence use and uptake are discernible

1. The Personal pathway: self-efficacy and motivation, credibility and influence

2. The Technocratic pathway: sign-posting solutions for policy makers and practitioners

3. The Democratic pathway: informing opinions, generating engagement

1. The ‘Personal’ pathway – self-efficacy and motivation, credibility and influence

“Access to information is empowering ....We used to be able to talk only about Tanzania, but now we can go to a conference and talk about... many countries” Tanzania Think Tank

“As long as the data is credible, it’s a good thing and it causes dialogue between people on techniques as well as data. Tanzania research institute

“I can download and then when I am on the plane 35,000 ft high, I can read it offline…”Nigerian consultant

“Nowadays, no one can deceive me; I am able to cross check information..to see if it is authentic”. Nigerian NGO

* Individuals have a wide range of internet skills and degrees of confidence in them

*But all find it saves them time and opens up sources to them

*They generally therefore feel more effective influencers

2. The ‘Technocratic’ pathway– sign-posting solutions for policy makers and practitioners

“It’s rare to hear that they [Government] have been reading to understand the underlying problem.” Tanzanian think-tank

Politicians want steps for how to change, and unfortunately researchers are not doing that. Tanzanian NGO

*Policy makers and practitioners are very time poor [Case studies]

*Evidence needs are pragmatic, not academic –

*search terms are ‘how to...’ ‘manual’[Case studies]

* Donor-funded TA has privileged access and remit to work with Government

3. The ‘Democratic’ pathway – informing opinions, generating engagement

“It’s forcing the credibility issue on us that we must look for the right information if we really want to debate, otherwise you’d better keep your mouth shut” Ghana MP.

“For example, in transport sectors, we wouldn’t know how the Tanzanian port is regarded, e.g. corruption, speed of handling....’ Tanzanian consultant

They [politicians] are very strategic, for example looking through the references to check their credibility. Tanzanian research institute

It has also made verification of information easier [especially] by accessing information from reliable local internet sites. Nigerian media professional

* Users are extremely time aware – speed, relevance, real time feedback

* They seek and check stats/facts - not extended rationales

* Trust in the source is important

* Other media and devices are employed – radio, facebook, mobiles

* Not all directed at policy makers but also at the public

*Uptake is not immediate or direct: ‘They will remember us at election time… and listen’.

Key Lessonsfor online research dissemination

Policy actors are characterised by pressure of time; don’t have the luxury to assess quality in depth when online

There are concerns about quality of research online but policy makers are happy to use proxies and short cuts to assess them

Trusted sources are paramount

Optimising the discoverability of the portals and their content through general search engines is key to being found

Users of all types want access to statistics – local, national and international

A wide range of formats and media are used by people searching for research evidence online

North – South divides between user behaviours are not strongly evident

Handheld devices should be taken seriously to disseminate research

Distinguishing between use and uptake, defining uptake as behaviour change, and thinking in terms of pathways help unravel ‘impact’.