The Evidence Map...4 2019 Evidence Map Final eport The Evidence Initiative 2019 The Economist Intelligence Unit, on behalf of The Economist Group, convened a distinguished panel of

The Evidence Map:The Data Imperative

12019 Evidence Map

Final Report

© The Evidence Initiative 2019

Contents2 About the Evidence Initiative and the Evidence Map4 Acknowledgements5 Introduction6 Chapter 1 Data and evidence in context6 Framework for the Evidence Map9 Chapter 2 Key findings on overall data availability and characteristics9 None of the G20 countries collected data on all of the expert-

recommended indicators across the five key policy domains9 A nation’s economic development level does not appear to be the main

determinant of data coverage or characteristics10 Data gaps are best identified by looking at multiple related indicators10 International data sources often stand out in terms of accessibility and

ease of use11 Data is often free and well documented, but visualisation tools could be

improved12 Chapter 3 Key findings by policy domain12 3.1 Ageing and retirement12 In focus: forward-looking data on pension system resilience in Japan

and the United States14 3.2 Digital inclusion16 In focus: the role of global data institutions17 3.3 Disaster risk18 In focus: making the best of a crisis19 3.4 Financial inclusion21 In focus: Mexico’s policy focus shows up in the data21 3.5 Youth unemployment23 In focus: unemployment data in France, the US and South Korea24 Conclusion 25 Methodology Appendix

22019 Evidence MapFinal Report


About the Evidence InitiativeThe Evidence Initiative, a project of The Economist Group and The Pew Charitable Trusts, explores the use of facts and data in decision-making today and presents the case for evidence-based policymaking. The initiative leverages Pew’s rigorous, nonpartisan, fact-based approach and The Economist Group’s clear-headed analysis and convening power to foster discussion around the need for public discourse that is founded on facts.

www.evidenceiniative.org

About this reportAs part of the wider Evidence Initiative, The Economist Intelligence Unit (EIU)—the research and policy arm of The Economist Group—has produced a first-of-its-kind Evidence Map to track the availability and characteristics of data in Group of 20 (G20) countries that policymakers need to make sound decisions. The Evidence Map assesses data across five policy domains: ageing and retirement, digital inclusion, disaster risk, financial inclusion, and youth unemployment. Within these domains, it analyses the availability, accessibility and core characteristics of data on expert-defined indicators at the international, national and sub-national levels. The results have been used to assemble an interactive tool that allows comparisons, country profiles, and thematic analysis.

www.evidenceinitiative.org

Most of the research for this report, which included interviews and desk analysis, was conducted between August and December 2018.

Please use the following when citing this report:

The Economist Group and The Pew Charitable Trusts (2019). The Evidence Map: The Data Imperative. London and Washington, D.C.: The Evidence Initiative

32019 Evidence Map

Final Report


About the Economist GroupThe Economist Group is the leading source of analysis on international business and world affairs. It delivers information across a variety of platforms through a range of formats, from video, audio and print to conferences and digital services. What ties the Economist Group together is the objectivity of our opinion, the originality of our insight and our advocacy of economic and political freedom around the world. The Economist Group aims to offer insight, analysis and services that its customers value. Underpinning The Economist Group’s ability to fulfil this objective is its commitment to independence, integrity and delivering high quality in everything it does.

www.economist.com

About The Pew Charitable TrustsThe Pew Charitable Trusts is a global research and non-partisan public policy organisation that works to encourage responsive government and support scientific research, using data to make a difference. Our mission is to improve public policy, inform the public, and invigorate civic life. As part of that mission, the Pew Research Center, a subsidiary, studies global attitudes and trends and helps policymakers and the public prepare for future challenges. Throughout our history, The Pew Charitable Trusts has turned indifference into action—asking tough questions, studying problems, working with strong partners, and striving for effective solutions that bring diverse stakeholders together.

www.pewtrusts.org



The Economist Intelligence Unit, on behalf of The Economist Group, convened a distinguished panel of experts to discuss the Evidence Map and the project’s goals. We would like to thank the participants for their advice and guidance throughout the project (listed alphabetically by surname):

Daniel Acquah Organisation for Economic Co-operation and Development (OECD) Maria Baghramian University College Dublin, IrelandPaul Cairney University of Stirling, United KingdomMartha Chen Women in Informal Employment Globalising and Organising (WIEGO)Alison Fahey The Abdul Latif Jameel Poverty Action Lab ( JPAL)Maia Jachimowicz Results for AmericaStephane Jacobzone OECDArianna Legovini World Bank, Development Impact Evaluation Initiative (DIME)Carlos Santiso Inter-American Development Bank (IDB)Louise Shaxson Overseas Development Institute (ODI)Abeba Taddese Results for AllPiret Tõnurist OECD

We would also like to thank the policy experts who participated in our Delphi process, which helped to identify the datasets that featured in our analysis.

Evidence Map team

The Evidence Map was developed by teams from The Economist Group and The Pew Charitable Trusts, as part of the Evidence Initiative. The Economist Group team was represented by The Economist Intelligence Unit and included Leo Abruzzese, Monica Ballesteros, Sumana Rajarethnam, Stefano Scuratti, Richard Pedersen, Ankita Banerjea and Shubha Baradwaj.

Adam Green and Emma Ruckley contributed to the development and editing of the report and we thank them for their support.

The Pew team was led by James Bell, Emma Gilpin Jacobs, Molly Irwin, Sally O’Brien and Alan van der Hilst.

For enquiries about the report, please contact: Monica Ballesteros, The Economist Intelligence Unit, Washington, D.C., USA. Email: [email protected] Tel: +1(202) 650-6732

Acknowledgements

52019 Evidence Map

Final Report


There has been a great deal of discussion about the value of “evidence-based” policymaking, and few would disagree that evidence can play a vital role in identifying effective and enduring solutions to social, economic and environmental problems. But where does this evidence come from? The answer is data. Without data, robust analysis is impossible, which in turn means that no conclusions or evidence can be marshalled either in support of or opposition to policy choices and decisions.

Given the fundamental importance of data, The Economist Group and The Pew Charitable Trusts launched the Evidence Initiative in 2018, a project that explores the use of facts and data to generate evidence and inform sound decisions in the 21st century.

The Evidence Map—a free, online, interactive tool—enables users to assess the availability and characteristics of data in the public domain for the Group of 20 countries (G20; excluding the European Union). The G20 includes the world’s largest economies and is expressly focused on identifying and addressing pressing cross-national issues. The Economist Intelligence Unit (EIU) reviewed some of the highest-priority issues tackled by the G20 in recent years and identified five key policy domains, which it then used to populate the Evidence Map:

• Ageing and retirement

• Digital inclusion

• Disaster risk

• Financial inclusion

• Youth unemployment

Although these domains do not cover the entire policy landscape and may not reflect the highest policy priorities for each G20 country, taken together they provide a window into some of the collective priorities of the G20 and create a useful comparative framework for mapping the breadth and depth of data available to support evidence-based policymaking.

In the months ahead, the team will consult users and experts to help enhance both the content and the features of the Evidence Map. The map’s success will be measured by its ability to enable critical engagement with “the facts” and facilitate open, constructive debate about the data and evidence that is available to inform major policy decisions.

Introduction



Chapter 1Data and evidence in context

Governments have long harnessed data to inform decisions. Japan began collecting earthquake data in the 15th century, and Sweden created the world’s first national statistics office in the mid-1700s.1 In 1858, during the Crimean War, Florence Nightingale charted new ground with her collection and visualisation of British army mortality rates—data that led to new insights and policies aimed at improving nursing, hygiene and hospital management.

After the second world war, the US and the UK, among others, invested considerable sums in the development of modern statistical operations and the establishment of rigorous protocols for data acquisition and analysis.2, 3 Since then, the collection of more extensive and comprehensive statistics has become an essential component of government operations. In addition to data actively collected through statistical systems, surveys and other traditional methods, international organisations and private corporations are collecting a growing body of administrative, transactional and other passive data.

The Evidence Map provides a tool to help visualise the availability and accessibility of data across five domains to support evidence-based policymaking. In developing this map, The Economist Group and The Pew Charitable Trusts hope it will both illuminate the valuable data resources currently available to analysts

and government leaders, and highlight important gaps in the data needed to generate sound evidence and support effective policies.

Framework for the Evidence Map

The Evidence Map is designed to answer three core questions in the five prioritised policy domains:

• Is the data available?

• Is the data accessible?

• Can the data be easily analysed?

In this context, availability refers simply to the presence or absence of data. Accessibility is a composite measure that assesses how easily and readily policymakers, experts and citizens can consult data that is relevant to a policy domain. The degree to which data is ready for analysis is tied to the existence of metadata and the level of pre-programmed disaggregation in the dataset.

To understand the evolution of evidence-based decision-making, the team conducted a comprehensive literature review, assessing nearly 100 academic papers and reports on the subject. The EIU also convened an international panel of data experts in June 2018 to help guide the research (see Acknowledgements).

The Evidence Map’s framework assumes that data fuels analysis which produces

1 Stockholm University, Department of Statistics. “The history of official statistics in Sweden.” [https://www.statistics.su.se/english/research/official-statistics/the-history-of-official-statistics-in-sweden-1.263422]. 2 United States Census Bureau. “Data collection.” [https://www.census.gov/history/www/innovations/data_collection/].3 UK Statistics Authority, Government of the United Kingdom. “History.” [https://www.statisticsauthority.gov.uk/archive/about-the-authority/uk-statistical-system/history/index.html].

72019 Evidence Map

Final Report


evidence, which in turn informs decision-making. Importantly, the Evidence Map does not evaluate the use of data, nor does it evaluate eventual policy outcomes. It is strictly concerned with the data inputs that can lead to evidence-based decisions.

This framework applies to each of the five policy domains included in the initial Evidence Map: 1) ageing and retirement, 2) digital

inclusion, 3) disaster risk, 4) financial inclusion, and 5) youth unemployment. The Evidence Map was developed through a multi stage data and evaluation process, which identified the critical policy domains (Level 1), the universe of standard and aspirational policy inputs for each of those policy domains (Level 2), indicators or data that correspond to each policy input (Level 3) and characteristics for those indicators or data (Level 4; see Figure 1).

Figure 1: Analytical framework for the Evidence Map

Level 1: Policy domains Level 2: Policy inputs Level 3: Policyindicators

Level 4: Indicator characteristics

Policy domains: To identify G20 policy priorities, The EIU assessed G20 joint communiqués and public statements (the primary policy guidance documents produced by G20 summits) from the past five years and developed a list of policy domains for possible inclusion in the Evidence Map. This list was then revised to focus only on policy issues most frequently discussed by the G20. In consultation with the Pew team, The EIU further reduced the list to only include policy domains with the longest time horizons and greatest consequences for future societies.

Policy inputs: Based on desk research, and in consultation with subject matter experts and practitioners, The EIU identified policy inputs for each of the five domains covered by the Evidence Map. Policy inputs are sub-categories (or components) of each policy domain.

Policy indicators: Each policy input consists of a series of relevant policy indicators. The EIU identified these indicators by examining the types of data typically collected by governments and international standard-setting bodies in each of the five policy domains. The EIU also asked outside experts to recommend additional indicators for each domain. The EIU ultimately identified 268 policy indicators across the five domains. The first core question of the Evidence Map could then be asked: Is the data available?

Indicator characteristics: In consultation with outside experts, The EIU team developed a schedule of characteristics on which to score policy indicators (Table 1). These characteristics connect with other two core questions asked by the Evidence Map: Is the data accessible? Can data be easily analysed?



For more details regarding the role of experts in designing the Evidence Map, data collection methods, indicator weighting and final scoring, please consult the Methodology Appendix.

Chapter 2 details some of the key findings that emerged from the initial Evidence Map:

• None of the G20 countries collected all of the expert-recommended data in these five policy domains.

• A nation’s economic development level does not appear to be the main determinant of data coverage or characteristics.

• Data gaps are best identified by looking at multiple related indicators.

• International data sources often excel in accessibility and ease of use.

• Data is often free and well documented, but visualisation tools could be improved.

Chapter 3 highlights policy-specific findings regarding the breadth and depth of data available for analysis and evidence-based decision-making:

Table 1. Level 4 data characteristics

Core Question Data characteristics Characteristic detail

Accessible?Free Is the data available for free?

Downloadable Can the data be downloaded in an easily reusable format?

Ready for analysis?

Metadata Is comprehensive metadata available?

Recency What is the most recent year for which data are available?

Visualisation tools Does the database hosting the indicator offer visualisation functions?

Gender disaggregation Is the data series further disaggregated by gender?

Income disaggregation Is the data series further disaggregated by income bands?

Urban/rural disaggregation Is the data series further disaggregated by urban/rural location?

• The ageing and retirement domain had a high level of data coverage on average, with data available for 67% of the relevant indicators—the second-highest percentage across policy domains. However, data availability varied greatly across policy inputs.

• Digital inclusion was the most data-rich policy domain (with a G20 average of 73% data availability), driven primarily by international sources.

• Disaster risk data varied widely by country, with most collecting data on an average of 61% of the relevant indicators.

• The financial inclusion domain had relatively low data availability on average (54%), even in countries most affected by limited access to financing.

• Youth unemployment was the weakest of the five domains in terms of average data availability (48%), but it scored highly on data characteristics.

92019 Evidence Map

Final Report


Chapter 2Key findings on overall data availability and characteristics

The EIU’s analysis of the Evidence Map’s 268 policy indicators provides insight into the availability and characteristics of data across the five policy domains and between countries. The analysis led to some important findings and observations.

None of the G20 countries collected data on all of the expert-recommended indicators across the five key policy domains

None of the G20 countries collected data on more than 60% of the 268 indicators.

Youth unemployment was by far the weakest policy domain in terms of data coverage. On average, G20 countries had access to data on just 48% of the indicators that experts consider vital for addressing young people’s workforce participation. Data on apprenticeships was particularly scarce. This may represent a missed opportunity given the important role apprenticeships have played in shaping successful labour market outcomes in countries such as Germany and Switzerland.

Financial inclusion was the second-weakest policy domain in terms of data coverage, with only 54% of the relevant data available overall. Within this domain, data coverage was just 28% for the financial literacy policy input and 35% for the market conduct and consumer protection policy input. This points to a potential gap in governments’ knowledge about the factors that shape access to the

financial system, particularly for lower-income and disadvantaged citizens.

The digital inclusion domain had the most comprehensive data coverage with 73% of the relevant data available across G20 countries on average. It is likely that this reflects the data-centric nature of the digital industry and the important role played by international organisations.

A nation’s economic development level does not appear to be the main determinant of data coverage or characteristics

Although data gathering is often a resource-intensive process, the most economically developed countries did not uniformly have more comprehensive data coverage across the five policy domains. Indeed, Mexico, Turkey and Argentina had some of the highest levels of availability for domain-specific data, none of which are classified as “high-income” countries according to the World Bank’s definitions. In these three countries, data availability seems to be driven (at least in part) by national policy priorities. All three countries had comparatively high data coverage in areas that their governments, often over many years, have decided are particularly important: financial inclusion (Mexico), disaster risk (Turkey) and digital inclusion (Argentina). Conversely, some high-income countries had relatively weak data coverage in these domains, such as the UK and Italy (financial inclusion) and Germany (disaster risk).



In some instances, these gaps may exist because a government has decided that a particular domain is not a policy priority. This may be a rational approach in the short term, but over the longer term it could place countries at a disadvantage if faced with a crisis (e.g., a natural disaster) or deep structural challenges (e.g., the compounded costs of ageing and retiring populations). The value of data—whether at the international, national or sub-national level—should be assessed not only on current and historical needs but also from a forward-looking perspective.

Data gaps are best identified by looking at multiple related indicators

Few of the pressing challenges facing G20 members and other countries can be adequately understood or addressed based on a single indicator or data input. Multiple indicators, sometimes from seemingly disparate sources, are essential to assess strategic needs and guide policy decisions. For instance, countries that lack sufficient data on either the projected viability of pension funds or long-range demographic forecasts risk being ill prepared for future fiscal burdens associated with an ageing population. Data on financial inclusion is also vital. Collecting data on access to financial services without collecting corresponding data on financial literacy and safeguards could mean that large numbers of people enter the financial system in the future without governments providing proper education or establishing regulatory protections.

International data sources often stand out in terms of accessibility and ease of use

The Evidence Map includes a “data characteristics” score, which assesses a composite of attributes covering ease of access and user-friendliness. Analysis revealed that international data sources frequently earned higher data characteristics scores than national data sources. Across all five domains, the average scores (marked out of 100) for international and national data sources, respectively, consistently favoured the former: 83 versus 73 in the ageing and retirement domain, 85 versus 73 in the digital inclusion domain, 85 versus 75 in the disaster risk domain, 86 versus 69 in the financial inclusion domain, and 81 versus 73 in the youth unemployment domain.

This finding does not indicate an inherent flaw in national-level data generation. It does, however, indicate that international organisations such as the United Nations and the World Bank often have greater resources, and sometimes clearer mandates, to collect, standardise and make data accessible to the public on major issues such as digital and financial inclusion. This suggests that international institutions can be a valuable source of data for national governments and can encourage investment in capacity-building and data collection by national-level agencies.

International actors are also crucial stakeholders in standardising data, especially for benchmarking and country comparisons. They often provide tools for such efforts,

112019 Evidence Map

Final Report


such as the G20 Financial Inclusion Indicators, launched in 2012, which provide 15 detailed measurement metrics; and a disaster resilience scorecard developed by the United Nations Agency for Disaster Risk Reduction (UNISDR), which national agencies can populate with their own data and use to inform policy.4

Data is often free and well documented, but visualisation tools could be improved

On average, 98% of international data sources and 97% of national data sources could be accessed free of charge across G20 countries. Metadata (data about data) was also widely

available, including for 98% of international data series and 76% of national data series. Downloadable formats were widely available (89% of international data series and 67% of national data series).

However, data visualisation tools—which are increasingly important for making data useful—were only present in 54% of international data sources and 41% of national data sources. Visualisation functions were most commonly available for digital inclusion indicators (83% of international sources and 42% of national sources) and were least likely to be available for financial inclusion indicators (17% of international data sources and 36% of national data sources).

4 GPFI. “G20 Financial Inclusion Indicators.” [http://www.gpfi.org/sites/default/files/G20%20Set%20of%20Financial%20Inclusion%20Indicators.pdf].



Chapter 3Key findings by policy domainAnalysis of each policy domain allowed a closer look at the types of indicators that are more regularly measured by various international organisations and national and sub-national agencies, as well as differences in data coverage between countries and by policy area.

3.1 Ageing and retirement

The number of people aged 60 years or above is predicted to more than double worldwide by 2050 and to more than triple by 2100, increasing from 962m in 2017 to 2.1bn in 2050 and 3.1bn by 2100.5 Greying economies pose a threat to public finances, not just in developed economies but also in “prematurely ageing” middle-income countries such as China. This demographic shift puts pressure on governments to think about the long-term viability of public and private welfare provision, including health care. For example, governments are already debating the value of various policies designed to ensure that elderly populations are adequately cared for without weakening national economies.

In this domain, the Evidence Map includes 40 indicators across five policy inputs: 1) future size and composition of the workforce and retirees, 2) financial security of pension funds, 3) coverage of pension systems, 4) housing, and 5) publicly provided services. Indicators included the disaggregation of poverty rates by age cohort, as well as labour force participation above the age of 65.

Availability

In this domain, 41% of the indicators were available at the national level (Table 2) and 13% were available at the sub-national level. International sources aggregated, published and/or collected 54% of this data. The Organisation for Economic Co-operation and Development (OECD, a think-tank for developed countries) has some of the most comprehensive datasets in this space, drawn from its long-running research portfolio on labour markets and demographic change. These datasets are of substantial interest due to the greying of wealthy economies.

Across the five policy inputs within the ageing and retirement domain, data coverage was strongest for future size and composition of the workforce and retirees, for which G20 countries had 80% data availability on average. This was followed by financial security of pension funds, for which G20 countries had 75% data availability. Both datasets rely heavily on governments as census-takers and regulators and managers of public pension funds.

Coverage was weakest for the housing policy input, where countries had data for only 38% of the measured indicators; and for the public services policy input, where data was available for just 45% of the relevant indicators. Eleven countries had no national data covering publicly provided services for old age, which included data points such as the percentage

5 United Nations. “Ageing.” [http://www.un.org/en/sections/issues-depth/ageing/].

132019 Evidence Map

Final Report


Table 2. Ageing and retirement data availability and characteristics, G20 average

What % of expert-recommended indicators

are publicly available?

What % of publicly available indicators:

Are accompanied by metadata? Are provided for free? Can be downloaded? Are accompanied by

visualisation tools?

International source (e.g., World Bank) 54% 97% 100% 89% 61%

National source (e.g., National Statistical Agency)

41% 77% 99% 79% 40%

of retirees using government services such as food aid and transportation. This means that half of the G20 countries have no data on how their elderly populations use vital public services and welfare supports as measured by the Evidence Map indicators.

Characteristics

The most prevalent data characteristic was free access, with 100% of international datasets and 99% of national datasets available at no charge (Table 2). The vast majority of the data also included metadata—both for international and national sources—and was downloadable. However, both international and national sources were most likely to fall short in providing tools for visualisation.

Within the ageing and retirement domain, projections for the future size and composition of the workforce and retirees policy input had relatively high metadata availability overall (100% at the international level and 85% at the national level), as well as high levels of free access to data (100% at both the international and the national level) and access to data in downloadable formats (100% at the international level and 91% at the national level). Data characteristics scores were lower for the financial security of pension funds policy input, particularly for availability in downloadable formats (63% and 56% at the international and national levels, respectively).



Japan had 100% data coverage for the indicators measuring pension systems and informal support mechanisms, and 89% coverage for the financial security of pension funds. The high level of data coverage in this domain reflects the country’s unique demographics as one of the world’s few shrinking populations. Japan’s Statistics Bureau forecasts that the population will fall to just over 100m by 2050, from 127m today.6, 7

Japan has crafted a strong policy response to these challenges, including launching a long-term care insurance plan in 2000 that featured mandatory, public enrolment with premiums paid by everyone over the age of 40.8 Many of Japan’s institutions collect statistics that are relevant to ageing and retirement, including the Statistics Bureau, the Ministry of Internal

Affairs and Communications, and the Pension Fund Association.

The United States also has good data coverage in this domain, especially demographic projections (100%) and financial security of pension funds (100%). Several agencies create and/or analyse retirement statistics, including the US Census Bureau, the Bureau of Economic Analysis, the Bureau of Labour Statistics, the American Community Survey and the Centres for Disease Control and Prevention. The US Congressional Budget Office produces ultra-long-term budget projections, including spending projections for the government’s Social Security and Medicare programmes reaching to 2049. Additionally, US universities contribute data through specific departments such as the Centre of Retirement Research at Boston College.

In focus: forward-looking data on pension system resilience in Japan and the United States

6 Kopf, Dan. 2018. “The world is running out of Japanese people.” [https://qz.com/1295721/the-japanese-population-is-shrinking-faster-than-every-other-big-country/].7 Statistics Bureau, Ministry of Internal Affairs and Communications Government of Japan. 2017. “Statistical handbook of Japan.” [http://www.stat.go.jp/english/data/handbook/pdf/2017all.pdf].8 Creighton Campbell, John, and Naoki Ikegami. 2000. “Long-term care insurance comes to Japan.” [https://keio.pure.elsevier.com/en/publications/long-term-care-insurance-comes-to-japan].9 Solomon, Jessie. 2014. “Japan’s government pension investment.” CNN Money. [https://money.cnn.com/gallery/investing/2014/06/16/largest-investors-wall-street/2.html].10 G20. 2018. “G20 digital economy ministerial declaration.” [http://www.g20.utoronto.ca/2018/2018-08-24-digital.html].11 ITU. “Digital inclusion for all.” [https://www.itu.int/web/pp-18/en/backgrounder/digital-inclusion-of-all].

3.2. Digital inclusion

The digital economy is increasingly synonymous with the economy itself as more and more goods and services are made, sold and consumed via digital media and tools. The G20’s 2018 ministerial statement on the digital economy placed digital technology at the centre of a wide range of policy goals,

including gender equity, open government and support for underserved communities.10 Despite the size and economic sophistication of G20 economies, however, their digital inclusion rates vary widely. Broadly speaking, men and young people are more likely to be online than women, rural dwellers and older people, and internet access varies considerably across the G20.11

152019 Evidence Map

Final Report


The Evidence Map featured 60 indicators across seven digital inclusion policy inputs: 1) usage of digital technology, 2) digital technology’s contribution to the economy, 3) digital technology infrastructure, 4) network availability, 5) affordability of access to the internet, 6) competitive environment for the provision of internet services, and 7) readiness of the population to use the internet.

Availability

Digital inclusion was the most data-rich policy domain, with an average of 73% data coverage across the G20. Global data resources were much more prevalent than national ones, which may explain why the digital inclusion domain enjoyed higher data coverage overall than the other four domains in the Evidence Map. On average, 64% of the available data for this domain came from international organisations, notably the World Bank and the United Nations’ (UN) International Telecommunications Union (ITU), with just 37% coming from national sources. Mexico, Argentina and Australia had above-average

shares of digital inclusion data originating from national sources (60%, 58% and 57%, respectively), and Argentina and Australia had the highest data availability and data characteristics score overall.

Among the policy inputs for this domain, competitive environment for the provision of internet services and quality of digital infrastructure had the most comprehensive data coverage: 100% in all G20 countries, mostly obtained from international data sources. Usage of digital technology—which included factors such as the use of e-services—also had strong data coverage, with a G20 average of 90%. The policy input with the least coverage was digital technology’s contribution to the economy, measured using data points such as employment in the information and communications technology (ICT) sector, growth of the ICT industry and ICT investment as a percentage of GDP. On average, the G20 had 48% data coverage for this policy input. Argentina and Mexico had much higher coverage (90% and 80%, respectively).

Table 3. Digital inclusion data availability and characteristics, G20 average








37% 82% 99% 68% 42%



Characteristics

The most prevalent data characteristic was free access, with 95% of international datasets and 99% of national datasets available at no charge (Table 3). The lowest score was for data visualisation, which was available for 83% of international datasets and 42% of national

datasets. International sources performed better on the characteristics dimension than national sources: metadata was available for 97% of data published internationally, compared with 82% of data published nationally; and 91% of data published internationally could be downloaded, compared with 68% of national data.

The characteristics of international data sources were strongest in the digital inclusion domain, as international groups are involved in aggregating and standardising data in national systems. The ITU, a specialised UN agency, is a particularly important institution in this respect, as its mandate is to identify, define and produce official statistics. Gathered through annual questionnaires, its data cover telecommunications infrastructure, pricing and household-level access and usage.12 The ITU also plays a role in verifying and harmonising countries’ data, including collecting missing data for countries that do not provide answers to its questionnaires. Data points such as the number of mobile cellular subscriptions are widely collected by countries, but less data is provided in areas such as pricing. The ITU fills these

gaps, including by engaging directly with telecommunications operators.

To support the development of in-country statistical capacity, the ITU also runs training courses and technical workshops,13 and in 2014 it produced a manual to help countries collect high-quality and internationally comparable data on ICT access and use at the household and individual level.14 The ITU is also on the steering committee for the Partnership on Measuring ICT for Development, along with the United Nations Conference on Trade and Development (UNCTAD) and the United Nations Educational, Scientific and Cultural Organisation’s (UNESCO) Institute for Statistics (UIS). This partnership was launched in 2004 to improve the availability and characteristics of ICT data.

In focus: the role of global data institutions

12 ITU. “ITU data collection and questionnaires.” [https://www.itu.int/en/ITU-D/Statistics/Pages/datacollection/default.aspx].13 ITU. “Capacity development.” [https://www.itu.int/en/ITU-D/Statistics/Pages/capacitydev/default.aspx].14 ITU. 2014. “Manual for measuring ICT access and use by households and individuals.” [https://www.itu.int/en/ITU-D/Statistics/Pages/publications/manual2014.aspx].

172019 Evidence Map

Final Report


3.3 Disaster risk

Both acute and chronic natural disasters threaten G20 nations. Drought and water shortages are affecting Australia, Saudi Arabia, India, the United States and South Africa (where Cape Town came close to running out of water entirely in 2018), and earthquakes and tsunamis have plagued both Japan (notably Tōhoku in 2011) and Indonesia (most recently in 2018). As climate change increases the frequency of extreme weather events, disaster management is becoming a prominent concern for governments across the globe.

The Evidence Map measured disaster risk using 40 indicators across five policy inputs: 1) national-level risk assessments, 2) measures of susceptibility to damage, 3) disaster risk financing, 4) resilient infrastructure, and 5) external drivers.

Availability

The Evidence Map found that G20 members had data for 61% of the 40 disaster risk

indicators, with similar contributions from international and national sources (45% and 42% average data availability, respectively; Table 4). Data availability was strongest for external drivers, at 80% average availability. This policy input included indicators such as population density, the availability of flood-risk maps, and weather data. It was closely trailed by national-level risk assessments, for which countries had an average of 78% data coverage. This policy input included critical data points such as economic loss estimates, risk maps at the national and sub-national levels and probability figures for natural disasters.

Across the five main policy domains, disaster risk had the greatest coverage in sub-national data within the G20 (at 18%), followed by ageing and retirement (13%). This reflects the often localised nature of disaster risk, such as low-lying coastal areas or earthquake-prone zones, and the ability of disaster risk stakeholders to more precisely map risk areas.

Table 4. Disaster risk data availability and characteristics, G20 average








42% 68% 95% 59% 57%

Characteristics

The characteristics scores for international data were generally high in the disaster risk domain: Nearly 100% of published data included in our study was freely available and downloadable, with 94% of international

sources providing metadata and 64% including visualisation functions. At the national level, the most robust data characteristic was free access, with 95% of data published without charge. However, only 68% of national metrics included metadata—a low figure—and just 59% were available in downloadable formats.



France had high levels of data coverage overall for the disaster risk domain (80% of the measured indicators), and its data were easy to use and manipulate. The country’s online tools were particularly notable, such as the Georisques and Catastrophe Naturelles portals.15 The Georisques portal allows users to search for risks by address and GPS

points and then download results. The portal covers a range of risks including forest fires, floods and terrain movements. The three countries that performed well on both data availability and characteristics in the disaster risk domain—France, the United States and Japan—all had larger shares of sub-national data than other countries.

Disasters often prompt countries to improve data gathering, and this is reflected in their scores in the Evidence Map. In Japan, national-level risk assessment data were available for 83% of the measured indicators, particularly for those most closely related to the country’s vulnerabilities, such as earthquakes and typhoons, and to its prioritisation of emergency response. (Japan reportedly gathered earthquake disaster data as far back as the 15th century.)16 The country has expanded its capacities markedly in recent years, including the 2017 launch of a system called Monitoring of Waves on Land and Seafloor (MOWLAS). This observation network for earthquakes, tsunamis and volcanos, which was two decades in the making, covers all land and sea in Japan.17 That effort was enhanced after a large tsunami struck in the aftermath of the Great East Earthquake in 2011.

Turkey has also devoted significant resources to addressing its exposure

to natural hazards, including flooding, earthquakes and landslides, which have caused tens of thousands of deaths and trillions of dollars of cumulative damage, in the country. These vulnerabilities have led to improved data management protocols around disaster-related indicators. Turkey’s disaster risk management approach includes data components such as human and capital risk estimates,18 as well as efforts to improve technical capacity and build analytical foundations to inform decision-making.19 The country has also established a general directorate for emergency management, attached to the prime ministry; set up regional emergency operations offices; formed a national earthquake council; and extended local authorities’ responsibilities, with support from international partners.20 This provides a collective stakeholder group that can both generate and make sense of disaster-related data.

In focus: making the best of a crisis

15 Catastrophe Naturelles. [https://www.catnat.net/].16 Cabinet Office, Government of Japan. 2013. “Making use of disaster data: Examples of the Government of Japan.” [https://www.unescap.org/sites/default/files/S1-1_Japan.pdf].17 Public Relations Office, Government of Japan. “New network speeds up disaster detection.” [https://www.gov-online.go.jp/eng/publicity/book/hlj/html/201803/201803_02_en.html].18 Teker, Yeliz. “Disaster risk management in Turkey.” [https://www.unisdr.org/files/22160_tekerdisasterriskmanagementinturkey.pdf].19 Global Facility for Disaster Reduction and Recovery. “Turkey.” [https://www.gfdrr.org/en/turkey].20 Teker, Yeliz. “Disaster risk management in Turkey.” [https://www.unisdr.org/files/22160_tekerdisasterriskmanagementinturkey.pdf].

192019 Evidence Map

Final Report


Disaggregated data is particularly important in the economic realm because macro statistics can easily create a misleading impression of progress. To be properly understood, any economic or social improvement needs to be measured in ways that cover specific demographic sub-groups, taking gender, age, race and ethnicity into consideration, as well as location (urban and rural differences can be significant). For instance, the Global Findex initiative, launched in 2011, provides nationally representative demand-side data on access to, and use of, accounts, credit, payments and savings by adults over the age of 15 in 144 economies, some of which is disaggregated by gender, income level, employment status and rural residency.

Gender-disaggregated data provides a diagnostic starting point for governments and financial service providers to develop tailored products that can increase female economic autonomy. The benefits of women-centred products can include time savings and increased personal security (from being able to make digital payments instead of in-person transactions), greater investment allocation into household health and child education (compared with men), and more autonomy over household decisions. Digital financial services also allow women to receive welfare or government payments directly, which is crucial as governments increasingly shift to digital transfers.23

The importance of disaggregated data

3.4 Financial inclusion

Innovations in digital technology have dramatically increased financial inclusion. This is especially true in emerging economies, where millions of previously unbanked citizens have been able to use payment products thanks to mobile money, biometric and electronic identification systems, and improvements in services such as credit scoring. By 2018 52% of adults globally had sent or received digital payments during the previous 12 months, up from 42% in 2014.

Countries are also reducing the gender divide in financial inclusion. In India, for example, men were much more likely than women to have a financial account in 2015 (by a margin of 20 percentage points), but this gap has now narrowed (to 6 percentage points) thanks to a government push to increase account ownership, including through biometric identification.21 However, there is still much progress to be made. Nearly half of all unbanked adults live in seven countries, two of which are G20 members: India (accounting for 11% of unbanked adults) and China (accounting for 13%).22

21 Global Findex. 2018. “The little data book on financial inclusion.” [https://openknowledge.worldbank.org/bitstream/handle/10986/29654/LDB-FinInclusion2018.pdf].22 World Bank. “The Global Findex database 2017.” [https://globalfindex.worldbank.org/]23 Klapper, Leora. “Gender and the Global Findex: Collecting demand-side data on women’s financial inclusion.” [https://www.microfinancegateway.org/sites/default/files/global_findex_overview_presentation.pdf].



The Evidence Map included 65 policy indicators across five policy inputs for the financial inclusion domain: 1) access and usage of financial products, 2) points of access, 3) financial literacy and capabilities, 4) market conduct and consumer protection, and 5) barriers to use.

Availability

The majority of financial inclusion data was found in international datasets, which had 43% coverage, compared with 30% coverage at the national level. International sources also had better data coverage for adults’ use of financial products (56%, compared with 32% at the national level).

Of the five policy inputs, access and usage of financial products had the broadest data coverage, with data available for 65% of the measured indicators.

By contrast, data coverage for the financial literacy and capability policy input was just 28% across G20 countries, on average. Financial literacy data are important because weak financial management and susceptibility to fraud or the misuse of microfinance loans has affected several G20 economies, including India24 and South Africa.25 Data on barriers to financial use were also less available, with only 47% coverage on average. This included data on “enabling environment” issues, such as the presence and coverage of credit scoring and the collateral rules for small and medium-sized enterprises (SMEs). Coverage was also weaker for the market conduct and consumer protection policy input (35% average availability), which included data points such as the presence of dispute resolution and complaint procedures and the scale of personal identity coverage.

24 Hickel, Jason. 2015. “The microfinance delusion: who really wins?” [https://www.theguardian.com/global-development-professionals-network/2015/jun/10/the-microfinance-delusion-who-really-wins].25 Bateman, Milford. 2013. “Microcredit has been a disaster for the poorest in South Africa.” [https://www.theguardian.com/global-development-professionals-network/2013/nov/19/microcredit-south-africa-loans-disaster].

Table 5. Financial inclusion data availability and characteristics, G20 average








30% 72% 95% 59% 36%

Characteristics

Data characteristics scores were highest for international datasets, where 100% of published data was available for free and 100% of metadata could be accessed. National datasets scored somewhat lower for both

characteristics. In addition, just under 90% of international data could be downloaded, compared with 59% of data held by national agencies (see Table 5). National bodies scored better for data visualisation, however, which was available for 36% of national data, compared with 17% of international data.

212019 Evidence Map

Final Report


Mexico had the highest data coverage for financial inclusion indicators (75%, compared with a G20 average of 54%). This is driven largely by its coordinated national efforts, with Mexico’s national database holding data for 62% of financial inclusion indicators, compared with a G20 average of 30%. National data characteristics scores were also higher than the G20 average for some attributes. For example, 88% of national data included metadata, compared with a G20 average of 72%.

Mexico’s strong data coverage for financial inclusion reflects its policy priorities. The 2016 national financial inclusion strategy identified data generation and measurement as one of its pillars, leading to the continuation of the National Survey of Financial Inclusion, which provides valuable insights into financial access dynamics. Other data-gathering efforts include the Bank of Mexico’s Multidimensional Financial Inclusion Index.26

In focus: Mexico’s policy focus shows up in the data

26 Peña, Ximena, Carmen Hoyo, and David Tuesta. 2014. “Determinants of financial inclusion in Mexico based on the 2012 National Financial Inclusion Survey (ENIF).” [https://www.bbvaresearch.com/wp-content/uploads/2014/06/WP_1415.pdf].27 Office for National Statistics, Government of the United Kingdom. 2018. “UK labour market: September 2018.” [https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/bulletins/uklabourmarket/september2018].28 CNBC. “Unemployment.” [https://www.cnbc.com/unemployment/].29 United Nations. 2018. “World economic situation and prospects: February 2018 briefing, no. 111.” [https://www.un.org/development/desa/dpad/publication/world-economic-situation-and-prospects-february-2018-briefing-no-111/].30 Global News. 2018. “Canada’s unemployment rate fell to 5.6% last month—the lowest since 1976.” [https://globalnews.ca/news/4739555/canada-unemployment-rate-low-statistics-canada/].31 International Labour Organisation. 2018. “World employment social outlook. Trends 2018.” [https://ilo.org/wcmsp5/groups/public/---dgreports/---dcomm/---publ/documents/publication/wcms_615594.pdf].32 OECD. “Youth unemployment rate.” [https://data.oecd.org/unemp/youth-unemployment-rate.htm].33 United Nations. 2018. “World economic situation and prospects: February 2018 briefing, no. 111.” [https://www.un.org/development/desa/dpad/publication/world-economic-situation-and-prospects-february-2018-briefing-no-111/#_ftn5].

3.5 Youth unemployment

More than a decade on from the 2008-09 financial crisis, unemployment rates in many G20 member countries are quite low (in some cases, at multi decade lows) despite modest wage growth. This is especially true in the United Kingdom,27 the United States,28 Germany, Japan, Mexico29 and Canada.30 However, even as global unemployment falls, the share of young people who are out of work is rising. In 2018 global youth unemployment (i.e., among those aged 25 and under) was 13%—three times higher than the adult rate of 4.3%.31 Three G20 members face particularly high youth unemployment: South Africa (where 53.4% of the youth labour

force is unemployed), Italy (34.8%), and France (22.3%).32

The policy toolbox for reducing youth unemployment includes broadening and equalising access to primary and secondary education, apprenticeship programmes, entrepreneurship support and technical training facilities, and upskilling certain job categories.33 However, labour market interventions are among the toughest policy interventions to calibrate well, which means that they can sometimes lead to unintended outcomes. For example, providing companies with subsidies to hire younger workers can result in deadweight costs if



employers pocket those subsidies for hires they would have made anyway.34 Benefits can also accrue to the wrong segments of the target group in some instances. One evaluation in Morocco, known as the Taqeem Initiative (taqeem means “evidence” in Arabic), found that entrepreneurship interventions disproportionately advantaged those from more affluent backgrounds.35

For this domain, the Evidence Map included 63 policy indicators across six inputs: 1) closing the skills gap, 2) vocational training, 3) job availability, 4) apprenticeships, 5) government subsidies/financial support and 6) entrepreneurship.

Availability

Youth unemployment data was the weakest of the five policy domains in terms of data coverage, with low availability across both national and international sources. On average, G20 countries only had 48% data coverage for the 63 indicators examined. More youth unemployment data was held nationally than internationally (33% coverage and 25% coverage, respectively), with sub-national data available for 8% of indicators. The highest

national data coverage rates were found in France (86%) and Australia (54%). Seven countries had less than 25% of data available sub-nationally—an important gap given the localised nature of youth unemployment.

Despite some strong datasets for individual policy inputs such as closing the skills gap, the availability of data was weak overall. For example, data coverage for the apprenticeship policy input was just 21%. Among individual G20 countries, Australia placed in the upper ranks for apprenticeship data coverage (50%); five countries had no data on apprenticeships. This lack of coverage may be the product of several factors including scepticism in some countries about the effectiveness of apprenticeships following decades of uneven results. It is also challenging to measure indicators such as the number of completed apprenticeships or the percentage of apprentices who were offered full-time employment. High dropout rates, for example, can make these numbers hard to track. (In the United Kingdom, over 30% of apprentices withdraw from their schemes.)36 Entrepreneurship data covers factors such as the percentage of young people who own

34 Eccleston, John. 2011. “Government unveils £1 billion youth employment package.” Personnel Today. [https://www.personneltoday.com/hr/government-unveils-1-billion-youth-employment-package/].35 International Labour Office. 2017. “The impact of skills training on the financial behaviour, employability and educational choices of rural young people.” [https://www.ilo.org/wcmsp5/groups/public/---ed_emp/documents/publication/wcms_565085.pdf].36 Feldman, Paul. 2018. “Why are apprenticeship drop-out rates so high.” FEWeek. [https://feweek.co.uk/2018/03/20/why-are-apprenticeship-drop-out-rates-so-high/].

Table 6. Youth unemployment data availability and characteristics, G20 average






International source (e.g.., World Bank) 25% 100% 100% 92% 32%


33% 78% 98% 71% 35%

232019 Evidence Map

Final Report


a new business, the presence of start-up subsidies for youth and the average number of employees in youth-owned businesses.

Characteristics

Youth unemployment data received high data characteristics scores despite the overall lack of data. Metadata was available for

100% of published series from international sources, and for 78% of nationally held data. All international data were available free of charge, with 98% available nationally at no cost. Around 92% of international data were available for download, along with 71% of data from national agencies.

France has the most robust youth unemployment data among G20 countries, reflecting longstanding unemployment challenges. The country had 90% data coverage overall for this domain—far higher than the G20 average of 48%. France also had the highest data coverage for the apprenticeship policy input, at 63%. Australia placed in the upper ranks for apprenticeship data as well, with 50% coverage. (Five countries lacked any data on apprenticeships.) France also had the highest coverage for vocational training data (89%, tied with the United States).

In the United States, strong data coverage for vocational training reflects the efforts of the country’s Census Bureau, which tracks enrolment in vocational courses; and the National Centre for Education and

Statistics, which monitors the activities of post-secondary school institutions, participation in work experience programmes and the number of science, technology, engineering and maths (STEM) degrees conferred on citizens.

South Korea’s strong data coverage (67% when international and domestic sources were combined) is helped by the existence of several agencies that track relevant trends, including the Ministry of Education, the Ministry of Employment and Labour, and the Korea Educational Development Institute. These agencies monitor the number of schools with vocational training plans, in-house company vocational training and the continuing education participation rate, respectively.

In focus: unemployment data in France, the United States and South Korea



ConclusionThe Evidence Map examines the availability and key characteristics of international, national and sub-national data across five policy domains that present G20 governments with serious, long-term challenges. The ability of national governments to craft effective policies in response to these challenges depends on access to data that can fuel analysis, which in turn can provide evidence to guide decision-making. Without publicly available, easily accessible data—supported by metadata and built-in visualisation and analytic tools—government officials, experts in the civil and private sectors and citizens themselves may lack adequate information to judge and debate the merits of competing policy options.

The insights provided by the Evidence Map encourage discussion about whether governments are equipped with the necessary data to tackle the most pressing issues of the 21st century. Although the map’s initial focus was on the G20, the same question applies to all governments: Do countries have the data and the evidence they need to face the future?

252019 Evidence Map

Final Report


Methodology appendixThis methodology appendix describes the role of various experts in the design of the Evidence Map, the approach to data collection, and the map’s quality control and quality assurance procedures. It also outlines the scoring procedures used in the Evidence Map, and the underlying Excel model. It concludes with a table capturing the complete list of 268 indicators, broken down by policy domain.

Role of experts in designing the Evidence Map

A panel of 14 data science and evidence-based policy experts reviewed the initial framework for the Evidence Map to help clarify the kind of data that is necessary for good evidence-based policymaking. The panel assessed the intellectual rigour of the process for selecting policy domains (Level 1) and recommended policy inputs (Level 2) to help ensure that the map addressed the full scope of information relevant to each domains.

The expert panel then helped The EIU team identify three core questions for each policy indicator, regarding data availability, accessibility and suitability for analysis. Building on these core questions, the panel advised The EIU on constructing the final schedule of indicator characteristics to be assessed, including: metadata, which is necessary to guide proper data analysis; visualisation tools, which help to ensure that data can be understood and analysed; and disaggregation (by gender, income, location), which helps to illuminate patterns and differences beneath surface findings.

Having established the policy domains, The EIU team consulted with a second group of experts

to confirm the relevance and importance of the agreed policy inputs, and to guide the identification of policy indicators for each input (Level 3). The EIU employed a modified Delphi process to engage this second set of experts. (The Delphi process is a method of converging expert opinion through a series of iterative surveys, with the goal of coming to a group consensus.)

In Round 1 of the Delphi process, the experts were presented with a list of policy inputs and potential policy indicators, based on a literature review. They were asked to independently score each of the policy inputs and policy indicators on a scale from 1 to 3. Policy inputs and indicators that received a score of 1 were considered to offer the “most important” information and evidence for policymakers. Those that received scores of 2 or 3 were considered “important” and “less important,” respectively. Experts were also prompted to identify any additional policy inputs or indicators that had not been included in the initial list of inputs or indicators but were considered important for policymakers.

In Round 2 of the process, The EIU narrowed down the policy inputs (Level 2 in the Evidence Map) and policy indicators (Level 3) based on the experts’ responses in Round 1. This involved eliminating policy inputs and policy indicators that experts considered unimportant to policymaking ( i.e., those that scored less than 2, on average). The EIU also adopted additional inputs and policy indicators that experts suggested were necessary for each of the domains. Experts then scored the policy inputs and policy indicators again, on a scale from 1 to 3, to assess their importance.



Once the two rounds of the Delphi process had been completed, The EIU team consolidated the expert input. Policy inputs and policy indicators that had an average score below 2 in either the first or second round of the process were eliminated from the Evidence Map. Policy inputs and policy indicators that experts suggested after the first round of the process, and that scored well in the second round (i.e., above 2), were added to the Evidence Map.

Data collection

After confirming the policy domains (Level 1 in the Evidence Map), policy inputs (Level 2), policy indicators (Level 3) and indicator characteristics (Level 4), The EIU began data collection at both the international and national level. This process was organised by G20 member state and by policy domain within countries.

For each country and domain, The EIU team began by combing through international databases—such as those operated by the World Bank, the United Nations and the World Health Organisation—for each of the policy indicators (Level 3 in the Evidence Map). These searches were initially performed with a single pilot country to refine the search process and establish best practices before wider data collection began. The data collection was then expanded to each of the 19 countries included in the Evidence Map. If data for a country was found in an international database, that data was considered available and its characteristics were assessed.

The EIU team then repeated this process, but with a focus on data collected at the national level (e.g., by census bureaus or statistical agencies). Data collection at the national

level required local knowledge to identify appropriate sources, as well as local language skills in many instances. In addition, many national data sources did not have searchable or interactive datasets (e.g., those in Indonesia), which meant that extensive manual searches were required, along with more advanced local language skills. To mitigate these obstacles, each country was assigned an EIU researcher who could speak the local language and had some experience with data collection in that country.

The data search for each indicator was limited to 20 minutes. This was based on the assumption that data at the international level that could not be found within 20 minutes using specific English-language searches, and national-level data that could not be found within 20 minutes using specific local-language research, would be too obscure to be useful to researchers. If indicator data could not be found within that time frame, the country was recorded as lacking data for that indicator.

Quality assurance and data confirmation

Throughout the data collection process, The EIU instituted systems to ensure that any data collected was assessed repeatedly to ensure the accuracy of the results. First, each of the results from the pilot country data collection (at both the national and international level) was confirmed by a second researcher. This second researcher ensured that data scored as “available” for each policy indicator was indeed readily accessible to researchers, that the data matched the indicator, and that the data characteristics indicators were scored correctly. If data was not found, the second researcher spent 20 minutes looking for it to confirm that it was not available.

272019 Evidence Map

Final Report


After the pilot country was scored and reviewed, research on other countries began. Each country’s scorecard went through two rounds of checks across all five policy domains, ensuring that the identified data matched the indicator, and that indicator characteristics were scored correctly. Additionally, the scorecards were each reviewed twice for anomalies (e.g., where data that is typically collected across G20 countries had been marked as absent, or where data that was not available elsewhere had been found in one country).

Finally, after completing the research, The EIU contacted the national statistical offices of the G20 countries to provide them with an opportunity to review and comment on the preliminary results. The purpose of this data review was to ensure the accuracy of the data included in the Evidence Map. To make this process as efficient and rigorous as possible, The EIU developed documents that presented the data collected for each indicator. Figure A1 provides an example of one of these documents. These data review and confirmation forms listed which policy indicators had been found for a given country, and which had not been found. These forms allowed reviewers from each country’s statistical office to either agree or disagree with The EIU’s findings. A comment box was provided so that reviewers who disagreed with The EIU’s findings could offer an alternative answer and justification.

Country representatives had two months to respond to the data review and confirmation request. The EIU received responses from four countries: Argentina, Canada, Mexico and South Africa.

Scoring

The EIU built an Excel model to score countries and display the results. Scoring was broken down into two sections: (1) data availability, and (2) data accessibility and readiness for analysis. The data availability section measured the percentage of data that was obtainable in each country, by domain and input. It was scored as a simple binary, with all data weighted equally across international and national sources, five policy domains, and all 268 policy indicators. Availability scoring measured the percentage of data available in each country, by policy domain and policy input. The results of this scoring system can be seen in Figure A2.

If data was found for a certain policy indicator, its accessibility and readiness for analysis was assessed as a second overall dimension. Indicator characteristics were weighted (Table A1), providing each policy indicator with a score out of 100. These scores were averaged across all indicators within a given policy domain to provide a summary score by country.

In addition to tabulating data availability and aggregate characteristics scores for each country, the Excel model allowed the results to be analysed by policy domain, policy input, source (national or international) and individual characteristics. It also provided graphics such as data availability maps (Figure A2), availability pie charts, and graphs that plotted availability scores against characteristics scores (Figure A3). These features can help users to identify data challenges or gaps in particular policy domains or countries.



Figu

re A

1. E

xam

ple

of d

ata

confi

rmat

ion

scor

ecar

ds

Leve

l 1Le

vel 2

Leve

l 3N

atio

nal

Subn

atio

nal

Coun

try

Mex

ico

Polic

ygo

al

Redu

cing

disa

ster

risk t

om

itiga

teth

epo

tent

ial

loss

of

life,

inju

ry,

or d

estro

yed

or d

amag

edas

sets

as a

resu

lt of

natu

ral

haza

rds

Polic

yin

put

Nat

iona

l le

vel r

iskas

sess

men

t

Avai

labilit

yof

the

follo

win

gin

dica

tors

Econ

omic

loss

es fr

omdi

sast

er(v

alue/

perc

enta

geof

GDP

) ove

rth

e pas

t20

year

s

Wha

t is t

hefu

ll nam

eof

the

indi

cato

r?

Annu

alec

onom

icim

pact

of

disa

ster

s20

00-2

016

Com

men

tsfro

m th

est

atist

ical

o�ce

Avai

lable

repo

rts p

erye

ar, f

rom

2000

to

2016

The R

isk

Atlas

isco

nsta

ntly

upda

ted.

Link

http

://w

ww

.cena

pred

.gob.

mx/

es/

Publ

icacio

nes/

arch

ivos/

340-

NO

.17-

RESU

MEN

EJE

CUTI

VOIM

PAC

TO20

15.P

DFDo

wnl

oada

ble

Sour

ce

CEN

APRE

D

Sour

ce

Not

av

ailab

le

Link

Not

av

ailab

le

Year

it w

aslas

t upd

ated

2016

Is it

avai

lable

in

a nat

iona

lda

taba

se?

0 =

No

1 =

Yes

Year

Is th

ein

dica

tor

avai

lable

at th

esu

b-na

tiona

l(st

ate)

leve

l

Com

men

tsfro

m th

est

atist

ical

o�ce

00 =

No

1 =

Yes

No.

11

Mex

icoRe

ducin

gdi

sast

erris

k to

miti

gate

the

pote

ntia

llo

ss o

flif

e, in

jury

,or

des

troye

dor

dam

aged

asse

ts as

are

sult

of

natu

ral

haza

rds

Nat

iona

l le

vel r

iskas

sess

men

t

Num

ber o

fde

aths

from

disa

ster

ove

rth

e pas

t 20

year

s

Sum

mar

y of

dam

ages

by

natu

ral a

ndan

thro

pic

phen

omen

ain

201

5 -

Deat

hs

http

://w

ww

.cena

pred

.gob.

mx/

es/

Publ

icacio

nes/

arch

ivos/

340-

NO

.17-

RESU

MEN

EJE

CUTI

VOIM

PAC

TO20

15.P

DF

CEN

APRE

DN

ot

avai

lable

Not

av

ailab

le20

160

21

Mex

icoRe

ducin

gdi

sast

erris

k to

miti

gate

the

pote

ntia

llo

ss o

flif

e, in

jury

,or

des

troye

dor

dam

aged

asse

ts as

are

sult

of

natu

ral

haza

rds

Nat

iona

l le

vel r

iskas

sess

men

t

Risk

map

sat

the

natio

nal

leve

l

Nat

iona

lRi

sk A

tlas

http

://im

gir.p

roy

ecto

mes

oam

eri

ca.o

rg/A

NR/

apps

/feno

men

os/

CEN

APRE

DN

atio

nal

Risk

Atla

sCE

NAP

RED

2019

13

1

292019 Evidence Map

Final Report


Table A1. Data characteristics scoring

Indicator Description Weight Outcome Points scored

Metadata Is comprehensive metadata available? 25% “Yes” 25

Free Is the data available for free? 20% “Yes” 20

Year updated What is the most recent year for which data were available? 20%

2017 or 2018 20

2016 10

2015 5

Downloadable Can the data be downloaded in an easily reusable format? 15% “Yes” 15

Visualisation tools Does the database hosting the indicator offer visualisation functions? 5% “Yes” 5

Disaggregation

Is the data series disaggregated?

Three possible disaggregations are evaluated:

Gender (male/female) Income band Urban/rural

15%

All disaggregations are available 15

⅔ disaggregations are available

⅓ disaggregations are available

½ disaggregations are available 15

Figure A2. Data availability maps

Ageing and retirement

Digital inclusion

Disaster risk

Youth unemployment

Financial inclusion

67%

73%

61%

48%

54%

G20 Average(Average availability across 19 member countries)

Ageing and retirement(40 indicators)

Digital inclusion(60 indicators)

Disaster risk(40 indicators)

Financial inclusion(65 indicators)

Youth unemployment(63 indicators)



Figure A3. Policy indicator availability versus policy indicator characteristics

LOWER AVAILABILITY, WEAKER CHARACTERISTICS HIGHER AVAILABILITY, WEAKER CHARACTERISTICS

LOWER AVAILABILITY, STRONGER CHARACTERISTICS HIGHER AVAILABILITY, STRONGER CHARACTERISTICS

COUNTRIES IN THIS QUADRANT HAVE LOWER DATA AVAILABILITY

AND STRONGER CHARACTERISTICS COMPARED WITH G20 AVERAGE

KEYCharts show availability and characteristics relative to the G20 average.Availability represents the % of data indicators located in a national source.Characteristics is scored out of 100 where 100 represents data with the strongest characteristics.The centre of each chart represents the average availability and characteristics score across the G20.

COUNTRIES IN THIS QUADRANT HAVE LOWER DATA AVAILABILITY AND WEAKER CHARACTERISTICS COMPARED WITH G20 AVERAGE

COUNTRIES IN THIS QUADRANT HAVE HIGHER DATA AVAILABILITY AND WEAKER CHARACTERISTICS COMPARED WITH G20 AVERAGE

AVAILABILITY ->

CHAR

ACTE

RIST

ICS

->

G20 AVERAGE

LOWER AVAILABILITY, WEAKER CHARACTERISTICS HIGHER AVAILABILITY, WEAKER CHARACTERISTICS

LOWER AVAILABILITY, STRONGER CHARACTERISTICS HIGHER AVAILABILITY, STRONGER CHARACTERISTICS

COUNTRIES IN THIS QUADRANT HAVE HIGHER AVAILABILITY AND

STRONGER CHARACTERISTICS COMPARED WITH G20 AVERAGE

AGGREGATE268 indicators.G20 average availability: 36%.G20 average characteristics score: 73/100

INDIA

SOUTH AFRICASAUDI ARABIACHINA

GERMANYINDONESIA

UKITALY

RUSSIABRAZILCANADA

TURKEYAUSTRALIA

SOUTH KOREA

JAPANMEXICO

USAFRANCE

ARGENTINA

312019 Evidence Map

Final Report


Table A2. Complete indicator list by policy domain


Level 1 Level 2 Level 3

Policy goal Policy input Availability of the following policy indicators

Maintaining retirement income adequacy to provide a financially secure retirement for ageing populations

Projecting future size and composition of workforce and retirees

Life expectancy

Old-age dependency ratio

Disposable income disaggregated by age

Source rates by age cohort

Labour-force participation of people above 65 as a % of population

Effective labour market exit age

Expected years in retirement

Working-age population

Educational attainment by age group

Job openings by education level

Financially secure pension funds

Occupational pension fund assets as a % of GDP

Personal pension fund assets as a % of GDP

Public expenditure on old-age and survivors’ benefits

Social security contribution rates

Defined benefit (DB) funding ratios

Pension replacement rate (ratio of pension to final earnings before retirement)

Present value of contributions to pension scheme

Gross pension replacement rates (gross pension entitlement divided by gross pre-retirement earnings)

Net pension replacement rates

Coverage of pension systems and informal support

Pension eligibility ages

Active members as a % of labour force / % of working-age population

Recipients as a % of total and over-65 population

Coverage of private pension plans

Coverage of public pension systems

Share of people entitled to receive pensions at the current retiring age

Share of people who are not covered by pension funds (public and/or private)

Average lifetime contribution to social security funds

Employment rate by age cohort






Maintaining retirement income adequacy to provide a financially secure retirement for ageing populations

Housing

Percentage of households aged over 65 who are homeowners and pay a mortgage

Percentage of households aged over 65 who are homeowners outright

Homeownership among the over-65s by income decile

Rental rates for senior housing

Percentage of population over 65 in group care facility (nursing home, assisted living facility, etc.)

Percentage of retirees living with children/in multigenerational homes

Percentage of retirees living alone

Publicly provided services

Percentage of retired persons who have government-sponsored health care

Percentage of retired persons without health insurance

Percentage of retired persons with government-sponsored long-term care

Percentage of retirees using other government services (food aid, transportation supplements, etc.)

Digital inclusion



Promoting accessibility and use of informational and communication technologies

Use of digital technology by individuals/households

Percentage of households that use the internet

Fixed-line broadband subscriptions per 100 inhabitants

Mobile subscribers per 100 inhabitants

Households with a computer

Per-capita internet users

Percentage of schools connected to the internet

Number of internet exchange points

Percentage difference in male versus female internet users

Household income of internet users

Percentage of internet users by types of services used

332019 Evidence Map

Final Report


Digital inclusion




Digital technology’s contribution to the economy and productivity

Enterprises’ broadband connectivity, by firm size

Employment in the information and communication technology (ICT) sector and sub-sectors

Growth of employment in the ICT sector and its sub-sectors

Value added of ICT sector and sub-sectors

ICT investment by capital asset, as a percentage of GDP

Contribution to GDP of internet connectivity

Percentage of firms involved in ICT

Percentage of digital workers in the economy

Gender distribution of ICT-sector employees

Number of ICT start-ups over the past 12 months

Quality of digital technology infrastructure

Average fixed broadband upload speed, Kbps

Average fixed broadband download speed, Kbps

Average fixed broadband latency, ms

Average mobile upload speed, Kbps

Average mobile download speed, Kbps

Average mobile latency, ms

Bandwidth capacity, bit/s per internet user

Percentage of households with electricity in urban areas

Percentage of households with electricity in rural areas

Affordability of access to the internet

Smartphone cost (handset) as a percentage of monthly gross national income (GNI) per capita

Mobile phone cost (pre-paid tariff) as a percentage of monthly GNI per capita

Mobile phone cost (post-paid tariff) as a percentage of monthly GNI per capita

Fixed line monthly broadband cost as percentage of monthly GNI per capita

Cost or time spent charging electronic devices

Average cost per gb of data (mobile)

Average cost per gb of data (broadband)



Digital inclusion




Use of digital technology by individuals/households

Average revenue per user, annualised, USD

Wireless operators market share, Herfindahi-Hirschman Index (HHI) score

Broadband operators market share, HHI score

Readiness of the population to use the Interal

Literacy rate as percentage of population

Primary, secondary and tertiary gross enrolment rates

Education attainment rate, years of schooling

Level of web accessibility

Internet skills

Self-rating of ability to use the Internet

Self-rating of confidence as an Internet user

Proportion that describe the Internet as important to them

Percentage of classrooms with digital technology

Level of web accessibility, as rated by W3C guidelines

English literacy rate

352019 Evidence Map

Final Report


Disaster risk



Reducing disaster risk to mitigate the potential loss of life, injury or destroyed or damaged assets as a result of natural hazards

National-level risk assessment

Economic losses from disaster (value/percentage of GDP) over the past 20 years

Number of deaths from disasters over the past 20 years

Risk maps at the national level

Risk maps at the sub-national level

Probability of natural disasters per type of disaster mapped

Frequency of disasters per type of disaster mapped

Intensity of disasters per type of disaster mapped

Duration of disaster per type of disaster mapped

Estimates of potential physical (infrastructure) loss

Estimates of potential human loss

Estimates of potential economic loss

Estimates of potential environmental impacts

Percentage of government funds available for disaster response

Measures of susceptibility to damage during a disaster

Map of housing stock’s susceptibility to damage

Map of flood defences

Local vulnerability and capacity assessments

Income inequality (Gini coefficient)

National capital stock by asset type

GDP per capita

Disaster-risk financing

Insurance coverage: total insured value, by sector

Percentage of government funds dedicated to disaster risk mitigation

Percentage of private-sector funds dedicated to mitigation

Percentage of investment from country in water and sanitation

Percentage of investment from country in communications infrastructure

Percentage of the population with access to disaster-risk financing



Disaster risk



Reducing disaster risk to mitigate the potential loss of life, injury or destroyed or damaged assets as a result of natural hazards

Resilient infrastructure

Level of infrastructure investment (overall/by sector, per capita/as a proportion of GDP)

Average age of critical infrastructure

Annual expenditure on infrastrcture maintenance

Percentage of buildings that conform to disaster resiliency standards

Ratio of full-time-equivalent (FTE) professional building inspection labour to construction labour (both measured in units of time)

Annual hours of professional training for building inspectors

Percentage of new buildings and other infrastructure built at elevation above potential flood levels

External drivers

Change in average temperature over 20 years

Level of deforestation

Level of urbanisation

Population density

Maps of flood hazard at various probability levels

Maps of earthquake ground motion and ground failure at several probability levels

Percentage of population in wildfire risk areas

Population living in buildings that pre date disaster-resistant design standards

372019 Evidence Map

Final Report


Financial inclusion



Increasing access to and use of quality financial services for underserved and excluded households and enterprises

Use of financial products by adults

Percentage of adults who report having an account with a formal financial institution or a mobile money provider

E-money accounts per 1,000 adults

Mobile money transactions per 100,000 adults

Percentage of adults with at least one loan outstanding from a bank or other formal financial institution

Outstanding loans per 1,000 adults

Percentage of adults using the Internet to pay bills or make purchases online

Percentage of adults using a phone to pay bills, make purchases, or send or receive money from an account

Percentage of adults using a debit or credit card to make a direct payment from an account

Percentage of adults sending or receiving remittances to/from an account

Percentage of adults receiving wages, government transfer payments or agricultural payments to an account

Percentage of aduls sending utility or school fees from an account

Percentage of adults who saved at a bank or other formal financial institution in the past year

Number of live mobile money services

Number of registered and active agents

Number of registered and active mobile money accounts

Number of registered financial technology (fintech) firms

Volume of transactions for different mobile money products

Value of transactions for different mobile money products

Percentage of adults who saved at all in the past year

Percentage of adults who saved semi formally in the past year

Percentage of adults with at least one loan outstanding

Percentage of mobile money accounts with a non-zero balance

Percentage of adults with mobile credit

Percentage of adults accessing short-term credit (i.e., payday loans) in the past year

Percentage of adults able to raise emergency funds equal to 1/20 GNI per capita



Financial inclusion




Use of financial products by enterprises

Percentage of small or medium-sized enterprises (SMEs) with an account at a bank or other formal financial institution

Number of SME deposit accounts (as a % of non-financial corporation borrowers)

Percentage of SMEs with an outstanding loan or line of credit from a bank or other formal financial institution

Number of SME loan accounts (as a % of non-financial corporation borrowers)

Percentage of SMEs that send or receive digital payments from an account

Percentage of SMEs that identify access to finance as a major constraint

Volume of mobile payments versus volume of non-mobile payments

Points of access

Number of branches per 100,000 adults

Number of ATMs per 100,000 adults

Agent of payment service providers per 100,000 adults

Mobile agent outlets per 100,000 adults

Point of sale (POS) terminals per 100,000 adults

Percentage of adults with access to a mobile phone or device or internet access in the home

Percentage of SMEs that have a POS terminal

Interoperability of ATM networks and interoperability of POS terminals

Percentage of firms that accept mobile payments

Percentage of population covered by mobile networks

Percentage of administrative units with at least one access point

Number of interoperable payment platforms

Financial literacy and capability

Financial knowledge score

Percentage of entrepreneurs with basic accounting knowledge

Percentage of adults who could come up with 5% of median per capita GDP in the event of an emergency

Percentage of adults using an overdraft in the past year

Percentage of households that report being over-indebted

Measures of female decision-making power

392019 Evidence Map

Final Report


Financial inclusion




Market conduct and consumer protection

Number of disclosure requirements for financial services firms

Number of people who have issued a claim in dispute resolution mechanisms

Percentage of lenders that offer financial counselling at time of loan

Number of document requirements to open an account at a formal institution

Number of document requirements to take out a loan from a formal institution

Percentage of adults with government-issued identification

Average wait time on consumer complaint/support hotlines

Percentage of deposits covered by federal or private deposit insurance

Barriers to use

Percentage of SMEs required to provide collateral on their last bank loan

Coverage of credit reporting systems

Percentage of administrative units with at least one access point

Percentage of adults whose primary mobile device is shared

Average distance to nearest point of access

Average lending rate for loans

Average fees charged for opening an account



Youth unemployment



Reducing the share of young people who are most at risk of being permanently left behind in the labour market

Closing the skills gap

Share of youth not in employment, education or training (NEET)

Educational attainment of unemployed youth (primary, secondary, undergraduate, graduate)

Average length of time unemployed

Employment rate of recent university graduates

Involuntary part-timers (measured in FTEs)

Percentage of over-educated university graduates

Share of early school leavers

Share of high school dropouts

Share of inactive NEET

Youth unemployment by family income

Incidence of over-qualification

Incidence of under-qualification

Incidence of field-of-study mismatch

Percentage of youth with low literacy skills

Share of population who are recent university graduates

Percentage of employers reporting difficulty filling jobs

Share of youth in total employment

Percentage of young workers reporting they need more training to do current jobs

Vocational training

Compulsory science, technology, engineering and mathematics (STEM) curriculum in secondary schools

Share of secondary school students in vocational training programmes

Share of tertiary school students in vocational training programmes

Proportion of enterprises providing/not providing continued vocational training

Continued education programmes for young adults/retraining

Share of high school dropouts engaged in formal or non-formal job training

Growth rate of adults participating in formal, job-related adult learning

Share of college graduates with STEM degrees

Number of government-issued and nationally recognised certifications of vocational training

412019 Evidence Map

Final Report


Youth unemployment




Job availability

Job creation by sector or skill type

Job loss by sector or skill type

Reporting mechanism from private sector on skills gap

Job openings, hires and separations by sector

Occuptations with the most job growth

Industries with the largest wage and salary employment growth and declines

Total value of tax incentives for employing youth

Percentage of enterprises reporting availability of staff with the right skills as a major obstacle to long-term investment decisions

Percentage of workers facing a significant risk of replacement by automation

Skills in shortage or surplus

Public unemployment spending as a percentage of GDP

Unemployment benefit generosity (net replacement rate of unemployment benefits, % of previous net incomes)

Apprenticeship

Total number of new apprentices in current year

Total number of completed apprenticeships in current year

Number of unsuccessful apprenticeship seekers

Percentage of successful apprenticeship graduates being offered permanent contracts by their training companies

Availability of incentives for employers accepting apprenticeship programmes

Distribution of apprenticeship lengths

Long-term employment rates of apprenticeship graduates

Employment rate of apprentices one year after graduation

Government subsidies/financial support

Public spending in government-provided job search support, activation programmes and employment subsidies targeted at young jobseekers as a percentage of GDP

Public spending in government-provided employment subsidies targeted at young jobseekers as a percentage of GDP

Share of unemployed youth registered with national job search services

Public expenditure on active labour market policies (ALMP) training as a percentage of GDP

Participation in ALMPs, share of unemployed population

Public spending in subsidies for adult learning

Percentage of adults who wanted to participate in training, but did not because it was prohibitively expensive



Youth unemployment




Entrepreneurship

Percentage of youth indicating a plan to start a business in the next three years

Percentage of youth who own a new business (<3 years)

Percentage of youth declaring access to capital to start or grow a business

Percentage of youth declaring access to training on how to start a business

Number of new jobs by self-employed workers by occupational group

Percent of working population who work on their own business as their primary job

Start-up subsidies for youth

Average size (number of employees) of youth-owned businesses

Percentage of firms reporting taxes as a constraint for doing business

Disclaimer: While every effort has been taken to verify the accuracy of this information, The Economist Intelligence Unit Ltd. cannot accept any responsibility or liability for reliance by any person on this report or any of the information, opinions or conclusions set out in this report. The findings and views expressed in the report do not necessarily reflect the views of the sponsor.

LONDON20 Cabot SquareLondon, E14 4QWUnited KingdomTel: (44.20) 7576 8000Fax: (44.20) 7576 8500Email: [email protected]

Documents

The Evidence Map...4 2019 Evidence Map Final eport The Evidence Initiative 2019 The Economist Intelligence Unit, on behalf of The Economist Group, convened a distinguished panel of