Microdata Management Microdata Management ToolkitToolkit

Tools to facilitate archive and dissemination Tools to facilitate archive and dissemination of surveysof surveys

Pascal HeusPascal Heuspascal.heus@gmail.compascal.heus@gmail.com

International Household Survey NetworkInternational Household Survey Networkhttp://www.surveynetwork.orghttp://www.surveynetwork.org

Session C3 - Wednesday, May 24

Effective Strategies for Metadata Management

BackgroundBackground• International Household Survey NetworkInternational Household Survey Network

– Marrakech Action Plan for Statistics (MAPS) (Feb 2004) Marrakech Action Plan for Statistics (MAPS) (Feb 2004) – Established in September 2004Established in September 2004– International organizations actively sponsoring household surveys International organizations actively sponsoring household surveys

+ NSO+ NSO– Coordination: ILO, DFID, Paris 21, UNICEF, UNSD, WHO, World Coordination: ILO, DFID, Paris 21, UNICEF, UNSD, WHO, World

BankBank– http://www.surveynetwork.orghttp://www.surveynetwork.org

• Survey under-use: limited access, poor return on Survey under-use: limited access, poor return on investment limited impact, difficulties in policy makinginvestment limited impact, difficulties in policy making

• Common obstacles: quality, technical capacity, Common obstacles: quality, technical capacity, legal/political issueslegal/political issues

• Common problems: Accessibility, Timeliness, Coherence, Common problems: Accessibility, Timeliness, Coherence, Lack of metadata / documentation / data, poorly Lack of metadata / documentation / data, poorly organized archivesorganized archives

• Need for new tools and guidelinesNeed for new tools and guidelines Microdata Management Toolkit Microdata Management Toolkit

Toolkit RequirementsToolkit Requirements• User friendly software for microdataUser friendly software for microdata• Facilitate metadata exchange (DDI, Facilitate metadata exchange (DDI,

Dublin Core)Dublin Core)• Facilitate archiving (metadata and data, Facilitate archiving (metadata and data,

quality control)quality control)• Facilitate preservation/dissemination: Facilitate preservation/dissemination:

network, CD / DVD, web sitesnetwork, CD / DVD, web sites• Works with common data formats Works with common data formats • Multilingual supportMultilingual support• Free or InexpensiveFree or Inexpensive• Availability of technical support and Availability of technical support and

trainingtraining• Supported by national, international and Supported by national, international and

research communitiesresearch communities

Generate HTML based CD-ROM

Import metadata and prepare CD-ROM

Import data and compile metadata

The Toolkit ProcessThe Toolkit Process1

2

3

Toolkit ComponentsToolkit Components• Archiving: Metadata Editor (World Bank / Nesstar Archiving: Metadata Editor (World Bank / Nesstar

Ltd.)Ltd.)– To compile survey data, documentation and metadata in a To compile survey data, documentation and metadata in a

standard format (Nesstar/DDI). Free data reader for users.standard format (Nesstar/DDI). Free data reader for users.– Built on Nesstar PublisherBuilt on Nesstar Publisher

• Dissemination: CD Builder (World Bank / Mark Dissemination: CD Builder (World Bank / Mark Diggory)Diggory)– To facilitate the publication of survey data, documentation To facilitate the publication of survey data, documentation

and metadata on CD-ROM and on the web (transforms DDI and metadata on CD-ROM and on the web (transforms DDI into HTML based navigation)into HTML based navigation)

– Based on Eclipse Platform, open sourceBased on Eclipse Platform, open source

• Nesstar ExplorerNesstar Explorer– Free softwareFree software– Access metadata & data and export to common statistical Access metadata & data and export to common statistical

formatsformats

• IHSN ToolsIHSN Tools– Reporting, diagnostics and other utilitiesReporting, diagnostics and other utilities

What is the Metadata What is the Metadata Editor?Editor?

• DDI / Dublin Core specialized editorDDI / Dublin Core specialized editor• Template drivenTemplate driven• Enhanced version of the Nesstar Enhanced version of the Nesstar

PublisherPublisher• Import/Export common data filesImport/Export common data files• Integrated interface, multilingual Integrated interface, multilingual

supportsupport• Metadata and data in single fileMetadata and data in single file• Export to DDI / DCExport to DDI / DC• Licensing agreementLicensing agreement

for developing countriesfor developing countriesand IHSN membersand IHSN members

What is the Metadata What is the Metadata Editor?Editor?

What is the CD Builder?What is the CD Builder?• Publish survey Publish survey

metadata, documents and data metadata, documents and data on a CD-Rom or web site)on a CD-Rom or web site)

• Transforms DDI into an HTML Transforms DDI into an HTML based interfacebased interface

• User can customize the layout User can customize the layout (branding) and content of the CD (single (branding) and content of the CD (single or multi-surveys)or multi-surveys)

• Open source applicationOpen source application• Build on the Eclipse FrameworkBuild on the Eclipse Framework• Based on DDI / Dublin CoreBased on DDI / Dublin Core• Integrates with Metadata EditorIntegrates with Metadata Editor

CD Builder ProcessCD Builder ProcessCreate new CD-ROM

Project

Create new CD-ROM Project

Add a survey to the project and select its

type and branding

Add a survey to the project and select its

type and branding

1

2• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

• Selecting a survey consist in opening the DDI-XML or Nesstar file• The survey “branding” determines the overall look and feel of the CD• The survey “type” determines the default metadata content

Click the “Save” button to generate the HTML

interface

Click the “Save” button to generate the HTML

interface

3

After a few minutes, your CD Project is ready

for publishing!

After a few minutes, your CD Project is ready

for publishing!

4

Sample outputSample output

Nesstar ExplorerNesstar Explorer• Free softwareFree software• PDF philosophyPDF philosophy• Access to survey Access to survey

metadatametadata• Access to data (no need Access to data (no need

for specialized for specialized software)software)

• Export to common Export to common formatsformats

• Single file holds data Single file holds data and metadataand metadata

IHSN ToolsIHSN Tools• Free / Open SourceFree / Open Source• Utilities to Utilities to

complement complement existing softwareexisting software

• Integrates in EditorIntegrates in Editor• DiagnosticsDiagnostics• ReportingReporting• Metadata sharingMetadata sharing

Benefits and UsersBenefits and Users

• Improves and facilitates Improves and facilitates documentation preservation, documentation preservation, cataloguing, metadata exchange, cataloguing, metadata exchange, dissemination, quality, etc.dissemination, quality, etc.

• Users: survey producers, survey Users: survey producers, survey sponsors, data archivessponsors, data archives

• Benefits: data producers, national & Benefits: data producers, national & international survey sponsors, survey international survey sponsors, survey data repositories, data analysts data repositories, data analysts policy makers, DDI Communitypolicy makers, DDI Community

Status & AvailabilityStatus & Availability• Metadata Editor: Metadata Editor:

– Available in English, Commercial release in Available in English, Commercial release in Nesstar 4.0 (end 2006?)Nesstar 4.0 (end 2006?)

• CD BuilderCD Builder– Available in English & FrenchAvailable in English & French

• IHSN ToolsIHSN Tools– Diagnostics reports available, Survey Diagnostics reports available, Survey

Documentation in PDF Documentation in PDF • OtherOther

– I18n: working on French, Spanish, Bosnian, I18n: working on French, Spanish, Bosnian, Russian, Portuguese,…Russian, Portuguese,…

– User’s Guide in EnglishUser’s Guide in English– Open source project components hosted on Open source project components hosted on

SourceForge at SourceForge at http://www.sourceforge.net/projects/toolkithttp://www.sourceforge.net/projects/toolkit

Roll-outRoll-out• Roll-out in in 2006/2007 (global)Roll-out in in 2006/2007 (global)• Completed training / pilot in about 20 Completed training / pilot in about 20

countries, mainly in the Africa regioncountries, mainly in the Africa region• Expected use by UNICEF for next round Expected use by UNICEF for next round

of Multiple Indicators Cluster Surveys of Multiple Indicators Cluster Surveys (MICS, 55 countries)(MICS, 55 countries)

• Asia: Partnership with United Nations Asia: Partnership with United Nations Economic and Social Commission for Asia Economic and Social Commission for Asia and the Pacific (ESCAP)and the Pacific (ESCAP)

• Latin America: partnership with Inter-Latin America: partnership with Inter-American Development Bank (IADB) and American Development Bank (IADB) and Pan-American health Organization Pan-American health Organization (PAHO)(PAHO)

• Used by IHSN member agencies (WHO, Used by IHSN member agencies (WHO, ILO, etc.)ILO, etc.)

Surveys and censuses in low-income countries in Africa

Health survey in last 5 years

Coverage:82 % of countries81 % of population

Poverty survey in last 5 years

Coverage:56 % of countries

62 % of population

Available Not conducted /unavailable Non-IDA country

Pop. census in last 10 years

Coverage:67 % of countries53 % of population

99 % in Europe,95% in Asia,only

53% in Africa

Accelerated Data ProgramAccelerated Data Program• Growing demand for data for MDG, PRSP, Growing demand for data for MDG, PRSP,

IDAIDA• Must improve timing, sequencing and Must improve timing, sequencing and

frequency of surveys and censuses frequency of surveys and censuses • More surveys is not enough, must address More surveys is not enough, must address

reliability, comparability, relevance & use reliability, comparability, relevance & use • Activities:Activities:

– Inventory and dissemination of existing dataInventory and dissemination of existing data– Analysis of existing data Analysis of existing data – Assess past and on-going survey program; Assess past and on-going survey program;

review and improve survey methodsreview and improve survey methods– Design and implement improved medium-term Design and implement improved medium-term

survey program (within NSDS)survey program (within NSDS)

Accelerated Data ProgramAccelerated Data Program• ImplementationImplementation

– A MAPS activity (with Advisory Board)A MAPS activity (with Advisory Board)– Six pilot countries in Sub-Saharan Africa (12 Six pilot countries in Sub-Saharan Africa (12

pilot countries total)pilot countries total)– Development Grant Facility (DGF) ($2m per Development Grant Facility (DGF) ($2m per

year)year)– Likely to be extended to all African countriesLikely to be extended to all African countries– 3 to 5 years (from 2006)3 to 5 years (from 2006)– Cooperation with other donorsCooperation with other donors– Coordination with IHSNCoordination with IHSN

• Customized to countries needs/capacityCustomized to countries needs/capacity

Next?Next?• Continue roll-out programContinue roll-out program

– Packaging and Users ManualPackaging and Users Manual– Regional and national trainings, training of trainersRegional and national trainings, training of trainers– Country level capacity building, Accelerated Data Country level capacity building, Accelerated Data

ProgramProgram

• Translations and support for additional Translations and support for additional languageslanguages

• Maintenance, bug fixes and new featuresMaintenance, bug fixes and new features• Open source light web based cataloguing toolOpen source light web based cataloguing tool• Guidelines for archiving, preservation and Guidelines for archiving, preservation and

dissemination in developing countriesdissemination in developing countries• Platform harmonization (Eclipse, XForms, DDI Platform harmonization (Eclipse, XForms, DDI

3.0, XML DB)3.0, XML DB)

Thank you!Thank you!

QUESTION / ANSWERQUESTION / ANSWER