View
620
Download
0
Category
Preview:
Citation preview
Documentation and Metadata
Sherry Lake
Data Life Cycle
Re-Purpose
Re-Use Deposit
Data
Collection
Data
Analysis
Data
Sharing
Proposal
Planning
Writing
Data
Discovery
End of
Project
Data
Archive
Project
Start Up
Andrea Denton
We’ll Explore
• Why is documenting your research important?
• What do you document (files? datasets? projects? Hands-on
• What are the common types of documentation?
• Metadata: What is it? Why is it important? Hands-on
• Q & A
You’re already documenting your data
• Notebook
– Paper
– Digital
– Lab
• Folders with notes, text files
• Sources, experiments or surveys,
procedures, etc.
Critical roles of data documentation
• Data Use
– To know enough details about how the how the data were collected and stored
• Data Discovery
– To be able to identify important data sets
• Data Retrieval
– To know how and where to access data
• Data Archiving
– Data can grow more valuable with time, but only if the critical information required to retrieve and interpret the data remains available
Information EntropyIn
form
atio
n C
on
ten
t o
f D
ata
and
Met
adat
a
Time of data development
Specific details about problems with individual items or specific dates are lost relatively rapidly
General details about datasets are lost through time
Accident or technology change may make data unusable
Retirement or career change makes access to “mental storage” difficult or unlikely
Loss of investigator leads to loss of remaining information
TIME
From Michener et al 1997
http://dx.doi.org/10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2)
Elements of Documentation
Good data documentation answers these
basic questions:
• Why were the data created?
• What is the data about?
• What is the content of the data? The
structure?
• Who created the data?
• Who maintains it?
Elements of Documentation, continued
• How were the data created?
• How were the data produced/analyzed?
• Where was it collected (geographic
location)?
• When were the data collected? When
were they published?
• How should the data be cited?
Documentation throughout your research
Variable or Item Level File or Dataset Level Project or Study Level
• Labels, codes, classifications
• Missing values (andhow they are represented)
• Inventory of data files
• Relationship between those files
• Records, cases, etc.
• What the study set out to do; researchquestions
• How it contributes new knowledge to the field
• Methodologies used, instruments and measures
UK Data Service: http://ukdataservice.ac.uk/media/440277/documentingdata.pdf/
Exercise 1: Exploring Documentation
• Refer to the files on the Data Management
Bootcamp site, either
– http://guides.lib.odu.edu/VADMBC/materials
• In the section Documentation and Metadata
Exercise_1_Data_Documentation Worksheet
– Or, you may have a handout “Exercise 1”
Exercise 1: Exploring Documentation
• For Column 1, take 2-3 minutes and, for each
row, write down what general concept (who,
what, when, where, how, or why, or a combination
of these) that field describes about data, if
applicable.
• Now take 2-3 minutes to complete Column 2.
Considering your research data, what
information would you provide for each field?
• Don’t have research data? Use the file
DailyWeather to fill in Column 2.
Exercise 1 continued
• Take 2 minutes
• There is a blank row under each category for any
information specific to your field, e.g. latitude and
longitude, species, etc.
• Please share an example with the class in the
Google doc “Questions: Ask them here”
Wrapping up: elements of documentation
• We’ve looked at commonly used fields
• What does your discipline say about
what you should document?
• The answers you’ve provided could be
used to create a data dictionary
– we’ll examine next
Types of Documentation
• ReadMe File
• Data Dictionary
• Codebook
ReadMe
• Describes the core documentation about
an investigation and its data files
• Typically a simple text file
• Can describe the individual file(s) and/or
data package as a whole
ReadMe Example - File
ReadMe Example - File
ReadMe Example - Dataset
Data Dictionary
• Provides definitions of the data fields in a
data file
• More details on the variables, observations
of a file
Data Dictionary
• Used to understand the data and the
databases that contain it
• Identifies data elements and their
attributes including names, definitions and
units of measure and other information
• Often they are organized as a table
http://www.pnamp.org/sites/default/files/best_practices_for_data_dictionary_definitions_and_usage_version_1.1_2006-11-14.pdf
Data Dictionary Example: the dataset
http://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/GetPdf.cgi?document_name=HowToSubmit.pdf
Data Dictionary Example: the dictionary
Exercise 2: Data Dictionary
• Refer to the files on the Data Management
Bootcamp site, either
– http://guides.lib.odu.edu/VADMBC/materials
• In the section Documentation and Metadata
Exercise_2_DataDictionaryTemplate
– Or, you may have a handout “Exercise 2”
• Open the file DailyWeather
Weather data source:http://www.ncdc.noaa.gov/cdo-
web/search?datasetid=GHCND
• Use the Daily Weather dataset
– Two worksheets (tabs)
• Data
• Definitions
• Start by answering the questions
• Fill out a data dictionary for this dataset
Exercise 2: Data Dictionary Creation
Exercise 2 Discussion
What is a Codebook?• Typical in social sciences research
• Includes elements similar to readme and
dictionary
– Project level information (e.g. survey design
and methodology)
– Response codes for each variable
– Codes used to indicate nonresponse and
missing data
http://www.icpsr.umich.edu/icpsrweb/ICPSR/support/faqs/2006/01/what-is-codebook
What is a Codebook?
• Additionally, codebooks may also contain:
– A copy of the survey questionnaire (if applicable)
– Exact questions and skip patterns used in a
survey
– Frequencies of response
• Quite long!
http://www.icpsr.umich.edu/icpsrweb/ICPSR/support/faqs/2006/01/what-is-codebook
Codebook Example
http://www.icpsr.umich.edu/icpsrweb/ICPSR/help/cb9721.jsp
Codebook Example
http://dataarchives.ss.ucla.edu/archive%20tutorial/aboutcodebooks.html
Other Examples of Data Documentation
• Lab notebooks
• Software syntax
• Programming code
• Instrument settings and/or calibration
• Provenance of sources of data
• Embedded metadata (e.g. EXIF, FITS)
Metadata
• What is it?
– Information that describes a resource
– NISO: “metadata is structured information that
describes, explains, locates, or otherwise makes it
easier to retrieve, use, or manage an information
resource”
• Why is it important?
– Enables a resource or data to be easily
discovered
– Good metadata will help others understand and
use your data
Metadata in Everyday Life
DataONE Education Module: Metadata. DataONE. Retrieved Nov 12, 2012. From http://www.dataone.org/sites/all/documents/L07_Metadata.pptx
Author(s) Boullosa, Carmen. Title(s) They're cows, we're pigs /
by Carmen Boullosa Place New York : Grove Press, 1997.
Physical Descr viii, 180 p ; 22 cm. Subject(s) Pirates Caribbean Area Fiction.
Format Fiction
Metadata Formats
• Documentation for understanding & re-use
– Readme File
– Data Dictionary
– Codebook
• Structured documentation in XML format for use in programs (few examples)
– DDI
– FGDC
– EML
Exercise 3: XML File Creation
• Refer to the files on the Data Management
Bootcamp site, either
– http://guides.lib.odu.edu/VADMBC/materials
• In the section Documentation and Metadata
Exercise_3_Weather-DDI-XML-FillinBlanks
– Or, you may have a handout “Exercise 3”
Exercise 3: XML File Creation
• Take the file Weather-DDI-XML and fill in
the blanks (as best you can) using:
• the file DailyWeather
• and/or Exercise 2 Data Dictionary
Exercise 3 Discussion
Exercise 3 Discussion
Exercise 3 Discussion
Structured XML
A Few Standard Schemes (XML)
– DDI– Data Document Initiativehttp://www.ddialliance.org/
– FGDC– Geospatial Metadata Standardhttp://www.fgdc.gov/metadata/geospatial-metadata-standards
– EML– Ecological Metadata Languagehttp://knb.ecoinformatics.org/software/eml/
FGDC Example
Structured Metadata Tools
Tools
– Colectica add-on for Excel (DDI)
– Nesstar (DDI)
– Metavist (FGDC)
– ArcGIS (FGDC) *
– Morpho (EML)
http://data.library.virginia.edu/data-management/plan/metadata/metadata-workshop/
Example 1: Nesstar DDI Tool
Example 2: Metavist FGDC Tool
Metadata Concept Map by Amanda Tarbet is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Metadata Standards
Metadata Wrap-up
How to chose a metadata standard or
documentation format?
• What does your discipline use?
• Look at what depositing repository requires
Research Life Cycle
Data Life Cycle
Re-
Purpose
Re-
Use
Deposit
Data
Collection
Data
Analysis
Data
Sharing
Proposal
Planning
Writing
Data
Discovery
End of
Project
Data
Archive
Project
Start Up
QUESTIONS?
Recommended