Dagstuhl Presentationn 2012 - Larry Hoyle 1
Documenting Research Project Process for Reproducibility
Larry HoyleInstitute for Policy & Social Research
University of Kansas
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 2
The challenges
• Large (or complex) multi-disciplinary projects– Multiple sites, data streams, standards, and
practices– Complex data preparation procedures
• Point and click software used• Documenting as overhead
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 3
Example Project
• Farmer's land use decisions related to climate change (e.g. biofuel related crops)
• One component of larger NSF grant • Multiple teams, multiple universities – The two main sites are 135 km apart
• Multi-disciplinary– Economists, geographers, agronomists, biologists, engineers,
climate scientists, anthropologist, sociologist, political scientists, urban planner, GIS experts, photographer
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 4
Example Project Data– Develop substantial geodatabase (ARC SDE)• ground cover, soils, crop statistics, facility locations (e.g.
purchaser, processing plant). Weather, climate, watershed and aquifer models, • Sub-(farmer’s) field geographic level
– Climate models at different scales– Focus groups and multi wave survey (geocoded)– Interviews coded in NVIVO (geocoded)– Photographs– Large proprietary dataset with time-limited use
10/22/2012
Challenge - put it all together and document how it was done and how everything relates.
Other example: Iassist posting
Dagstuhl Presentationn 2012 - Larry Hoyle 5
Spatial Aspects
• Reconciling different spatial schemes at multiple scales across time– Raster images, – model grids at different scales, – weather point sources, other point locations (e.g. biorefineries), – political entity polygons (state, county), – farm field and sub-field polygons, – Attribute data at all these levels, imputed and aggregated data
• Harmonizing data from different geographic schemes• Producing new spatial objects
– E.G. corners as separate from circle with center-pivot irrigation
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 6
New Polygons
10/22/2012
Polygons to be extracted from remote sensing imagery
Subfield areas sometimes growdifferent crops(corners are 21% of the square)
Dagstuhl Presentationn 2012 - Larry Hoyle 7
Need to Capture Process Example 1
• Project member with expertise volunteered to process data to produce a spatial dataset (soils data).
• Users of the dataset discover anomalies• Expert no longer available, can’t remember
quite what he did and has no documentation (used point and click tools)
• Ouch
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 8
Process Example 2
• Qualitative analysis– Transcription– Multiple coders, common coding scheme– Coding scheme evolves (capture this?)– Training– Paired coders code each interview– Testing of coder reliability
• Integrate this after the fact with geodatabase
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 9
Point and Click
• Some tools are only point and click and don’t create a log.– E.g. Some procedures in ArcGIS
• How do you document process– Screen capture pasted into Word?– Action recording software– Discoverable? Machine actionable?
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 10
An ArcGIS process (different project)
10/22/2012
NSFCHEMAnnualDataProcedure.docx
AnnualLinksByTime4.avi
Dagstuhl Presentationn 2012 - Larry Hoyle 11
Need Tools
• There is a need for tools built on top of standards that make it easy to capture and annotate process
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 12
Need Tools to Capture ProcessOne example – SAS Enterprise Guide
10/22/2012
• Can modify nodes during development. • Can run the process from any point
• But – overall process may involve multiple tools - in this case also R and ArcGIS. In other cases, multiple people in different settings.
Scott Long - The Workflow of Data Analysis Using Statahttp://www.indiana.edu/~jslsoc/web_workflow/wf_home.htm
Datasets – Permanent and temporary
Dagstuhl Presentationn 2012 - Larry Hoyle 13
Capturing Process as it is Being Developed• False starts and blind alleys– Does the whole process matter or only a process that
reproduces the final result? (learn from my mistakes?)– Description of process gets edited as it evolves
• Adding minimal overhead– If the tool requires a lot of attention it won’t get used.
• Combining sub-processes• Filling in pieces of overall planned project• Parallel parts• Time as ordinal or interval (or ratio?)
10/22/2012
Dagstuhl Presentationn 2012 - Larry Hoyle 14
• Annotated screen capture – works on top of any software– Text (or audio/video?) annotation– Dealing with IP in captured images– Flow diagram with popups?– Editable– Time stamped
Tools – The Fantasy
10/22/2012
Sub process edited separately
Planned overall process
Persistent identifiers allow (re-)linking
Dagstuhl Presentationn 2012 - Larry Hoyle 15
Final thoughts
• Metadata for the audience– Documentation for reproducibility– Documentation in cases of disputed results
• Sometimes the researcher is the audience– One researcher commented that having
documentation at this level would be very helpful in writing methods sections of papers.
– Teaching tool - critique students process– Assists refining methods– Also useful in future similar projects
10/22/2012