Making scientific computations reproducible

  • Published on

  • View

  • Download


  • In the mid-1980s, we realized that our labo-ratorys researchers often had difculty re-producing their own computations withoutconsiderable agony. We also noticed thatjunior students, who typically build on the workof more advanced students, frequently spent agreat deal of time and effort just to reproducetheir colleagues computational results.

    Reproducing computational research poseschallenges in many environments. Indeed, theproblem occurs wherever people use the tradi-tional methods of scientific publication to de-scribe computational research. For example, in atraditional article, the author simply outlines therelevant computationsthe limitations of a pa-per medium prohibit complete documentation,which would ideally include experimental data,parameter values, and the authors programs.Readers who wish to use and verify the workmust reimplement it, which is often a painfulprocess. Even if readers have access to the au-thors source les (a feasible assumption given re-

    cent progress in electronic publishing) they canonly recompute the results by invoking the vari-ous programs exactly as the author invoked them;such information is something that is usually un-documented and difcult to reconstruct.

    To address these problems, we developed Re-Doc, a system for reproducing scientic compu-tations in electronic documents. Since imple-menting it in the early 1990s, ReDoc has becomeour principal means for organizing and transfer-ring our laboratorys scientic computational re-search.

    ReDocs are best dened operationally: Afteran author completes a ReDoc, readers can de-stroy all existing resultsprincipally the illustra-tionsand rebuild them using the authors un-derlying programs and raw data. Using ReDoc,authors describe their computations and preservetheir details in fully functional examples. Authorscan also test their archived research software byoccasionally removing and regenerating the doc-uments results using ReDocs standardized in-terface commands. ReDoc also lets authors de-velop automatic scripts to verify any documentscompleteness and reproducibility before its pub-lication. Scientic journal publishers might alsouse ReDoc in the referee process to test the re-producibility of illustrations.

    ReDoc benefits readers in several ways. Justas a driver wants to nd the brake pedal in the


    MAKING SCIENTIFIC COMPUTATIONSREPRODUCIBLETo verify a research papers computational results, readers typically have to recreate themfrom scratch. ReDoc is a simple software filing system for authors that lets readers easilyreproduce computational results using standardized rules and commands.

    S C I E N T I F I CC O M P U T A T I O N S


    1521-9615/00/$10.00 2000 IEEE


    same location in every car, readers benefit byhaving a few standard commands that let themexplore any documents scientic contents. Con-sistent, standard commands to remove and re-produce a documents result les help readers ac-cess and study an unknown document.

    ReDoc overview

    In essence, ReDoc is a ling system that makesresearch cooperation effortless: Researchers canpublish their results and colleagues can imme-diately build upon them, adding improvementsand innovations. To create a ReDoc, authorscombine project-specific rules with general,standard rules and naming conventions to offerreaders a simple set of commands to remove andrebuild the authors results. ReDocs becomereservoirs of easily maintained, reusable soft-ware, and thus authors, along with readers, ben-et from using the system.

    In our laboratory, using ReDoc has tremen-dously increased the amount of software that isreadily accessible to researchers. Students cannow take up former students projects and eas-ily remove and recompute the original resultles. Students who graduate and leave our lab-oratory can seamlessly continue their researchat new locations. Although the effort to gener-ate reproducible research documents is typicallydirected at helping other people use existingwork to further their own, authors also benet.Because many researchers typically forget detailsof their own work, they are not unlike strangerswhen returning to projects after time away.Thus, efforts to communicate your work tostrangers can actually help you communicatewith yourself over time.

    ReDocs current reader interface is implementedin the platform-independent GNU make, whichexcels in efciently maintaining even complex soft-ware packages. Conceptually, the ReDoc reader interface is independent of both document format(TeX, HTML, and so on) and the underlyingcomputational software, whether it be Matlab orMathematica, or C and Fortran programs. Al-though we restrict our focus here to Unix systemsand the make utility, the conceptusing a readerinterface to reproduce a documents computationalresultsshould apply to electronic documents inother computer environments as well. The GNU code that implements the ReDoc rules, this arti-cle, and the accompanying example is freely avail-able on our Web site at

    ReDoc features

    In addition to a traditional article and the ap-plications source code, a ReDoc features threemain components: makeles, a small set of uni-versal make rules (less than 100 lines), and nam-ing conventions for les. ReDoc offers a readerfour standard commands:

    burn the gures, build the gures, view the gures, and clean up and remove all intermediate results.

    These four commands are centered around threetypes of les:

    Fundamental les are data sets, programs,scripts, parameter les, and makeles (any lenot generated by some computer process).

    Result les are typically plot les, which cancome in various formats including postscriptor gif les.

    Intermediate les are the les that a computerprocess generates when computing the re-sult files from the corresponding funda-mental les. Examples include object les,executables, and partially processed data.

    ReDocs naming conventions let a communityformulate universal rules that recognize a lestype and thus permit the system to handle it ap-propriately. For example, a cleaning rule uses acommunity naming convention to identify andremove all intermediate les, such as those withsufx .0 (object les) or les with the name stemjunk (temporary les). The clean rule leaves atidy directory without any distracting clutter. Wealso use naming conventions for result les. Webase our rules for displaying, removing, or re-computing a result le on the result les name.For example, our result les are always gure lesand can have various formats, such as PostScriptor GIF. Our naming conventions require that au-thors indicate the result les format by a sufx,such as .ps or .gif. Consequently, we can supply auniversal format-independent rule for displayingresult les: The rule identies the result les suf-x, concludes the les format, and invokes theappropriate viewing program, such as Ghostviewfor PostScript or Xview for GIF les.

    ReDoc rules are easy to implement. As the ex-ample in the box, Creating a ReDoc explains,authors who already use makeles need only ad-here to the ReDoc naming conventions and in-clude the ReDoc rules to make a traditional doc-


    ument reproducible. We now describe ReDocmakeles and rules in more detail.

    MakefileA makele is a standard Unix utility for soft-

    ware maintenance and contains the commandsto build a software package. More powerful thana simple command script, a makefile indicatesthat result files (targets) are up-to-date whenthey are younger than their correspondingsource les (dependencies). ( For more informa-tion, see the clean, up-to-date support side-bar.) Because maintaining a reproducible re-search project resembles maintaining software,the make utility solves our problem elegantly.

    Fortunately, ne tutorial books exist on the makelanguage,1,2 and students can easily begin usingmake without a formal introduction.

    Because our laboratory deals with computa-tional problems of various sizes that use a diversecollection of software and hardware tools, notevery reader will be able to easily reproduce allresult les. Consequently, authors typically de-fine their application makefiles as one of threeresult list variablesRESULTSER, RESULTSCR,and RESULTSNRwith the ending letters indi-cating the degree of reproducibility:

    ER: Easily reproducible result les that canbe regenerated within 10 minutes on a stan-

    Creating a ReDocBecause an authors rules deal with a documents appli-

    cation, we can best illustrate ReDoc creation using anactual example. The electronic version of this article is ac-companied by a subdirectory called Frog. Frog contains acomplete, albeit small, ReDoc that discusses a finite-differ-ence approximation of the 2D surface waves caused by afrog hopping around a rectangular pond. The files paper.latex and contain two formats of the short scientificarticle describing the finite-difference approximation. SomeRatfor files implement the 2D wave propagation code (Rat-for is a Fortran preprocessor that provides control flow con-structs similar to C; Ratfor is freely available at our Website). The Fig directory contains the result files: a figure(PostScript and GIF versions) of the pond after some wildfrog hops, and the output (two float numbers) of a dot-product test of the linear finite-difference operator andits adjoint. To organize the documents files, the Frog ex-amples author wrote the makefile shown in Figure A.

    The variable RESULTSER contains the list of the docu-ments easily reproducible results, frog, and dot.

    The next rule contains the commands to build thePostScript and GIF versions of the frog result. Such a ruleis application-specific and cannot be supplied by existingdefault rules. The target names comprise the directoryRESDIR, which contains the result files and file suffixes(.ps and .gif), which indicate the file formats. The ruledepends on an executable frog.x, which it executesduring result computations.

    A shared include file, Prg.rules.std, supplies defaultrules for compilation and linking of executables such asfrog.x. Compilation and link rules are compiler-depen-dent. In this example, we include some generic Fortranrules. At our laboratory, compilation rules depend on anenvironment variable that indicates compiler type.The author must define the executables dependency

    on its subroutine object files (as in the case of frog.x) be-cause it depends on the application-specific file names.

    In the case of the dot result file, the author-suppliedrules reflect the reader-interface commands: cre-ates the result file, dot.view displays it, and dot.burnremoves it.

    Finally, the target clean invokes the included defaulttarget jclean, which removes intermediate files based onour laboratorys naming conventions.

    SEPINC = ../rulesinclude ${SEPINC}/

    RESULTSER = frog dot

    col = 0.,0.,0.-1.,1.,1.${RESDIR}/ ${RESDIR}/frog.gif: frog.x

    frog.x > junk.pgmpgmtoppm ${col} junk.pgm > junk.ppmppmtogif junk.ppm > ${RESDIR}/frog.gifpnmtops junk.pgm > ${RESDIR}/

    objs = copy.o adjnull.o rand01.o wavecat.o \pressure.o velocity.o viscosity.o wavesteps.o

    frog.x: ${objs} ${RESDIR}/dot.txt : dot.xdot.x dummy > ${RESDIR}/dot.txt

    dot.view: ${RESDIR}/dot.txtcat ${RESDIR}/dot.txt

    dot.burn: rm ${RESDIR}/dot.txt

    dot.x : ${objs}

    clean: jclean

    include ${SEPINC}/Doc.rules.redinclude ${SEPINC}/Doc.rules.idocinclude ${SEPINC}/Prg.rules.std

    Figure A. The makefile that organizes the Frog files.


    dard workstation. NR: Nonreproducible result les that cannot

    be recalculated (examples include hand-drawn illustrations or scanned gures).

    CR: Conditionally reproducible result lesthat require proprietary data, licensed soft-ware, or more than 10 minutes for recompu-

    tation. The author nevertheless supplies acomplete set of source les; this ensures thatreaders can reproduce the results if they pos-sess the necessary resources. Authors list therequired resources in a warning le (such asmyresult.warning), which accompaniesthe CR result le ( In indus-trial-scale geophysical research, many inter-esting results are conditionally reproducible.

    Following this usage, the standard make targetsburn and build, which we describe below, arecomplemented by targets burnER, burnCR,burnNR, and buildER, buildCR, buildNR. Forexample, burnCR burns all conditionally repro-ducible result les. The target burn defaults toburnER to restrict the standard removal of resultles to easily reproducible ones. The target builddefaults to buildER to recompute the result lesthat make burn removes.

    Make rulesReDocs third component is a small, consis-

    tently available set of standard make rules. Theserules let readers interact with the documentwithout knowing the underlying application-specic commands or les. In the Frog example(see the box, Creating a ReDoc), a reader in-vokes targets, such as build, that are not listedin the authors application makefile. Standardlaboratory rules supply these targets, which theresearcher includes in the documents makele(,, andDoc.rules.idoc).

    Standard rules ensure a consistent reader in-terface and free authors from the need to reim-plement the ReDoc rules in every makele, let-ting them concentrate exclusively on themakeles application-specic aspects. Althoughwe formulated the rules so that individual au-thors can override them, in our experience this israrely necessary or desirable. Furthermore, acentral rules set accumulates the communityswisdom on how to organize a ReDoc.

    Our laboratory offers about 100 lines of GNUmake code that constitutes our ReDoc rules. TheReDoc rules facilitate four commands: makeburn, which removes the result les (usually g-ures); make build, which recomputes them;make view, which displays the gures; and makeclean, which removes all les other than sourceor result les. Although authors and readers donot need to understand the implementation de-tails of the ReDoc rules to use them (most re-searchers at our laboratory have never inspected

    burn: burnER

    burnER: ${addsufx .burn, ${RESULTSER}}burnCR: ${addsufx .burn, ${RESULTSCR}}

    %.burn: ${foreach sfx, ${RES_SUFFIXES} , \

    if ${EXIST} ${RESDIR}/$*${sfx} ; then\${RM} ${RESDIR}/$*${sfx} ; ; \


    Figure 1. make burn finds and removes all easily reproducibleresult files.

    view : ${addsufx .view, ${RESULTSALL}}

    %.view: FORCEif ${CANDO_GIF} ; then \

    ${MAKE} $*.viewgif ; \elif ${CANDO_PS} ; then \

    ${MAKE} $*.viewps ; \else \echo cant make $*.viewps $*viewgif;\


    %.viewgif : ${RESDIR}/%.gif FORCE${XVIEW} ${UXVIEWFLAGS} ${RESDIR}/$*.gif

    %.viewps : ${RESDIR}/ FORCE${GVIEW} ${UGVIEWFLAGS} ${RESDIR}/$*.ps


    Figure 3. The view rule (a) updates and displays results. When the%.view rule finds a workable version, it invokes %.viewgif or%.viewps, which updates and displays the result f...


View more >