The Insight Journal

Kitware Inc.

Credits

Julien Jomier – UNC Caddlab

Stephen Aylward – UNC Caddlab

Zack Galbreath – Kitware

Andy Cedilnik – Kitware

Bill Hoffman - Kitware

The Insight Journal

Technical work must be reproducible

Papers should be publicly accessible

The Peer-Review process must be open

Let’s Implement this paper….

Happy Grad Student, unaware

of her destiny

Supervisor

“You just need to implement the code

and try it…”

Let’s Implement this paper….

Unhappy Grad Student, aware of

her destiny

Six months later…

• Parameters are missing

• Images are not available

• Implementation details are missing

• Abundance of pretty equations

• Abundance of pretty screenshots

Time to face the

Truth

Publications are a measure

of scientific productivity

They disseminate knowledge

They allow others to reproduce our results

They are validated by the peer-review process

Why do we really

want to publish ?

Because we need

to have publications

in our CV

We have met the enemy…

and he is us !

“Publish or Perish”

Who invented this ?

and Why ?

“Publish or Perish”

Was invented by those who needed to evaluate

researcher’s productivity.

“Publish or Perish”

Empowers those who read

your CV to grade you by

simply counting lines in the

“Publications” section.

“Publish or Perish”

The group of best educated

people in the world has been

alienated with a simple trick

The Scientific Method

Hypothesis

OutcomeExperiment

Galileo Galilei

HypothesisRejection

Confidence

The Scientific Method

The Logic of Scientific Discovery

by Karl Popper

Image Processing Method

AlgorithmInput

Image(s)

Parameters

OutputImage(s)

Insight JournalManager

Author

SubmitsProject

DSpace

StoresProject

AutomaticTesting

Environment

PostsResults

Dashboard

SubmitsEntry

Reviewer

ReviewsProject

Process Flow

SendsCode

Submission

Code

InputData

JournalRepository

WebSite

ResultsData

Author

BuildMachines

PDF doc

• Comparison of output images

• Publicly available

• Tests for non-trivial images

• Reviewers and readers can use it as a first view of the paper.

Results Verification

DSpace

• Open Source

• Developed by MIT & HP

• Persistant URIs

• Prevents data from becoming obsolete

Insight JournalManager

DSpace

StoresProject

Insight Journal Manager

• Open Access

• Peer Review

• Searchable

Insight JournalManager

Author

ReviewerReviewerReviewerReviewer

Peer-Review is ONLY useful if it enforces reproducibility

Testing Environment

• Security Issues

• Open source tools

• Dashboard

Insight JournalManager

AutomaticTesting

Environment

Dashboard

Enjoy the Insight Journal !

Implementation Details

Insight Journal Manager

http://insight-journal.org Web interface: PHP/Java/HTML Central place for submissions, reviews,

browsing/searching No delay between submissions & reviews Public reviews

Issues

Title, comments. Creation/Paper Due/Notification/Publication

dates Tied to a DSpace collection Submission through DSpace + comments

Adding reviews

Rating Level of Affiliation Expertise Anonymous Review Title Review

Browsing reviews

No delays Users can post comments Authors can reply to comments PDF can be downloaded

Scoring

Weighted average of: Sensitivity (STAPLE algorithm) Helpfulness # submissions/max submissions # reviews/max reviews # anonymous reviews

What’s next?

Moderated submissions SVN repository for code fixing …

Safe Testing

Virtualization!

Lots of options: User-Mode-Linux, coLinux, VMWare, chroot

Xen: http://www.cl.cam.ac.uk/Research/SRG/netos/xen/

Tools used in the testing process

Xen Debian CMake Python Dashboards

Testing Details

two CPUs and two partitions, so two tests can be run simultaneously

VMs can only access their own partition; no network access

Mutexes prevent filesystem corruption Submit test results to dashboard Restore pristine filesystem after every test

Enjoy the Insight Journal !