The Bioinformatics Testing Consortium: Improving Bioinformatics software one manuscript at a timeBetter Science through Superior Software - Titus Brown

Ben TempertonGiovannoni LabDepartment of MicrobiologyOregon State Universitybtemperton@gmail.com@bentemperton

Data problem identified

Bioinformatics - Dealing with the data deluge

Data problem identified

Bioinformatics - Dealing with the data deluge

Software written to analyze data

Data problem identified

Bioinformatics - Dealing with the data deluge

Software written to analyze data

Solution published and shared

Data problem identified

Bioinformatics - Dealing with the data deluge

Software written to analyze data

Solution published and shared

The Problem - Insufficient codebase quality checks

• Software is released having been tested only by those who wrote it

The Problem - Insufficient codebase quality checks

• Software is released having been tested only by those who wrote it

The Problem - Insufficient codebase quality checks

• Software is released having been tested only by those who wrote it

• Software rarely checked as part of the manuscript review process

Not a new problem in IT

‘Client’Developer

Unit testingIntegration testing

System testingUser Acceptance testing

Not a new problem in IT

‘Client’Developer

Unit testing

TesterIntegration testing

System testingUser Acceptance testing

Not a new problem in IT

‘Client’Developer

Unit testing

TesterIntegration testing

System testingUser Acceptance testing

InstallsRunsMeets Requirements

Not a new problem in IT

‘Client’Developer

Unit testing

TesterIntegration testing

System testingUser Acceptance testing

InstallsRunsMeets Requirements

Not a new problem in Science - Peer Review

Manuscript Peer Review:Improving quality for free*!

* not free

The Bioinformatics Testing Consortium

Testers run through standard

tests

Minimum standards

met?

Testers raise bugs/issues in discussion with

developer

Developer fixes bugs/issues

BTC

Developer places code in Open Source Repository

and registers project

Testers assigned to project by BTC

NO

YES

BTCAPPROVED

Does it compile?Does it run?Does the test data output match?...

Who Benefits?

• New Users

• Reduced activation energy when using BTC-approved software

• Compulsory inclusion of a test data set with known results

Who Benefits?

• New Users

• Reduced activation energy when using BTC-approved software

• Compulsory inclusion of a test data set with known results

• Developers

• Rapid awareness of issues

• Bugs raised in a consistent manner with enough details to replicate

Who Benefits?

• New Users

• Reduced activation energy when using BTC-approved software

• Compulsory inclusion of a test data set with known results

• Developers

• Rapid awareness of issues

• Bugs raised in a consistent manner with enough details to replicate

• BTC Testers

• Using students as testers teaches standards in early career

Who Benefits?

• New Users

• Reduced activation energy when using BTC-approved software

• Compulsory inclusion of a test data set with known results

• Developers

• Rapid awareness of issues

• Bugs raised in a consistent manner with enough details to replicate

• BTC Testers

• Using students as testers teaches standards in early career

• Journal Editors

• Assurance that software described in a manuscript is fit-for-purpose and key analyses can be reproduced

What’s Needed?

Feedback

What’s Needed?

Feedback

Participation

What’s Needed?

Feedback

Participation

Test Cases

What’s Needed?

Feedback

Participation

Test Cases

@BioTestConsort

With thanks to:

Iddo Friedberg

J. Cameron Thrash

Jonathan Eisen

Pascale Gaudet

Titus Brown

Cameron Neylon

Chris Sullivan

Jason Stajich

Stephen J. Giovannoni

Bank Beszteri

Hilmar Lapp