EMEN2Steve Ludtke

NCMIBaylor College of Medicine

NCRR

Current EMEN Database

• Diverse data requirements:

purification data collection reconstruction • OODB (Zope based)• Direct equipment interface• 290 users from dozens of labs• ~7 Tb image data• 2-3 Tb/year of image data • 385,000 total records

LIMS

• Centralized databases (PDB, EBI, etc.) vs. in-house archives with detailed information

• Scientific Database vs. Electronic Notebook

Film/Frame Id

Defocus MagTotal Dose

ExpTime

ccd_20040314193633 #1

0.8 um 138.44k 20.0 1.0 s

ccd_20040314193634 #2

2.5 um 138.44k 20.0 1.0 s

ccd_20040314194642 #1

0.8 um 138.44k 20.0 1.0 s

ccd_20040314194643 #2

2.5 um 138.44k 20.0 1.0 s

ccd_20040314194917 #1

0.8 um 138.44k 20.0 1.0 s

ccd_20040314194918 #2

2.5 um 138.44k 20.0 1.0 s

ccd_20040314195514 #1

0.8 um 138.44k 20.0 1.0 s

ccd_20040314195515 #2

2.5 um 138.44k 20.0 1.0 s

ccd_search1 #1

0.8 um 138.44k 20.0 1.0 s

ccd_search2 #2

2.5 um 138.44k 20.0 1.0 s

ccd_search3 #1

0.8 um 138.44k 20.0 1.0 s

ccd_search4 #2

2.5 um 138.44k 20.0 1.0 s

ccd_search5 #1

0.8 um 138.44k 20.0 1.0 s

ccd_search6 #2

2.5 um 138.44k 20.0 1.0 s

• Excellent mineability• Limited flexibility• Good for centralized databases (standards)

Excellent flexibilityRich information contentLimited mineability

Goals

• BOTH flexibility and mineability

• KISS

• Database should think like the scientist, not the other way around

• Archive detailed experimental protocols

• Association of databases

New Concepts

• Object Oriented Databases (OODB)

• Web Ontologies (semantic web)

• Evolving collaborative environment (wikipedia)

• XML

• Peer to Peer Networking

• Blogging

Experimental Protocol

Experimental Parameter

Record

DescriptiveText

Experimental ParameterExperimental

ParameterExperimental Parameter

Experimental Parameter

Experimental Parameter

Temperature

Parameter Ontology

Ambient_Conditions

Ambient_Temperature Ambient_Pressure

Ambient_RelativeHumidity

Specimen_Temperature

Grid_Temperature

Grid_Temperature_Imaging

Grid_Temperature_Previtrification

grid

negative_stain_gridvitrified_grid

manually_vitrified_grid vitrobot_vitrified_grid

manually_vitrified_grid_flash_photolysis

TEM_specimen

holey_carbon_grid

quantifoil_grid

Protocol Ontology

manually_vitrified_grid

A preprepared #grid was placed in a pair of forceps and loaded into the plunger. $cryogen was preprepared below the plunger (ethane and other cryogens which may become solid are reliquefied using a room temperature copper rod immediately prior to plunging). $grid_volume of specimen was deposited on the front of the grid using a pipette. The grid was then blotted on $grid_blot_side using $filter_paper_type and the plunger was triggered after a $grid_plunge_delay to rapidly submerse the grid in the cryogen. The forceps were then released from the guillotine and the grid was placed in $grid_storage_id in $grid_storage_slot. The grid storage button was then placed in $cryofreezer_id for storage until imaging.

EMEN2

• Ease of use (new protocols without DBA)• Protocol archival• Protocol and Parameter Ontologies (P2P)• ‘Blogging’• Traceability• Workflow• Dissemination (mirroring)• Data Mining

plot bfactor vs truedefocus where truedefocus is between 0.1 and 5.0 and bfactor is between 1 and 1000

Microscopy Session

CCD FrameMicrograph

Scan

Particles

Project Microscope

Particles

plot bfactor vs truedefocus where truedefocus is between 0.1 and 5.0 and bfactor is between 1 and 1000

split by protocol

plot bfactor vs truedefocus where truedefocus is between 0.1 and 5.0 and bfactor is between 1 and 1000

split by creator

plot bfactor vs truedefocus where truedefocus is between 0.1 and 5.0 and bfactor is between 1 and 1000

split by microscope

Microscopy Session

CCD FrameMicrograph

Scan

Particles

Project Microscope

Particles

EMEN2 Status

• Core library functional– BerkeleyDB + Python

• All EMEN data EMEN2• Begun work on Web-based front-end

– Apache + Cheetah

• P2P incorporated into design, but implementation incomplete

• Formal XML interfaces (OWL for exchange incomplete)

Acknowledgements

Thanks to:National Center for Research ResourcesAgouron Institute

• Haili Tu• Runsun Pan

Relational vs. OODB

Tables

Tabular Storage

Fixed Records

No Table-mixing

Classes

Hierarchical Storage

Flexible Records

Mixed Class Reports

0

1000

2000

3000

4000

5000

6000

7000

Scanned Micrographs

CCD Frames

EMEN Goals

• Project Management• Data Archival• Data Mining• Automation• Flexibility• Communication with Collaborators• Dissemination• Portability

plot intendeddefocus vs truedefocus where truedefocus>0 and intendeddefocus>0

Group     Address      Affiliations      City      Contact Email      Contact Name      Fax #      Group Name      Institution     Phone #      State      Support Sources      Website URL      Zip Code

Microscope     CCD Camera Type      CCD Model #      CCD Serial #      CCD Size X      CCD Size Y      CC      CS      Microscope Serial #      Pole Piece      Sensitivity      Title     Voltage

Project     Axis Codes      Bio-hazard Codes      Biomedical Properties      Biophysical Properties     Biochemical Properties      Genetic Characterization      Goals of Project      Height (of specimen)      Keywords for Project      Length (of specimen)      Mass (of specimen)     Particle Diameter      Project Description      Project Title      Sequence      Specimen      Storage Location (Data)      Symmetry (of specimen)

User     Address      Degrees      Department      Email      First Name      Institution      Last Name      Login Name      Phone

Aliquot     Buffer      Concentration      Date Received      Identifiers      Received By      Storage Loc.      Volume

Freezing Session     Aliquots Used      Apparatus      Blotting      Concentration      Frozen By      Grid Batch      Grids Used      Grid Type      Hole Size      Mesh Size      Number of Grids      Post Freezing      Pre-Treatment      Storage Loc.      Substrate      Substrate Prep.      Freezing Tech.      Vitrobot Parm.

Purification     Buffer      Concentration      Description      Purification Meth.      Spec. Stability      Storage Condn.

Labnotebook     Links (web)      Notebook Text

Structure Factor     From Whom      Processed      Source

Reference     Comments

CCD     Anti Blooming      Astigmatism Parm.     Beam Diameter      B Factor      Binning      Camera      Camera Length      Camera Units      Dose      Drift Parm.      Energy Filter      Exposure #      Exposure Time      Frame ID      Ice Comments      Ice Thickness      Identifier      Intended Defocus      Lens Current      Magnification      Peak S/N Ratio      Screen Current      Screen Magnification      Tilt Angle      True Defocus      True Magnification      (X,Y) Coordinates      Zoom Factor

Micrograph     Amplitude Cont.      Astig. Parm.     Beam Diameter      B Factor      Camera Length      Camera Units      Contamination      Dose      Drift Parm.     Energy Filter      Exposure #      Exposure Time      Film ID      Ice Comments      Ice Thickness      Illum. Angle      Intended Defocus      Lens Current      Magnification      Maximum Res.      Micrograph Qual.      Peak S/N Ratio      Screen Current      Screen Mag.      Tilt Angle      True Defocus      True Mag.      (X,Y) Coordinates

Microscopy Sess.     Apertures      Camera Length      Camera Units      Condenser      CS      Develop Time     Film Type      Freezing Session      Magnification      Microscope      Room Humidity      Room Temp.      Specimen Temp.      Spot Size      Voltage Scan

     Averaging Fac.      Brightness      Contrast      Exposure Time      Parameters      Scanned By      Scanner Used      Scan Step