1November 17, 2009Jim Russell - EOPM Review

GATS

AIM Data Accessibility

John McNabb,

AIM Project Data Center Manager

Hampton University

2November 17, 2009Jim Russell - EOPM Review

GATS

AIM Data Accessibility

• AIM Science Data System (SDS)

• AIM Project Data Center (PDC)

• Data Availability and Access Statistics

• Lessons Learned

3November 17, 2009Jim Russell - EOPM Review

GATS

AIM Science Data System

• Distributed Data Processing and Delivery– Instrument Data Processing Centers (DPC):

• Process, store and disseminate data products for their instrument

• Centralized Meta-data, Search, and Archiving– Project Data Center (PDC):

• Provide search capability for instrument data products

• Interfaces to:– Affiliated Systems(Co-Investigators, NSSDC, E/PO K-14)– Non-affiliated Systems (Science community, Public)

4November 17, 2009Jim Russell - EOPM Review

GATS

Command Flow

Telemetry Flow

Post MissionMission Operations Elements

Science Processing Elements

Affiliated Systems

Non-affiliated Systems

MOC Mission Operations Center DPC Data Processing Center PDC Project Data CenterPOC Payload Operation Center

GroundStation

MOCLASP / CU

Co-

Investigators

AIM PDCHU

K-14Community

NSSDC

SOFIE

DPCGATS

CIPS/CDE

DPCLASP / CU

E/POHU

Public &Scientific

Community

POC

POC

5November 17, 2009Jim Russell - EOPM Review

GATS

MOCLASP

DPCCIPS/CDE

DPCSOFIE

Data Users

PDCHU

Raw Data Raw Data

GATS FTP

Meta-Data Meta-Data

Data Distribution

Search Queries

LASP FTP

Data Products Data Products

6November 17, 2009Jim Russell - EOPM Review

GATS

MOCLASP

DPCCIPS/CDE

DPCSOFIE

PDCHU

Raw Data Raw Data

Data Archiving

Data Products Data Products

NSSDC

Final Archive

LASP GATS

7November 17, 2009Jim Russell - EOPM Review

GATS

AIM Project Data Center

• Primary source of information to outside groups– Online Data Search tools– Preliminary Archive– Orbital Predictions

aim.hamptonu.edu

8November 17, 2009Jim Russell - EOPM Review

GATS

Online data search tool

9November 17, 2009Jim Russell - EOPM Review

GATS

Preliminary Archive

• Provides ongoing development and testing platform for final archive procedures.

• Collect all publicly available data as soon as it is published.

• Collect all data products and documentation in preparation for final archive.

10November 17, 2009Jim Russell - EOPM Review

GATS

Orbital Predictions

• Have provided regular orbital predictions for ground based sensors, sounding rocket launches, and Space Shuttle missions

• Online on-demand predictions available

11November 17, 2009Jim Russell - EOPM Review

GATS

Data Availability: CDE• CDE has released two levels of science

products:– Level 1: Charge

• Amount of charge generated by PVDF upon particles impact

– Level 2: Mass• Calculated mass of impacting particle

– All data available on Hampton Univ. AIM website• Currently at final version, V5

– Data available from June 1, 2007 to June 1, 2009

12November 17, 2009Jim Russell - EOPM Review

GATS

Data Availability: CIPS

• Preliminary data is available within 24 hours to Science Team and others by special request.

• After 5 days definitive revision of current version of data is created and published.

13November 17, 2009Jim Russell - EOPM Review

GATS

Data Availability: SOFIE• Preliminary data is available within 24

hours to Science Team

• After review (typically 2-3 days) definitive revision of current version of data is published.

• Online tools provide powerful plotting options

14November 17, 2009Jim Russell - EOPM Review

GATS

SOFIE Online Plotting Tools

http://sofie.gats-inc.com/sofie

15November 17, 2009Jim Russell - EOPM Review

GATS

Data Access Statistics: PDC

Average unique IP addresses per month based on Fall 2009 (excluding bots and local users)

• ~1500 to AIM web site

• ~700 to AIM homepage

• ~260 to outreach web pages

• ~75 to Data Search Tool

16November 17, 2009Jim Russell - EOPM Review

GATS

Data Access Statistics: CIPS/CDE

• Based on Google Analytics

• 175 Unique visitors over 3 months

• 41% new visitors

• Referrers:– 50% direct access– 25% google.com– 12% aim.hamptonu.edu

17November 17, 2009Jim Russell - EOPM Review

GATS

Data Access Statistics: SOFIE

• Based on Google Analytics and Webalizer data

• 25% new visitors

• Referrers:– 86% direct access. – 11% referred by aim.hamptonu.edu

• ~50 daily average visits

18November 17, 2009Jim Russell - EOPM Review

GATS

Data Access Statistics: SOFIE

19November 17, 2009Jim Russell - EOPM Review

GATS

Lessons Learned

• Product Availability Notices (PANs) sent by email can fall prey to changing SPAM filters. Realistic test cases for sending PANs need to include large numbers of files in a short period of time representing reprocessing of the whole data set.

• In file naming, version numbering system should include at least two integer places, one decimal, and a two character revision:  "v02.1_rev03"

• NetCDF-3 is not well suited for storing CIPS data. Official file types should be chosen that works best for the type of data needing to be stored, and that might be different for different data sets on a project.

• A distributed data dissemination system that takes advantage of shared data servers can work, however decentralized anonymous FTP makes tracking data downloads difficult. People will not generally register for update notifications unless required to in order to get access to the data.

• Overlapping transition period is very helpful when personnel change