John Kusterer Head, ASDC

ASDC UPDATE CERES SCIENCE TEAM MEETING

MAY 5, 2015

May 5, 2015 ASDC Update for CERES STM Page: 2

Agenda

•  CERES User Metrics •  ASDC Private Cloud Pilot •  Big Earth Data Initiative (BEDI) and Data Services

ASDC CERES USER METRICS

May 5, 2015 ASDC Update for CERES STM Page: 4

0

10

20

30

40

50

60

70

80

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Volu

me

(TB

)

TERRA AQUA NPP FLASH

CERES and FLASHFlux Archive Volume By Data Date Through March 2015

Total Volume Archived ~ 1,104 TB NPP Volume Archived ~ 14.1 TB

May 5, 2015 ASDC Update for CERES STM Page: 5

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

2010 2011 2012 2013 2014 2015

Num

ber o

f Ord

ers

CERES Data Orders (June 2010 – March 2015)

Total Orders: 27,242

May 5, 2015 ASDC Update for CERES STM Page: 6

CERES Data Distribution (June 2010 – March 2015)

0

10

20

30

40

50

60

70

80

90

100

2010 2011 2012 2013 2014 2015

Volu

me

(TB

)

May 5, 2015 ASDC Update for CERES STM Page: 7

Number of Users by Product (June 2010 – March 2015)

39 19

87 52 58

438 38 35

90 99

50 9

32 49

31 49

66 256

334 423

83 377

831

0 100 200 300 400 500 600 700 800 900

BDS CERES-MISR-MODIS

CER-NEWS ES8

FLASH-SSF SSF CRS ES9 ES4

ISCCP-D2like SFC

ZAVG FSW

FLASH-TISA SYN

SRBAVG AVG

SYN1deg-lite SSF1deg SYN1deg

ESG EBAF-Surface

EBAF-TOA

May 5, 2015 ASDC Update for CERES STM Page: 8

User Affiliations

30%

29%

22%

19%

Commercial Educational Government Other

Users by Country (June 2010 – March 2015)

ASDC PRIVATE CLOUD PILOT

May 5, 2015 ASDC Update for CERES STM Page: 11

ASDC Private Cloud Pilot

•  ASDC has participated in numerous cloud experiments and research efforts •  Cloudbursting, experiment with LaRC OCIO, others

•  Learning has been exceptional, progress not so much •  IT Security has been a challenge •  Cost models evolving •  Different groups leading different efforts

•  Often looks like “solution looking for a problem” •  Application hosting •  Data storage •  Collaborative processing

May 5, 2015 ASDC Update for CERES STM Page: 12

ASDC Private Cloud Pilot

•  ASDC setting up a private cloud in ASDC facility initially only available to internal (LaRC) users •  Engineers out IT Security issues •  Allows us to collectively learn about cloud realities •  Addresses real-world challenges (i.e., episodic/surge

processing requirements)

•  Private cloud system being setup initially •  Received from NCCS •  480 node 3 year old Dell system •  All x86 based •  Pilot targeted to be operational in September 2015 •  Eucalyptus and Openstack cloud technologies being utilized

May 5, 2015 ASDC Update for CERES STM Page: 13

ASDC Private Cloud Pilot

296 206 288 768

5,760

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

AMI AMI-P ASDC Cloud Pilot:

Eucalyptus

ASDC Cloud Pilot:

Openstack

Total Capacity (Potential)

X86

Cor

es

x86 Cores

May 5, 2015 ASDC Update for CERES STM Page: 14

ASDC Private Cloud Pilot

•  Will identify potential processing candidates to run on pilot cloud in coming months

•  Will evaluate as we exercise cloud and adjust plans accordingly

BIG EARTH DATA INITIATIVE (BEDI) AND DATA SERVICES

May 5, 2015 ASDC Update for CERES STM Page: 16 ASDC Update for CERES STM

ASDC  has  four  BEDI  groups  based  on  BEDI  assigned  tasks  working  on  most  current  versions  of  CERES,  CALIPSO,  MISR,  MOPITT,  TES  1.  OPeNDAP  -­‐  con&igure  hardware  and  software  to  implement  data  

for  availability  through  the  OPeNDAP  protocol  and  other  Open  Geospatial  Consortium  (OGC)  services  as  appropriate.    

2.  GCMD  and  ECHO  Reconciliation  -­‐  provide  a  thorough  review  of  current  metadata  and  ensure  all  required  metadata  for  identi&ied  ASDC  data  products  is  reconciled.    

 

GIBS

3.  GIBS  -­‐  coordinate  with  science  teams  to  create  and  offer  subsets  identi&ied  images  via  GIBS.  Coordination  includes  discussions  with  science  teams  to  de&ine  and  validate  color  scales  and  ingest  testing  with  GIBS.    

4.  DOI  -­‐  assess  the  current  DOI  registration  status  for  the  most  current  versions  of  identi&ied  products  and  provide  the  required  metadata  for  each  product  to  be  registered  (each  DOI  is  updated  based  on  Earth  Science  Data  Types  (ESDTs),  EOSWEB  Landing  pages,  Information  Management  System  (IMS),  GCMD,  and  ECHO  as  appropriate.    

Big Earth Data Initiative (BEDI)

May 5, 2015 ASDC Update for CERES STM Page: 17

ASDC Data Services •  Standardized services that can be accessed in many

different applications

•  ArcGIS Image Service •  Imagery and data service •  Processing templates and functions •  NASA Earthdata Webinar: https://www.youtube.com/watch?

v=N_BC7ZrWUwY

•  OPeNDAP •  provides subsetting, dataset aggregations, and conversion

to/from various data formats

•  Web Mapping Service (WMS) •  Imagery service with basic data access •  Will serve imagery to GIBS

May 5, 2015 ASDC Update for CERES STM Page: 18

Questions?