AMSR-E and AMSR-E Validation Status at NSIDC Amanda Leon, NSIDC AMSR-E Lead Joint AMSR-E Science Team Meeting Asheville, NC 28-29 June 2011

AMSR-E and AMSR-E ValidationStatus at NSIDC

Amanda Leon, NSIDC AMSR-E LeadJoint AMSR-E Science Team MeetingAsheville, NC28-29 June 2011

Joint AMSR-E Science Team Meeting

Amanda Leon - NSIDC DAAC 28-29 June 20112

Outline

• Accomplishments• Data Support • Activities at NSIDC• Metrics

AMSR-E AMSR-E Validation

Sea ice derived from the AMSR-E 89 GHz Tbs. Image courtesy

of NASA/GSFC Visualization Studio



Accomplishments

• Release of Sea Ice Drift Version 1 product (24 June 2011) Documentation & metadata creation End-to-end ingest testing with the SIPS Comprehensive distribution testing

o Enabled spatial and parameter subsetting via WIST/Reverb

• Improved product latency Level-2 product latency reduced from 5 days to 20 hours!!! Level-3 daily product latency reduced from 5 days to 4 days

AMSR-E Data Versions: http://nsidc.org/data/amsre/versions.html

Accomplishments

• LANCE: SIPS-NSIDC collaboration Close coordination to ensure Preliminary data from the SIPS phased out in lieu of LANCE Joint poster presented at A-Train meeting and AGU

• Google Earth Animations using LANCE data 12 km Sea Ice & Daily Soil Moisture nsidc.org/data/virtual_globes/index.html

• Upgrade to EOSDIS Core System (ECS)



http://nsidc.org/data/virtual_globes/index.html





Data Support: Data Access

Data Pool• On-line access to entire archive nsidc.org/data/data_pool/• Subsetting, reformatting, reprojection

Warehouse Inventory Search Tool (WIST)• Search/order from ECHO wist.echo.nasa.gov• Subsetting

Subscriptions• Automatic delivery of new data nsidc.org/opendap/

through FTP push or pull

http://nsidc.org/data/amsre/order_data.html

http://nsidc.org/data/data_pool/



https://wist.echo.nasa.gov/

http://nsidc.org/opendap/





Data Support: New Data Access

REVERB• WIST replacement reverb.echo.nasa.gov/reverb/• Improved user interface and search/order workflow

OpenDAP• Provides simple access to remote nsidc.org/opendap/

data• Currently limited use and not publicized

Data Casting (RSS Feed)• Automatic delivery of new data nsidc.org/feeds/data/AE_Ocean.xml

through an RSS feed• Currently available for L2 Ocean only

http://nsidc.org/data/amsre/order_data.html

http://reverb.echo.nasa.gov/reverb/granules?utf8=%E2%9C%93&spatial_map=satellite&spatial_type=rectangle&keywords=ae_l2a&datasets=C98817885-NSIDC_ECS




http://nsidc.org/feeds/data/AE_Ocean.xml



Data Support

User Services [email protected]

• Provide support of data products, in-house developed tools, and proprietary software

Documentation nsidc.org/data/amsre/data_summaries/

• Comprehensive product documentation• Algorithm version information nsidc.org/data/amsre/versions.html

Tools nsidc.org/data/amsre/data_summaries/

• General HDF and specific data set tools• New Data Viewer - interactive tool for viewing satellite passive

microwave data sets

Published Research nsidc.org/data/amsre/research.html

mailto:[email protected]

http://nsidc.org/data/amsre/data_summaries/index.html





http://nsidc.org/data/amsre/versions.html








http://nsidc.org/data/amsre/research.html



NSIDC Activities



Digital Object Identifiers (DOIs) & Versioning• Develop process for implementation of DOIs

•Recommendations for NSIDC•Major.Minor version schema (AMSR-E Version 2, V10 = 2.10)•Each major version of a data set gets a separate DOI

•Recommendation for AMSR-E data managementIncrement the ESDT (major) version number when a change to an

algorithm warrants reprocessing the entire time seriesCurrently archiving both Version 1 & 2. Will need to discuss version

retention policy moving forward.

HDF4 content maps• Collaborative study between NASA Earth Science Data Centers & The HDF

Group focused on ensuring long term readability of HDF4 data• XML file containing a “map” to the contents of the HDF4 file

Enables reading of the data without HDF4 libraries• ECS 8.1 will generate, archive, and distribute the HDF4 content maps

NSIDC Activities



Metadata Project• Provide overall improvements to NSIDC’s metadata acquisition,

creation, management, and dissemination • Replace or enhance existing metadata infrastructure

NOAA Sea Ice Concentration CDR – W. Meier• Funded by the NOAA Climate Data Record grant• Integrates current SSM/I-SSMIS algorithms (NASA Team and

Bootstrap) distributed by NSIDC into one combined product• NetCDF4 file will include data quality fields, metadata, & provenance

Green Data Center• $525,000 NSF research instrumentation grant• Data center will transition power consumption from 100kw (42kw load for

data center equipment, 58kw for cooling) to ~45-50kw (cooling reduced by 95%)

NSIDC Activities



• Four Phase project: 1. Machine room

consolidation2. Server Virtualization

(60% reduction in servers)

3. High efficiency cooling system

4. Solar photo-voltaic power supplementation system with battery array

Coolerado Units

http://news.nationalgeographic.com/news/news/energy/2010/11/101227-national-snow-and-ice-data-center-makeoverNational Geographic Online Article



Archive Metrics: Volume1 June 2002 – 1 June 2011

Stats from EMS

Version 2 Archive Volume

Total Archive: 20 TBs

Total Version 2 Archive: 11 TB

ProductVolume (GB)

AE_5DSNO 1.28

AE_DYOCN 44.04

AE_DYSNO 6.49

AE_L2A 4795.08

AE_LAND 65.48

AE_LAND3 195.56

AE_MOOCN 1.46

AE_MOSNO 0.22

AE_OCEAN 1119.35

AE_RAIN 1001.46

AE_RNGD 0.02

AE_SI12 187.50

AE_SI25 61.79

AE_SI6 139.93

AE_WKOCN 6.32

AMSREL1A 3456.872002 2003 2004 2005 2006 2007 2008 2009 2010 2011

0.00

1000.00

2000.00

3000.00

4000.00

5000.00

6000.00

Yearly Total Archive(excludes deleted data)



Distribution Metrics: Total

1 June 2002 – 1 June 2011

Stats from EMS

Total Distribution:454 TBs

13.8 million files

Data Type Files Volume (TB)

AE_5DSNO 54331 0.09

AE_DYOCN 72224 0.80

AE_DYSNO 360128 0.63

AE_L2A 6444515 330.61

AE_LAND 1246548 0.75

AE_LAND3 897300 43.06

AE_MOOCN 2717 0.03

AE_MOSNO 11517 0.02

AE_OCEAN 1330601 14.39

AE_RAIN 2188690 18.56

AE_RNGD 3980 0.00

AE_SI12 435767 20.62

AE_SI25 179977 2.95

AE_SI6 168152 6.25

AE_WKOCN 7086 0.08

AMSREL1A 448112 15.12

Total 13851645 453.97



Distribution Metrics: Files2003 - 2011 File Distribution

Stats from EMS

2003 2004 2005 2006 2007 2008 2009 2010 20110

500000

1000000

1500000

2000000

2500000

3000000

3500000

332369

874314

561729

704166

13370341555637

752830

457444

307185

222507

537633349628

434835

397739 468197

AMSREL1A

AE_WKOCN

AE_SI6

AE_SI25

AE_SI12

AE_RNGD

AE_RAIN

AE_OCEAN

AE_MOSNO

AE_MOOCN

AE_LAND3

AE_LAND

AE_L2A

AE_DYSNO

AE_DYOCN

AE_5DSNO

3 million files

2 million files

3 million files



Distribution Metrics: Volume2003 – 2011 Volume Distribution

Stats from EMS

2003 2004 2005 2006 2007 2008 2009 2010 20110.00

20000.00

40000.00

60000.00

80000.00

100000.00

120000.00

15177.22

37136.91 34356.60

55704.22

76351.78

72975.47

34232.06

6859.83

11013.6211448.51

AMSREL1A

AE_WKOCN

AE_SI6

AE_SI25

AE_SI12

AE_RNGD

AE_RAIN

AE_OCEAN

AE_MOSNO

AE_MOOCN

AE_LAND3

AE_LAND

AE_L2A

AE_DYSNO

AE_DYOCN

AE_5DSNO

100 TB

50 TB



Distribution Metrics: Data Access Method

Stats from EMS

2003 – 2011 Data access method based on data volume

Overall Average: Past 3 Year Average:

ECHO/EDG orders = 52 % ECHO = 28%DataPool = 43% DataPool = 70%Subscriptions = 5% Subscriptions = 2%

2003 2004 2005 2006 2007 2008 2009 2010 20110%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

81%

19%

48%

36%

77%

61%

73%

37%

50%

46%

35%

34%24%

18%

21% 18%

23%

18%

ECHO - SUBSETTED

ECHO - FTP PULL

ECHO - FTP PUSH

SUBSCRIPTION

FTP PUSH

FTP PULL

DVD

DLT

CD

DATAPOOL



Distribution Metrics: Users1 June 2002 – 1 June 2011

Stats from EMS & USO TRAK Database

2192 Distinct Users

Product Distinct Users

AE_5DSNO 167

AE_DYOCN 102

AE_DYSNO 305

AE_L2A 712

AE_LAND 305

AE_LAND3 719

AE_MOOCN 47

AE_MOSNO 124

AE_OCEAN 166

AE_RAIN 189

AE_RNGD 71

AE_SI12 339

AE_SI25 165

AE_SI6 120

AE_WKOCN 25

AMSREL1A 97

User Type Distinct Users

FOREIGN COMMERCIAL 28

FOREIGN GOVERNMENT 117

FOREIGN OTHER 647

FOREIGN UNIVERSITY 461

ECS OTHER 128

NASA 140

NOAA 29

US COMMERCIAL 35US FEDERAL GOVERNMENT 27

US GENERAL PUBLIC 8

US GOVERNMENT 14

US OTHER 89

US STATE/LOCAL GOVERNMENT 11

US UNIVERSITY 455

WDC 2

U.S. Users - 43%

Foreign Uses - 57%



Distribution Metrics: Users

Stats from USO TRAK Database

2003 – 2011 Total Active Distinct Users per Year

2003 2004 2005 2006 2007 2008 2009 2010 20110

100

200

300

400

500

600

52

198

336

429

493

510

466

414

228



AMSR-E Validation: Distribution Metrics

• No new AMSR-E Validation data sets delivered to NSIDC

• Published last data set in March 2010



Distribution Metrics: Archive volume

Rainfall Cryosphere Soil Moisture0

50

100

150

200

250

300

350

400

450

500

5.1

476

94

Archive volume (GB) through 1 June 2011Total Archive Volume = 575 GB



Distribution Metrics: UsersDistinct users through 1 June 2011.

Total Distinct Users = 4,95233% increase from last year

Statistics from our University FTP logs


500

1000

1500

2000

2500

3000

435

1780

2737



AMSR-E Validation: MetricsFiles distributed through 1 June 2011

Total Files Distributed = 772,36423% increase from last year

Statistics from our University FTP logs


50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

23870

273349

475145



Metrics: VolumeVolume (GB) distributed through 1 June 2011

Total Volume Distributed = 2 TB25% increase from last year

Statistics from FTP logs

Rainfall Cryosphere Soil Moisture0.00

200.00

400.00

600.00

800.00

1000.00

1200.00

121.88

846.16

1093.41



AMSR-E Team at NSIDC

Thanks to the AMSR-E Teams at NSIDC!

AMSR-E

Molly Thomas – Operations

Karla LeFevre – Technical Writer

Lisa Booker – User Services

Sean Mallory – Programmer

Siri Jodha Khalsa – ECS Science Coordinator

Walt Meier – Scientist

Doug Fowler – Test Engineer

Cathy Fowler – Database Administrator

AMSR-E Validation

Molly Thomas – Operations

Karla LaFevre – Technical Writer

Dan Webster – Technical Writer

Lisa Booker – User Services

Sean Mallory – Programmer



Documents

AMSR-E and AMSR-E Validation Status at NSIDC Amanda Leon, NSIDC AMSR-E Lead Joint AMSR-E Science Team Meeting Asheville, NC 28-29 June 2011