SSEC Storage Systems Panel: Solving the Data Deluge

1

Scott Nolin 20 November 2013

About SSEC

The University of Wisconsin Space Science and Engineering center (SSEC) is a research and development center focusing on geophysical research and technology to enhance our understanding of the atmosphere of Earth, the other planets in our Solar System, and the cosmos. Major SSEC Initiatives §  atmospheric studies of Earth and other planets §  interactive computing, data access, and image processing §  spaceflight hardware development and fabrication Noted Scientific Work §  satellite-based and other weather observing instruments §  remote sensing applications in earth and atmospheric science §  spaceflight instrumentation §  planetary meteorology §  data analysis and visualization §  diagnostic and numerical studies of the atmosphere

2

What Shapes our Storage Solutions?

§  The SSEC contains many groups and projects, often with different funding sources. Individual groups and projects ultimately control their own expenditures. §  The Technical Computing Group provides an infrastructure to share and combine

these resources. §  Atmospheric research via satellite meteorology is the major focus of SSEC.

§  This has large data needs. Files are typically fairly large – we use 100MB files as a generic representation of a lot of our data.

§  Most projects are research, and not strictly operational so high availability is usually not required, but data integrity is critical.

3

Uniform Components,Varied Solutions

NAS

4

SAS  Cables

SAS  Cables

Split  Mode

MDS/MDT

OSS  1

OSS  2

FDR-­‐10

FDR-­‐10

FDR-­‐10

OST0  OST1  OST2

OST3  OST4  OST5

High Performance Storage Infrastructure

SSEC has recently created a generally available storage infrastructure, largely focused on Lustre. Making large file systems available generally, and not just on compute clusters has changed how people work with large data. Key components include: §  Infiniband

§  Once you have taken the step to infiniband infrastructure (which was required by some of our NWP needs) it’s hard to pay more for lower performance with ethernet

§  Lustre Routers §  Allow connecting the infiniband resources (storage, compute clusters) to 10G and

1G ethernet systems. §  Patchless Lustre Clients

§  This is particularly useful for extending lustre filesystems to non-cluster nodes. Linux is the main focus here, our ‘clients’ are what many would consider servers, typically ethernet connected.

§  BETA - Providing a mountable file system aggregating all resources (NAS, Gluster, Lustre) as a single file system to desktops. §  Work in progress – custom application in house by a research group

5