DReplOptimizing Access to Application Data for

Analysis and Visualization

L a t c h e s a r I o n k o vM i c h a e l L a n g

L A N L

C a r l o s M a l t z a h nU C S C

HPC Cluster

Head Node

CN1 CN2 CNk...

CNk+1 CNk+2 CNl...

CNl+1 CNl+2 CNm...

CNm+1 CNm+2 CNn...

IO1

IO2

IO3

IOp

FS

FS

FS

FS

FS

Desktop Desktop Desktop Desktop

Data Storage

Data stored in files

Many applications use legacy formats

Data is stored in format, convenient for the producer

In-situ and in-transit data analysis slow

Objective

Decouple storage data layout from application data layout(s)

Make replicas with different data layouts

Each application working with the data can use a layout that is optimized for it

Allow both materialized (on-storage) and on-the-fly data layouts

DesignDefinitions

Dataset -- abstract data model

Views -- how applications see the data

Replicas -- how the data is stored

Provision of an easy way to express how data is used by the applications

View View

Dataset

Replica Replica Replica

Example

pressure=5.1temp=33.1density=0.4

N.........

......

Mpsim

Dataset

pressure=5.1temp=33.1density=0.4

N.........

......

Mpsim

N.........

......

Mpressure

N.........

......

Mtemp

N.........

......

Mdensity

N.........

......

M

pressure

N.........

......

M

tempN

...

...

......

...

Mdensity

pressure=5.1density=0.4N

...

...

......

...

Mpsim

View 1 View 2

Replica 1

Replica 2

DRepl

Dataset Language

Parser

Replication Engine

File Server

DRepl

dataset { var p struct { a, b, c float32 }}

view default { var p = p }view viz { var pa { a } = p var pba { b, a } = p}

replica default { view default }replica viz { view viz }

Parser

Replication EngineDReplFS

ReplicasReplicas

ReplicasOS

SimulationVizualization

viz

default

S viz:000000

S default:000004

dest

S viz:000004

S default:000000

dest

S viz:000000dest

destT viz:000004

field

field

T viz:000000dest

T default:000000

dest

fielddest

dest

S default:000008dest

dest

field

field

field

DatasetDescription

ConversionMap

Configurations

Burst Buffer Node

Parallel FS

Replica 1

Replica 2

DRepl

...

...

Parallel FS

Replica 1

Replica 2Replica 3

ApplicationDRepl

Node 2

ApplicationDRepl

Node 1

ApplicationDRepl

Node N

ApplicationNode 1

ApplicationNode 2

ApplicationNode N

Application

DR

epl

Node 1

Application

DR

epl

Node 2

Application

DR

epl

Node N

Parallel FS

Replica 1

Replica 2...

Embedded Separate

Burst Buffer

Dataset Language

Syntax Similar to C, C++, Java

Dataset

define data types (structs, arrays)

define named data of the types

View(s)

define substructs and subarrays

define named data based on the dataset data

Replica(s)

Dataset LanguagePrimary types - int8, int16, int32, int64, float32, float64, stringN

Structsstruct { a, b, c float64}

Multidimensional arrays[50,40,21] Point

Custom typestype int64 Point

Arithmetic expressions in the subarray definitionsa[i*3, j + 2] = aa[j, i - 1]

Support for different array orders -- row-major, row-minor, in future Hilbert and z-order

Language Exampledataset {const N = 500

type Data struct {a, b, c float32

}

var data [N]Data}

view array-of-structs { var ds = data}

view struct-of-arrays { var a[i]{a} = data[i] var b[i]{b} = data[i] var c[i]{c} = data[i]}

view ab rowmajor { var ab[i]{a,b} = data[i]}

replica array-of-structs { view array-of-structs}

replica struct-of-arrays { view struct-of-arrays}

replica other { view array-of-structs view ab}

Subarray Examplesdataset {const N = 500const M = 200

var data [N, M]float32}

view v { // flip dimensions var flip[i,j] = data[j,i]

// middle row var mr[i] = data[N/2, i]

// each third element var te[i, j] = data[i*3, j*3]}

DReplFS

Represent the application data formats (views) as virtual files

Stored data formats (replicas) -- collection of replicas

DReplFS

ABC

BAC

B

CD

ABCD AB CD

Sim2

Sim1

Viz1 A1

Transformation Rules

dataset { var p struct { a, b, c float32 }}

view default { var p = p

}

view viz { var pa { a } = p var pba { b, a } = p

}

default

viz

T 0004pba

T 0000pa S 0000

S 0008

S 0004

T 0000p

S 0000

S 0004

S 0008

field a

field b

field a

field a

field b

field c

destdest

dest

dest

dest

dest

dest

DReplFS -- Parser, Replication Engine, File Server in Go

KDreplFS -- Parser in Go, Replication Engine and File Server in the Linux kernel

Implementation

ExperimentsDataset

const N = 176160768 type Data struct {

a, b, c float32 } var data [N]Data

Views

array of structs (AOS)

struct of arrays (SOA)

partial (only b)

Replicas

three replicas (AOS, SOA, b)

two replicas (AOS, b)

one replica (AOS)

Each replica on separate SSD

File Servers

pass-through (POSIX)

kdreplfs

Results:Read

0

200

400

600

800

1000

Array of Structs

3 Replicas

Struct of Arrays

3 Replicas

Partial3 Replicas

Array of Structs

2 Replicas

Struct of Arrays

2 Replicas

Partial2 Replicas

Array of Structs

1 Replica

Struct of Arrays

1 Replica

Partial1 Replica

Band

wid

th (M

B/s)

kdreplfsPOSIX

Results:Write

0

200

400

600

800

1000

Array of Structs

3 Replicas

Struct of Arrays

3 Replicas

Partial3 Replicas

Array of Structs

2 Replicas

Struct of Arrays

2 Replicas

Partial2 Replicas

Array of Structs

1 Replica

Struct of Arrays

1 Replica

Partial1 Replica

Band

wid

th (M

B/s)

kdreplfs-synckdreplfs-async

POSIX

Results:Combined

0

200

400

600

800

1000

1200

1400

kdreplfs 1 replica kdreplfs 2 replicas POSIX

Band

wid

th (M

B/s)

ReadWrite

Future Work

Variable-sized arrays

More array element orders (z-order, Hilbert)

Optimizations

Endianness for primary types

Support for HDF5 replicas

Implementation that doesn’t use file servers

Automatic generation of dataset definition from standard data formats (HDF5, NetCDF)