May 30-31, 2012 HDF5 Workshop at PSIMay 30-31

HDF5 File Image Operations

Dana RobinsonThe HDF Group

Efficient Use of HDF5 With High Data Rate X-Ray DetectorsPaul Scherrer Institut

May 30-31, 2012 HDF5 Workshop at PSI

Purpose

Allow users to work with files like how they do on disk.

No disk I/O when file images are opened, created, read from, or written to.

Faster access to data.

Need to be careful about minimizing your memory footprint when working with large files.

May 30-31, 2012 HDF5 Workshop at PSI

Use in pipelines

Processing 1

HDF5 file

1) Processing step 1 opens the HDF5 file with the core VFD.

HDF5 file

May 30-31, 2012 HDF5 Workshop at PSI

Process 2

Use in pipelines

Process 1

2) Buffer is sent to processing step 2.

This saves writing the file to the filesystem.

HDF5 file

IPC

May 30-31, 2012 HDF5 Workshop at PSI

Useful for MPI

Process 0 Process 1 Process n

config file

1) Process 0 reads configuration file.

2) File image is broadcast to other processes to open in memory.

Process 2

May 30-31, 2012 HDF5 Workshop at PSI

New API Functions

H5Pget/set_file_image

H5Pget/set_file_image_callbacks

H5Fget_file_image

H5LTopen_file_image (high-level convenience function)

May 30-31, 2012 HDF5 Workshop at PSI

H5Pget/set_file_imageherr_tH5Pget/set_file_image(hid_t fapl_id,

void *buffer, size_t buf_len)

Requires allocating a buffer, which is passed to the function.

Designed for the core VFD but most VFDs can be modified to support using an initial image file.

The buffer is copied in/out of the HDF5 library, not passed by reference!

We will discuss ways around this…

May 30-31, 2012 HDF5 Workshop at PSI

Digression: Property Lists

Used frequently in the HDF5 library.

Allow us to be flexible in passing parameters to functions without breaking the API.

Use "pass by value" semantics, not "pass by reference"

- Designed for moving small amounts of data around.

- Greatly simplifies copying and deletion of property lists.

May 30-31, 2012 HDF5 Workshop at PSI

Digression: Property Lists

Unfortunately, passing very large buffers by value can bring a severe performance penalty.

We have a few ways around this:

- Use the H5LTopen_image_file call (discussed later)

- Ignore the problem if the overhead is low (small files)

- Use file image callbacks to implement call-by-reference semantics (discussed later)

May 30-31, 2012 HDF5 Workshop at PSI

H5Fget_file_imagessize_tH5Fget_file_image(hid_t fapl_id,

void *buffer, size_t buf_len)

Provides a simple way to retrieve a copy of the image of an open file.

You provide the buffer and the size.

You can call the function with a null buffer pointer to get the current file size.

Can be used with files opened with the SEC2, STDIO, and core VFDs.

May 30-31, 2012 HDF5 Workshop at PSI

H5P_get/set_file_image_callbacksherr_tH5Pget/set_file_image_callbacks(hid_t fapl_id,

H5_file_image_callbacks_t *callbacks_ptr)

H5_file_image_callbacks_t is a struct containing function pointers which can be invoked on:

image re/allocation image freeimage copy

It also contains a void pointer for user-specific data and function pointers which can be invoked on:

user data copy user data free

May 30-31, 2012 HDF5 Workshop at PSI

H5P_get/set_file_image_callbacks

The purpose of the callback functions are twofold:

1) Allow the user more careful control over the image in memory when resources are scarce (e.g.: when handling large files).

2) Allow the user to implement pass-by-reference semantics for the buffer.

May 30-31, 2012 HDF5 Workshop at PSI

H5_file_image_callbacks_ttypedef struct {

void *(image_malloc)(size_t size,H5_file_image_op_t file_image_op, void *udata),

void *(image_memcpy)(void *dest, void *src,size_t size, H5_file_image_op_t

file_image_op,void *udata),

void *(image_realloc)(void *ptr, size_t size,H5_file_image_op_t file_image_op, void *udata),

herr_t *(image_free)(void *ptr,H5_file_image_op_t file_image_op, void *udata),

void *(udata_copy)(void *udata),herr_t *(udata_free)(void *udata),void *udata;

} H5_file_image_callbacks_t;

May 30-31, 2012 HDF5 Workshop at PSI

H5P_get/set_file_image_callbacksvoid *(image_malloc)(size_t size,

H5_file_image_op_t file_image_op, void *udata),

The file_image_op parameter indicates which operation is taking place when the function was invoked.

The image_malloc, realloc, free, and memcpy functions must have the same semantics as their C counterparts.

Note that the memcpy and free calls can indicate failure in their return values, unlike their C counterparts.

May 30-31, 2012 HDF5 Workshop at PSI

H5_file_image_op_t

typedef enum {H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,H5_FILE_IMAGE_OP_FILE_OPEN,H5_FILE_IMAGE_OP_FILE_RESIZE,H5_FILE_IMAGE_OP_FILE_CLOSE,

} H5_file_image_op_t;

These values indicate the operation in progress when the callback was invoked.

May 30-31, 2012 HDF5 Workshop at PSI

Warning!!!

Creating a full set of callback functions to control the behavior of the file image is likely to be a difficult process!

The high-level function H5LTopen_file_image (discussed later) should be suitable in most cases.

May 30-31, 2012 HDF5 Workshop at PSI

H5LTopen_file_imagehid_tH5LTopen_file_image (void *buffer,

size_t buf_len, unsigned flags)

High-level convenience function to simplify buffer use.

Allows the library to take control of the buffer with reasonable default behavior.

Suitable for most uses.

May 30-31, 2012 HDF5 Workshop at PSI

H5LTopen_file_imageBitwise flags:

H5LT_FILE_IMAGE_OPEN_RWOpen the image file with read/write permissions.

H5LT_FILE_IMAGE_DONT_COPYPass the buffer by reference instead of by value.The library will call free on the buffer.

H5LT_FILE_IMAGE_DONT_RELEASEDo not free the image when the library is done with it.Also prohibits resizing/reallocating the buffer.

May 30-31, 2012 HDF5 Workshop at PSI

Reading an in-memory image

<allocate and initialize the buffer><allocate the fapl><set fapl to use the core VFD>

H5Pset_file_image(fapl_id, buffer, buf_len);

<discard buffer any time after this point>

<open file>

<discard fapl any time after this point><read/write file as desired, close>

May 30-31, 2012 HDF5 Workshop at PSI

H5LTopen_file_image example

<allocate and initialize the buffer>

hid_t file_id;unsigned flags = H5LT_FILE_IMAGE_DONT_COPY;

file_id = H5LTopen_file_image(buffer, buf_len, flags);

<read/write file as desired, close>

May 30-31, 2012 HDF5 Workshop at PSI

Status

Complete. Appeared in HDF5 1.8.9 (May 2012).

Java and Fortran equivalents are not available at this time.