Paweł Lebioda, Tomasz Kapela

2

Agenda

Introduction:

• Persistent Memory

• NVML

• PMEMCHECK

Tutorial:

• Kernel support

• NVML overview

• libpmem

• libpmemblk

• libpmemobj C API

• libpmemobj C++ API

• PMEMCHECK

Just a little bit of theory...

What is it ?

5

Persistent Memory

What is Persistent Memory ?

Non-volatile

Retains data without power

Byte addressable

Load/Store access

Memory-like performance

DMA-able

NVDIMM

6

Persistent Memory use cases

Block storage

Volatile Memory

Persistent Memory aware filesystem

Persistent Memory aware applications

7

Persistent Memory characteristics

Stores are not durable

The data must be flushed

8-byte atomicity

Virtual address may change

8

Persistent Memory kernel support

NVDIMM subsystem

DAX (Direct Access)

ndctl – github.com/pmem/ndctl

DAX support in ext2, ext4, xfs

Non-Volatile Memory Libraries

https://github.com/pmem/nvml

10

Persistent Memory Programming Model

11

NVML introduction

Set of user-space libraries

PMEM-aware applications

Source code on github - github.com/pmem/nvml

BSD license

Web page - pmem.io

Distros (Fedora, openSUSE)

12

NVML libraries

Library Description

libvmem Volatile memory allocator

libvmmalloc malloc() replacement using libvmem

libpmem Low-level support for persistence

libpmemlog Append only log structure

libpmemblk Atomic access to memory blocks

libpmemobj Transactional object store

libpmempool Managing library

„In theory, theory and practice are the same. In practice, they are not”

Albert Einstein

How to configure my Linux OS for persistent memory ?

15

Kernel support

Kernel config: NVDIMM, DAX

Persistent memory emulation:

– http://pmem.io/2016/02/22/pm-emulation.html

– X86_PMEM_LEGACY

– Memmap=<size>!<offset>

Create filesystem

Mount with DAX

16

Persistent Memory configuration

Quick insight into source tree

18

NVML overview

Source tree organization

Building

Running tests

Examples

Packages

19

NVML packages

Low level persistent memory support

21

libpmem examples

How to check if memory is Persistent Memory ?

How to access Persistent Memory ?

Debug library and log messages

How to make memory persistent ?

Running examples under Valgrind pmemcheck

22

How to check if memory is Persistent Memory ?

23

How to access Persistent Memory ?

24

Running pmem_is_pmem example

25

libpmem: turning on log messages

26

libpmem: pmem_persist - 1

27

libpmem: pmem_persist - 2

Persistent Memory aware tool for Valgrind

github.com/pmem/valgrind

29

PMEMCHECK

Valgrind tool

Need to build nvml with special flags

Checks correctness of persistent memory usage

Checking data persistence

Support for transactions (libpmemobj)

NVML: make EXTRA_CFLAGS=-DUSE_VALGRIND

30

libpmem: valgrind support

31

libpmem: pmem_persist + valgrind pmemcheck

32

libpmem: pmem_persist - 2

Persistent Memory pools utils

34

pmempool: available commands

Persistent Memory resident array of blocks

36

Libpmemblk: examples

Pmempool introduction

Creating pmemblk pool with pmempool

Opening pmemblk pool

Reading pool parameters

Reading and writing blocks

37

pmempool: create command

38

Opening pmemblk pool

39

Reading and writing block

40

libpmempool: running example

Persistent Memory Transactional Object Store

42

libpmemobj

Creating pmemobj pool with pmempool

Opening pmemobj pool

PMEMoid – persistent memory pointer

Root object

Allocations

43

Libpmemobj: simple example

Application which stores list of people

Stores first and last name

Constant size array

44

Data structures

45

Main function

46

List of people

47

Insert function - 1

48

Insert function - 2

49

Object’s constructor

50

Clear function

51

Recovery

52

Running example

53

Missing persist

54

Valgrind pmemcheck

55

libpmemobj: type safety macros

PMEMoid as void *

Type safety macros for convenience

No need to track persistent and volatile pointers

56

Type safety macros: data structures

57

Type safety macros: insert function

58

Type safety macros: clear function

59

libpmemobj: foreach macros

API for traversing through all objects

Foreach object per type

Useful for recovery actions

Avoids persistent memory leaks

60

Foreach type macro

61

libpmemobj: atomic lists API

Circular doubly linked list

Atomic operations

sys/queue – like API

Useful for hybrid persistent-volatile data structures

62

atomic lists API: data structures

63

Atomic lists API: insert function

64

Atomic lists: clear function

65

Atomic lists API: listing people

66

Atomic lists API: recovery

67

libpmemobj: transactions

Most generic API

No need for recovery

Convenient macros

Operations:

– Alloc

– Free

– Modify

68

Transactions: data structures

69

Transactions: insert function

70

Transactions: entry initialization

71

Transactions: inserting elements

72

Transactions: clear function

73

Transactions: listing people

74

Transactions: valgrind pmemcheck

75

Transactions: valgrind pmemcheck

76

C++ API

Using language features

Overloading operators

Hiding required operations

Scope transactions

Lambda

77

C++ API: main function

78

C++ API: list of people class

79

C++ API: listing people

80

C++ API: insert function

81

C++ API: clear function

82

C++ API: list entry class

83

C++ API: persistent string

84

C++ API: valgrind pmemcheck

85

C++ API: valgrind pmemcheck - 1

86

C++ API: valgrind pmemcheck - 2

87

C++ STL

STL allocator concept for libpmemobj

Using STL containers

Need some changes in std libraries

88

C++ STL: list of people class

89

C++ STL: list entry class