Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc.  All rights reserved.Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc.  All rights reserved.

HBase on FlashMatt Kennedy

HBaseCon June 13, 2013

Switch your database to flash now. Or you’re doing it wrong.Brian Bulkowski, Aerospike CTO and co-founder

April 11, 2023 2

http://highscalability.com/blog/2012/12/10/switch-your-databases-to-flash-storage-now-or-youre-doing-it.html

April 11, 2023 3

NAND Flash

+

April 11, 2023 #Cassandra13 4

NAND Flash Memory

April 11, 2023 5

NAND Flash Memory

Flash is a persistent memory technology invented by Dr. Fujio Masuoka at Toshiba in 1980.

BitLine

Source Line Word Line

Control Gate

Float Gate

NPN

April 12, 2023 6

Flash in Servers

04/12/2023 Fusion-io Confidential 7

Direct Cut Through Architecture

PC

Ie

DRAM

Host CPU

AppOS

LEGACY APPROACH FUSION DIRECT APPROACH

PC

IeS

AS

DRAM

Data path Controller

NAND

Host CPU

RAIDController

AppOS

Goal of every I/O operation to move data to/from DRAM and flash.

SC

SC

Super Capacitors

April 12, 2023 8

NAND Flash

+

April 12, 2023 9

HBase Options

1. What do we do today?

2. What does HBase look like on flash?

3. What if we can not go all flash?

April 12, 2023 10

Conventional HBase Node

▸Key Design Principle:

▸Working Set < DRAM

April 12, 2023 11

Working Set Getting Bigger

▸Key Design Principle:

▸Working Set < DRAM

April 12, 2023 12

EEP

▸Key Design Principle:

▸Working Set < DRAM

April 12, 2023 13

DO

LL

AR

S

Cost of DRAM Modules

4 G B 8 G B 1 6 G B 3 2 G B0

200

400

600

800

1000

1200

1400

1600

$ $$$$$

$$$$$$

April 12, 2023 14

HBase Server

▸A typical server…

CPU Cores: 32 with HTMemory: 128 GB

Is your working set larger than 128GB?

April 12, 2023 15

HBase Cluster

▸With NoSQL Databases, we tend to scale out for DRAM

Combined ResourcesCPU Cores: 96Memory: 384 GB

More cores than needed to serve reads and writes.

The All Flash Option

April 12, 2023 16

HBase Node

Data NodeDaemon

Region ServerDaemon

Memory Store Storage Directories

Disk Flash

HBase Node

Data Node Daemon

Storage Directories

April 12, 2023 Fusion-io Confidential 17

Short-Circuit Reads Matter on Flash!R

EA

D O

PS

/SE

C

02

04

06

08

01

00

12

01

40

16

01

80

20

02

20

24

02

60

28

03

00

32

03

40

36

03

80

40

04

20

44

04

60

48

05

00

52

05

40

56

05

80

60

06

20

64

06

60

68

07

00

72

07

40

76

07

80

80

0

0

1000

2000

3000

4000

5000

6000

7000

DataNode reads Short-Circuit Reads

April 12, 2023 18

Short-Circuit Reads (Datanode Bypass)

▸ In hdfs-site.xml▸ <property>    <name>dfs.block.local-path-access.user</name>    <value>hbase</value> </property>

In hbase-site.xml

<property>    <name>dfs.client.read.shortcircuit</name>    <value>true</value>  </property>

<property>    <name>hbase.regionserver.checksum.verify</name>    <value>true</value>  </property>

April 12, 2023 Fusion-io Confidential 19

YCSB Suite – Uniform DistributionM

IXE

D W

OR

KL

OA

D O

PE

RA

TIO

NS

1 0 4 0 7 01 0 0

1 3 01 6 0

1 9 02 2 0

2 5 02 8 0

3 1 03 4 0

3 7 04 0 0

4 3 04 6 0

4 9 05 2 0

5 5 05 8 0

6 1 06 4 0

6 7 07 0 0

7 3 07 6 0

7 9 08 2 0

8 5 08 8 0

0

20000

40000

60000

80000

100000

120000

140000

50/50 R/W 95/5 R/W Read-only

April 12, 2023 20

YCSB Suite Latency – Uniform Distribution

Workload Average Latency

95th Percentile Latency

99th Percentile Latency

50/50 Read/WriteUpdate Latency

81 µs 0 ms 0 ms

50/50 Read/WriteRead Latency

13.5 ms 34ms 128ms

95/5 Read/WriteUpdate Latency

69.3 µs 0 ms 0 ms

95/5 Read/WriteRead Latency

8.5 ms 26 ms 39 ms

Read-only 9.2 ms 26 ms 38 ms

April 12, 2023 21

Write Amplification

Workload Type Amplification Factor

Bulk Load 14.8

Normal Operations (80/20 update/insert split)

4.2

Amplification Factor = Physical Bytes WrittenWorkload Bytes Written

April 12, 2023 22

The HBase BucketCache (HBase-7404)Committed to HBase trunk. Will be in 0.96 release, backport patch for 0.94 available.

+

https://issues.apache.org/jira/browse/HBASE-7404

April 12, 2023 23

BucketCache Configuration

▸ In hbase-site.xml <property>    <name>hbase.bucketcache.ioengine</name>    <value>file:/path/to/bucketcache.dat</value>  </property>

<property>    <name>hbase.bucketcache.size</name> <!-- 2TB: unit is MB -->

    <value>2097152</value>

  </property>

April 12, 2023 Fusion-io Confidential 24

BucketCache Warm-upR

EA

D O

PS

DU

RIN

G C

AC

HE

WA

RM

-UP

30

51

09

90

14

70

19

50

24

30

29

10

33

90

38

70

43

50

48

30

53

10

57

90

62

70

67

50

72

30

77

10

81

90

86

70

91

50

96

30

10

11

01

05

90

11

07

01

15

50

12

03

01

25

10

12

99

01

34

70

13

95

01

44

30

14

91

01

53

90

15

87

01

63

50

16

83

01

73

10

17

79

01

82

70

18

75

01

92

30

19

71

02

01

91

20

67

12

11

51

21

63

12

21

11

22

59

12

30

71

23

55

12

40

31

24

51

12

49

91

25

47

12

59

51

26

43

12

69

11

27

39

1

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

read ops/sec

April 12, 2023 Fusion-io Confidential 25

BucketCache Steady-StateR

EA

D O

PS

/SE

C S

TE

AD

Y-S

TA

TE

10

80

15

02

20

29

03

60

43

05

00

57

06

40

71

07

80

85

09

20

99

01

06

01

13

01

20

01

27

01

34

01

41

01

48

01

55

01

62

01

69

01

76

01

83

01

90

01

97

02

04

02

11

02

18

02

25

02

32

02

39

02

46

02

53

02

60

02

67

02

74

02

81

02

88

02

95

03

02

03

09

03

16

03

23

03

30

03

37

03

44

03

51

03

58

03

65

03

72

03

79

03

86

03

93

04

00

04

07

0

0

10000

20000

30000

40000

50000

60000

read ops/sec

Avg Latency: 5.2 ms95th Percentile: 21 ms99th Percentile: 24 ms

April 12, 2023 Fusion-io Confidential 26

BucketCache 50% Read, 50% UpdateO

PS

/SE

C M

IXE

D W

OR

KL

OA

D

10

70

13

01

90

25

03

10

37

44

34

49

65

56

61

66

76

73

67

96

85

69

16

97

71

03

71

09

71

15

71

21

71

27

71

33

71

39

71

45

71

51

71

58

11

64

11

70

11

76

11

82

11

88

11

94

42

00

42

06

42

12

42

18

42

24

42

30

42

36

42

42

42

48

42

54

42

60

42

66

42

72

42

78

42

84

42

90

42

96

43

02

43

08

4

0

20000

40000

60000

80000

100000

120000

read ops/sec

Update LatencyAverage: 11.7 µs95th Pctl:0 ms99th Pctl: 0 ms

Read LatencyAverage: 7.9 ms95th Pctl: 34 ms99th Pctl: 65 ms

April 12, 2023 Fusion-io Confidential 27

BucketCache during CompactionR

EA

D O

PS

/SE

C U

ND

ER

CO

MP

AC

TIO

N

10

13

02

50

37

04

90

61

07

30

85

09

70

10

90

12

10

13

30

14

50

15

70

16

90

18

10

19

30

20

50

21

70

22

90

24

10

25

30

26

50

27

70

28

90

30

10

31

30

32

50

33

70

34

90

36

10

37

30

38

50

39

70

40

90

42

10

43

30

44

50

45

70

46

90

48

10

49

30

50

50

51

70

52

90

54

10

55

30

56

50

57

70

58

90

60

10

0

10000

20000

30000

40000

50000

60000

read ops/sec

Avg Latency: 7.8 ms95th Percentile: 37 ms99th Percentile: 61 ms

April 12, 2023 28

What Next?

1. Can we do something about that write amplification?

2. There is minimal penalty to in-place updates in flash on modern FTLs; can we devise a way to do this compatible with HDFS?

3. Does HDFS need to be more aware of different storage technologies? (DRAM, Flash, PCM)

f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E

T H A N K Y O U

f u s i o n i o . c o m | R E D E F I N E W H A T ’ S P O S S I B L E

T H A N K Y O U

April 12, 2023 30

Performance

April 12, 2023 31

Performance