JBoss Data Grid Tuning

7/23/2019 JBoss Data Grid Tuning

1/24

JBoss Data Grid 6 Implementations with OpenJDK

John OsborneMiddleware Solutions Architect

Red Hatosborne!redhat"com

#o$ember %&'(

'
mailto:[email protected]:[email protected]


2/24

Executive Summary

In recent )ears* I+ departments ha$e been to tas,ed to deal with the per-ormance and scalabilit)

concerns related to the e.ponential /rowth in data" Man) enterprise I+ departments are turnin/ to in0

memor) distributed /rid 1IMDG2 technolo/ies such as JBoss Data Grid in order to meet their Ser$ice

3e$el A/reements 1S3As2" JBoss Data Grid is an enterprise IMDG built o-- o- the communit) open

source communit) proect In-inispan4 it has helped enterprise I+ departments meet their data

per-ormance and scalabilit) re5uirements b) pro$idin/ -eatures li,e e.tremel) -ast 1 O1'2 2 readwrite

access to data* elastic scalin/* hi/h a$ailabilit)* -ault tolerance* and much more" 7hen implementin/ an

IMDG technolo/) li,e JBoss Data Grid per-ormance is essential to e$er) step in the process -rom

de$elopment to production"

+here are se$eral -actors when ta,in/ into consideration the per-ormance o- a distributed cache"

At the hardware le$el it is now commonplace to see $er) lar/e machines capable o- runnin/ J8Ms withlar/e heaps" Ja$a has also e$ol$ed to help to meet the demands o- lar/e heaps throu/h J8M

enhancements includin/ the release o- its new /arba/e collector 1G'2 in Ja$a 9" +he core o- JBoss Data

Grid is a Ja$a application and will re5uire J8M tunin/ whether runnin/ in ser$er mode 1accessin/ the

cache remotel) throu/h protocols li,e R:S+* memcached* Hot Rod2 or in librar) mode 1accessin/ the

cache $ia a local Ja$a A;I2" 7hen usin/ lar/e heaps in Ja$a* the J8M itsel- re5uires a lot o- tunin/ to

-ull) optimi


3/24

Introduction

JBoss Data Grid is an enterprise class* open source* in memor) data /rid plat-orm" It is

implemented as a distributed in0memor) ,e)0$alue #oS@3 store" JBoss Data Grid can be con-i/ured to

run in se$eral di--erence architectures4 the cache supports local* in$alidation* replicated* and distributed

modes" Most -re5uentl)* it is deplo)ed as a data tier to impro$e throu/hput to an application b)

o--loadin/ IO -rom a persistence data store" JBoss Data Grid supports se$eral S@3 and #oS@3 data

persistence options" +he data /rid itsel- acts as a schema0less ,e)$alue store allowin/ the abilit) -or

applications to store distinct obects without ha$in/ a ri/id -i.ed data model"

+he data /rid is t)picall) run on commodit) ser$ers to -orm a cluster that can scale up and

down elasticall) based upon the -luctuatin/ needs o- the application" B) scalin/ elasticall) in replicated

or distributed mode* the /rid is hi/h scalable and -ault tolerant b) ha$in/ no sin/le point o- -ailure" Inthe e$ent o- a ser$er -ailure* JBoss Data Grid pro$ides -ull data inte/rit) throu/h its support -or the

Ja$a +ransaction A;I 1J+A2" It also supports parallel processin/ throu/h its Map Reduce ramewor,

and the abilit) to run tas,s on all or some o- the nodes throu/h its Distributed :.ecution ramewor,"

+here ha$e been a broad ran/e o- sectors that ha$e implemented JBoss Data Grid technolo/) to

power applications* includin/ telecommunications* e0commerce* mobile* /amin/* and -inancial

ser$ices" Re/ardless o- the sector* the main reason to consider a data /rid is per-ormance" 7hen

per-ormance is critical it is necessar) to pa) close attention to the run0time components in order to

ma.imi


4/24

b) pro$idin/ /uidance tips and lessons learned"

Capacity Planning and Sizing The Java Heap

;roper si


5/24

o- a lar/e heap could cause the application to be completel) halted -or a lon/ period o-

time"

Ade5uate heap space is le-t -or JBoss Data Grid and the J8M" JBoss Data Grid will

re5uire additional space -or computations and searches" +he J8M will re5uire additional

space -or critical tas,s li,e /arba/e collection* compaction* temporar) obects* networ,trans-er bu--ers* etc" A /ood rule o- thumb is to ,eep the heap si


6/24

Why Tiered Compilation Is Critical

+he Ja$a compiler has two main tas,s4 enable the code to be plat-orm independent and ensure

hi/h per-ormance" :nablin/ plat-orm independence is accomplish throu/h the use o- a static compiler

li,e a$ac which ta,es Ja$a code and translates it into b)tecode which can be e.ecuted an)where b) the

Ja$a 8irtual Machine" :nsurin/ hi/h per-ormance is not no simple" It is accomplished -irst b)

optimi' compiler which is enabled the F0client J8M -la/" +he >' compiler

is desi/ned -or smaller applications that ha$e -ew resources and are usuall) short li$ed" +he >'

compiler is o-ten bene-icial -or applications that need to -ast start0up times because it d)namicall)

compiles b)tecode $er) earl) on in the applicationLs li-ec)cle* without waitin/ -or e.tensi$e pro-ilin/

and en$ironment data" Since the >' compiler compiles b)te into machine lan/ua/e $er) earl) in the

process* applications usuall) per-orm optimal with the >' compiler soon a-ter the application is

deplo)ed"

Most JBoss Data Grid applications are lon/0runnin/ ser$er0side enterprise applications which

can bene-it -rom e$en -urther compilations" +)picall) -or these t)pes o- applications* it ma,es more

6


7/24

sense to use the >% compiler which waits lon/er to /ather more pro-ilin/ data be-ore optimi' compiler )ields the best results initiall)* the >% compiler t)picall) pro$ides much

better per-ormance in a lon/ runnin/ Ja$a application" +he >% compiler is enabled usin/ the F0ser$er

J8M -la/" +his is the de-ault settin/ -or 6(0bit $ersions o- Ja$a 9" As a matter o- -act* passin/ F0client

to a 6(0bit J8M on 3inu. or 7indows 1not true on Solaris2 will still result in the J8M usin/ the >%

compiler"

One issue with the >% compiler is that it is not tri//ered until the code is e.ecuted '&*&&& times

which is o-ten the reason -or a warm0up period when runnin/ benchmar, tests" +he >' compiler

maintains a lower settin/ o- '*E&&" or both compilers it can be modi-ied with the

0C>ompile+hreshold -la/" +he J8M itsel- uses a counter to ,eep trac, o- when the code should be

compiled" +hat counter decrements o$er time as it mainl) measures how man) times the code has been

e.ecuted recentl)" One option is to simpl) lower the compile threshold but that ris,s -illin/ up the code

cache with machine lan/ua/e that has not been -ull) optimiompilation" +iered compilation combines the best -eatures o- both compilers"

+he >' compiler is most acti$e durin/ application start0up and handles optimi% compiler

deli$ers more ad$anced optimi% compiler alone

9


8/24

and 6MB when usin/ tiered compilation" or Ja$a 9 implementations it is best to increase the code

cache which has a -i.ed si


9/24

collection* stop0the0world pauses can still occur due to -ra/mentation'or when the /arba/e collector

can no lon/er ,eep up with the application obect allocation" Durin/ stop0the0world /arba/e collection

pauses* JBoss Data Grid is completel) halted" I- this occurs durin/ a data trans-er between nodes then

the data bein/ sent across the networ, will start to -ill the networ, bu--ers 1the ne.t section will show

how to modi-) these $alues2" I- the bu--ers -ill up completel)* be-ore the application is resumed* thenpac,et loss will occur which could result in JBoss Data Grid timeouts%" In the worst case* the stabilit)

o- the entire could be a--ected due to especiall) lon/ /arba/e collections" +)picall)* /arba/e collection

per-ormance scales worse than linearl)* meanin/ that when a heap is doubled it would ta,e more than

twice as lon/ -or a -ull /arba/e collection" 7ith $er) lar/e heaps* a -ull /arba/e collection could pause

the application threads -or so lon/ that it would cause the cluster to act as i- the paused node has been

terminated"

+unin/ /arba/e collection -or JBoss Data Grid re5uires care-ul attention to detail and some

anal)sis to see how the application beha$es" ?n-ortunatel)* there is no sil$er bullet to /arba/e

collection as the correct tunin/ will be dependent on application wor,loads" +he /oal o- this section is

to assist de$elopers and architects with the tunin/ and per-ormance considerations o- /arba/e

collection with JBoss Data Grid"

OpenJDK 9 has -our a$ailable /arba/e collectors" +he -irst is the serial /arba/e collector 10C

?seSerialG>2 which uses a sin/le thread to process /arba/e collection and should ne$er be used -or a

hi/h per-ormance Ja$a application" Ma,in/ a decision between the other three /arba/e collectors will

depend on the con-i/uration* re5uirements* and per-ormance testin/"

+he -irst decision to ma,e is whether )ou want to use a throu/hput collector or a concurrent

collector" OpenJDK 9 onl) has one throu/hput collector and it is enabled b) settin/ 0C

?se;arallelG> 0C?se;arallelOldG>" +his is the de-ault collector -or 6(0bit OpenJDK 9 as it

pro$ides the hi/hest throu/hput in an isolated Ja$a application" +he throu/hput collector is e.tremel)

e--icient because it pauses all application threads durin/ minor and -ull /arba/e collections and utili;?s -or ma.imum /arba/e collection throu/hput" 7hen /arba/e collection is completed*

all o- the machine >;?s are used -or application processin/" +he throu/hput collector -ull) compacts

the old /eneration durin/ a -ull G> in order to pre$ent -ra/mentation" JBoss Data Grid applications

'" ra/mentation occurs when the -ree space in the heap is -ra/mented into small pieces o- memor) spread in $arious locations" 7hen-ra/mentation occurs there is o-ten a drastic decrease in per-ormance as the s)stem cannot allocate enou/h conti/uous space -or theapplication" +his causes the /arba/e collector to wor, harder and can o-ten lead to -ull stop0the0world /arba/e collection"

%"JBoss Data Grid timeouts can be modi-ied in the cache con-i/uration"


10/24

usin/ the throu/hput collector will o-ten notice a maor drawbac,4 since it is constantl) stoppin/ all

application threads this will o-ten lead to -re5uent latenc) outliers* especiall) -or lar/er heap si


11/24

application threads durin/ a minor /arba/e collection o- the )oun/ /eneration" >MS collector tries to

a$oid a -ull /arba/e collection b) usin/ bac,/round threads to periodicall) scan throu/h the old

/eneration and discard unused obects" +)picall)* >MS will onl) stop application threads durin/ a

minor /arba/e collection and -or $er) short periods while collectin/ the old /eneration" Howe$er* a

poor tuned >MS collector can still result in e.tremel) lon/ pauses which can hurt the per-ormance o-JBoss Data Grid" In >MS* the bac,/round threads need to scan the entire old /eneration be-ore -reein/

obects" +he time to scan the old /eneration is dependent on heap siMS implementation ma) pro$ide better throu/hput than the G' collector" Howe$er* the worst

case per-ormance with >MS will o-ten be worse than usin/ G' due to ris, o- a concurrent mode

-ailure"

ortunatel)* OpenJDK 9 -ull) supports the G' collector which was desi/ned to do a better ob

handlin/ bac,/round /arba/e collection -or lar/e heaps" Heap -ra/mentation can still occur with the G'

collector* but it is much less li,el) since the G' collector can compact the heap in the bac,/round" +he

G' collector is enabled with the -ollowin/ -la/* 0C?seG'G>" 3i,e* >MS* G' still stops all the

application the threads -or a minor /arba/e collection o- the )oun/ /eneration" G' di$ides the old

/eneration into re/ions and utili


12/24

latenc) 1NCMa.G>;auseMillis2 which de-aults to %&&ms" or lar/e heaps* its best to increase the

latenc) to E&&ms or '&&&ms" +he G' collector should be able to ,eep up with that con-i/uration" One

-inal bene-it to the G' collector o$er the >MS collector is that it re5uires -ar less tunin/" As pre$iousl)

mentioned* JBoss Data Grid does not )et support Ja$a =* but there ha$e been man) impro$ements in

the G' collector -or Ja$a = as well that will be ta,en ad$anta/e o- in -uture release"+he best choice o- a /arba/e collector -or a JBoss Data Grid implementation will $ar) per use

case and can onl) be -ound b) e.perimentin/" Howe$er* most de$elopers and architects o- JBoss Data

Grid are /oin/ to be runnin/ lar/er heaps and there-ore the G' collector should be the collector o-

choice -or hi/h o$erall throu/hput and /ood worst case per-ormance" I- the JBoss Data Grid

implementer is not as concerned with worst case per-ormance and is more concerned with total

throu/hput than a well tuned >MS implementation ma) be a better -it" A /ood wa) to start tunin/ G' is

to set the ma. NCMa.G>;auseMillis to )our pause time -or the worst case S3A" +he throu/hput

collector will be bene-icial in limited use cases and the >MS collector will be bene-icial -or small to

medium si


13/24

increase how man) threads 1and how o-ten the) run2 to ,eep the /arba/e collector one step ahead o- the

application" +he easiest and most e--ecti$e wa) to a$oid concurrent mode -ailures is to ma,e the old

/eneration lar/er" +his can be done b) tunin/ the 0C#ewRatio -la/ which is the ratio o- )oun/

/enerationCold /eneration" +he de-ault settin/ is % which indicates that the old /eneration is % times the

)oun/ /eneration* or % o- the heap siMS collector is o-ten a lar/e balancin/ act" B) settin/ a lar/er )oun/ /eneration siMS b) increasin/ the :den space siMS collector does not collect perm/en" ;erm/en o-ten -ills up when the same

'


14/24

archi$e 1JBoss Data Grid application2 is deplo)ed multiple times to the same runnin/ en$ironment" I-

the perm/en space -ills up a -ull /arba/e collection will be e.ecuted" In order to turn on perm/en

/arba/e collection in >MS use the -ollowin/ -la/sC 0C>MS;ermGenSweepin/:nabled 0C

>MS>lass?nloadin/:nabled"

+he G' collector* i- tuned properl)* should not e.perience a -ull /arba/e collection" G' will stillstop JBoss Data Grid application threads -or minor /arba/e collection and -or some o- the concurrent

/arba/e collection c)cles to compact re/ions in the old /eneration" Similar to >MS* increasin/ the si;auseMillis to a $alue that the G' collector

cannot ,eep up with then it will reduce the )oun/ /eneration si;auseMillis will allow the heap to be si


15/24

Historicall)* OpenJDK has been ,eepin/ class de-initions and class metadata in the ;ermanent

Generation 1;ermGen2 space" +he ;ermGen space is bein/ phased out o- Ja$a 9 and in Ja$a = it has

been completel) replaced with Metaspace" 7hen si


16/24

e$en more important than with an a$era/e siompressin/ the Ja$a obect pointers 10C?se>ompressedOops2 can also help limit the

memor) -ootprint o- obect printers and reduce /arba/e collection" In OpenJDK 9* this settin/ is

alread) enabled when the ma. heap space 1m.2 is set to less than %GB" Since >ompressedOops onl)

compresses re-erences up to %GB* it will not be applicable -or heaps lar/er than %GB"

Additionall)* modi-)in/ soc,et bu--er si


17/24

s)sctl 0w net"core"rmemma.%6%'((&&

s)sctl 0w net"core"wmemma.'&(=E96

JBoss Data Grid utili; connection and handsha,e separatel) -or

each node" JGroups will also handle messa/e deli$er) /uarantee e$en with an unreliable protocol li,e

?D;" Modi-) the JGroups settin/s to match the operatin/ s)stem settin/s" In the JGroups con-i/uration

-ile ensure the -ollowin/ settin/sC

?D;

ucastrec$bu-si;

rec$bu-sis2 to send -rames

that are lar/er than the 'E&& b)te standard and ta,e ad$anta/e o- /reater bandwidth" Man) JBoss Data

Grid implementations will be sendin/ lar/er pac,ets and b) increasin/ the pac,et si


18/24

those runnin/ on '&G networ,s"

:nablin/ Jumbo rames will onl) help JBoss Data Grid per-ormance" Jumbo rames onl)

chan/es the upper limit on the -rame si; and ?D;" ;M+?D mustnot be bloc,ed b) -irewalls or pac,ets could be dropped b) switches that do not support the M+?" On

3inu. )ou can test the si


19/24

onl) be $aluable -or duplicate Strin/s on the same nodes" B) de-ault* JBoss Data Grid stores

obects unseriali


20/24

J8M Health" +he two most important aspects o- J8M Health with respect to JBoss Data Grid is

memor) consumption and /arba/e collection per-ormance" OpenJDK 9 includes se$eral use-ul

monitorin/ tools includin/ console* stat* $isual$m* hat* and map" +hese are critical tools -or

monitorin/ the health o- JBoss Data Grid"

Memor) consumption needs to be closel) monitored" Red Hat recommends that at most E& o-memor) is consumed b) application data so that the rest o- the space can be optimiMS collector which re5uires more manual tunin/" In that case* i- )ou are usin/ the >MS collector*

decrease the 0C#ewRatio settin/ to allow -or a lar/er )oun/ /eneration" I- the application itsel- is

simpl) creatin/ too man) temporar) obects* this ma) also be a /ood time -or a code re$iew"

A /ood wa) to 5uic,l) loo, at the health o- the JBoss Data Grid application is to loo, at a

histo/ram o- the heap distribution" OpenJDK 9 has a couple wa)s to do that -rom the console usin/

%&


21/24

cmd and map" Generatin/ a histo/ram will show the instance count and total si


22/24

console" Jstat also e.poses the collection times usin/ JM so that it can be monitored with other JM

solutions" Jstat wor,s -rom the command line and allows users to see se$eral di--erent $iews o- /arba/e

collection durin/ pro/ram e.ecution" +he command Fstat 0/ccause pid '&&& will continuousl)

print $aluable /arba/e collection in-ormation to the console" I- /arba/e collection needs to be -orced* it

can be done with cmd usin/ the command Fcmd pid G>"run" I- 8isual8M is )our pre-erred toolthen add the 8isualG> plu/in -or $isual /arba/e collection monitorin/"

Additionall)* )ou can collect /arba/e collection lo/s with Ja$a -la/s" +his process has $er) low

o$erhead and can pro$ide a lot o- insi/ht into JBoss Data Grid per-ormance so to enable /arba/e

collection lo//in/ e$en on a production s)stem" or use-ul lo//in/* add the -ollowin/ Ja$a -la/sC F0

$erboseC/c 0lo//cC-ilename 0C;rintG>Details 0C;rintG>+imeStamps* 0C

;rint+enurin/Distribution* 0C;rintG>ApplicationStopped+ime 0C

;rintG>Application>oncurrent+ime " :nsure to update the output -ile in-ormation with the lo//c

-la/" Althou/h )ou can manuall) re$iew the /arba/e collection lo/s* the best wa) to anal) Histo/ram or G>8IW which are separate downloads" I- )ou obser$e a hi/h rate o-

/arba/e collection ma,e sure to re$iew the /arba/e collection tunin/ section"

Althou/h sli/htl) out o- the scope o- this white paper* it is important to also monitor hi/h >;?

utili usin/ open source

tools li,e nicstat which will automaticall) print the utili


23/24

-irst /uideline is to benchmar, usin/ the tar/et con-i/uration" I- JBoss Data Grid will be in production

as a distributed cache* ma,e sure to benchmar, it as a distributed cache as it will beha$e di--erentl)

than as a local cache" +he second /uideline is to ma,e sure )ou ha$e a warm0up period" As noted in the

compilation section* the HotSpot J8M will /enerate machine lan/ua/e -or the -re5uentl) e.ecuted

portions o- the application" As a /eneral rule )ou will want wait at least & seconds and run throu/h atleast '&*&&& iterations o- the code be-ore benchmar,in/ it" +hird* ma,e sure that memor) and /arba/e

collection parameters ha$e been tuned appropriatel)" Additionall)* alwa)s ensure that )our tests are

multi0threaded4 this is especiall) true -or bloc,in/ operations li,e s)nchronous puts into the cache"

Determinin/ the ideal number o- threads will be trial and error but usin/ '&& threads is a /ood startin/

point -or ser$ers with '6 cores" I- the >;? is not near ma.imum capacit) then tr) addin/ more threads"

inall)* it is best to use bul, operations li,e putAll12 as opposed to put12" +his will reduce the number

o- messa/es that need to tra$erse the networ, which will /reatl) increase the throu/hput" Adustin/ the

siloud Ser$in/ Benchmar,in/

1Q>SB2 can also be used to benchmar, JBoss Data Grid in clientser$er mode" Q>SB will test -or

latenc) on the remote JBoss Data Grid ser$er" Apache JMeter can also be used to /enerate load on

JBoss Data Grid in clientser$er mode"

BenchmarC JBoss Data Grid applications using the recommend guidelines' hen possile, use opensource enchmarCing tools to generate load on the data grid'

Conclusion

De$elopin/ and tunin/ -or an in memor) distributed cache re5uires a lot more attention to detail

than a standard Ja$a application" In order to ma.imi


24/24

ma.imum per-ormance usin/ either the >MS or G' collectors" +he >MS collector* i- tuned properl)*

ma) pro$ide the hi/hest throu/hput" Howe$er* the G' collector will pro$ide hi/h throu/hput without

all the tunin/ headaches and will also pro$ide a better worst case scenario than the >MS collector" A

-inel) tuned G' collector will ne$er stop JBoss Data Grid -or a -ull /arba/e collection* and there-ore is

the recommend /arba/e collector o- choice"A JBoss Data Grid application that is producin/ lar/e outliers in response time probabl) needs

tunin/ to the heap si

Documents

JBoss Data Grid Tuning