26
Copyright © 2012 Splunk Inc. Thinking Beyond Applica=on Monitoring Jus=n Hardeman, Unix Administrator Availity Health Informa=on Networks

Splunk Beyond application monitoring PDF

Embed Size (px)

Citation preview

Page 1: Splunk Beyond application monitoring PDF

Copyright*©*2012*Splunk*Inc.*

Thinking*Beyond*Applica=on*Monitoring*Jus=n*Hardeman,*Unix*Administrator*Availity*Health*Informa=on*Networks*

Page 2: Splunk Beyond application monitoring PDF

Legal*No=ces*During*the*course*of*this*presenta=on,*we*may*make*forwardLlooking*statements*regarding*future*events*or*the*expected*performance*of*the*company.*We*cau=on*you*that*such*statements*reflect*our*current*expecta=ons*and*es=mates*based*on*factors*currently*known*to*us*and*that*actual*events*or*results*could*differ*materially.*For*important*factors*that*may*cause*actual*results*to*differ*from*those*contained*in*our*forwardLlooking*statements,*please*review*our*filings*with*the*SEC.**The*forwardLlooking*statements*made*in*this*presenta=on*are*being*made*as*of*the*=me*and*date*of*its*live*presenta=on.**If*reviewed*aTer*its*live*presenta=on,*this*presenta=on*may*not*contain*current*or*accurate*informa=on.***We*do*not*assume*any*obliga=on*to*update*any*forwardLlooking*statements*we*may*make.**In*addi=on,*any*informa=on*about*our*roadmap*outlines*our*general*product*direc=on*and*is*subject*to*change*at*any*=me*without*no=ce.**It*is*for*informa=onal*purposes*only*and*shall*not,*be*incorporated*into*any*contract*or*other*commitment.**Splunk*undertakes*no*obliga=on*either*to*develop*the*features*or*func=onality*described*or*to*include*any*such*feature*or*func=onality*in*a*future*release.**

Splunk,(the(engine(for(machine(data(is(a(registered(trademark(of(Splunk(Inc.(and/or(its(subsidiaries(and/or(affiliates(in(the(United(States(and/or(other(jurisdic=ons.(*All(other(brand(names,(product(names(or(trademarks(belong(to(their(respec=ve(holders.((

©2012(Splunk(Inc.(All(rights(reserved.*

2*

Page 3: Splunk Beyond application monitoring PDF

About*Availity*

3*

Pa=ents.*Not*Paperwork.®"•  A*na=onal*leading*health*informa=on*network*(HIN)*•  Op=mizing*informa=on*exchange*through*a*single,*secure*network*•  200,000+*physicians*and*1,000+*hospitals*

Mul=ple*payers.*One*website.*One*login.*•  1300+*health*plans*and*450+*industry*partners*in*one*place*�  Aetna,*20+*Blue*Cross*and*Blue*Shield*plans,*CIGNA,*Humana*and*more*

•  RealL=me*info*exchange*means*highest*accuracy*of*health*plan*info**

HIPAA*transac=ons*and*proprietary*products.*•  File*claims,*check*eligibility,*review*clinical*history,*process*payments*•  Products*include:**Availity*CareCollect®,CareProfile®,*CareRead®,*CareCost*Es=mator®,*

Remit*Reader®,"and*more*

Page 4: Splunk Beyond application monitoring PDF

About*Me*

4*

Jus=n*Hardeman*Unix*Administrator*Availity*Health*Informa=on*Networks**

Core*Func=ons*•  Produc*on"Opera*ons"(4"years)"

L  Monitoring*L  Incident*response*L  24/7*Support*on*Unix*systems,*authen=ca=on*systems*and*webLportal*

•  Resident"Splunk"user"(3"years)"L  BeLall*and*endLall*of*Splunk*

Page 5: Splunk Beyond application monitoring PDF

Agenda*

•  Going*beyond*system*monitoring*

•  Challenges*we’ve*faced*at*Availity*

•  Senior*leadership*views*–*how’s*the*business*doing*overall?*

•  System*opera=ons*–providing*insights*into*the*nuts*and*bolts,*

and*maybe*even*the*broken*gears*

•  Marke=ng*and*training*–*are*people*using*our*demos?*

•  Wins!*

•  Takeaways?*

*

Page 6: Splunk Beyond application monitoring PDF

Ask*Yourself*

I*already*have*monitoring,*so*that*should*be*good*enough,*right?*

6*

Page 7: Splunk Beyond application monitoring PDF

What*Do*I*Mean*by*Monitoring?*"   The*easy*stuff*–  Is*a*process*up?**Ac=ve*or*stale?*–  What’s*my*CPU/disk/RAM*usage?*–  I*need*to*monitor*a*log*file*for*a*par=cular*error*message*

"   What*about*the*more*difficult*measurements*–  How*fast*is*my*average*user*login*=mes?**What*was*it*last*week?*–  What*is*the*top*error*code*for*a*par=cular*payer*and/or*transac=on*type?*

–  How*many*hits*did*we*receive*to*a*training*demo*during*the*=meframe*of*last*week?**Last*month?*

** 7*

Page 8: Splunk Beyond application monitoring PDF

Before*Splunk*

8*

Reac=ve*approach*to*problems*

Lack*of*visibility*into*root*cause*of*failures*/*issues*

Long*resolu=on*=mes*–*always*needed*the*availability*of*the*right*person*at*the*right*=me*

•  RealL=me*visibility*into*issues*with*proac=ve*aler=ng*

•  Reports*and*dashboards*with*realL=me*transac=ons*and*B2B*workflows*

•  Combining*mul=ple*sources*of*informa=on*in*varying*formats*into*a*single*view*for*capacity*planning*

•  Secure,*roleLbased*views*for*IT*teams*

OPERATIONAL*CHALLENGES* MONITORING*REQUIREMENTS*

Page 9: Splunk Beyond application monitoring PDF

Transforming*Machine*Data*with*Splunk*

9*

Proac*ve"monitoring"

Real?*me"visibility"

Opera*onal"insight"

Enterprise"adop*on"

Availity’s*data* …*for*applica=on*intelligence*

Red"Hat"Linux"Servers" Oracle"Iden*ty"

Manager"

Custom"metrics"from"NFS"Servers"

PlaLorm"independent"SAN"systems"

Homegrown"Applica*ons""

Oracle"Databases"

Page 10: Splunk Beyond application monitoring PDF

Scheduled*Searches*Become*Alerts*

Recent*Request:*I(need(to(be(alerted(if(the(batch(aggrega=on(queue(backs(up*

!  I*should*have*no*more*then*500*items*for*longer*than*30*minutes*

!  I*need*to*receive*an*email*!  I*need*this*yesterday*!  You*should*be*able*to*find*what*you*need*in*aries.log*

*

10*

Page 11: Splunk Beyond application monitoring PDF

Scheduled*Search*Demo/Example*"   Let’s*build*an*alert!**

11*

Page 12: Splunk Beyond application monitoring PDF

Use*Your*Alerts*to*Drive*Dashboard*Usage*

FollowLup*Request:*Can(you(add(some(visualiza=ons?(

!  Can*you*provide*me*with*some*history?*

!  Can*you*show*me*what*my*counts*are*in*rela=on*to*the*same*=me*last*week?*

!  Can*you*build*it*so*that*I*can*easily*edit*and*add*to*it*later?*

*

12*

Page 13: Splunk Beyond application monitoring PDF

Dashboard*Demo/Example*"   Let’s*build*a*dashboard!**

13*

Page 14: Splunk Beyond application monitoring PDF

Team*Dashboards*•  Senior*Leadership**

–  Overall*health*of*the*business*•  Account*Management**

–  Payer*stats*–  Portal*stats*

•  Opera=ons*–  Transac=on*processing*measurements*–  Job*tracking*

•  Marke=ng*and*Training*–  Page*hits*–  File*downloads*

14*

Page 15: Splunk Beyond application monitoring PDF

Senior*Leadership*"   What’s*really*going*on?*–  I*don’t*need*to*see*the*dayLtoLday*issues,*I*need*the*big*picture.*

"   Don’t*kill*me*with*data!*–  I*should*receive*what*I*need*in*a*quick*glance.*

"   “X”*number*of*events*isn’t*good*enough!*–  There*are*5,000*events.**Is*that*good*or*bad?**Is*that*what*we*normally*run?*

15*

Page 16: Splunk Beyond application monitoring PDF

16*

Page 17: Splunk Beyond application monitoring PDF

Account*Management*"   How*is*my*account*doing?*–  Are*transac=ons*comple=ng*successfully?*–  Have*response*=mes*increased/decreased?*

"   What*do*I*need*to*report*back*to*my*account*contacts?*–  Trouble*transac=on*types*–  Regional*outages*

"   What*might*cause*an*increase*in*call*volume*to*the*help*desk?*

17*

Page 18: Splunk Beyond application monitoring PDF

18*

Page 19: Splunk Beyond application monitoring PDF

Opera=ons*•  How*do*the*transac=on*volumes*look?*•  RealL=me,*B2B,*and*Batch*

•  Are*automated*connec=on*jobs*occurring?*•  Job*Scheduler*–*Tidal*•  Example*log*entry*L*07/31 15:05:29:740[16:MD-5]: (mem=1609407968/2142568448)

ServerNode: Event = SystemActivityMessage: Job FTP MEDICAREB277IN[189] completed status [Completed Abnormally].*

•  Any*issues*with*our*batch*processing*engine?*•  Homegrown*Applica=on*–*Aries*•  Example*log*entry*L*08/10/2012 13:41:47 [IR-0] INFO c.a.aries.gateway.GatewayScanner

- [splunk] tpt="66",qn="ARIES.INTERNAL.REQUESTS",ts="08-10-2012 13:41:47.723",di="9d531343-1d56-417c-abac-8c367e87ef99",psz="957",nn="10.xx.xx.xx",qt="76871",ct="16",si="Availity",evi="1636814868"!

19*

Page 20: Splunk Beyond application monitoring PDF

20*

Page 21: Splunk Beyond application monitoring PDF

Marke=ng*and*Training*"   Who*uses*our*product?*–  Page*hits*–  Browser*usage*–  Loca=on*–  Market*penetra=on*

"   ATer*hos=ng*a*webinar,*do*we*see*an*increase*in*demo*hits?*"   Which*part*of*the*FAQ*receives*the*most*views?*"   Who*is*reading*the*links*in*the*“Availity*News”*sec=on?*

**

21*

Page 22: Splunk Beyond application monitoring PDF

22*

Page 23: Splunk Beyond application monitoring PDF

23*

Page 24: Splunk Beyond application monitoring PDF

Availity’s*Top*Wins*with*Splunk*

24*

Ease*of*data*consolida=on*

From*IT*to*the*enterprise*

From*reac=ve*to*proac=ve*

"   Find*clues*easily*with*simple*searches*

"   Transform*the*data*to*eliminate*IT*problems*proac=vely*

"   Trace*transac=ons*quickly*across*the*applica=on*and**infrastructure*stack*

"   Incrementally*grow**based*on*need*

"   Mine*exis=ng*structured*and*unstructured*data*

"   Consolidate*and*integrate*mul=ple*sources*easily*without*any*prior*knowledge*on*the*data*

"   Customized*roleLbased*views*in*less*than*10*minutes*

"   RealL=me*views*for*our*execu=ve*management*(CTO)**

"   Viral*adop=on*across*the*enterprise*–*over*126*ac=ve*users*

Page 25: Splunk Beyond application monitoring PDF

What*Should*I*Do*with*This?*"   Don’t*stop*at*simple*shell*scripts.**Paint*a*picture*with*your*data.*"   Encourage*your*users*to*build*their*own*dashboards.*"   Try*to*replace*a*piece*of*monitoring*with*Splunk.**Improve*on*the**process,*and*offer*more*then*the*consumer*originally*asked*for.*

25*

Page 26: Splunk Beyond application monitoring PDF

Ques=ons?*

26*