Pre-Con Ed: Who's minding the SSO store?

World®’16

Who’sMindingyourSSOStore?

JasonWilcoxSr.ServicesArchitect

SCX16E

SECURITY

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

AttheheartofyourenterprisesecurityinfrastructureisyourCASingleSignOnwebaccessmanagementenvironment.Sometakeitssimplicityforgranted,thesilentworkhorsethatprovidesagreatuserexperienceacrossyourappswithhighperformanceandreliability.ButwhoandwhatiskeepingtabsonyourCASSOapplication?

Inthissession,wewillexplorebestpractices,methodsandtoolsthatcanbedeployedtomonitorthehealthofyourmissioncriticalwebaccessmanagementsolution.Wewillcover:

ThekeyaspectsofwhatimpactstheperformanceandstabilityofyourCASingleSignOnsolutionHowtotraceaproblemtorootcauseusingtoolslikeCATraceLogReader,APMforSSO,Spylogix,andSplunk.HowtocreateamonitoringandalertingstrategyforyourCASingleSingOnSolution.HowtousemonitoringdatatotuneandoptimizeyourCASingleSignOnSolutioncomponents

JasonWilcox

CAtechnologiesSr.ServicesArchitect


Agenda

WHO’STALKINGTOMYPOLICYSERVER,ANDWHYDOICARE?

MST=T/ATT(S),YESMATHMATTERS

ESTABLISHINGAPROACTIVEMONITORINGPROGRAM

WHATARETHESSOKPI’S

HOWDOIMONITORTHOSEKPI’S

BUYWHYDOESITMATTER?

1

2

3

4

5

6


AgentsConnection

§ WebAgentopenstheTCPconnectiontothePolicyServer– Bydefault,underload,theweb

agentwillopenamaxof20sockets– Connectionsare“long-lived”

§ PolicyServerclosesconnections– IdleTimeout(minutes)in

SMConsole– Non-IdleTimeoutvia

AgentConnectionMaxLifetime inXPSObject

AgentsOpentheConnection,thePolicyServerCanClosetheConnection

Authoriza

tion

Authen

tication

Administratio

n

Accoun

ting

AdministrativeUIUserStore

PolicyServer

Protected Resources

PolicyStore

WebServer

WebAgent


AgentConversation

§ Individualsocketsaresynchronousconnections

§ OncetheWebAgent“asksaquestion”tothepolicyserver,thatsocketconnectionis“busy”untilthePolicyServerresponds

TheConversationisaRequest/ResponseModel,NoInterruptionsAllowed

Authoriza

tion

Authen

tication

Administratio

n

Accoun

ting


PolicyServer

Protected Resources

PolicyStore

WebServer

WebAgent


AgentSecurity

§ AgentAPIinitiatesaconnectionwiththeTrustedHostNameandSharedSecret– Handshakeincludesestablishing

encryptedchannel– Retrievesoperationalparameters

fromtheHCO– Retrievesconfigurationparameters

fromtheACO(OrLocalConfig)

TheConversationStartswithAuthentication,andCreatedanEncryptedChannel

Authoriza

tion

Authen

tication

Administratio

n

Accoun

ting


PolicyServer

Protected Resources

PolicyStore

WebServer

WebAgent


TopicstheyDiscuss

§ OverthisTCPconnectiontheWebAgentsendsthefollowingAgentAPIcommands:– isProtected()– isAuthenticated()– isAuthorized()

§ NottheonlycommandsbutthosearetheprimaryfunctionsoftheAgent.

AgenttoPolicyServerCommunicationisPrimarilyRequest/Response

Authoriza

tion

Authen

tication

Administratio

n

Accoun

ting


PolicyServer

Protected Resources

PolicyStore

WebServer

WebAgent


ThePolicyServerresponds

§ Wecanseethereactorthreadtakingtheserequestsandputtingtheminthequeue

§ Nowontheotherendofthequeue,wecanseewhatarecalled“Workerthreads”tohandlethework

WhentheAgentComesKnocking,WhoAnswers?

WebAgent

ReactorThread

WorkerThread

PolicyServer


ConfiguringWorkerThreads

§ Theworkerthreadsareconfiguredhereinthemanagementconsole– defaultisfor20worker

threadsinCASiteMinderR12.5x

§ Howmanydoyouneed?

ThreadsNeverDie,OnceReachedTheyWillAlwaysAtMax


WhatDoWorkerThreadsDo?

§ Workerthreadsdothework!

§ Theworkerthreadstaketheitemoffthequeue,andgototheUserStore,PolicyStore,SessionStore,cache,etc…

§ Workerthreadsgenerateassertions

§ Workerthreadsprocessxml

§ Workerthreadsdoeverythingthepolicyserverneedstodo

BeforeICantellYouHowManyYouNeed,WeNeedtoKnowWhattheyareDoing?


ThreadLocking

§ SimilartotheAgentsocketrequest,theworkerthreadwillcontinuehandlingtherequestuntilitiscomplete.– Forexample,iftheworkerthreadsneedtodoanisAuthenticate()call,

itwillgoouttotheLDAPdirectoryserver.Theworkerthreadwillbeblockeduntiltheldapsearch andbindiscomplete.

– IfanindividualworkerthreadneedstomakemultipleLDAPcalls,thosecallsareprocessedinasynchronousmannerwithinthatthread

– Thisthreadcannotbeusedforanythingelsewhileblocked

WorkerThreadsStartaTaskandareBusyUntiltheTaskisCompletedorTimesOut


ControlWehavecontrolovertheagentconnections,thepolicyserversockets,andthenumberandlifetimeofeachofthem.

Wedon’thavecontroloverhowlongatasktakestocomplete.

AgentsAgentsinitiateasecureconnectiontothepolicyserver.

Eachagentconnectiontakesupasocketonthepolicyserver.

Eachagentconnectionperformsatask,andisbusyuntilthattaskiscompleted.

PolicyServerPolicyServersreceiverequestsfromtheagentswithareactorthread.Thereactorthreadputsrequestsinthequeueforworkerthreadstowork.Workerthreadsperformataskandareblockeduntilthetaskiscompleted

CheckpointWhatDoWeKnowSoFar?


MST=t/att(s),YesMathMatters

§ Withtheinformationwehave,wecanbuildapredictivemodelforperformanceandcapacity.

§ Wemustalsounderstandtheimpactsofthroughputonthatmodel,andtheimpactsoflatencyonthroughput.

§ UsingthiswecanidentifyKeyPerformanceIndicatorsthatshouldbeproactivelymonitored,managed,andreportedon.

Rememberthatteacherwhosaidsomedaythiswillsaveyourlife?Yeahitwon’tbutmathstillmatters


BuildingOurModel

§ Throughput– Totaltransactionspersecondthepolicyserverisfulfilling(persecond)

§ Latency– Howlongdoeseachtransactiontaketobeprocessed

§ ThreadLatency – howlongbeforeaworkerthreadpullstherequestfromthequeue

§ ExecutionLatency– howmuchtimedoesthatworkerthreadtakeinprocessingtherequest


TheRelationshipofThreads,ThroughputandLatency

§ Onanysystemwithasetnumberofthreads,throughputandlatencyareinterrelated– Aslatencygoesupthethroughputgoesdown– Asthroughputgoesdownadditionalrequestsarequeuedcausing

increasedlatency

MaximumServerThroughput= !"#$%&'()*!,-!./$('$1)


WeControlThreads,butNotThreadorExecutionLatency

§ TherearetwoprimaryreasonsforPolicyServerslowdown1. ToomanyAgentAPIrequestscominginforthePolicyServer2. Responsetimefromtheuserdirectory

§ IftoomanyAgentAPIrequestsarecomingin,thethreadlatencywillincreaseiftherearen’tenoughthreadstoservicetheminatimelymanner.

§ Iftheresponsetimefortheuserdirectoryincreasesexecutionlatencyincreases,whichinturncausesthreadlatencytoincrease.


TooManyAgentAPIRequests

§ Asinglewebpagedoesn’tmeanasinglerequest.

§ HCOsettingslimitingthenumberofagentconnectionsaren’tapplicabledependingontheapachethreadingmodel.

§ Iftheapplicationteamshaveconfiguredtoallow2000maxclients,butyouaresaying20maxconnections….itwillbe2000maxconnections.

§ Atpeaktimes,ifnotproperlymanaged,yourwebserverscanoverloadyourpolicyserverandsignificantlyincreasethreadlatency.


DirectoryLatencyAffectsThroughputwhichAffectsSingleSign-OnPerformance§ Assumptions

– 15threads

– Averageof7LDAPqueriespertransaction– AverageLDAP(includingthenetwork)latencyis10ms

– Goal:125transactions/sec

§ Averagetransactiontimemustbeatleast70Ms(LDAP)+30msprocessing=100ms (0.1sec)

§ 15threads/0.1seconds=150transactions/secmaximum

§ WhenLDAPgoesto15msthemaximumthroughputdropsto111txns/sec

Using:MaximumServerThroughput=


KeepingLowTransactionalLatency

§ Rightsizeconnections,threads,andagentratio’sforloadandhardware.

§ Minimizecustom/thirdpartycodeandoptimizeanycalloutsthatcodemakestoremotesystems

§ UsesmartLDAPsearchesandoptimizeddatabasequeries

§ KeepAgenttoPolicyserverandPolicyservertouserdirectoryconnectionsoverfastconnections– possiblyinsamedatacenter

§ Workwithuserdirectoryteamstoensurethedirectoriesareperformingasrequired


ThroughputWecanmodelthroughputtoproperlyplanforcapacity.

Wecanmodelwhatevenasmallchangeinthenetworkordirectoryperformancewilldo.

Wecanusethesemodelstobecomeproactiveandpredictive.

ThreadLatencyCanbeaffectedbyalargevolumeofrequests

Canbeaffectedbyhighexecutionlatency

Youmustdothemathinadvanceandrightsizefortheexpectedpeakloads

ExecutionLatencyPoorPolicyDesigncanincreasecallstothedirectorySSOisusuallythevictim,butunlessyoucanproveit,thatdoesn’tmatter.Increasingthenumberofthreadsisnotalwaystheanswer.

CheckpointWhatDoWeKnowSoFar?


WhataretheSSOKPI’s

§ Withtheinformationwehave,wecanbuildapredictivemodelforperformanceandcapacity.

§ Wemustalsounderstandtheimpactsofthroughputonthatmodel,andtheimpactsoflatencyonthroughput.

§ UsingthiswecanidentifyKeyPerformanceIndicatorsthatshouldbeproactivelymonitored,managed,andreportedon.

Knowledgeishalfthebattle,butyouwillstillloseifthat’sallyouhave.


WhatdoyouNeedTrack?

UserStoreAccess PolicyStoreAccess Session StoreAccess

CacheSuccessRate CacheMissRate SocketCounts

Max QueueLength CurrentNormalQueueLength HighPriority QueueLength

Avg Authorization Time Avg AuthenticationTime Avg ValidationTime

Avg IsProtected Time MaxSockets TransactionCounts

AgentTransactionTimes AgentCachesettings


HowCanWeMonitorSSOKPI’s

§ CASMTraceTool

§ CAOneView Monitor

§ CAAPMforSSO

§ SNMP(Splunk,UIM,AnytoolthatcanissueSNMPGET)

§ Spylogix forCASSO

It’snotabouthowyougetthedata,butwhatyoudowithit


SMTraceToolUnlockingtheDatainYourCASSOLogs

§ LoadandparselogsfromallCASSOcomponents

§ Generatesreportsdetailingperformanceforthatpointintime

§ Identifiespotentialbottlenecks

§ Let’stakealook!!!


OneView Monitor

§ CentrallyreportsKPI’s

§ Gathersdatafromagentsandpolicyservers

§ Youneedtorecordandgatherthedata


TheDataisthere,ifYouGoandGetit


CAAPMWhenYouPreferthoseKPI’sWrappedupinaBow

§ BuiltinCASSODashboards

§ Knowinstantlyifthereisaproblem,drilldowntoRCA

§ Dataandanalysiscomestoyou.



SNMPIntegratewithYourPreferredTool

§ 60ObjectsaccessiblebySNMPGET

§ 17EventsavailabletobesentviaSNMPTrap

§ Presentthedatahowyouwantitpresented

§ Let’slookatsomesamplesfromSplunk!


SpyLogix forCASSOCertifiedCASolution

§ FullAnalyticsplatformforCASSO

§ Builtindashboardsforperformance,systemsmanagementandutilization

§ FocusedonMTTRandMTBSI


WhyDoesitMatter?HowwillthisDataHelpMe?

§ User’sarecomplainingabout‘slow’SSOperformance– Whatisthedefinitionofslow?– CanyoushowwhattheperformanceofSSOistocombatthat

impression?– Doestheirdefinitionofslowincludetheloadtimefortheapplication

page?Howdoyoushowifyouareaffectingthat?– Canyoushowhistoricalevidenceofperformanceandstability?

§ MaybeSSOishavinganissue,canyoupinpointwhereitis?

Realworldscenario’s


HavingAccesstotheDataImprovesYourResponse

§ Herearegraphsshowingthecustomerqueuedepth

§ Fromthesechartswecanseethequeuekeepsgrowing– Eithertheloadhasincreased,

or– Thebackendcannotkeepup


NowIKnowthereisanIssue,What’sCausingit?

§ Needtoanswertwoquestions– HowdowedeterminewhichofthetwoconditionsthePolicyServeris

in?§ Arewequeuingtherequestbecausetherearetoomanyincomingrequests?

§ Arewequeuingtherequestbecausethetransactionsaretakingtoolongtoprocess?

– Howcanweeasilyanswerthosequestions?


SMTraceToolisAbletoIdentifySlowLDAPResponseTimes§ Wecanseethedistribution

ofLDAPresponseswhichhaveamajorinfluenceonCASiteMinderthroughput

§ Wehadtogogetthisdata,whatifwehadbeenalertedwhentheaveragestartedincreasing,beforeuserscomplained?


APMforCASSOShowstheSameTypeofSlowdowns

§ Weseeadifferentrepresentationofthesameproblem.

§ Butinsteadofbeingalertedtotheproblembyusers,APMforSSOcanalertusbeforebecomesaproblem.


EstablishingaProactiveMonitoringProgram

§ Wecannotrelyonusererrorstotelluswhenthereisaproblem,itsoftentoolate

§ YoumusthavethedatareadilyavailabletoadvertiseSSO’ssuccess

§ YoumusthavethedatareadilyavailabletoactwhenSSOhasanissue.

Usersarenotyourfirstlineofalerting


SingleSign-OnisMissionCritical

§ SingleSign-Ontouchesmanyapplicationsacrosstheenterprise– BothinternalemployeeandConsumertransactions

§ IfSingleSign-Onstops,theapplicationsstopaswell

§ Whenaproblemoccurswemustknowwhysoactioncanbetaken– Needtoidentifyproblemsthatareintermittent– Needtoidentifypossibleproblemsbeforetheycauseoutages


OrganizationsNeedtoIdentifyProblemsQuickly

§ SingleSign-Oncancrossmanyorganizations– Applicationteams– Directoryteams– SingleSign-Onteams

§ Whenaproblemoccurswetendtoplayorganizationalblamegames

§ SinceSingleSign-Ontouchesmanycomponentsitoftengetsblamedevenifitisnotatfault


ManyDifferentWaysto“Monitor”

§ “Monitor”canmeanmanydifferentthings– ComponentsUp/Down– System“health”– Useractivity– AdministrativeActivity

§ Yourprogrammustincorporateelementsofalltobemosteffective


SyntheticTransactions(CAAppSyntheticMonitor)

§ ToolstoAutomatically“login”andaccessapage

§ Seesthesitefromanenduserperspective

§ Becarefulwhengeographicallydistributed– Themonitorbecomesthefailurepoint

§ Becarefulofmonitoringbecomingthebiggestuserofthesystem.


SyntheticTransactions(CAAppSyntheticMonitor)

§ Whatittellsyou– Isyourwebsiterespondingtologins– Logintransactionandfirstpageloadtimes.

§ Benefits– Looksacrossentiresite

§ Drawbacks– Unknownwhatthepathisforthetransaction

§ Failover,roundrobin,internalcomponentfailuresarehidden– Cancreateextraloadonsystem

Tip:asinglewebsiteoneachpolicyserverwithasingleagentthatonlycommunicatestothatpolicyserver


ServerErrorFile ACOSetting

§ ExistingACOsettingtoeitherdisplayafriendlyHTMLpageorredirectonaWebAgent Error

§ Usetheredirectabilitytoredirectuserstoafriendlypageonaseparatewebserver– Createaseparatelogforerrorsforallagentsinasinglespot– Collecttheerrorcode(Querystring)– Collectthereferrer(HTTPheaders)

§ Logtheseandanalyzeweekly.


ServerErrorFile ACOSetting

§ Whatittellsyou– HasaWebAgentencounteredanerror

§ Whattheerrorcodeis§ Whichwebsite

§ Benefits– Realtimeinformation– cantriggeranalert– Usefulincalculatingintermittentissues– Canalsodisplayafriendlyerrorpage

§ Drawbacks– Ifyouaren’tanalyzingthedata,thereisnovalue


NetworkVisualization(CAApplicationDeliveryAnalysis(ADA))

§ NetworkLayerMonitoringtool

§ PlugsintonetworkswitchesandlooksatTCPTraffic

§ Canexaminecommunicationsto/frommultiplesystemsandunderstandlatencyofthesecomponents


NetworkVisualization(CAApplicationDeliveryAnalysis(ADA))§ Whatittellsyou

– Latencyofcommunicationsbetweenmultiplecomponents

§ Benefits– Canquicklyidentifycomponenthavetrouble– Canidentifyifitisthenetworkortheapplication

§ Drawbacks– NotincludedinCoreSingleSign-OnLicense– NotaSingleSign-Onspecificsolution– Overkillinsmallenvironments


KeyPerformanceIndicators

§ Oneview/APM/SNMP/SpyLogix

§ Onecomponentofacomprehensivesolution

§ Oftenthisisthemissingcomponentinacomprehensivemonitoringsolution.


Step1– BaselineYourEnvironment

§ Onceyouhavechosenyourtoolsetidentifyyourbaseline– Capturedatawithoutalertingfor2– 4weeks

§ Focusonatimeframethatspanskeypeakusageperiods– Monthlyorquarterlyspikes

§ Discussthegoalswithyourcustomersandstakeholders– Theyoftenhaveinsightsintotheirusagethatyoumaynotknow.


Step2– Createhabitstoreviewdata

§ Createhabitsandcarveouttimefordatatobereviewed– Thishelpsidentifynewarea’sforcoverage– Identifiespreviouslyunknownandunmonitorederrors– Themoreyoureviewasystem,thebetteryouknowit

§ Asnewerrorsarefound,createaknowledgebaseandkeepitupdated– Sharethatdataandstepstoresolve.Themorepeoplethatknowthebetter.

Thisisaboutensuringagoodcustomerexperience.

§ Incentivizefindingnewitems.Rewardyourteamforfindingnewissuesandtheirresolution,makeitcultural


Step3– Createalertsbasedonthebaselinedata

§ Whenperformancedegradedmorethanx%warn

§ WhenperformancedegradesmorethanXX%alert

§ Warnyourteamsotheycanactbeforeaproblem

§ Alertbroadandwide.Transparencybuildstrust,trustbuildsconfidence.

§ Continuetofinetunealerts


Step4– CreateDashboards

§ Createdashboardstohelpyourteamstayontopofthesolution.

§ Identifythemostcriticalitemsandputtheminfirst

§ Identifythemosttroublesomeandputtheminsecond

§ Makesureeveryoneknowshowtogettothemandhowtoreadthem.


Step5– Advertise,Engage,andSell

§ Createdashboardsforyour‘customer’tosee– Generalonesthatareacrossthewholesolution– Specificapplicationbaseddashboardsforlargerapps

§ Createanexecutivereportandsenditoutregularlytoyourcustomers,theirchainofcommandandyourchainofcomment.– Advertiseyoursuccess.

§ Considerasubscriptionmodeltoyoursuccessesandchallenges– Internaltwitterfeedswherecustomerscansubscribeandjustseewhatis

goingon.


RecommendedSessions

SESSION# TITLE DATE/TIME

SCX12EPre-ConEd:FiveEasyStepsforMigratingtoCADirectory

11/15/2016at3:30pm

SCT44TWebAccessManagementandFederation–TwoGreatTastesthatTasteGoodTogether

11/16/2016at11:30am

SCX20SCARoadmap:Authentication,SingleSign-On,Directory

11/17/2016 at 01:45pm


WeWanttoHearFromYou!

§ ITCentralisaleadingtechnologyreviewsite.CAhasthemtohelpgenerateproductreviewsforourSecurityproducts.

§ ITCSstaffwillbeatmostsessions.Ifyouwouldliketoofferaproductreview,pleaseaskthemaftertheclass,orgobytheirbooth.

Note:§ Onlytakes5-7mins§ Youhavetotalcontroloverthereview§ Itcanbeanonymous,ifrequired


Questions?


Thankyou.

Stayconnectedatcommunities.ca.com


Security

FormoreinformationonSecurity,pleasevisit:http://cainc.to/EtfYyw

Technology

Pre-Con Ed: Who's minding the SSO store?