ArcGIS Enterprise: Performance and Scalability Testing … · 2018-04-10 · Administration...

ArcGIS Enterprise:

Performance and Scalability

Testing MethodologiesAndrew Sakowicz, asakowicz@esri.com, Professional

Services

Motivation

Growing complexity of ArcGIS Enterprise

Certificates

Load balancer

Firewall

ArcGIS Web Adaptor

Portal for ArcGIS

Storage with immediate consistency

Requires dependable infrastructure

ArcGIS Server

ArcGIS Data Store

Database

-users

-services

Overload System:

Identify typical cases

-Network

-NAS

-VMWare

Unstable

Infrastructure:

Identify typical cases

-configuration

-maintenance

-workflows

Bottlenecks:

Identify typical cases

Administration Challenges

• Multiple administrators

• Multiple disparate monitoring/diagnostic tools

• Data collected in a reactive fashion: on demand and for limited time

• Correlation of data with different timestamp is difficult

• ArcGIS administrators do not have access to all tools, data and reports

• Challenging to quickly identify the root cause and take appropriate measures

When problems arise, what is the root cause?

Definitions

Performance

• Speed, e.g. response time (seconds)

Scalability

• The ability to increase output and maintain acceptable performance

Capacity

• The maximum level of output the system can produce, e.g.

• X cars/sec

• X maps/sec

At capacity Over capacity

Bottleneck

• Resource(s) limiting the performance or capacity

Not bottleneck bottleneckThink of :

Lanes -as CPU processor

Toll -as ArcGIS Server instances

Cars -as map requests

Step Load and Response Time

Response Time (sec)

time

Step Load (users)

Throughput (request/hr)

Throughput(req/hr)

Response Time (sec)

time

Step Load (users)

Resource utilization: CPU, Memory, Network

Throughput(req/hr)

CPU Utilization (%)

Memory used (Mb)

Network used (Mbps)

Response Time (sec)

time

Step Load (users)

Capacity

Throughput(req/hr)

CPU Utilization (%)

Memory used (Mb)

Network used (Mbps)

Content length (bytes)

Response Time (sec)

Time

User load

Capacity (~ 85% utilization)

Process and tools

Process and toolsEsri tools

Process and toolsEsri tools

Tools download location

• ArcGIS Monitor

- https://my.esri.com/

• Others

- http://www.arcgis.com

- owner:EnterpriseImp

- Show ArcGIS Desktop Content

Performance Testing

Tuning methodologyProfile each tier starting from the top

Browser

Web Server

ArcGIS Server

ArcSOC

Total Response

Time (t1-t2)

Wait Time

Search &

Retrieval Time

Usage Time

ArcSDE/DBMS

t1 t2

FiddlerFiddler measurement approximately 5.2 seconds

Mxdperfstathttp://www.arcgis.com/home/item.html?id=a269d03aa1c840638680e2902dadecac

Oracle TraceCompare elapsed time

Elapsed time slightly changed due to different test runs

Oracle Execution plan

Inefficient spatial index

Testing process

Application

GIS Services

Infrastructure: Hardware and Software

Test tools feature comparison

Tool Cost Learning

Curve

OS Metrics GIS Data

Generation

GIS Test

Automation

Load Runner High High Windows/Linux No No

Visual Studio Medium High Windows No No

JMeter Free High Requires additional

plugin

No No

System Test Free Low Windows/Linux Yes Yes

Tech Support by Esri PS as part of consulting support

Performance testing

• Identify bottlenecks

• Determine system capacity

• Demonstrate performance SLA

Value

System Test Tool features

End to end

Actionable information

ArcGIS Monitor

Alerts: actionable informationStarting point for troubleshooting

Available in 10.6.1Charts and Stats SourceDetailsAnalyze

alert groups

Charts and stats

Details

Availability

• Downtime defined only by Critical alerts

Aggregated view based on Critical alerts

36

Categories End to end

Web

• Response Time

• HTTP code

ArcGIS

• Server

• Portal

• GeoEvent

• GeoAnalytics

• ArcSOC Optimizer

Additional portal stats coming soon, e.g. dependent items, views per item, users

ArcGIS: WebGIS Health

• Severity

• Healthy

ArcSOC OptimizerSetting min / max instances across 100 to 1000s of services

in dynamic environments is challenging

Database

• ArcGIS Datastore

• Oracle

• SQL Server

AWS RDS and SQL Azure coming soon

InfrastructureCPU, Memory, Disk, Network, Load Balancer stats

Monitor network between ArcGIS componentsCritical to have communication to config store and nodes

Web Server

Web Adapter

ArcGIS Server

Usage

• Transactions

• Response Time

• HTTP Codes

User stats

GeoInfoRequests per user IP

Get Startedhttps://enterprise.arcgis.com/en/monitor/latest/get-started/arcgis-monitor-system-requirements.htm

Demo site – the latest featureshttps://arcgismonitor.esri.com/

Gallery: https://arcgismonitor.maps.arcgis.com

• Tutorials

• Video

• Extensions / Tasks

https://www.arcgis.com/home/group.html?id=58d996e9b40d45439d298d14fa309534&start=1&view=list&sortOrder=asc&sortField=owner#content

Use Case:

NOAA National Weather

Service IDP GIS Program

NOAA National Weather Service IDP GIS Program

• 24 x 7 High Availability (HA)

• Service Level Agreement (SLA)

• High volume of requests

High volume of requests

Service Update Frequency Performance

Workflow

Nominal Map/Image requests (Est) Peak

Map/Image requests (Est)

Watches\Warnings and

Advisories

1 minute \

10 minutes

Light Vector 10K Hr 100K Hr

Daily Global Precipitation Daily Light Raster 1K Hr 10K Hr

Radar (1x1 km base reflectivity) 5 minutes Light Raster 20K Hr 200K Hr

NDFD Temperature 15 minutes

1 hour

Heavy Raster 1K Hr 20K Hr

Hurricane Tracks/Wind/Surge 10 minutes or less Light Vector 1K Hr 100K Hr

AHPS gauges 15 minutes Light Vector 1K Hr 10K Hr

Flood Outlook Product Daily Light Vector 1K Hr 10K Hr

CPC Weather Hazards Daily Light Vector 10K Hr 100K Hr

Quantitative Precipitation

Forecast (QPF)

15 minutes Light Vector 1K Hr 10K Hr

Weather Features Daily Light Raster 1K Hr 10K Hr

Sea Surface Temp Daily Light Raster 1K Hr 10K Hr

NDFD Wind Velocity Forecasts 1 hour Light Raster 1K Hr 10K Hr

Tools

• Big Brother (NOAA internal system of record)

• Nagios Check MK (SA’s system of ‘preference’)

• System Monitor

• System Test

• JMeter

• mxdperfstat

Nagios - Check MK

System Monitor KPI

GIS Server/Service Key Performance Indicators (KPI)

currently released as ArcGIS Monitor

System Monitor (Reports)

• Tr/Sec

• Free

currently released as ArcGIS Monitor

System Monitor (Reports)

• CPU

• Mem

currently released as ArcGIS Monitor

System Test

• Individually test services that we expect to receive high traffic to define a service

throughput baseline given a certain min/max instance configuration

• Investigate/analyze performance for services that have been alerting as slow or non-

responsive in Big Brother

• Develop test extents/parameter files to be exported using custom python scripts so

that they can be configured in long running system wide load tests performed by

the SA’s using JMeter

System Test

System Test

System Test

System Test

JMeter

mxdperfstat

• Been around for a while and still very useful

• Easy to batch script

• All service sponsor provided map documents were tested using mxdperfstat to

identify cases where poor configuration could impact server performance

mxdperfstat

• 1.59 seconds may not seem horrible but this service is

expected to see very heavy traffic and accounts for 6

instances on 8 servers.

• Improvements to the cartography could make a noticeable

difference in overall server performance and/or free resources

for other services

Summary

Process and toolsEsri tools

ArcGIS Monitor: Optimize Your Enterprise GIS Deployments

• Monitor the health, usage, SLA

• Improve performance

• Reduce administration costs

• Quickly diagnose:

- Unstable infrastructure

- Overloaded system

- Bottlenecks

Scalable and non-intrusive: Used by Esri Managed Cloud Services to Monitor 500+ GIS Servers

Print Your Certificate of AttendancePrint stations located in the 140 Concourse

Tuesday Wednesday12:30 pm – 6:30 pm GIS Solutions Expo Hall B

5:00 pm – 6:30 pm GIS Solutions Expo SocialHall B

10:30 am – 5:15 pm GIS Solutions Expo Hall B

6:30 pm – 9:00 pm Networking ReceptionSmithsonian National Portrait Gallery

Download the Esri Events

app and find your eventSelect the session

you attended

Scroll down to find the

feedback section

Complete answers

and select “Submit”

Please Take Our Survey in the Esri Events App