Download pdf - SER1504BU Extreme Performance vCenter Performance or ...€¦ · PSC 1 2 Replication and Failover VC1 VC2 VC3 PSC 1 and Failover Replication NO FAILOVER Site 1 Web Site 2 Client External

Ravi Soundararajan, VMware, IncEric Saxe, VMware, Inc

SER1504BU

#VMworld #SER1504BU

Extreme Performance Series:vCenter Performance Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#SER1504BU CONFIDENTIAL 2



bution

Our Goals

• Help you understand vCenter architecture

• Help you understand what factors influence vCenter performance

• Help you use this knowledge to guide vCenter deployment




bution

vCenter 6.5: Appliance vs. Windows

• Performance: Appliance at Parity with (or better than) Windows

• Migration tool for Windows to Appliance: 6.0u2m (and 6.5)

CONFIDENTIAL: The information in this presentation is intended to outline our general product direction, and it should not be

relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract.4

0

0.2

0.4

0.6

0.8

1

1.2

Cluster, light Cluster, heavy Datacenter, light Datacenter, heavy Datacenter, heavy (Windowsmodified settings)

Normalized vCenter throughput benchmark operations per minute

Windows Appliance



bution

vCenter 6.5: Appliance vs. Windows

• Performance: Appliance at Parity with (or better than) Windows

• Migration tool for Windows to Appliance: 6.0u2m (and 6.5)

• https://blogs.vmware.com/performance/2017/04/vcenter-6-5-performance-what-does-6x-mean.html



0

0.2

0.4

0.6

0.8

1

1.2

Cluster, light Cluster, heavy Datacenter, light Datacenter, heavy Datacenter, heavy (Windowsmodified settings)

Normalized vCenter throughput benchmark operations per minute

Windows Appliance



bution

Agenda

1 vCenter and Web Client Deep Dive

2 Multi-Node Deployment Strategies

3 Performance Considerations

4 Concluding Remarks




bution

10,000 foot view of vCenter

7

UI clients

API clientsvpxd DB

vCenter server

#SER1504BU CONFIDENTIAL



bution

vCenter: Under the Hood (We Will Dissect This…)


VPXD

DB

vsphere-client

Web Browser

AD

vCenter server

DIRECTORY SERVICE

SSO

(idmd,stsd,etc.)

vSphere hosts

vpxd-svcs vmware-sps

Dataservice

perfcharts

eam

vmon,

rhttpproxy,health,

content library,

licensing …



bution

Single Node Deployments: Embedded DB (Windows, Appliance)

9

VPXD

DB

vsphere-client

AD

vCenter server

DIRECTORY SERVICE

SSO

(idmd,stsd,etc.)

vpxd-svcs

Dataservice

perfcharts

vmon,

rhttpproxy,health,

content library,

licensing …

CONFIDENTIAL: The information in this presentation is intended to outline our general product direction, and it should not be relied on in

making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract.: :

Web Browser

vSphere hosts

PerfchartsOverview chartsRealtime: vpxdHistorical: DB

2015: vpxd-svcs = invsvcKV store frontend to VCDB TaggingAuthorization

(e.g., content library)

Directory ServiceRoles/PrivilegesLicensesTags

ESX1ESX2

vmware-spsVMworld 2017 Content: N

ot for publicatio

n or distribution

Deep Dive: VC + Extensions

10

VPXD

DB

vsphere-client

Web Browser

AD

vCenter server

DIRECTORY SERVICE

SSO

(idmd,stsd,etc.)

vSphere hosts


Dataservice

perfcharts

NSX Manager

Plugin installed inside vsphere-client

NSX uses EAM for vibs

eam

vmon,

rhttpproxy,health,

content library,

licensing …





bution

Host Operation 101

11

Client vCenter ESXi hosts

CreateVM

ReserveName

Provision VM

Host Sync “GetChanges”

Host Sync Response

DB

DONE

Storage, etc.

Vpxa/hostd

ROBO viable: http://www.vmware.com/resources/techresources/10165Minimize VC-to-DB latency



bution

Agenda








bution

Multiple vCenters: Terminology

• vCenter node

– Management Node

– vCenter minus licensing service, directory service, and SSO

• PSC node

– Licensing service

– Directory service (stores licenses, tags, roles/privileges)

– SSO

13#SER1504BU CONFIDENTIAL



bution

vCenter node vs. PSC node

14

VPXD

DB

vsphere-client

AD

vCenter node

SSO

(idmd,stsd,etc.)


Dataservice

perfcharts

eam

PSC node

DIRECTORY SERVICE

vmon,

rhttpproxy,health,

content library…

Licensing





bution

Hypothetical Configuration Showing Sharing

• Single Domain, single site

• PSC node allows VCs to share

– Roles/Privileges

– Licenses

– Tags



VC1 VC2 VC3

PSC1

Directory

SSO

vsphere-

client

vsphere-

client

vsphere-

client

Web Browser

Tags

Tag associationsin VCDB





bution

Single Site, PSC-HA configuration

• 6.0: <10ms between nodes

• 6.5: <30ms between nodes

• External LB for failover

• Embedded to External? Manual repointing required



VC1 VC2 VC3

PSC1 PSC2Replication

External Load Balancer for PSC Failover



bution

Single Site, PSC-HA configuration, Search



VC1 VC2 VC3

PSC1 PSC2Replication

Web Browser

Tag: DSL” Tag: DSL”

dsl-n-2001, …” dsl-0508, … dsl-hugeHeap-1,…

SEARCH FOR “DSL”




bution

Multi-Site

• Use when high latency between VCs or PSCs (e.g., > 10ms in 6.0; > 30ms in 6.5)

– 6.0/6.5: < 100ms between sites recommended

18

VC1 VC2 VC3

PSC

1

PSC

2

Replication and Failover

VC1 VC2 VC3

PSC

1

PSC

2


ReplicationNO FAILOVER

Site 1 Site 2



External Load Balancer for PSC Failover External Load Balancer for PSC Failover



bution


Multi-Site, Search for “new”

• Roles, Privileges, Tags, Licenses replicated

• No repointing across sites

• No failover



VC1 VC2 VC3

PSC

1

PSC

2


VC1 VC2 VC3

PSC

1

PSC

2


ReplicationNO FAILOVER

Site 1 Site 2Web

Client




bution

Agenda








bution

Performance Considerations and Best Practices

• Concurrency: How many concurrent ops can I really do?

• CPU and Memory

• DB Performance Monitoring

• UI Performance




bution



• CPU and Memory


• UI Performance




bution

How Many Concurrent Operations Can I Perform?

• vCenter hard limits

– 640 concurrent operations before incoming requests are queued

– 2000 concurrent sessions (incoming requests plus remote console sessions)

• Per-host limits (ESX6.0+)

– Max cost per host: 16

– Host cost per clone/relocate/vMotion: 2

– Host cost per Storage vMotion: 8

– Host cost per linked clone: 1 [requires pre-existing snapshot]

– Cloning powered-on VM: VC takes snapshot first




bution

vCenter Concurrency

• Clone VM from host A to host B • Clone VM from host A to host A

24

vCenter

Host A

VM 1

Host B

VM 2

Cost to A: 2 Cost to B: 2

vCenter

Host A

VM 1 VM 2

Cost to A: 4

Do not use a single host as the source of all clones (i.e., spread out templates)

Better disk performance and better concurrency

A

B

A

BFree Slots Free Slots




bution

How Many Concurrent Operations Can I Perform?

• Per-datastore limits

– Max cost per datastore: 128

– Datastore cost per vMotion: 1

– Datastore cost per Storage vMotion: 16

• Per-NIC limits

– Max cost per 1Gb NIC: 4

– Max cost per 10Gb NIC: 8

– Network cost per vMotion: 1

Best practice: vMotion NIC (portgroup) and vSphere Provisioning NIC (portgroup)

• Limits can be changed, but changes are not officially supported




bution



• CPU and Memory


• UI Performance




bution

CPU/Memory Notes

• VC CPU Rule of thumb: typically does not exceed 70% on average

– Peaks are OK, but consistently > 70% is usually not OK

– If vCenter Server CPU is over 70%...

• Check which process (vpxd, vsphere-client, etc.)

• If vpxd, consider may need more CPU

• If Java services, may need more memory (GC activity)

• May need to examine plugins/extensions/custom API code talking to vCenter

• Memory rule of thumb: do not exceed 70% on average

– If memory is 70%, you may need more memory

– Avoid swapping of vCenter server

– If running VC in a VM, set reservation = VM size




bution

VCSA: Appliance Management Interface (CPU view)

https://<vCenter IP>:5480 “CPU and Memory”




bution

Windows VC: TaskManager


VDCS:pid20744

? pid 7228



bution

Windows VC: ProcessExplorer (1/2)


pid7228



bution

Windows VC: ProcessExplorer…name of service

31

Command line view for pid 7244:invsc = Inventory Service (vCenter 6.0)




bution

VCSA: vimtop (type “vimtop” on command line)

32

Enable troubleshooting modeEnable ssh

CPU: 100%: 1 core200%: 2 cores

MEM: currently allocated




bution

API Calls and Tasks: 6.0

• (6.5 also has it, but under ‘Monitor’)




bution

API Calls and Tasks – More Details: Profiler Log

• Profiler log

• Appliance: /var/log/vmware/vpxd/vpxd-profiler-<NNN>.log

• Windows: C:/ProgramData/VMware/vCenterServer/logs/vmware-vpx/vpxd-profiler-<NNN>.log

• Number of sessions

• --> /SessionStats/SessionPool/NumSessions/total 73

• Running out of sessions to vCenter:

– NumSessions 2000

– Vpxd.log: “Out of HTTP sessions: reached the limit of 2000 sessions”




bution

API Calls and Tasks – More Details: Profiler Log

• Profiler log

• Appliance: /var/log/vmware/vpxd/vpxd-profiler-<NNN>.log

• Windows: C:/ProgramData/VMware/vCenterServer/logs/vmware-vpx/vpxd-profiler-<NNN>.log

• Number of sessions

• --> /SessionStats/SessionPool/NumSessions/total 73

• Mapping of SessionID to Username and IP address of client

• --> /SessionStats/SessionPool/Session/Id='5275dfb0-cc10-38d2-6a44-

04ea824193c0'/Username='VSPHERE.LOCAL\vra'/ClientIP='10.153.10.53'/HttpSession

Object/Hidden/total 0




bution

API Calls and Tasks – More Details

• Anatomy of a vpxd.log message in vCenter: SessionID

– Appliance: /var/log/vmware/vpxd/vpxd.log

– Windows: C:/ProgramData/VMware/vCenterServer/logs/vmware-vpx/vpxd.log

– SessionID: (maps to vRA session from previous slide)

– 2017-08-01T14:27:05.200-07:00 info vpxd[17208] [Originator@6876 sub=vpxLro

opID=28e69da] [VpxLRO] -- BEGIN task-internal-5280 -- EventManager --

vim.event.EventManager.QueryEvent -- 5275dfb0-cc10-38d2-6a44-

04ea824193c0(52b4aeed-eacb-3aa1-2177-8efbd21e57eb)




bution

API Calls and Tasks – More Details

• Anatomy of a vpxd.log message in vCenter: API Call

– Appliance: /var/log/vmware/vpxd/vpxd.log

– Windows: C:/ProgramData/VMware/vCenterServer/logs/vmware-vpx/vpxd.log

– API call: (maps to vRA session from previous slide)

– 2017-08-01T14:27:05.200-07:00 info vpxd[17208] [Originator@6876 sub=vpxLro

opID=28e69da] [VpxLRO] -- BEGIN task-internal-5280 -- EventManager --

vim.event.EventManager.QueryEvent -- 5275dfb0-cc10-38d2-6a44-

04ea824193c0(52b4aeed-eacb-3aa1-2177-8efbd21e57eb)




bution



• CPU and Memory


• UI Performance




bution

DB Tasks

• Stats rollups, top-10 calculations

• Host syncs

• Stats queries

• Events/Tasks




bution

DB Notes and Best Practices

• Database is write-intensive

– Database log (dblog), Database (db), and stats (SEAT) are write-intensive

– Put DB partitions on SSD

• Slow API calls (e.g., stats queries, query events)

– Check DB Resources (CPU/Memory/Disk)

– Check DB space

• Use VC alarm to avoid running out of space

– Use VAMI to see DB utilization

– Reduce Event/Task retention time if possible

– Reduce stats lower stats level if possible

– Truncate tables (with GSS support) if necessary




bution

DB Monitoring: Appliance Management (Utilization View)

41

Transaction log, events, stats largest consumers of space




bution

DB Appliance Disk Usage from Command Line




bution

DB Space Utilization Alarm (New in 6.5)

43

Warning: 80%; Stop vpxd at 95%.




bution

Service Status (DB)

44

Make sure rollups are occurring




bution

DB Space Breakdown Events: UI View

45

PowerCLI: Get-VIEvent




bution

Events using SQL

46

Appliance instructions:1. ssh2. /opt/vmware/vpostgres/current/bin/psql –U postgres –d VCDB3. same queries as above

100M rows or more: may impact query performance




bution

Impact of Changing Stats Level on DB/NW Traffic

• Sample hardware setup

• Number of stats collected per stats level


Hardware VM 1 VM 2 ESX Host (6.0u2)

CPUs 2 2 48 (logical)

Local/Remote Disks 11 1 13

Datastores 1 1 9

NICs 1 1 3

Stats Level VM 1 VM 2 ESX Host

1 67 34 223

2 231 148 858

3 263 184 1779

4 348 196 1967

4x(storage/nw)



bution

Windows MS-SQL Performance Deep Dive: SQL Profiler


SQL Profiler shows high number of reads and high CPU UsageVMworld 2017 Content: Not fo


bution

DB Profiling in Postgres

• /opt/vmware/vpostgres/current/bin/pg_top –U postgres –d VCDB

• DECLARE CURSOR: usually stats query

– ‘Q’ and PID to see query

– ‘R’ to see stats (reads/writes/inserts)


autovacuum: does DB maintenance



bution

DB Performance: Log analysis

• Vpxd log diagnosis:

– “[VdbStatement] Execution elapsed time: …” and “SQL execution took too long”

– Occasional 3-4s OK (often for TOP_N, Rollups)

– Extremely frequent 3-4s: may want to check resources (IO, CPU, mem)

– 10s or more: usually BAD

• Slow historical queries?

– Purge DB tables; lower retention policy; lower stats level

– Recompute stats on DB tables

– DBLog, DB partition, and SEAT tables on robust storage (e.g., 3K IOPs)




bution



• CPU and Memory Investigation


• UI Performance




bution

UI Performance

• Ensure proper heap size

– Multiple vCenters with Enhanced Linked Mode

– Multiple clients

– Large Inventories

• Load Certificates into Browser

• Memory/CPU of machine running web browser

• Network connection from browser to vCenter

• Network between vCenter and extension server




bution

UI: Heap Issues

• CPU consistently high? Consider increased heap size

vimtop

53

cloudvm-ram-size -l

Changing heap size: cloudvm-ram-size –C <value in MB> <name of service>Windows Path: C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin




bution

Agenda








bution

Concluding Remarks

• vCenter Appliance meets or exceeds Windows vCenter performance

• For best performance, vCenter needs sufficient resources

– CPU: scales with inventory size and churn

– Memory: scales with inventory size

– IO: scales with inventory size, churn, and stats level

– Network: low-latency between VC and DB and between M/N nodes recommended

• vCenter Appliance has rich set of debugging tools




bution

Extreme Performance Series – Las Vegas

• SER2724BU Performance Best Practices

• SER2723BU Benchmarking 101

• SER2343BU vSphere Compute & Memory Schedulers

• SER1504BU vCenter Performance Deep Dive

• SER2734BU Byte Addressable Non-Volatile Memory in vSphere

• SER2849BU Predictive DRS – Performance & Best Practices

• SER1494BU Encrypted vMotion Architecture, Performance, & Futures

• STO1515BU vSAN Performance Troubleshooting

• VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks

• VIRT1397BU Optimize & Increase Performance Using VMware NSX

• VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX

• VIRT1052BU Monster VM Database Performance

• VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling

• VIRT1997BU Machine Learning and Deep Learning on VMware vSphere

• FUT2020BU Wringing Max Perf from vSphere for Extremely Demanding Workloads

• FUT2761BU Sharing High Performance Interconnects across Multiple VMs




bution

Extreme Performance Series - Hand on Labs

Don’t miss these popular Extreme Performance labs:

• HOL-1804-01-SDC: vSphere 6.5 Performance Diagnostics & Benchmarking

– Each module dives deep into vSphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools.

• HOL-1804-02-CHG: vSphere Challenge Lab

– Each module places you in a different fictional scenario to fix common vSphere operational and performance problems.




bution

Performance Survey


The VMware Performance Engineeringteam is always looking for feedback about your experience with theperformance of our products, ourvarious tools, interfaces and wherewe can improve.

Scan this QR code to access ashort survey and provide us directfeedback.

Alternatively: www.vmware.com/go/perf

Thank you!



bution



bution



bution