Vsphere Esxi Vcenter Server 60 Monitoring Performance Guide

vSphere Monitoring and PerformancevSphere 6.0

vCenter Server 6.0ESXi 6.0

This document supports the version of each product listed andsupports all subsequent versions until the document isreplaced by a new edition. To check for more recent editionsof this document, see http://www.vmware.com/support/pubs.

EN-001403-01

vSphere Monitoring and Performance

2 VMware, Inc.

You can find the most up-to-date technical documentation on the VMware Web site at:

http://www.vmware.com/support/The VMware Web site also provides the latest product updates.

If you have comments about this documentation, submit your feedback to:

[email protected]

Copyright 20102015 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc.3401 Hillview Ave.Palo Alto, CA 94304www.vmware.com

Contents

About vSphere Monitoring and Performance 5Updated Information 7

1 Monitoring Inventory Objects with Performance Charts 9Performance Chart Types 10Data Counters 10Metric Groups in vSphere 11Data Collection Intervals 12Data Collection Levels 13View Performance Charts 14Performance Charts Options Available Under the View Menu 14Overview Performance Charts 15Working with Advanced and Custom Charts 92Troubleshoot and Enhance Performance 94

2 Monitoring Guest Operating System Performance 101Enable Statistics Collection for Guest Operating System Performance Analysis 101View Performance Statistics for Windows Guest Operating Systems 101

3 Monitoring Host Health Status 103Monitor Hardware Health Status 104Reset Health Status Sensors 104

4 Monitoring Events, Alarms, and Automated Actions 105View Events 106View System Logs 107Export Events Data 107View Triggered Alarms and Alarm Definitions 107Set an Alarm 108Acknowledge Triggered Alarms 116Reset Triggered Event Alarms 116Preconfigured vSphere Alarms 117

5 Monitoring Solutions with the vCenter Solutions Manager 123View Solutions and vServices 123Monitoring Agents 124Monitoring vServices 124

6 Monitoring the Health of Services and Nodes 125View the Health Status of Services and Nodes 125

VMware, Inc. 3

7 Performance Monitoring Utilities: resxtop and esxtop 127Using the esxtop Utility 127Using the resxtop Utility 128Using esxtop or resxtop in Interactive Mode 128Using Batch Mode 142Using Replay Mode 143

8 Using the vimtop Plug-In to Monitor the Resource Usage of Services 145Monitor Services by Using vimtop in Interactive Mode 145Interactive Mode Command-Line Options 145Interactive Mode Single-Key Commands for vimtop 146

9 Monitoring Networked Devices with SNMP and vSphere 149Using SNMP Traps with vCenter Server 149Configure SNMP for ESXi 150SNMP Diagnostics 159Monitor Guest Operating Systems with SNMP 159VMware MIB Files 160SNMPv2 Diagnostic Counters 161

10 System Log Files 163View System Log Entries 163View System Logs on an ESXi Host 163External System Logs 164Export System Log Files 164Managing ESXi Log Files 165Configure Syslog on ESXi Hosts 166Configuring Logging Levels for the Guest Operating System 167Collecting Log Files 168Viewing Log Files with the Log Browser 170

Index 175


4 VMware, Inc.

About vSphere Monitoring and Performance

VMware provides several tools to help you monitor your virtual environment and to locate the source ofpotential issues and current problems.Performance charts Allow you to see performance data on a variety of system resources

including CPU, Memory, Storage, and so on.Performance monitoringcommand-line utilities

Allow you to access detailed information on system performance throughthe command line.

Host health Allows you to quickly identify which hosts are healthy and which areexperiencing problems.

Events, alerts, andalarms

Allow you to configure alerts and alarms and to specify the actions thesystem should take when they are triggered.

System Log Files System logs contain additional information about activities in your vSphereenvironment.

Intended AudienceThe content in this section is intended for vSphere administrators who perform the following tasks:n Monitor the health and performance of physical hardware backings for the virtual environment.n Monitor the health and performance of virtual devices in the virtual environment.n Troubleshoot problems in the system.n Configure alarms.n Configure SNMP messages.Virtual machine administrators also might find the section on Chapter 2, Monitoring Guest OperatingSystem Performance, on page 101 helpful.

VMware, Inc. 5


6 VMware, Inc.

Updated Information

This vSphere Monitoring and Performance documentation is updated with each release of the product or whennecessary.This table provides the update history of the vSphere Monitoring and Performance documentation.Revision Description001403-01 Removed information related to Storage Views and Storage Reports. This functionality is deprecated in

vSphere 6.0 release.001403-00 Initial release.

VMware, Inc. 7


8 VMware, Inc.

Monitoring Inventory Objects withPerformance Charts 1

The vSphere statistics subsystem collects data on the resource usage of inventory objects. Data on a widerange of metrics is collected at frequent intervals, processed, and archived in the vCenter Server database.You can access statistical information through command-line monitoring utilities or by viewingperformance charts in the vSphere Web Client.

Counters and Metric GroupsvCenter Server systems and hosts use data counters to query for statistics. A data counter is a unit ofinformation relevant to a given inventory object or device. Each counter collects data for a different statisticin a metric group. For example, the disk metric group includes separate data counters to collect data for diskread rate, disk write rate, and disk usage. Statistics for each counter are rolled up after a specified collectioninterval. Each data counter consists of several attributes that are used to determine the statistical valuecollected.For a complete list and description of performance metrics, see the vSphere API Reference.NOTE Counters that are introduced in later versions might not contain data from hosts of earlier versions.For details, see the VMware Knowledge Base.

Collection Levels and Collection IntervalsCollection levels determine the number of counters for which data is gathered during each collectioninterval. Collection intervals determine the time period during which statistics are aggregated, calculated,rolled up, and archived in the vCenter Server database. Together, the collection interval and collection leveldetermine how much statistical data is collected and stored in your vCenter Server database.

Data AvailabilityReal-time data appears in the performance charts only for hosts and virtual machines that are powered on.Historical data appears for all supported inventory objects, but might be unavailable during certaincircumstances.This chapter includes the following topics:n Performance Chart Types, on page 10n Data Counters, on page 10n Metric Groups in vSphere, on page 11n Data Collection Intervals, on page 12n Data Collection Levels, on page 13

VMware, Inc. 9

n View Performance Charts, on page 14n Performance Charts Options Available Under the View Menu, on page 14n Overview Performance Charts, on page 15n Working with Advanced and Custom Charts, on page 92n Troubleshoot and Enhance Performance, on page 94

Performance Chart TypesPerformance metrics are displayed in different types of charts, depending on the metric type and object.Table 11. Performance Chart TypesChart Type DescriptionLine chart Displays metrics for a single inventory object. The data for each performance counter is plotted on a

separate line in the chart. For example, a network chart for a host can contain two lines: one showingthe number of packets received, and one showing the number of packets transmitted.

Bar chart Displays storage metrics for datastores in a selected data center. Each datastore is represented as a barin the chart. Each bar displays metrics based on the file type: virtual disks, snapshots, swap files, andother files.

Pie chart Displays storage metrics for a single object, based on the file types or virtual machines. For example, apie chart for a datastore can display the amount of storage space occupied by the virtual machinestaking up the largest space.

Stacked chart Displays metrics for the child objects that have the highest statistical values. All other objects areaggregated, and the sum value is displayed with the term Other. For example, a host's stacked CPUusage chart displays CPU usage metrics for the ten virtual machines on the host that are consumingthe most CPU. The Other amount contains the total CPU usage of the remaining virtual machines.The metrics for the host itself are displayed in separate line charts.Stacked charts are useful in comparing resource allocation and usage across multiple hosts or virtualmachines. By default, the ten child objects with the highest data counter values are displayed.

Data CountersEach data counter includes several attributes that are used to determine the statistical value collected. Seethe vSphere API Reference for a complete list and description of supported counters.Table 12. Data Counter AttributesAttribute DescriptionUnit ofMeasurement

Standard in which the statistic quantity is measured.n Kilobytes (KB) 1024 bytesn Kilobytes per second (KBps) 1024 bytes per secondn Kilobits (kb) 1000 bitsn Kilobits per second (kbps) 1000 bits per secondn Megabytes (MB)n Megabytes per second (MBps)n Megabits (Mb), megabits per second (Mbps)n Megahertz (MHz)n Microseconds (s)n Milliseconds (ms)n Number (#)n Percent (%)n Seconds (s)

Description Text description of the data counter.


10 VMware, Inc.

Table 12. Data Counter Attributes (Continued)Attribute DescriptionStatistics Type Measurement used during the statistics interval. Related to the unit of measurement.

n Rate Value over the current statistics intervaln Delta Change from previous statistics interval.n Absolute Absolute value (independent of the statistics interval).

Rollup Type Calculation method used during the statistics interval to roll up data. Determines the type ofstatistical values that are returned for the counter.n Average Data collected during the interval is aggregated and averaged.n Minimum The minimum value is rolled up.n Maximum The maximum value is rolled up.

The Minimum and Maximum values are collected and displayed only in statistics level 4.Minimum and maximum rollup types are used to capture peaks in data during the interval.For real-time data, the value is the current minimum or current maximum. For historical data,the value is the average minimum or average maximum.For example, the following information for the CPU usage chart shows that the average iscollected at statistics level 1 and the minimum and maximum values are collected at statisticslevel 4.n Counter: usagen Unit: Percentage (%)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

n Summation Data collected is summed. The measurement displayed in the chart representsthe sum of data collected during the interval.

n Latest Data collected during the interval is a set value. The value displayed in theperformance charts represents the current value.

Collection level Number of data counters used to collect statistics. Collection levels range from 1 to 4, with 4having the most counters.NOTE Be careful when you set a higher collection level, as the process requires significant increaseof resource usage. For more information, see Data Collection Levels, on page 13.

Metric Groups in vSphereThe performance data collection subsystem for vSphere collects performance data on a variety of inventoryitems and their devices. Data counters define individual performance metrics. Performance metrics areorganized into logical groups based on the object or object device. Statistics for one or more metrics can bedisplayed in a chart.Table 13. Metric GroupsMetric group DescriptionClusterServices

Performance statistics for clusters configured by using vSphere Distributed Resource Scheduler,vSphere High Availability, or both.

CPU CPU utilization per host, virtual machine, resource pool, or compute resource.Datastore Statistics for datastore utilization.Disk Disk utilization per host, virtual machine, or datastore. Disk metrics include I/O performance, such as

latency and read/write speeds, and utilization metrics for storage as a finite resource.Memory Memory utilization per host, virtual machine, resource pool, or compute resource. The value obtained

is one of the following:n For virtual machines, memory refers to the guest physical memory. Guest physical memory is the

amount of physical memory presented as a virtual-hardware component to the virtual machine, atcreation time, and made available when the virtual machine is running.

n For hosts, memory refers to the machine memory. Machine memory is the RAM that is installed onthe hardware that comprises the host.

Chapter 1 Monitoring Inventory Objects with Performance Charts

VMware, Inc. 11

Table 13. Metric Groups (Continued)Metric group DescriptionNetwork Network utilization for both physical and virtual network interface controllers (NICs) and other

network devices, such as the virtual switches that support connectivity among all components, such ashosts, virtual machines, VMkernel, and so on.

Power Energy usage statistics per host.StorageAdapter

Data traffic statistics per host bus adapter (HBA).

Storage Path Data traffic statistics per path.System Overall system availability, such as system heartbeat and uptime. These counters are available directly

from hosts and from vCenter Server.Virtual Disk Disk utilization and disk performance metrics for virtual machines.Virtual Flash Virtual flash counters.VirtualMachineOperations

Virtual machine power and provisioning operations in a cluster or data center.

vSphereReplication

Statistics for virtual machine replication performed by VMware vCenter Site Recovery Manager.

Data Collection IntervalsCollection intervals determine the duration for which statistics are aggregated, calculated, rolled up, andarchived. Together, the collection interval and collection level determine how much statistical data isgathered and stored in your vCenter Server database.Table 14. Collection IntervalsCollectionInterval/ArchiveLength

CollectionFrequency Default Behavior

1 Day 5 Minutes Real-time statistics are rolled up to create one data point every 5 minutes.The result is 12 data points every hour and 288 data points every day. After30 minutes, the six data points collected are aggregated and rolled up as adata point for the 1 Week time range.You can change the interval duration and archive length of the 1 Daycollection interval by configuring the statistics settings.

1 Week 30 Minutes 1 Day statistics are rolled up to create one data point every 30 minutes. Theresult is 48 data points every day and 336 data points every week. Every 2hours, the 12 data points collected are aggregated and rolled up as a datapoint for the 1 Month time range.You cannot change the default settings of the 1 Week collection interval.

1 Month 2 Hours 1 Week statistics are rolled up to create one data point every 2 hours. Theresult is 12 data points every day and 360 data points every month(assuming a 30-day month). After 24 hours, the 12 data points collected areaggregated and rolled up as a data point for the 1 Year time range.You cannot change the default settings of the 1 Month collection interval.

1 Year 1 Day 1 Month statistics are rolled up to create one data point every day. Theresult is 365 data points each year.You can change the archive length of the 1 Year collection interval byconfiguring the statistics settings.

NOTE If you change the duration of data collection intervals you might need to allocate more storageresources.


12 VMware, Inc.

Data Collection LevelsEach collection interval has a default collection level that determines the amount of data gathered and whichcounters are available for display in charts. Collection levels are also referred to as statistics levels.Table 15. Statistics LevelsLevel Metrics Best PracticeLevel 1 n Cluster Services (VMware Distributed Resource Scheduler)

all metricsn CPU cpuentitlement, totalmhz, usage (average), usagemhzn Disk capacity, maxTotalLatency, provisioned, unshared,

usage (average), usedn Memory consumed, mementitlement, overhead,

swapinRate, swapoutRate, swapused, totalmb, usage(average), vmmemctl (balloon)

n Network usage (average), IPv6n System heartbeat, uptimen Virtual Machine Operations numChangeDS,

numChangeHost, numChangeHostDS

Use for long-term performancemonitoring when device statistics arenot required.Level 1 is the default Collection Levelfor all Collection Intervals.

Level 2 n Level 1 metricsn CPU idle, reservedCapacityn Disk All metrics, excluding numberRead and

numberWrite.n Memory All metrics, excluding memUsed and maximum

and minimum rollup values.n Virtual Machine Operations All metrics

Use for long-term performancemonitoring when device statistics arenot required but you want to monitormore than the basic statistics.

Level 3 n Level 1 and Level 2 metricsn Metrics for all counters, excluding minimum and maximum

rollup values.n Device metrics

Use for short-term performancemonitoring after encounteringproblems or when device statistics arerequired.Because of the large quantity oftroubleshooting data retrieved andrecorded, use level 3 for the shortesttime period (Day or Week collectioninterval).

Level 4 All metrics supported by the vCenter Server, includingminimum and maximum rollup values.

Use for short-term performancemonitoring after encounteringproblems or when device statistics arerequired.Because of the large quantity oftroubleshooting data retrieved andrecorded, use level 4 for the shortestamount of time.

NOTE When you increase the collection level the storage and system requirements might change. Youmight need to allocate more system resources to avoid decrease in the performance.


VMware, Inc. 13

View Performance ChartsThe vCenter Server statistics settings, the type of object selected, and the features that are enabled on theselected object determine the amount of information displayed in charts. Charts are organized into views.You can select a view to see related data together on one screen. You can also specify the time range, or datacollection interval. The duration extends from the selected time range to the present time.Overview charts display multiple data sets in one panel to easily evaluate different resource statistics,display thumbnail charts for child objects, and display charts for a parent and a child object. Advancedcharts display more information than overview charts, are configurable, and can be printed or exported.You can export data in the PNG, JPEG, or CSV formats. See View Advanced Performance Charts, onpage 92.Procedure1 Select a valid inventory object in the vSphere Web Client.

Overview and advanced performance charts are available for datacenter, cluster, host, resource pool,vApp, and virtual machine objects. Overview charts are also available for datastores and datastoreclusters. Performance charts are not available for network objects.

2 Click the Monitor tab, and click Performance.3 Select a view.

Available views depend on the type of object. For views that might contain a large number of charts ina large environment, the vSphere Web Client displays the charts distributed on multiple pages. You canuse the arrow buttons to navigate between pages.

4 Select a predefined or custom time range.

Performance Charts Options Available Under the View MenuThe performance chart options that you can access under the View menu vary depending on the type ofinventory object you select.For example, the Virtual Machines view is available when you view host performance charts only if thereare virtual machines on the selected host. Likewise, the Fault Tolerance view for virtual machineperformance charts is available only when that feature is enabled for the selected virtual machine.Table 16. Performance Chart Views by Inventory ObjectObject View List ItemsData center n Storage - space utilization charts for datastores in the data center, including space by file type and

storage space used by each datastore in the data center.n Clusters - thumbnail CPU and memory charts for each cluster, and stacked charts for total CPU

and memory usage in the data center. This view is the default.Datastore anddatastorecluster

n Space - space utilization charts for the datastore:n space utilization by file typen space utilization by virtual machinen space usage

n Performance - performance charts for the datastore or datastore cluster and for virtual machinedisks on the resource.

NOTE The Performance view for datastores is only available when all hosts that are connected to thedatastores are ESX/ESXi 4.1 or greater. The Performance view for datastore clusters is only availablewhen the Storage DRS is enabled.


14 VMware, Inc.

Table 16. Performance Chart Views by Inventory Object (Continued)Object View List ItemsCluster n Home - CPU and memory charts for the cluster.

n Resource Pools & Virtual Machines - thumbnail charts for resource pools and virtual machines,and stacked charts for total CPU and memory usage in the cluster.

n Hosts - thumbnail charts for each host in the cluster, and stacked charts for total CPU, memory,disk usage, and network usage.

Host n Home - CPU, memory, disk, and network charts for the host.n Virtual Machines - thumbnail charts for virtual machines, and stacked charts for total CPU usage

and total memory usage on the host.Resource Pooland vApps

n Home - CPU and memory charts for the resource pool.n Resource Pools & Virtual Machines - thumbnail charts for resource pools, and virtual machines

and stacked charts for CPU and memory usage in the resource pool or vApp.VirtualMachine

n Storage - space utilization charts for the virtual machine: space by file type, space by datastore, andtotal gigabytes.

n Fault Tolerance - CPU and memory charts that display comparative metrics for the fault-tolerantprimary and secondary virtual machines.

n Home - CPU, memory, network, host (thumbnail charts), and disk usage charts for the virtualmachine.

Overview Performance ChartsThe overview performance charts display the most common metrics for an object in the inventory. Use thesecharts to monitor and troubleshoot performance problems.The metrics provided in Overview performance charts are a subset of those collected for hosts and thevCenter Server. For a complete list of all metrics collected by hosts and the vCenter Server, see the vSphereAPI Reference.

ClustersThe cluster charts contain information about CPU, disk, memory, and network usage for clusters. The helptopic for each chart contains information about the data counters displayed in that chart. The countersavailable are determined by the collection level set for vCenter Server.

CPU (MHz)The CPU (MHz) chart displays CPU usage for the cluster.Cluster CountersThis chart is located in the Home view of the Cluster Performance tab.


VMware, Inc. 15

Table 17. Data CountersChart Label DescriptionUsage Sum of the average CPU usage values, in Megahertz, of all virtual machines in the cluster.

n Counter: usagemhzn Stats Type: Raten Unit: Megahertz (MHz)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Total Total amount of CPU resources available in the cluster. The maximum value is equal to thenumber of cores multiplied by the frequency of the processors.For example, a cluster has two hosts, each of which has four CPUs that are 3GHz each, and onevirtual machine that has two virtual CPUs.VM totalmhz = 2 vCPUs * 3000MHz = 6000MHzHost totalmhz = 4 CPUs * 3000MHz = 12000MHzCluster totalmhz = 2 x 4 * 3000MHz = 24000MHzn Counter: totalmhzn Stats Type: Raten Unit: Megahertz (MHz)n Rollup Type: Summationn Collection Level: 1

Chart AnalysisA short spike in CPU usage indicates that you are making the best use of cluster resources. However, if thevalue is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high CPUusage value can lead to increased ready time and processor queuing of the virtual machines on the hosts inthe cluster.If performance is impacted, consider taking the following actions.Table 18. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 If the cluster is not a DRS cluster, enable DRS. Select the cluster, and click Manage > Settings > vSphere DRS >

Edit > Turn ON vSphere DRS.3 If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.n Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots

in the cluster.4 Migrate one or more virtual machines to a new host.5 Upgrade the physical CPUs or cores on each host in the cluster if necessary.6 Enable CPU-saving features, such as TCP Segmentation Offload.7 Replace software I/O with dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

CPU UsageThe cluster CPU Usage charts monitors the CPU utilization of the hosts, resource pools, and virtualmachines in the cluster. This chart displays the 10 child objects in the cluster with the most CPU usage.This chart is located in the Resource Pools and Virtual Machines view of the Cluster Performance tab.


16 VMware, Inc.

Table 19. Data CountersChart Label Description, , or

Amount of CPU actively used by the host, resource pool, or virtual machine in thecluster.n Counter: usagemhzn Stats Type: Raten Unit: MegaHertz (MHz)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA short spike in CPU usage indicates that you are making the best use of cluster resources. However, if thevalue is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high CPUusage value can lead to increased ready time and processor queuing of the virtual machines on the hosts inthe cluster.If performance is impacted, consider taking the following actions.Table 110. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 If the cluster is not a DRS cluster, enable DRS. Select the cluster, and click Manage > Settings > vSphere DRS >




Disk (KBps)The Disk (KBps) chart displays the disk I/O of the 10 hosts in the cluster with the most disk usage.This chart is located in the Hosts view of the cluster Performance tab.Table 111. Data CountersChart Label Descriptionhost_name Average data I/O rate across all hosts in the cluster.

n Counter: usagen Stats Type: Raten Unit: Kilobytes per second (KBps)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)


VMware, Inc. 17

Chart AnalysisUse the disk charts to monitor average disk loads and to determine trends in disk usage. For example, youmight notice a performance degradation with applications that frequently read from and write to the harddisk. If you see a spike in the number of disk read or write requests, check whether any such applicationswere running at that time.The best ways to determine if your vSphere environment is experiencing disk problems is to monitor thedisk latency data counters. You can use the advanced performance charts to view these statistics.n The kernelLatency data counter measures the average amount of time, in milliseconds, that the

VMkernel spends processing each SCSI command. For best performance, the value should be 0-1milliseconds. If the value is greater than 4ms, the virtual machines on the host are trying to send morethroughput to the storage system than the configuration supports. Check the CPU usage, and increasethe queue depth.

n The deviceLatency data counter measures the average amount of time, in milliseconds, to complete aSCSI command from the physical device. Depending on your hardware, a number greater than 15msindicates probable problems with the storage array. Move the active VMDK to a volume with morespindles or add disks to the LUN.

n The queueLatency data counter measures the average amount of time taken per SCSI command in theVMkernel queue. This value must always be zero. If not, the workload is too high and the array cannotprocess the data fast enough.

If the disk latency values are high, or if you notice other problems with disk I/O performance, considertaking the actions listed below.Table 112. Disk I/O Performance Enhancement Advice# Resolution1 Increase the virtual machine memory. This should allow for more operating system caching, which can reduce I/O

activity. Note that this might require you to also increase the host memory. Increasing memory might reduce theneed to store data because databases can utilize system memory to cache data and avoid disk access.To verify that virtual machines have adequate memory, check swap statistics in the guest operating system.Increase the guest memory, but not to an extent that leads to excessive host memory swapping. Install VMwareTools so that memory ballooning can occur.

2 Defragment the file systems on all guests.3 Disable antivirus on-demand scans on the VMDK and VMEM files.4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously

access common elements on an array, the disks might have trouble keeping up. Consider array-side improvementsto increase throughput.

5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.6 Balance the disk load across all physical resources available. Spread heavily used storage across LUNs that are

accessed by different adapters. Use separate queues for each adapter to improve disk efficiency.7 Configure the HBAs and RAID controllers for optimal use. Verify that the queue depths and cache settings on the

RAID controllers are adequate. If not, increase the number of outstanding disk requests for the virtual machine byadjusting the Disk.SchedNumReqOutstanding parameter. For more information, see vSphere Storage.

8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with thesystem page file. This alleviates disk spindle contention during periods of high use.

9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtualmachine's VMX file.

10 If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.


18 VMware, Inc.

Table 112. Disk I/O Performance Enhancement Advice (Continued)# Resolution11 For ESXi hosts, create virtual disks as preallocated. When you create a virtual disk for a guest operating system,

select Allocate all disk space now. The performance degradation associated with reassigning additional disk spacedoes not occur, and the disk is less likely to become fragmented.

12 Use the most current host hardware.

Memory (MB)The Memory (MB) chart displays consumed memory for the cluster. The chart appears only at collectionlevel 1.This chart is located in the Home view of the cluster Performance tab.Table 113. Data CountersChart Label DescriptionConsumed Amount of host machine memory used by all powered on virtual machines in the cluster. A

cluster's consumed memory consists of virtual machine consumed memory and overheadmemory. It does not include host-specific overhead memory, such as memory used by theservice console or VMkernel.n Counter: consumedn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Total Total amount of machine memory of all hosts in the cluster that is available for virtual machinememory (physical memory for use by the Guest OS) and virtual machine overhead memory.Memory Total = Aggregate host machine memory - (VMkernel memory + Service Consolememory + other service memory)NOTE The totalmb data counter is the same as the effectivemem data counter, which issupported only for backward compatibility.n Counter: totalmbn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisMemory usage is not an indicator of performance problems. Memory can be high if a host is swapping orballooning, which can result in virtual machine guest swapping. In such cases, check for other problems,such as CPU over-commitment or storage latencies.If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking thefollowing actions.Table 114. Memory Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to

performance. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping.Generally, this does not impact virtual machine performance.


VMware, Inc. 19

Table 114. Memory Performance Enhancement Advice (Continued)# Resolution3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource

pools on the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine.If free memory is available on the hosts and the virtual machines are experiencing high swap or balloon memory,the virtual machine (or resource pool, if it belongs to one) has reached its resource limit. Check the maximumresource limit set on that host.

4 If the cluster is not a DRS cluster, enable DRS. Select the cluster in the object navigator, and click Manage > Settings> vSphere DRS > Edit > Turn ON vSphere DRS.

5 If the cluster is a DRS cluster:n Increase the number of hosts, and migrate one or more virtual machines to the new host.n Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots

in the cluster.6 Add more physical memory to one or more hosts.

Memory (MB)The Memory (MB) chart displays memory data counters for clusters. The chart appears at all collectionlevels except level 1.DescriptionThis chart is located in the Home view of the cluster Performance tab.NOTE These data counter definitions are for hosts. At the cluster level, the values are collected and totaled.The counter values in the chart represent the aggregate amounts of the host data. The counters that appearin the chart depend on the collection level set for your vCenter Server.Table 115. Data Counters Chart Label DescriptionActive Sum of the active guest physical memory of all powered on virtual machines on the

host, plus memory used by basic VMkernel applications. Active memory isestimated by the VMkernel.n Counter: activen Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 2 (4)

Balloon Sum of the guest physical memory reclaimed by the balloon driver for all poweredon virtual machines on the host.n Counter: vmmemctln Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Consumed Amount of machine memory used on the host.Consumed memory includes virtual machine memory, service console memory, andVMkernel memory.consumed memory = total host memory - free host memoryn Counter: consumedn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)


20 VMware, Inc.

Table 115. Data Counters (Continued)Chart Label DescriptionGranted Sum of the guest physical memory granted for all powered on virtual machines.

Granted memory is mapped to the host's machine memory.Granted memory for a host includes the shared memory of each virtual machine onthe host.n Counter: grantedn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 2 (4)

Swap Used Sum of the memory swapped by all powered on virtual machines on the host.n Counter: swapusedn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 2 (4)

Total Aggregate total memory available to the cluster.n Counter: totalmbn Stats Type: Absoluten Unit: Megabytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisTo ensure best performance, the host memory must be large enough to accommodate the active memory ofthe virtual machines. The active memory can be smaller than the virtual machine memory size. This allowsyou to over-provision memory, but still ensures that the virtual machine active memory is smaller than thehost memory.Transient high-usage values usually do not cause performance degradation. For example, memory usagecan be high when several virtual machines are started at the same time or when a spike occurs in virtualmachine workload. However, a consistently high memory usage value (94% or greater) indicates that thehost is probably lacking the memory required to meet the demand. If the active memory size is the same asthe granted memory size, the demand for memory is greater than the memory resources available. If theactive memory is consistently low, the memory size might be too large.If the memory usage value is high, and the host has high ballooning or swapping, check the amount of freephysical memory on the host. A free memory value of 6% or less indicates that the host cannot handle thedemand for memory. This leads to memory reclamation, which might degrade performance.If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtualmachines and resource pools on the host. Verify that the host settings are adequate and not lower than thoseset for the virtual machines.If the host has little free memory available, or if you notice a degradation in performance, consider takingthe following actions.Table 116. Memory Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools

and is critical to performance.2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by

ballooning and swapping. Generally, this does not impact virtual machine performance.


VMware, Inc. 21

Table 116. Memory Performance Enhancement Advice (Continued)# Resolution3 Reduce the memory space on the virtual machine, and correct the cache size if it is too large. This frees up memory

for other virtual machines.4 If the memory reservation of the virtual machine is set to a value much higher than its active memory, decrease the

reservation setting so that the VMkernel can reclaim the idle memory for other virtual machines on the host.5 Migrate one or more virtual machines to a host in a DRS cluster.6 Add physical memory to the host.

Memory ConsumedThe Memory Consumed chart displays memory usage for the 10 child objects in the cluster with the mostconsumed memory.For resource pools and virtual machines in a cluster, this chart is located in the Resource Pools & VirtualMachines view of the cluster Performance tab. For hosts in a cluster, this chart is located in the Hosts viewof the cluster Performance tab.Table 117. Data CountersChart Label Descriptionresource_pool,virtual_machine, orhost

Amount of machine memory used by all resource pools and virtual machines in the cluster or byall hosts in the cluster, depending on the cluster view.Consumed memory includes virtual machine memory, service console memory, and VMkernelmemory.consumed memory = total host memory - free host memoryn Counter: consumedn Stats Type: Absoluten Unit: MegaBytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisMemory usage is not an indicator of performance problems. Memory can be high if a host is swapping orballooning, which can result in virtual machine guest swapping. In such cases, check for other problems,such as CPU over-commitment or storage latencies.If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking thefollowing actions.Table 118. Memory Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to

performance. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping.Generally, this does not impact virtual machine performance.

3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resourcepools on the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine.If free memory is available on the hosts and the virtual machines are experiencing high swap or balloon memory,the virtual machine (or resource pool, if it belongs to one) has reached its resource limit. Check the maximumresource limit set on that host.

4 If the cluster is not a DRS cluster, enable DRS. Select the cluster in the object navigator, and click Manage > Settings> vSphere DRS > Edit > Turn ON vSphere DRS.


22 VMware, Inc.

Table 118. Memory Performance Enhancement Advice (Continued)# Resolution5 If the cluster is a DRS cluster:


in the cluster.6 Add more physical memory to one or more hosts.

Network (Mbps)The Network (Mbps) chart displays network speed for the 10 hosts in the cluster with the most networkusage.This chart is located in the Hosts view of the Cluster Performance tab.Table 119. Data CountersChart Label Description Average rate at which data is transmitted and received across all NIC instances on the host.

n Counter: usagen Stats Type: Raten Unit: Megabits per second (Mbps)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisNetwork performance depends on application workload and network configuration. Dropped networkpackets indicate a bottleneck in the network. To determine whether packets are being dropped, use esxtopor the advanced performance charts to examine the droppedTx and droppedRx network counter values.If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check thesize of the network packets and the data receive and transfer rates. In general, the larger the networkpackets, the faster the network speed. When the packet size is large, fewer packets are transferred, whichreduces the amount of CPU required to process the data. When network packets are small, more packets aretransferred but the network speed is slower because more CPU is required to process the data.NOTE In some instances, large packets might result in high network latency. To check network latency, usethe VMware AppSpeed performance monitoring application or a third-party application.If packets are not being dropped and the data receive rate is slow, the host is probably lacking the CPUresources required to handle the load. Check the number of virtual machines assigned to each physical NIC.If necessary, perform load balancing by moving virtual machines to different vSwitches or by adding moreNICs to the host. You can also move virtual machines to another host or increase the host CPU or virtualmachine CPU.If you experience network-related performance problems, also consider taking the actions listed below.Table 120. Networking Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high

performance.3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to

avoid transferring packets over the physical network.


VMware, Inc. 23

Table 120. Networking Performance Enhancement Advice (Continued)# Resolution4 Assign each physical NIC to a port group and a vSwitch.5 Use separate physical NICs to handle the different traffic streams, such as network packets generated by virtual

machines, iSCSI protocols, vMotion tasks.6 Ensure that the physical NIC capacity is large enough to handle the network traffic on that vSwitch. If the capacity

is not enough, consider using a high-bandwidth physical NIC (10Gbps) or moving some virtual machines to avSwitch with a lighter load or to a new vSwitch.

7 If packets are being dropped at the vSwitch port, increase the virtual network driver ring buffers where applicable.8 Verify that the reported speed and duplex settings for the physical NIC match the hardware expectations and that

the hardware is configured to run at its maximum capability. For example, verify that NICs with 1Gbps are notreset to 100Mbps because they are connected to an older switch.

9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NICresetting itself to a lower speed or half duplex mode.

10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabledwhere possible.

DatacentersThe data center charts contain information about CPU, disk, memory, and storage usage for data centers.The help topic for each chart contains information about the data counters displayed in that chart. Thecounters available are determined by the collection level set for vCenter Server.

CPU (MHz)The CPU (MHz) chart displays CPU usage for the 10 clusters in the data center with the most CPU usage.This chart is located in the Clusters view of the Datacenters Performance tab.Table 121. Data CountersChart Label Description Amount of CPU currently in use by the cluster. The active CPU usage is approximately equal to the

ratio of the used CPU cycles to the available CPU cycles.The maximum possible value is the frequency of the processors multiplied by the number of cores.For example, a two-way SMP virtual machine using 4000MHz on a host that has four 2GHzprocessors is using 50% of the CPU (4000 4 2000) = 0.5).n Counter: usagemhzn Stats Type: Raten Unit: MegaHertz (MHz)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA short spike in CPU usage indicates that you are making the best use of cluster resources. However, if thevalue is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high CPUusage value can lead to increased ready time and processor queuing of the virtual machines on the hosts inthe cluster.If performance is impacted, consider taking the following actions.


24 VMware, Inc.

Table 122. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on each virtual machine.2 If the cluster is not a DRS cluster, enable DRS. Select the cluster, and click Manage > Settings > vSphere DRS >




Memory (MB)The Memory (MB) chart displays the average amount of consumed memory for the 10 clusters in the datacenter with the most consumed memory.This chart is located in the Clusters view of the Datacenters Performance tab.Table 123. Data CountersChart Label Description Amount of host machine memory used by all powered on virtual machines in the

cluster.n Counter: consumedn Stats Type: Absoluten Unit: MegaBytes (MB)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA cluster's consumed memory consists of virtual machine consumed memory and overhead memory. Itdoes not include host-specific overhead memory, such as memory used by the service console or VMkernel.If you experience problems with cluster memory usage, use the thumbnail cluster charts to examinememory usage for each cluster and increase memory resources if needed.If the cluster is a DRS cluster, check the aggressiveness threshold. If the value is low, increase the threshold.Increasing the threshold might help avoid hot spots in the cluster.

Space used in GBThe Space in GB chart displays the 10 datastores in the data center with the most used disk space.This chart is located in the Storage view of the Datacenter Performance tab.


VMware, Inc. 25

Table 124. Data CountersChart Label Description Amount of used storage space on the 10 datastores with the most used space.

n Counter: usedn Stats Type: Absoluten Unit: GigaBytes (GB)n Rollup Type: Latestn Collection Level: 1

Chart AnalysisThe datastore is at full capacity when the used space is equal to the capacity. Allocated space can be largerthan datastore capacity, for example, when you have snapshots and thin-provisioned disks. You canprovision more space to the datastore if possible, or you can add disks to the datastore or use shareddatastores.If snapshot files are consuming a lot of datastore space, consider consolidating them to the virtual disk whenthey are no longer needed. Consolidating the snapshots deletes the redo log files and removes the snapshotsfrom the vSphere Web Client user interface. For information about consolidating the data center, see thevSphere documentation.

Space Utilization By File TypeThe Space Utilization By File Type chart displays datastore space usage for virtual disks, swap files,snapshot files, and other virtual machine files.NOTE This chart does not show historical statistics. It only shows the most recently available data, whichmay be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statisticsare not collected across all datastores at one time. They are collected asynchronously.The Space Utilization by File Type chart is located in the Storage view of the data center Performance tab.Datastore CountersTable 125. Data CountersFile Type DescriptionVirtual Disks Amount of disk space used by virtual disk files.

Virtual disk files store the contents of the virtual machine's hard disk drive, including informationthat you write to a virtual machine's hard disk, such as the operating system, program files, anddata files. The files have the extension .vmdk and appear as a physical disk drive to a guestoperating system.NOTE Delta disks, which also have an extension .vmdk, are not included in this file type.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Swap Files Amount of disk space used by swap files.Swap files back up the virtual machine's physical memory.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)


26 VMware, Inc.

Table 125. Data Counters (Continued)File Type DescriptionSnapshots Amount of disk space used by virtual machine snapshot files.

Snapshot files store information about virtual machine snapshots. They include snapshot statefiles and delta disk files. A snapshot state file stores the running state of the virtual machine at thetime of the snapshot. It has the extension .vmsn. A delta disk file stores the updates made by thevirtual machine to the virtual disks after a snapshot is taken.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Other VM Files Amount of disk space used by all other virtual machine files, such as configuration files and logfiles.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Other Amount of disk space used by all other non-virtual machine files, such as documentation files andbackup files.

Free Space Amount of disk space not currently in use.Total Space Amount of disk space available to the datastore. This is the datastore capacity. The chart displays

the information for datastores but not for data centers.total space = virtual disk space + swap file space + snapshot space + other VM file space + otherspace + free space


DatastoresThe datastore charts contain information about disk usage for datastores. The help topic for each chartcontains information about the data counters displayed in that chart. The counters available are determinedby the collection level set for vCenter Server.

Space in GBThe Space in GB chart displays space usage data counters for datastores.This chart is located in the Space view of the datastore Performance tab.


VMware, Inc. 27

Table 126. Data CountersChart Label DescriptionAllocated Amount of physical space provisioned by an administrator for the datastore. It is the storage size

up to which files on the datastore can grow. Allocated space is not always in use.n Counter: provisionedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1

Used Amount of physical datastore space in use.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1

Capacity Maximum capacity of the datastore.capacity = virtual machine file space + non-virtual machine file space + free spaceNOTE Storage data is collected and updated in the overview charts every 30 minutes. Therefore,if you refresh the datastore, the capacity value might only be updated in the datastore Summarytab, and not in the overview charts.n Counter: capacityn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1


Space Utilization By File TypeThe Space Utilization by File Type chart displays space used by virtual disks, swap files, snapshot files, andother virtual machine files on the the datastore.NOTE This chart does not show historical statistics. It only shows the most recently available data, whichmay be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statisticsare not collected across all datastores at one time. They are collected asynchronously.The Space Utilization by File Type chart is located in the Space view of the datastore Performance tab.


28 VMware, Inc.

Datastore CountersTable 127. Data CountersFile Type DescriptionVirtual Disks Amount of disk space used by virtual disk files.

Virtual disk files store the contents of the virtual machine's hard disk drive, including informationthat you write to a virtual machine's hard disk, such as the operating system, program files, anddata files. The files have the extension .vmdk and appear as a physical disk drive to a guestoperating system.NOTE Delta disks, which also have an extension .vmdk, are not included in this file type.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Swap Files Amount of disk space used by swap files.Swap files back up the virtual machine's physical memory.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Snapshots Amount of disk space used by virtual machine snapshot files.Snapshot files store information about virtual machine snapshots. They include snapshot statefiles and delta disk files. A snapshot state file stores the running state of the virtual machine at thetime of the snapshot. It has the extension .vmsn. A delta disk file stores the updates made by thevirtual machine to the virtual disks after a snapshot is taken.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Other VM Files Amount of disk space used by all other virtual machine files, such as configuration files and logfiles.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1 (4)

Other Amount of disk space used by all other non-virtual machine files, such as documentation files andbackup files.

Free Space Amount of disk space not currently in use.Total Space Amount of disk space available to the datastore. This is the datastore capacity. The chart displays

the information for datastores but not for data centers.total space = virtual disk space + swap file space + snapshot space + other VM file space + otherspace + free space

Chart AnalysisThe datastore is at full capacity when the used space is equal to the capacity. Allocated space can be largerthan datastore capacity, for example, when you have snapshots and thin-provisioned disks. You canprovision more space to the datastore if possible, or you can add disks to the datastore or use shareddatastores.


VMware, Inc. 29

If snapshot files are consuming a lot of datastore space, consider consolidating them to the virtual disk whenthey are no longer needed. Consolidating the snapshots deletes the redo log files and removes the snapshotsfrom the vSphere Web Client user interface. For information about consolidating the data center, see thevSphere documentation.

Space Utilization by Virtual MachineThe Space Utilization by Virtual Machine chart displays the amount of space used by the five virtualmachines with the most space used on the datastore.NOTE This chart does not show historical statistics. It only shows the most recently available data, whichmay be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statisticsare not collected across all datastores at one time. They are collected asynchronously.The Space Utilization by Virtual Machine chart is located in the Space view of the datastore Performancetab.Table 128. Data CountersChart Label Descriptionvirtual_machine Amount of datastore space used by the five virtual machines with the most used

datastore space.n Counter: usedn Stats Type: Absoluten Unit: Gigabytes (GB)n Rollup Type: Latestn Collection Level: 1


Storage I/O Control Normalized LatencyThis chart displays the normalized latency in microseconds on the datastore, which is the latency that ismonitored by Storage I/O Control to detect congestion on the datastore. This metric computes a weightedresponse time across all hosts and VMs accessing the datastore. I/O count is used as the weight for theresponse time. It captures the device level latency and does not include any queuing inside the hypervisorstorage stack or inside the VM. It is adjusted for I/O size, meaning that high latencies that are the result ofvery large I/Os are discounted so as not to make the datastore seem slower than it really is. Data for allvirtual machines is combined. This chart will display zero values when Storage I/O Control is disabled.This chart is located in the Performance view of the datastore Performance tab. ThesizeNormalizedDatastoreLatency counter can also be displayed for datastore cluster charts.


30 VMware, Inc.

Table 129. Data CountersChart Label DescriptionStorage I/O ControlNormalized Latency

This is the latency that is monitored by Storage I/O Control to detect congestion on thedatastore.n Counter: sizeNormalizedDatastoreLatencyn Stats Type: Absoluten Unit: Microsecondsn Rollup Type: Averagen Collection Level: 3

Storage I/O Control Aggregate IOPsThis chart displays the number of I/O operations per second on the datastore, aggregated across all hostsand virtual machines accessing the datastore. The chart displays zero values when Storage I/O Control isdisabled.This chart is located in the Performance view of the datastore Performance tab. The datastoreIops countercan also be displayed for datastore cluster charts.Table 130. Data CountersChart Label DescriptionStorage I/O Control AggregateIOPs

Number of I/O operations per second on the datastore, aggregated across all hostsand virtual machines accessing the datastore.n Counter: datastoreIopsn Stats Type: Absoluten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Storage I/O Control ActivityThis chart displays the percentage of time during which Storage I/O Control actively controlled thedatastore latency.This chart is located in the Performance views of the datastore and datastore cluster Performance tabs.Table 131. Data CountersChart Label DescriptionStorage I/O ControlActivity

This is the percentage of time during which the Storage I/O Control actively controlled the I/Olatency for the datastore.n Counter: siocActiveTimePercentagen Stats Type: Absoluten Unit: Percentn Rollup Type: Averagen Collection Level: 3

Average Device Latency per HostThis chart displays the average amount of latency on a host device. The chart displays the ten hosts with thehighest device latency.This chart is located in the Performance view of the datastore Performance tab.


VMware, Inc. 31

Table 132. Data CountersChart Label DescriptionAverage DeviceLatency per Host

Measures the amount of time, in milliseconds, to complete a SCSI command issued from thephysical device.n Counter: deviceLatencyn Stats Type: Absoluten Unit: Milliseconds (ms)n Rollup Type: Averagen Collection Level: 3

Maximum Queue Depth per HostThis chart displays maximum queue depth that hosts are currently maintaining for the datastore. WhenStorage I/O is enabled, queue depth can change over time when congestion is detected at the array.This chart is located in the Performance view of the datastore Performance tab. The chart displaysinformation about the ten hosts with the highest values.Table 133. Data CountersChart Label DescriptionMax Queue Depth per Host Maximum queue depth. Queue depth is the number of commands the SCSI driver

queues to the HBA.n Counter: maxQueueDepthn Stats Type: Absoluten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Read IOPs per HostThis chart displays the per-host disk read rates for a datastore. The chart displays information about the tenhosts with the highest values.This chart is located in the Performance view of the datastore Performance tab.Table 134. Data CountersChart Label DescriptionRead IOPs per Host Number of disk read commands completed on each disk on the host, per second.

Read rate = blocks read per second block sizen Counter: numberReadAveragedn Stats Type: Raten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Write IOPs per HostThis chart displays the per-host disk write rates for a datastore. The chart displays information about the tenhosts with the highest values.This chart is located in the Performance view of the datastore Performance tab.


32 VMware, Inc.

Table 135. Data CountersChart Label DescriptionWrite IOPs per Host Number of disk write commands completed on each disk on the host, per second.

Write rate = blocks written per second block sizen Counter: numberWriteAveragedn Stats Type: Raten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Average Read Latency per Virtual Machine DiskThis chart displays the top ten virtual machine disks with the highest average read latency in milliseconds.Data is not displayed when the virtual machine is powered off.This chart is located in the Performance view of the datastore Performance tab.Table 136. Data CountersChart Label DescriptionAverage Read Latencyper Virtual MachineDisk

Latency measures the time used to process a SCSI command issued by the guest OS to thevirtual machine. The kernel latency is the time VMkernel takes to process an I/O request. Thedevice latency is the time it takes the hardware to handle the request.Total latency = kernelLatency + deviceLatency.n Counter: totalReadLatencyn Stats Type: Absoluten Unit: Milliseconds (ms)n Rollup Type: Averagen Collection Level: 3

Average Write Latency per Virtual Machine DiskThis chart displays the top ten virtual machine disks with the highest average write latency in milliseconds.Data is not displayed when the virtual machine is powered off.This chart is located in the Performance view of the datastore Performance tab.Table 137. Data CountersChart Label DescriptionAverage Write Latencyper Virtual MachineDisk

Latency measures the time used to process a SCSI command issued by the guest OS to thevirtual machine. The kernel latency is the time VMkernel takes to process an I/O request. Thedevice latency is the time it takes the hardware to handle the request.Total latency = kernelLatency + deviceLatency.n Counter: totalWriteLatencyn Stats Type: Absoluten Unit: Milliseconds (ms)n Rollup Type: Averagen Collection Level: 3

Read IOPs per Virtual Machine DiskThis chart displays the top ten virtual machines with the highest number of read operations. Data is notdisplayed when the virtual machine is powered off.This chart is located in the Performance view of the datastore Performance tab.


VMware, Inc. 33

Table 138. Data CountersChart Label DescriptionRead IOPs per VirtualMachine Disk

Number of disk read commands completed on each virtual machine disk, per second.Read rate = blocks read per second block sizen Counter: numberReadAveragedn Stats Type: Raten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Write IOPs per Virtual Machine DiskThis chart displays the ten virtual machines with the highest number of write operations. Data is notdisplayed when the virtual machine is powered off.This chart is located in the Performance view of the datastore Performance tab.Table 139. Data CountersChart Label DescriptionWrite IOPs per VirtualMachine Disk

Number of disk write commands completed on each virtual machine disk on the host.Write rate = blocks read per second block sizen Counter: numberWriteAveragedn Stats Type: Raten Unit: Numbern Rollup Type: Averagen Collection Level: 3

Virtual Machine Observed Latency per DatastoreThis chart displays the average datastore latency as observed by the virtual machines.This chart is located in the Performance view of the datastore cluster Performance tab.Table 140. Data CountersChart Label DescriptionVM observed latencyreport per Datastore

This is the average datastore latency as observed by the virtual machines in the datastorecluster.n Counter: datastoreVMObservedLatencyn Stats Type: Absoluten Unit: Microsecondsn Rollup Type: Latestn Collection Level: 3

HostsThe hosts charts contain information about CPU, disk, memory, network, and storage usage for hosts. Thehelp topic for each chart contains information about the data counters displayed in that chart. The countersavailable are determined by the collection level set for vCenter Server.

CPU (%)The CPU (%) chart displays CPU usage for the host.This chart is located in the Home view of the Host Performance tab.


34 VMware, Inc.

Table 141. Data CountersChart Label DescriptionUsage Actively used CPU, as a percentage of the total available CPU, for each

physical CPU on the host.Active CPU is approximately equal to the ratio of the used CPU to theavailable CPU.Available CPU = # of physical CPUs clock rate.100% represents all CPUs on the host. For example, if a four-CPU host isrunning a virtual machine with two CPUs, and the usage is 50%, the host isusing two CPUs completely.n Counter: usagen Stats Type: Raten Unit: Percentage (%)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA short spike in CPU usage indicates that you are making the best use of the host resources. However, if thevalue is constantly high, the host is probably lacking the CPU required to meet the demand. A high CPUusage value can lead to increased ready time and processor queuing of the virtual machines on the host.If performance is impacted, consider taking the actions listed below.Table 142. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on every virtual machine on the host.2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles

required.3 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For

example, a single-threaded application on a four-way virtual machine only benefits from a single vCPU. But thehypervisor's maintenance of the three idle vCPUs takes CPU cycles that could be used for other work.

4 If the host is not already in a DRS cluster, add it to one. If the host is in a DRS cluster, increase the number of hostsand migrate one or more virtual machines onto the new host.

5 Upgrade the physical CPUs or cores on the host if necessary.6 Use the newest version of hypervisor software, and enable CPU-saving features such as TCP Segmentation Offload,

large memory pages, and jumbo frames.

CPU (MHz)The CPU (MHz) chart displays CPU usage for the host.This chart is located in the Home view of the Host Performance tab.


VMware, Inc. 35

Table 143. Data CountersChart Label DescriptionUsage The sum, in megahertz, of the actively used CPU of all powered on virtual

machines on a host.The maximum possible value is the frequency of the processors multiplied bythe number of processors. For example, if you have a host with four 2GHzCPUs running a virtual machine that is using 4000MHz, the host is using twoCPUs completely.4000 (4 2000) = 0.50n Counter: usagemhzn Stats Type: Raten Unit: MegaHertz (MHz)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA short spike in CPU usage indicates that you are making the best use of the host resources. However, if thevalue is constantly high, the host is probably lacking the CPU required to meet the demand. A high CPUusage value can lead to increased ready time and processor queuing of the virtual machines on the host.If performance is impacted, consider taking the actions listed below.Table 144. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on every virtual machine on the host.2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles

required.3 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For

example, a single-threaded application on a four-way virtual machine only benefits from a single vCPU. But thehypervisor's maintenance of the three idle vCPUs takes CPU cycles that could be used for other work.




CPU UsageThe CPU Usage chart displays CPU usage of the 10 virtual machines on the host with the most CPU usage.This chart is located in the Virtual Machines view of the host Performance tab.


36 VMware, Inc.

Table 145. CountersName Descriptionvirtual_machine Amount of CPU actively being used by each virtual machine on the host. 100% represents all

CPUs.For example, if a virtual machine has one virtual CPU that is running on a host with fourCPUs and the CPU usage is 100%, the virtual machine is using one CPU resource.virtual CPU usage = usagemhz (number of virtual CPUs core frequency)NOTE This is the host's view of the CPU usage, not the guest operating system view.n Counter: usagen Stats Type: Raten Unit: Percentage (%). Precision is to 1/100%. A value between 0 and 100.n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)

Chart AnalysisA short spike in CPU usage or CPU ready indicates that you are making the best use of the virtual machineresources. However, if the CPU usage value for a virtual machine is above 90% and the CPU ready value isabove 20%, performance is being impacted.If performance is impacted, consider taking the actions listed below.Table 146. CPU Performance Enhancement Advice# Resolution1 Verify that VMware Tools is installed on every virtual machine on the host.2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles

required.3 Compare the CPU usage value of a virtual machine with the CPU usage of other virtual machines on the host or in

the resource pool. The stacked line chart on the host's Virtual Machine view shows the CPU usage for virtualmachines on the host.

4 Determine whether the high ready time for the virtual machine resulted from its CPU usage time reaching the CPUlimit setting. If so, increase the CPU limit on the virtual machine.

5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the hostmight remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, setthe CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles.

6 Increase the amount of memory allocated to the virtual machine. This decreases disk and or network activity forapplications that cache. This might lower disk I/O and reduce the need for the host to virtualize the hardware.Virtual machines with smaller resource allocations generally accumulate more CPU ready time.

7 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. Forexample, a single-threaded application on a four-way virtual machine only benefits from a single vCPU. But thehypervisor's maintenance of the three idle vCPUs takes CPU cycles that could be used for other work.




Disk (KBps)The Disk (KBps) chart displays disk I/O of the host.This chart is located in the Home view of the host Performance tab.


VMware, Inc. 37

Table 147. Data CountersChart Label DescriptionUsage Average data I/O rate across all LUNs on the host.

n Counter: usagen Stats Type: Raten Unit: Kilobytes per second (KBps)n Rollup Type: Average (Minimum/Maximum)n Collection Level: 1 (4)













38 VMware, Inc.

Table 148. Disk I/O Performance Enhancement Advice (Continued)# Resolution8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the

system page file. This alleviates disk spindle contention during periods of high use.9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual

machine's VMX file.10 If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.11 For ESXi hosts, create virtual disks as preallocated. When you create a virtual disk for a guest operating system,

select Allocate all disk space now. The performance degradation associated with reassigning additional disk spacedoes not occur, and the disk is less likely to become fragmented.

12 Use the most current host hardware.

Disk Rate (KBps)The Disk Rate chart displays disk read and write rates for LUNs on a host, including average rates.This chart is located in the Home view of the host Performance tab.Table 149. Data CountersChart Label DescriptionRead Number of disk read commands completed on each disk on the host, per second. The

aggregate number of all disk read commands is also displayed in the chart.Read rate = blocksRead per second blockSizen Counter: readn Stats Type: Raten Unit: Kilobytes per second (KBps)n Rollup Type: Averagen Collection Level: 3

Write Number of disk write commands completed on each disk on the host, per second.The aggregate number of all disk write commands is also displayed in the chart.Write rate = blocksWritten per second blockSizen Counter: writen Stats Type: Raten Unit: Kilobytes per second (KBps)n Rollup Type: Averagen Collection Level: 3




VMware, Inc. 39










8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with thesystem page file. This alleviates disk spindle contention during periods of high use.

9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtualmachine's VMX file.

10 If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.11 For ESXi host

Documents

Vsphere Esxi Vcenter Server 60 Monitoring Performance Guide