SER1289BU vSphere Troubleshooting Tips and … Menendez & Francis Daly SER1289BU vSphere Troubleshooting Tips and Tricks #VMworld #SER1289BU VMworld 2017 Content: Not for publication

Javier Menendez & Francis Daly

SER1289BU

vSphere Troubleshooting Tips and Tricks

#VMworld #SER1289BU

VMworld 2017 Content: Not fo

r publication or distri

bution

Agenda

1Use of the VMware vSphere Command-line Interface (vCLI),

the ESXi Shell, and Log Files to Diagnose and Correct Problems

2 Troubleshooting Networking Issues

3 Troubleshooting Storage Issues

4 Troubleshooting vSphere HA Cluster and Performance Issues

2#SER1289BU CONFIDENTIAL



bution

3Certain troubleshooting topics are addressed in the VMware vSphere: Fast Track [6.5] course.

Scope

This Presentation covers portions of the Troubleshooting Workshop:

For more information, go to http://vmware.com/education



bution

Troubleshooting Overview



bution

Troubleshooting Process

#SER1289BU CONFIDENTIAL 5

The troubleshooting process consists of the following tasks:

Defining the problem

• Identifying symptoms

• Gathering information

Identifying the cause of the problem

• Identifying possible causes

• Determining the root cause

Solving the problem

• Identifying possible solutions

• Implementing the best solution



bution

Solving the Problem … Can I Do It with the Web Client or Should I Use the esxi Shell?


What are my choices:

• Can I fix the problem with the Web, Host or vSphere Client?

• Do I want or need to use the command line utilities?VMworld 2017 Content: N

ot for publicatio

n or distribution

Troubleshooting Toolsand Choices



bution

Command-line Troubleshooting Choices


Choices to obtain command-line access on a VMware® ESXi™ host:

vSphere Management Assistant (VMA)

VMware vSphere® ESXi™ Shell

• Can be access via:

• 1. The DCUI (if enabled via ALT + F1)

• 2. Via putty after enabling ssh.

• Appliance deployed via ovf

• Contains vCLI package

• Does NOT use port 22.VMworld 2017 Content: N

ot for publicatio

n or distribution

The esxcli Command


The esxcli command

offers the following

namespaces,

as well as many new esxcli functions added

in the vSphere 6.5.

If working with vSAN,

check out the new esxcli

vsan debug namespaceVMworld 2017 Content: N

ot for publicatio

n or distribution

Example: The esxcli vsan Namespace


The vSAN related

commands are found in

esxcli vsan



bution

New Commands Added with Every New vSphere Release. Ex: The esxcli vsan debug Namespace


The new vSAN debug related commands are amazing to discover the state of your objects, vms, etc.



bution

esxcli esxcli command list

displays all the command

options.

To quickly find the command

needed, combine it with the

grep command

Tip: Remember the esxcli esxcli command list Command




bution

The VMA’s vicfg-* Commands


vicfg- [tab] [tab]

• Commands with the vicfg- prefix enable you to manage

your storage, network, and host configuration.

• Type vicfg- followed by the TAB key twice so see all the available commands.

• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.



bution

The esxi shell’s esxcfg- Commands Are Still There


esxcfg- [tab] [tab]

• Commands with the esxcfg-

prefix enable you to manage

your storage, network, and

host configuration.

• Type esxcfg- followed by the TAB key twice so see all the available commands.

• Remember the –h (help) option to master these commands.



bution

Use the –h option (for help)

to see options for this

particular command and in

some cases, even

examples on how to use the

command.

How many times did you

struggle to create a

vmkernel port with the cli?

Tip: Remember the “-h” Option for esxcfg-* Commands




bution

Manipulating Virtual Machines with the cli from the V.M.A.?


• vmware-cmd

– The vmware-cmd command is exclusively used for virtual machines.

vmware-cmd

• The vmware-cmd command is used to interact with virtual machines.



bution

Example: Listing Your Virtual Machines


vmware-cmd -l

• Lists the virtual machines that are located on the target host. Lists virtual machines by path to the .vmx file.



bution

Tip: Learn More on Your Own

18

Type vmware-cmd --help to

see what the command allows

you to do.

vmware-cmd allows you to

start a vm, take snapshots,

register a vm, etc.

The esxi host has a similar

command to manipulate vms

called vim-cmd vmsvc.

#SER1289BU CONFIDENTIAL



bution

Manipulating Virtual Machines from the esxi shell

19

vim-cmd vmsvc can

perform similar

tasks from the

esxi shell.

Wondering what

the destroy

option does?

Better try it on

a test vm.




bution

Tip: Location of VMware vCenter Server 6.5 Log Files


The VMware vCenter Server

6.5 logs are located in the

%ALLUSERSPROFILE%\VM

Ware\vCenterServer\logs

folder.

The VMware vCenter Server

Appliance 6.5 logs are

located in the

/var/log/vmware folder.

Recommendation

See http://kb.vmware.com/kb/2110014 and 1021806 for additional details.



bution

Useful ESXi Host Logs for Troubleshooting

ESXi hosts write to multiple log files, depending on which action is being performed.

21

Log file Purpose

hostd.log Host Management service logs

syslog.logManagement service initialization, watchdogs, scheduled

tasks, and DCUI use

vmkernel.log

Core VMkernel logs, including device discovery, storage

and networking device and driver events, and virtual

machine startups

vmkwarning.logA summary of warning and alert log messages excerpted

from the VMKernel logs

vmksummary.log

A summary of ESXi host startup and shutdown, and an

hourly heartbeat with uptime, number of virtual machines

running, and service resource consumption




bution

Tip: Other Critical Logs to Be Aware of and How to View Them

22

/var/log/fdm.log displays HA

cluster related events and

notifications.

/var/log/shell.log displays

commands executed inside

of the esxi shell.

/var/log/clomd.log displays vSAN

health related information for

devices among the output




bution

Tip: Remember /var/log


Lots of logs available in

this directory



bution

Collecting Diagnostic Data for

VMware Technical Support

24

Methods for collecting diagnostic information to send to VMware Technical Support include the following:

• Use the GUI to export files to a log bundle.

− vSphere Client or vSphere Web Client

• Use the vm-support command to collect

information from an individual ESXi host.




bution

Tip: Running vm-support when Engaging Support




bution

Network Troubleshooting Commands



bution

Tip: Useful Networking Related Commands

27

Use esxcfg-vmknic –l or

vicfg-vmknic -l to view your

vmkernel port related

information. Are they still there

and enabled?

Use esxcfg-vswitch –l or

vicfg-vswitch –l to view virtual

switch related information. Is

the switch connected to at

least one uplink?

Use esxcfg-nics –l or vicfg-

nics –l to view uplink related

information. Are the nics up or

down?

esxcli also has options to

perform these tasks.

For example:

esxcli network nic list is similar

to esxcfg-nics -l




bution

Somebody could accidentally

down a nic?

If so, type esxcli network nic up

–n vmnic2.

Tip: Did You Know That…




bution

Somebody could disable a vmkernel port

with commands like esxcfg-vmknic -D?

If so, type esxcfg-vmknic -e to re-

enable the vmkernel port.





bution

Somebody could accidentally

disconnect an uplink from a

virtual switch?

ffaaa

Tip: Rebuilding the Management Network from Scratch




bution

There is a built-in packet

sniffer called pktcap-uw

and an older one called

tcpdump-uw.

Need a Packet Sniffer? Remember pktcap-uw




bution

Storage Troubleshooting



bution

You can see how many dastastores exist and how full

the datastores are? This command displays vmfs, nfs,

vsan and vvol datastores. There is also a vdf –h

command.





bution

You can easily view all

sorts of information

with esxcli?





bution

Use this command to

retrieve data about a

specific SSD device.

Critical for vSAN, Flash

Read Cache or any

datastore using SSDs.

vSphere 6.5

Includes

Tip: Use esxcli Storage Core Device Smart for SSD Information

35 35#SER1289BU CONFIDENTIAL



bution

Tip: Be Familiar with partedUtil

36

partedUtil is the command

that partitions luns. It can also

create, resize and delete

partitions.




bution

Tip: Suspect Metadata Corruption?


voma is the on-disk metadata

analyzer for vmfs datastores.



bution

Troubleshooting Clustersand Performance



bution

vSphere HA

39

vCenter

Server

ESXi host (slave)

FDM

ESXi host (master)

FDM

ESXi host (slave)

FDM

vpxd

hostdhostdhostd

Management

Network

vpxa vpxa vpxa

Heartbeat

Datastores

A reliable network connection between the hosts and VMware® vCenter Server™

is essential for enabling vSphere HA.




bution

There is a log for HA that

you can’t see with the Web

or vSphere Client?

Tip: Remember F.D.M




bution

You can stop, start or restart the HA service from the cli if needed?





bution

Tip: You Can Restart All Services at Once If Needed …

42

Remember services.sh restart




bution


43

esxtop is an amazing tool for

performance troubleshooting?esxtop has options for CPU

(c), RAM (m), Disk (d,u,v)

and Network (n) related

statistics?

esxtop has an option for vSAN

(-x option)?




bution

esxtop has built-in help?





bution

You can add more

output to esxtop by

pressing “f”?

For example, you can

add NUMA related

information (G Option)





bution

Tip: Learn the vsish Command and Its Capabilities

46

vsish has hundreds of

subcommands.

Just type the command and

take a look …

You can type commands like

cd to change directories and

get to view information.




bution

Tip: Learn the vsish Command and Its Capabilities

vsish has hundreds of

subcommands.

Just type the command and

take a look …




bution

Global Support Services

Learn more about how VMware is radically transforming Customer Support through VMware Skyline™ technology.

• See a demo in the VMware booth in the Solutions Exchange

• Sign up for a Meet the Experts roundtable in the Schedule Builder on the VMworld mobile app or visit the Meet the Experts, Level 2

• Visit www.vmware.com/support/service/skyline




bution

http://www.vmware.com/support/service/skyline

Additional Education Resources

At VMworld 2017

• Education & Certification Lounge: VM Village

• Certification Exam Center: Jasmine EFG, Level 3

Online

• vSphere Training: www.vmware.com/go/vsphere65training

• VMware Training: www.vmware.com/education

• VMware Certification: www.vmware.com/certification


Save 50% off VCP & VCAP

exams at

VMworld 2017



bution

http://www.vmware.com/go/vsphere65training

http://www.vmware.com/education

http://www.vmware.com/certification



bution



bution

Documents

SER1289BU vSphere Troubleshooting Tips and … Menendez & Francis Daly SER1289BU vSphere Troubleshooting Tips and Tricks #VMworld #SER1289BU VMworld 2017 Content: Not for publication