Networking (Standard and Distributed)

This section will cover both standard networking and distributed networking (vCenter server required for distributed networking), first I will start with standardnetworking.

Standard Networking

The VMkernel can create virtual switches called vSwitches, the virtual machines virtual NIC's are plugged into vSwitches these are then mapped to the physical NIC'son the ESXi server. This means many virtual machines can use the physical NIC's on the ESXI server. The other clever thing VMware can do is that if two virtualmachines communicate with each other on the same vSwitch no physical network traffic is generated. The VMkernel moves the data into memory seamlessly from onevirtual machine to another without ever involving the physical network interface. Knowing this knowledge you can design your VM's that communicate with eachother to use the same vSwitches thus increasing performance.

vSwitches are VLAN aware and can control outbound traffic using a VMware technology called traffic shaping, you can also impose security settings. A vSwitch cancontain 0,1 or many physical NIC's assigned to it, a vSwitch without any NIC's attached to it is called a internal vSwitch, as it only allows communication with thehost. The internal vSwitch could be used as staging area before moving it into Production, one note to remember is that vSwitches do not communicate with oneanother.

The only drawback of internal vSwitches is that you cannot carry out vMotion events without first disconnecting users from the virtual machine. Since they are internalto the ESXi server, we cannot guarantee that users would have a continuous connection to the virtual machine during the vMotion event.

A vSwitch with one physical NIC gives you basic connectivity to the outside world. This might be suitable for vMotion or one that does not require fault tolerance(FT), ideally this would be for testing, development environments, etc. A vSwitch with two physical NIC's gives you fault tolerance (FT) and load balancing, thiswould be idea for vMotion and could be used in Production environments.

You can have up to 20 physical NIC's in an ESXi server of any link speed, there are now 56 ports by default on a vSwitch and this is configurable for up to 4,088 ports.

Port Groups

vSwitches can be divided into smaller units called port groups, there are three types of port groups

Virtual MachineService ConsoleVMKernel (for vMotion, VMware FT logging and IP storage)

You could create one big fat vSwitch and connect all the NIC's to this but you should really separate the network traffic on separate NIC's. Try and name your portgroups with a meaningful name so that if support up the service they can see what the port group is used for, idea names would be vlan25, vlan26. An important noteto remember is that if you rename a port group virtual machines become orphaned from the switch because the name of the virtual switch to which the virtual machineis attached is held in the virtual machines configuration file (.vmx), so bear that in mind if you rename a port group that many have many virtual machines attached,that's a lot of work to resolve the problem and many even need a script to resolve it. An example of this that I change my iSCSI network port group to iSCSI networks(additional s in network) and as you can see in the image on the left had 6 virtual machines attached, a simple name change now has orphaned all 6 virtual machines,see the image on the right (the images were taken from the summary screen of the ESXi server)

Here is a guide on how to create the various standard vSwitch

Internal standard vSwitch

To create a internal vSwitch follow the notes below, remember to name your vSwitch appropriately

In vSphere Client, select the ESXi host1.Select the configuration tab2.In the hardware panel, select Networking3.Click the Add Network link4.Choose Virtual Machine and click next5.Make sure no network adapters are selected and then click Next6.In the Port Groups Properties dialog box, type a descriptive and unique name as internal0-vmware17.Click Finish8.

You should end up with something like below

Teamed Standard Switch

To create a teamed vSwitch it is the same as above, apart from selecting two or more physical adapters, you should end up withsomething like below

Standard vSwitch withVLAN support

ESXi supports two main methods of enabling access to VLAN's

External switch tagging (EST) - simply plug in the relevant NIC's to the relevant VLAN's and set the virtual machines IP settingsfor that network. This will consume a lot of NIC's for each VLAN you need one NIC.Virtual switch tagging (VST) - the network interfaces are plugged into truck ports on the physical switch, truck ports allowmany VLAN packets to traverse them, even with just one or two NIC's, ESXi can allow multiple virtual machines to access manyVLAN's

To create a VLAN vSwitch follow below

In vSphere Client, select the ESXi host1.

VMware - Networking

1 of 9

Select the configuration tab2.In the hardware panel, select Networking3.Click the Add Network link4.Choose Virtual Machine and click next5.Select the desired NIC's6.put a friendly name and the VLAN ID7.click Next and then click finish8.to add more VLAN ID's simply select properties9.select the add button and repeat the process as above10.

Hopefully you should have something like below

VMKernel standardvSwitch for vMotion

When you create a vMotion port group you will be asked for the following details

IP AddressSubnet MaskDefault gateway (this is optional)

You can use vMotion across routers using the gateway but this is not a preferred way, normally ESXi servers are attached to the samenetwork to increase performance and other problems.

To create a vMotion vSwitch follow below

In vSphere Client, select the ESXi host1.Select the configuration tab2.In the hardware panel, select Networking3.Click the Add Network link4.Choose VMKernel and click next5.select the desired NIC's6.In the port groups dialog box type a friendly name, in my case I type "vMotion"7.The select the "use this port group for vMotion"8.Set the IP address and subnet mask and gateway if required9.

Hopefully you should get something like below, you may get a license warning come up, it still will create the port group.

Configuration and Problems of vSwitches and Port Groups

To check to see if a NIC has a problem connecting to the network, go to the Networking screen and check if there is a red-cross against the NIC, see the below image,use standard networking fault finding to correct the problem.

There are a number of configuration settings on vSwitches and Port Groups

Increase the number ofports on a vSwitch

You can now change the number of ports up to 4088 with ESXi 4, if you have a large number of VM's you may have to increase this value

To increase the number of ports on a vSwitch

In vSphere Client, select the ESXi host1.Select the configuration tab2.In the hardware panel, select Networking3.Click the properties of the vSwitch that you wish to change4.make sure the vSwitch option is highlighted5.select the edit button6.then using the down arrow, select number ports that you desire7.click OK to implement8.

VMware - Networking

2 of 9

Setting speed and duplexon physical NIC's

You can change the speed and duplex of any on the physical NIC's in your ESXi server, generally they will auto-negotiate

In vSphere Client, select the ESXi host1.Select the configuration tab2.In the hardware panel, select Networking3.Click the properties of the vSwitch that you wish to change4.select the "Network Adapters" tab at the top5.select the physical NIC and click the edit button6.choose the desired speed and duplex7.click OK to implement8.

Setting vSwitch and PortGroup policies

I am going to cover these briefly, you have three additional areas that can be changed

Security

promiscuous mode - allows a NIC to collect all network packets including onesthat are not intended for itMAC address change - allow for the MAC address to change, this may be arequirement for things like Microsoft cluster and loading balancingForged Transmits - allows a VM to send traffic under a MAC address that isdifferent from that of the VM, Microsoft Network Loading balancing may breakif you reject this

Most systems the default option will be fine

Traffic Shaping

Traffic shaping is ESXi method of controlling outbound traffic generated by VM's youcan control the bandwidth, traffic shaping is not dynamic once set those values areset in stone. Not sure why you would use this feature as i like to give my servers asmuch bandwidth as possible but you can make use of it.

NIC Teaming

NIC teaming (bonding in the Linux world) is a fault tolerance, load balancing feature,you can configure the load balancing, what to detect to cause a failover, failbackwhen the problem has been solved and select particular NIC's to standby or activeNIC's

This is pretty much the same as all NIC teaming in other O/S

VMware - Networking

3 of 9

Service Console Network

ESXi 4 the networking architecture treats the Service Console as if it were just another VM connected to a vSwitch, this default vSwitch is called vSwitch0, the serviceconsole ports have a special name vswif which stands for virtual switch interface. You should try to have a backup service console port or try to protect the existing bymaking use it is fault tolerant to network failures. If you were to lose the service console to have to go to the command-line to either re-install it of repair it. Here aresome commands that will help in recovering a failed service console

Service Console troubleshooting

Correcting your NICselection

# Displaying the vSwitch configurations

esxcfg-vswitch -l

# Display all the NIC's

esxcfg-nics -l

# Link another NIC to the switch

esxcfg-vswitch -L vmnic1 vSwitch0

# Unlink a NIC from a Switch

esxcfg-vswitch -U vmnic0 vSwitch0

Correcting IP settings

# Display current IP address settings

esxcfg-vmknic -l

# Change the Service console network

esxcfg-vmknic "Management Network" -i 192.168.0.190 -n 255.255.255.0

# Restart the network services

service network restart

Configuring the CiscoDiscovery Protocol

# Display current CDP settings, it should return "listen"

esxcfg-vswitch -b vSwitch0

# Make the vSwitch bidirectional

esxcfg-vswitch -B both vSwitch0

# Display CDP data

esxcfg-info | more +/CDP\ Summary

Change the MTU value# Change the MTU value

esxcfg-vswitch -m 1500 vSwitch0

Distributed Switches

Distributed virtual networking (DVN) is completely new to version 4, it enhances the network layer of the VMkernel, in uses features such as below, however youmust be using vCenter in order to create DvSwitches

private VLAN'sNetwork vMotionAPI support for 3rd partiesVMDirectPath I/Onew NIC drivers within guest o/s using vmxnet3.sys

At a basic level a DvSwitch is a global switch, rather than configure vSwitches for each ESXi server, you create a DvSwitch that can be applied to every ESXi server,they also implement private VLAN's (PVLAN) within VMware. DvSwitches can do all the things a normal vSwitch can do, they also offer more ports than a standardvSwitch.

DvSwitches use distributed virtual uplink (DvUplink) ports, it is merely a container for holding a reference to a physical NIC. When you create a DvSwitch the systemattempts to set the correct number of DvUplink ports for you, it does this by looking at all the ESXi servers and searching for the ESXI server with the most uplinks.DvUplinks can be renamed this allows you given more meaningful names.

VMware - Networking

4 of 9

I am going to combine a number of items to explain how to create and configure a DvSwitch, I am going to create a DvSwitch, add a port group and then add anotherport group that uses vMotion. This should give you a fairly good understanding of DvSwitches, I will finish off looking at the advanced features (actually they arepretty much the same as a standard vSwitch).

First let me explain what I had already configure, I had a vSwitch configured with two port groups one for the private LAN and one port group for vMotion (don'tworry too much about vMotion for the moment I will be covering this in greater detail in a later section).

Current setup and freeingup some NIC's

This what I had setup before, a vSwitch with two port groups. Each ESXi server I have (vmware1 and vmware2) has this vSwitch setup.

The first thing I do is free up some NIC's by removing the Physical adapters from both ESXi servers, you should end up with something like below, you can keep the groups incase you want to restore back to what you had.

Creating a DvSwitch

To create a DvSwitch first go to the home page in the vCenter and select networking. You can either select the "New vNetwork Distributed switch" icon or select the "Add avNetwork Distributed Switch" in the main screen, you then get the the below screen

Select vNetwork Distributed Switch Version 4.1.0 (unless you need version 4.0) and click Next

I changed the name to "Private DvSwitch" but you can call it whatever you want, because I have only one NIC to offer from each of my ESXi servers (vmware1 and vmware2),I change the "Number of dvUplink ports" to one. In the real world you probably will have many NIC's to add so this can be increased, click next when finished.

Expand the ESXi server which then should display the available NIC's (if you remember these are the ones we free 'ed up earlier). Select all NICs and then click next tocontinue

You now get a summary screen, notice the "automatically create a default port group" is ticked, you can leave this as we will make use of this group. Click next to createthe DvSwitch. After renaming the dvSwitch to "dvswitch-NIC2" and renaming the port group to "Private" (you can do this by right-clicking each item and selecting editsettings, then change the name at the top", you should have something like below

VMware - Networking

5 of 9

If you select the dvSwitch-NIC2 icon you get a number of tabs appear, to view the ports select ports, remember we selected one dvUplink port for each ESXi server whichgives us two in total, you can also see that "Connectee" comes from each ESXi server

You can confirm the ESXi servers by then select the "Hosts" tab

So what you have now is a DvSwitch with two NIC's attached (one from vmware1 and one from vmware2), you can makes changes to the NIC's and it will beimplemented on both ESXi servers, this can be very handy if you have lots of ESXi servers to manage as most configurations only need slight tuning. The private portgroup will be empty as we have no VM's using it, I will be discussing how a VM can make use of this in my virtual machine section.

Next we will create a vMotion port group within our DvSwitch, follow below

Create a vMotion portgroup

Right click on the DvSwitch icon and select "New port group"

I have change the name to "vmotion", but again you can choose whatever you want

Once the port group is created you need to add the vMotion information for each ESXi server, you will need to provide an IP address and the subnet mask for each ESXiserver using the vMotion.

************** YOU WILL NEED TO REPEAT THE BELOW STEPS FOR EACH ESXI SERVER ************

So from the Home page select "Hosts and Clusters", from there select the first ESXi server (in my case it is vmware1) and select the configuration tab. Select Networkingfrom the hardware then select the "vNetwork Distributed Switch and you then should have a screen similar to below. Select the "Manage Virtual Adapters" ,then select addat the top

You should then have the screen below, select "New Virtual adapter"

then select "VMKernel" (this was the only choice for me)

Now select the vmotion port group from the first drop down list and also tick the "Use this virtual adapter for vMotion"

VMware - Networking

6 of 9

Now enter the IP address and subnet mask, when you repeat this step for the other ESXi server remember you use a different IP address but keep it on the same subnet

Next comes the summary screen, as you can see the vMotion port group now has a new VMKernel port with the IP address that you choose

************* NEW REPEAT THIS STEP OF OTHER ESXI SERVERS THAT YOU WANT TO USE VMOTION *********************

Once you have configured all the ESXi servers, if you select from the "home page" -> networking, then select the DvSwitch group icon, then select the configuration tab youwill see the screen below. I have two physical NIC's from the ESXi servers and two IP address for the vMotion (one each for each ESXi server). If you click on one of the IPaddresses you can see what NIC it goes to (follow the orange line), as you can see IP address 192.168.2.192 goes to vmnic2 vmware1, in this case. Also from this screen ifyou look in the top right hand corner you can manage the DvSwitch by removing it, adding hosts, managing hosts (use this option to add additional NIC's), creating a newport group and evening editing the existing DvSwitch.

You may have noticed that I have a VM using the private port group, I will cover this in greater detail in my virtual machine section.

Create a port group for theService console

This pretty much the same when you get to the "add Virtual Adapter" screen, I choose the service_console port group (this was a new port group that I added) and thenticked the last option "Use this virtual adapter for management traffic, you will then be ask to enter an IP address and subnet for the service console

Once you have finished the screen should look like below see my new service console port group.

As you see a DvSwitch is very similar to a normal standard vSwitch, have a look at each of them side by side, NIC's are represented on the right-hand side and the portgroups are on the left-hand side, there is very little difference in them.

VMware - Networking

7 of 9

Standard vSwitch DvSwitch

I am not going to discuss how to remove a DvSwitch as it is in reserve order as creating one, first remove from the VMKernel ports from each ESXi server first(otherwise when you try to remove the DvSwitch it will say it's in use), then remove each port group and lastly remove the DvSwitch. Most of the time you remove theitem by selecting it's icon and right-clicking, then selecting remove.....

The last topic to cover is the advanced sections of a DvSwitch, again there is very little difference than a normal vSwitch

Add/Remove NIC

You can add or remove additional NIC's form the DvSwitch view. From the home page select networking, then select the DvSwitch icon then select configuration fromthe tabs, you should see a screen like below

Select manage hosts from the top right-hand corner, then select the hosts from which you wish to add NIC's to then select next, you should see the hosts and availableNIC's

Select the additional NIC's and select next, if the NIC's already belong to an existing port group it will ask you if you wish to migrate them, as you can see the one iselect already belong to an existing group hence the warning message. Continue on and the NIC will be added to your DvSwitch.

To remove a NIC just deselect at the "Select Physical Adapters page" and continue through the rest of the screens.

Advanced settings for aDvSwitch

You can edit the setting of a DvSwitch by right-clicking it's icon and then select "edit settings", the first tab called properties you can change the name,increase/decrease the number of dvUplinks and even put notes for anyone.

VMware - Networking

8 of 9

The next tab "Network Adapters" just displays the available ESXi servers and the available dvUplinks

The last tab "private VLAN" you can add primary or secondary VLAN ID's

Advanced settings for theDvSwitch adapters

I am not going into much detail on this as it is pretty much the same as the standard vSwitch

Security - covers the promiscuous mode, MAC address change and the forged transitsTraffic shaping - covers Ingress traffic (traffic that comes from outside a ESXi server - inbound traffic management) and Egress activity that comes into the ESXiserver from either virtual machines or the ESXI server itselfVLAN - select the VLAN type and VLAN ID'sTeaming and Failover - the same as a standard vSwitch but the NIC references are related to dvUplinks port valuesMiscellaneous - allows you to enable port blockingAdvanced - allow override port polices (override the settings on the dvUplink group) and Configure reset at disconnect

Final Comments

DvSwitches are idea for large corporations, it can be a headache to manage lots of standard vSwitches, but bear in mind that you do require vCenter to use DvSwitcheswhich comes with a price and can be off putting to smaller corporations.

VMware - Networking

9 of 9