View
22
Download
0
Category
Preview:
Citation preview
NetXen, Inc.
Intelligent NIC™ Installation and Tuning Guide
Version 3.4
October 30, 2007
This document is applicable to releases 3.4.216 and up
NetXen, Inc.3965 Freedom CircleFourth FloorSanta Clara, CA 95054Telephone: (408) 330-0010Web Site: http://www.netxen.com
Copyright© Copyright 2007. NetXen, Inc. — All Rights Reserved.
DisclaimerThis is a preliminary document and is subject to change without prior notification and does not represent a commitment on the part of NetXen, Inc. The information contained herein is the proprietary and confidential information of NetXen or its licensors, and is supplied subject to, and may be used only by NetXen’s customer in accordance with, a written agreement between NetXen and its customer. Except as may be explicitly set forth in such agreement, NetXen does not make, and expressly disclaims, any representations or warranties as to the completeness, accuracy or usefulness of the information contained in this document.
Please contact NetXen, Inc. to obtain the latest revision of this document.
Trademarks NetXen, the NetXen logo, and Intelligent NIC are trademarks of NetXen, Inc. Third Party brands and names are the property of their respective owners.
NetXen, Inc
Revision History
Version Date Editor Notes
3.0 4/13/074/20/076/21/077/12/07
SR First release of document.Update screen shots and Advanced Properties table.Miscellaneous small edits from internal review.Updated selected screen shots to reflect new software version.
3.1 9/10/07 SR Updated Table 3, System and Software Requirements.Updated Table 4, New NIC Driver Installation.Updated Table 5, Updating from an Existing Driver and Firmware.Updated selected screen shots in Section 4.2.1, Windows NIC Driver Installation.Updated Table 6, Description of Advanced Properties Parameters.Updated Table 7, NetXen Flash Utility Commands.Updated Section 6, Linux Socket Accelerator (LSA).Added Section 7, Teaming Driver.Added Section 8.5, Additional Support.Updated Section A.1.1, Unidirectional: One Server to One Client.Added Appendix B, PXE Boot with the NetXen Card.Added Appendix C, Using Accelerated iSCSI Via the Linux Open iSCSI Initiator.Updated Appendix D, Glossary.Added Section 4.3.2, NIC Driver Options.Reworked Section 9, User Diagnostics.Added Table 12, NetXen Windows Diagnostic Utility Commands.Added Table 13, NetXen Linux Diagnostic Utility Commands.
3.2 9/18/07 SR Added Table 6, Enabling Windows Chimney and Linux LSA.Added Section 4.4.3, Uninstalling the LSA Driver in Linux.Updated Table 9, NetXen Flash Utility Commands.
3.3 10/03/07 SR Integrated miscellaneous edits from internal review cycle.Added TCPv4 Stateful Offload entry to Table 8, Description of Advanced Properties Parameters.Added Appendix Section B.2.2 Windows Remote Installation Services (RIS) Installation.Updated Table 3, NXB-10GXSR XFP Module Specifications.Reorganized material in Section B.2, ‘PXE Configuration — Setting Up Windows Server 2003’ into three subsections.
3.4 10/25/07 SR Updated Section 4.3, Linux Installation.Removed Section on Teaming Driver for Linux and Windows.Added footnote #1 to Table 5, Quick Start Instructions — New NIC Driver Installation.Remove appendix on PXE boot.
THIS PAGE INTENTIONALLY LEFT BLANK
NetXen, Inc
Table of Contents
1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 NIC Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Currently Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Software Release Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Hardware and Software Installation Requirements . . . . . . . . . . . . . . . . . . . 2
2.2.1 Cable Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2.2 XFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3
3 Quick-Start Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Detailed Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1 Installing and Connecting the Intelligent NIC Adapter. . . . . . . . . . . . . . . . . 74.2 Windows Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2.1 NIC Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2.2 Configuring the Network Interface Card . . . . . . . . . . . . . . . . . . . 16
4.3 Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.3.1 Linux NIC Driver Installation (TAR format) . . . . . . . . . . . . . . . . . 184.3.2 Linux NIC Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.3.3 Linux NIC Driver Installation (RPM format). . . . . . . . . . . . . . . . . 204.3.4 Configuring the NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.4 Uninstalling the Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.4.1 Uninstalling Windows Drivers for Both NIC and Chimney . . . . . 224.4.2 Uninstalling the NetXen NIC Driver in Linux. . . . . . . . . . . . . . . . 224.4.3 Uninstalling the LSA Driver in Linux . . . . . . . . . . . . . . . . . . . . . . 22
5 Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1 Using the NetXen Flash Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2 Windows Firmware Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.3 Linux Firmware Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Updating the Flash on Multiple Standalone Cards in Windows. . . . . . . . . 26
6 Linux Socket Accelerator (LSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1 LSA Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.1.1 LSA Driver Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . 276.1.2 TAR Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.2 LSA Offload Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.1 LSA Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.2 LSA Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.3 nxoffload Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.3 Running LSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.4 LSA Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.4.1 Example Read Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 337.2 Unable to ping the NIC Installed on the Host . . . . . . . . . . . . . . . . . . . . . . 337.3 Gather System Information and Check for Errors on Linux Host . . . . . . . 347.4 Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7.4.1 MSI Interrupts for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
NetXen, Inc. i
Table of Contents
7.4.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357.5 Additional Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
8 User Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378.1 Windows Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378.2 Linux Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Appendix A Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1 Windows Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1.1 NTttcp (Windows) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1.2 nttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . . . . 41A.2 Windows Optimization — Tcp1323Opts . . . . . . . . . . . . . . . . . . . . . . 42A.3 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42A.3.1 IPERF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43A.4 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Appendix B Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
ii NetXen, Inc.
1. Overview
1 OverviewThis guide describes the system requirements, installation, and setup required for the NetXen Intelligent NIC™ network interface card product release.
1.1 Related DocumentsRelease notes included in the software release.
Ethernet Driver Developer Reference Guide: For Open Source Software (on the web).
Check NetXen web site for updates.
1.2 NIC VersionsThe NIC products support Windows, Linux, and other operating systems. See the release notes for an updated list.
1.3 Currently Supported Features
Note: Check with NetXen periodically for software updates and additional features.
Product Number Product Definition
NXB-10GSXR Single 10-GE port (optical fiber)
NXB-10CX4 Single 10-GE port (copper)
Feature Windows Linux
TCP, UDP, and IP checksum offload Yes Yes
TCP segmentation offload (TSO / LSO) Yes Yes
Interrupt coalescing Yes Yes
Interrupt distribution No Yes
VLAN Yes Yes
SNMP Ethernet MIB Yes Yes
Jumbo frames Yes Yes
Teaming and Failover Yes Yes
Linux Socket Accelerator (LSA) N/A Yes
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 1
2. System Requirements
2 System RequirementsBefore installing the product, verify the minimum system configuration requirements outlined below.
2.1 Software Release Package Contents
2.2 Hardware and Software Installation Requirements
2.2.1 Cable Specifications
Contents Format Location
Documentation Release Notes NetXen web site
NIC Linux Driver [version].tar.gz or [version].rpm [version]/driver/nx_nic.ko or modprobe for nx_nic
NIC Windows Driver [version].zip [version]\win2k3\x64 or [version]\win2k3\x86
NX Intercept Driver [version].-lsa.tar.gz or [version]-lsa.rpm
[version]-lsa/driver/nx_intercept.ko or modprobe for nx_intercept
LSA Driver [version].-lsa.tar.gz or [version]-lsa.rpm
[version]-lsa/driver/nx_lsa.ko or modprobe for nx_lsa
Firmware cx4_romimage or xfp_romimage [version]\win2k3\utils or [version]/bin
Diagnostics nxudiag [version]\win2k3\utils or [version]/bin
Tools nxflash [version]\win2k3\utils or [version]/bin
Table 1 NXB-10GXSR Cable Specifications — Optical1
1. Check with NetXen for information on the NXB-10GXLR long range optical cable specifications.
ParameterOptical Core Thickness
50 μm 62.5 μm
Wavelength 850 nm 850 nm
Bandwidth (min) 500 MHz 500 MHz
Operating Range 300 m 300 m
Table 2 NXB-10CX4 Cable Specifications — Copper
802.3 Physical Layout Network Type Cable Length
10GBASE-CX4 LAN 15m (max)
2 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
2. System Requirements
2.2.2 XFP Module
2.2.3 System and Software Requirements
Table 3 NXB-10GXSR XFP Module Specifications
Manufacturer Name Part Number
JDS Uniphase Multi-rate 10 Gb/s Short Wave Optical XFP Transceiver
64P0194
PicoLight Multi-rate 10 Gb/s 850 nm Optical XFP Transceiver Module
PL-XXL-SC-S45-22
Table 4 System and Software Requirements
Hardware Systems Validated Platforms
Validated OS (32bit and x64) Firmware1
1. NIC and LSA use the same firmware.
Driver
PCIe: (x8)
Memory: Minimum
2GB
AMD, Intel Windows Server: 2003 SP1,SP2,R2
Linux Distros: RHEL4 U3,U4 SLES9 SP3,
SLES10, RHEL5
NICLSA
Linux:2 NIC, LSA
Windows3
NIC
2. For Linux NIC and LSA, install the kernel source and header files
3. For Windows NIC, SP1 must be installed. SP2 is recommended.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 3
3. Quick-Start Installation Instructions
3 Quick-Start Installation InstructionsThis section is for users already familiar with the Windows or Linux operating systems. For those user’s requiring more detailed information, refer to Section 4.
Table 5 Quick Start Instructions — New NIC Driver Installation
NetXen NIC Hardware Installation Windows NIC Driver Installation Linux NIC Driver Installation1
1.Certain Linux distributions may contain a NetXen GPL driver. This driver will be names netxen_nic.ko. This driver must be removed before using the NetXen commercial driver. Please refer to Section 4.4 of this document, as well as the NetXen release notes.
Pre-Installation1. Verify the system meets the requirements
described in Section 2.2. Power down the system.
Installing the NetXen NIC Adapter1. Insert the NetXen NIC adapter card into an
available PCI Express slot.2. Connect the appropriate cables to the
board.3. Power on the system.
Note that the NIC card comes with the firmware already installed.
Driver Installation1. Copy all of the provided NIC driver files
to a directory.2. From the Device Manager, scan for new
hardware.3. When prompted to connect to Windows
Update, Click No, not at this time, then click Next.
4. In the next window, click Install from a list of specific location and then click Next.
5. Select Search for the test driver in these locations. Select Include this location in the search.
6. Provide the path to the directory where the driver NXP2NIC.sys is located. For 32-bit operating systems, choose the x86 directory. For 64-bit operating systems, choose the x64 directory.
NIC Card Configuration1. Open the Network Connections window
from the Control Panel.2. Verify that NetXen NIC adapter icon says
Connected. If not, right-click on the NIC Adapter icon and select Properties, then assign the IP and subnet addresses and click OK.
Driver Installation (tar.gz)1. Untar the file, build, install, and load the
driver: tar xzf [version].tar.gz cd [version] make make install modprobe nx_nic
Driver Installation (source rpm)1. Install the RPM as follows2
rpm -iv nx_nic-[version].src.rpm2. Create the binary RPM as follows:
rpmbuild -bb /usr/src/redhat/SPECS/ nx_nic-[version].spec, (Redhat) or
rpmbuild -bb /usr/src/packages/SPECS nx_nic-[version].spec (SuSE)
3. Load the nx_nic driver. modprobe nx_nic
NIC Card Configuration1. Identify the NetXen Ethernet interface:
ethtool -i eth[n]2. Assign an IP address:
ifconfig eth[n] [IP address]
2. x.yy.zz is an example of [version]
4 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
3. Quick-Start Installation Instructions
If a NetXen Intelligent NIC card is already installed on the system and a driver and/or firmware update is required, refer to the procedure in Table 6 below.
Table 6 Quick Start Instructions — Updating from an Existing Driver and Firmware
Windows Driver/Firmware Update Linux Driver/Firmware Update
Verify Link Status1. With the existing driver and firmware installed,
verify that the interface is up before continuing with the firmware update.
Update to New Firmware1. Open a Command window.2. Go to the utils directory in the driver package
provided.3. The following binaries are required to be in the
same directory from where the Flash command will be executed: nxflash.exe xfp_romimage cx4_romimage
4. Run the following commands: nxflash -i NX_NIC --all1
5. Verify the firmware has been updated using the following command. nxflash -i NX_NIC --info
Update to the New Driver1. Install the new driver version.2. Verify the new driver version.
1. Writing of the Flash is a critical operation. Please do not interrupt or reboot during this time.
Verify Link Status1. With the existing driver and firmware installed, verify
that the interface is up using ifconfig before continuing with the firmware update.
Update to New Firmware1. Run the following command from the bin directory of
the driver: ./nxflash -i eth[n] --all1
Unload and Remove the Existing Driver1. Unload the existing driver.
ifconfig eth[n] down rmmod nx_nic2. Remove the existing driver module. rm -f /lib/modules/[kernel version]/kernel/
drivers/net/nx_nic.ko2
Reboot the System1. Once the existing driver has been removed, reboot the
system.
Update to the New Driver2. Update to the new driver as follows:
tar xzf [version].tar.gz cd [version] make make install modprobe nx_nic
3. Ensure that the Ethernet interface is up. ifconfig eth[n] <ip addr> up
2. For Linux kernel 2.4, the driver will appear as a nx_nic.o file.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 5
3. Quick-Start Installation Instructions
Once the WIndows or Linux NIC installation is complete as described in the previous tables, the following procedures in Table 7 can be used to enable Windows Chimney or Linux LSA. Table 7 Quick Start Instructions — Enabling WIndows Chimney and Linux LSA
Enabling Windows Chimney Enabling Linux LSA
The following steps can be used to enable Windows Chimney on the driver. Note that the Windows NIC driver must be installed before executing the procedure below.1
1. Open the Network Connections window.
2. Right click on the icon for the NetXen adapter and select Properties.
3. Click on the Configure button.4. Click on the Advanced tab.5. Change the “TCPv4 Stateful offload” from disabled
to enabled.6. To verify the offload has been enabled and is
operating: C:\>netsh int ip show offload
1.A key is required to be loaded onto the card before Chimney can be enabled. Please refer to the Release Notes for more information on how to obtain and verify this key.
The following steps can be used to enable Linux LSA on the driver. Note that the Linux NIC driver must be installed before executing the procedure below.
1. Untar the file, build, and load the drivers.tar xzf [version]-lsa.tar.gzcd [version]-lsamakecd driverinsmod ./nx_intercept.koinsmod ./nx_lsa.ko
2. To verify the LSA driver is operating, execute the following command:cat /proc/net/nx_nic/lsa_1/stats
3. Example for offloading iperf.cd ..\nxoffload./nxoffload –a –n iperf –z
4. To display the current offload rules execute the following:./nxoffload -s
6 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4 Detailed Installation and Configuration
4.1 Installing and Connecting the Intelligent NIC Adapter1 Verify that the system is powered off.
2 Remove the system cover.3 Locate the x8 PCIe slot if one is available.
4 Once the bracket is removed, insert the adapter into the PCI Express slot1. Apply uniform pressure on top of the board until it is firmly seated in the bus connector.
1. Due to space constraints in some systems, when installing an XFP board, it may be necessary to first remove theXPF module from the board, insert the board into the PCI slot, then reconnect the XFP module to the board.
Identify x8 slots for optimal performance
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 7
4. Detailed Installation and Configuration
5 For optical installations, once the board is inserted into the slot, connect the XFP module. Make sure the optical module clicks when inserted into the cage.
6 Secure the bracket.7 Connect the appropriate cables to the NIC and replace the cover.
8 Power on the system.
XFP Cable connection
CX4 Cable connection
XFP Module
CX4Connector
8 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4.2 Windows Driver InstallationThis section describes the Windows installation procedure. For the Linux driver installation procedure, refer to Section 4.3.
4.2.1 NIC Driver InstallationNote that the following procedure is for a new driver installation.
1 Copy the NIC release package to a directory on the host.
2 After the server system is powered on, Windows detects the new card and displays the Found New Hardware Wizard window.
When this window appears, select No, not at this time, and click Next.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 9
4. Detailed Installation and Configuration
3 Select Install from a list or specific location (Advanced) and click Next, as follows:
4 Select Search for the best driver in these locations.Select Include this location in the search.
Type the appropriate path to the NIC driver directory in the accompanying field and click Next. An example is shown below:
If the exact directory path is not known, click Browse and select the NIC driver directory.The NIC card supports both 32-bit and 64-bit operating systems. The x86 directory contains 32-bit driver and the x64 directory contains the 64-bit driver.
D:\3.4.216\SNP\x64
10 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
Select the appropriate directory (x86 or x64) and click OK. Then click Next in the Hardware Update Wizard window to install the driver.
5 The following screen is displayed while the driver is installed.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 11
4. Detailed Installation and Configuration
6 Once the driver installation is complete, the following window will appear. Click Finish to exit.
7 To verify that the driver was properly installed, Right-click on My Computer, then select Device Manager. A listing of devices appears in right side of the window. Under Network Adapters, the NetXen 10Gb Ethernet Adapter should be listed as shown below:
12 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
8 In the above window, right-click on the NetXen 10Gb Ethernet adapter and select Properties. The General tab will be displayed as shown.
9 There are four tabs — General, Advanced, Driver, and Resources. Click on the Advanced tab to display the driver properties as shown below.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 13
4. Detailed Installation and Configuration
The Advanced tab selections displayed above are defined as follows:Table 8 Description of Advanced Properties Parameters
Parameter Description
Flow Control Sets flow control parameters.Registry Key: FlowControlDefault: Enabled.Values: Enabled/Disabled
Large Send Offload TCP Large send offload – Enable/Disable options.Registry Key: LargeSendOffloadDefault: Enabled.Values: Enabled/DisabledSize of LSO: 64K
Locally Administered Address This is the Locally Administered Address (LAA) that users/administrators can set on this interface and this address overrides the permanent address of the adapter (that may have been flashed in to the hardware).Registry Key: NetworkAddressFormat: HexOptions: Value (a value is required, enter the 12 hex bytes of the MAC address to be used), or not present.
Max Ethernet Frame Size Ethernet frame size for packet transmission and receive (does not include MAC Header).Reg Key: MaxFrameSize)Default: 1500 (corresponds to 1514 bytes on the wire + 4 bytes of CRC)Max: 8000 Min: 1500
Max Jumbo Buffers Number of buffers per Jumbo frame.Reg Key: RxJumboRingSizeDefault: 4096 Values: 1024, 2048, 4096, 8192.
Maximum Receive buffers (in K) This is packet buffers used by the driver for receive (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: MaxRxBuffersMin: 1KMax: 16KDefault: 8192
802.1P/Q Packet Tagging For supporting 802.1p/q priority tagging. Should be enabled to set VLAN ID.Registry Key: PQTaggingDefault: EnabledValues: Enable/Disabled
14 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
10 Click on the Driver tab in the Advanced Properties window. The driver can be updated, rolled back, or uninstalled from this window.
Receive IP Checksum Offload This parameter is for enabling/disabling the receive IP checksum offload.Registry Key: RxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Receive TCP/UDP Checksum Offload This parameter is for enabling/disabling the receive TCP and receive UDP checksum offload.Registry Key: RxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Status Ring Size Receive status ring size from hardware.Registry Key: StsRingSizeDefault: 8192Values: 1024, 2048, 4096, 8192, 16384, 32768
TCPv4 Stateful Offload TCPv4 stateful offload.Registry Key: EnableTcpOffloadDefault: DisabledValues: Enabled/Disabled
Transmit IP Checksum Offload This parameter is for enabling/disabling the transmit IP checksum offload.Registry Key: TxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Transmit Ring Size Transmit command ring size provided to hardware.Registry Key: TxRingSizeDefault: 1024Values: 1024, 2048.
Transmit TCP/UDP Checksum Offload
This parameter is for enabling/disabling the transmit TCP and transmit UDP checksum offload.Registry Key: TxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
VLAN ID For setting the VLAN ID for this interface (also exposed through the standard OID).Registry Key: VlanIdDefault: 0 (no VLAN)Range: 0 - 4094
Table 8 Description of Advanced Properties Parameters (Continued)
Parameter Description
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 15
4. Detailed Installation and Configuration
This completes the installation of the NetXen NIC software. Refer to the next section for information on configuring the NIC.
4.2.2 Configuring the Network Interface Card
1 Configure the network connections by opening the Network Connections window in the control panel, or directly from the desktop if the icon exists. The following window is displayed.
16 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
Note that if an IP address has not been assigned, the NetXen 10Gb Ethernet adapter entry above will say “Limited or no connectivity”.If a red ‘X’ appears in the NetXen 10Gb Ethernet10 Adapter icon, this indicates that the link is down. If this occurs, check the following:
Verify that the XFP module is installed correctly.
Verify that the card is seated in the PCI slot properly.
Make sure the network cable is connected properly at both ends.
Is a known good cable being used?
Check the XFP module and replace if necessary.
Has an IP address been assigned to the interface?
Refer to Section 7, "Troubleshooting Guidelines" for more information.To check the status of the connection, right-click on the NetXen 10 Gb NIC adapter icon above and select properties. This causes the Local Area Connection Properties window to be displayed.2 To assign an IP address manually, select the Internet Protocol (TCP/IP) entry below
and click the Properties tab.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 17
4. Detailed Installation and Configuration
3 In the Internet Protocol (TCP/IP) Properties window, assign the appropriate IP, subnet mask, and default gateway values.
4 Using the ping utility, verify connectivity with another card or switch.
4.3 Linux InstallationThe Linux sources can be installed in either TAR or RPM format depending on the release package you received as described below.IMPORTANT NOTE: In the case of SLES 10.1 SP1 and RHEL5U1, the NetXen GPL driver netxen-nic.ko exists in the kernel and may be installed automatically when the O/S comes up. Other Linux distributions above kernel version 2.6.18 may also contain the GPL driver. This driver must be removed before installing the commercial driver from this release.
To check if the netxen-nic has been installed on the system:
1 Check the /lib/modules/<kernel-version>/kernel/net/netxen directory for the netxen-nic.ko driver and either remove or rename this file then reboot.
2 Or to uninstall the driver, perform a “rmmod netxen-nic” before continuing with the following steps in this section.
The following sections describe the procedure for a new driver installation.
4.3.1 Linux NIC Driver Installation (TAR format)
Note: Root level permissions are required for the driver installation.
18 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
1 Copy the file [version]1 tar.gz to the host system and follow the installation command sequence:
To build, install, and load the driver:
tar xvzf [version].tar.gz
cd [version]
make
make install
(driver module is installed in: /lib/modules/<kernel version>/kernel/drivers/net)
cd driver
modprobe nx_nicVerify that nx_nic driver is loaded:
lsmod | grep -i nx_nic
2 Identify the NetXen Ethernet interface:
ethtool –i eth[n]
3 Assign an IP address and bring up the Ethernet interface
ifconfig eth[n] [IPaddress] up
4.3.2 Linux NIC Driver OptionsStep 1 above uses the insmod command. The following optional parameters can be passed to the driver with the modprobe or insmod commands:
insmod nx_nic <parameter> or
modprobe nx_nic <parameter>
The optional parameters are listed below.
1. Where [version] is the numbered release version of the software.
Table 9 insmod and modprobe Command Optional Parameters
Parameter Description Default Range
tx_desc1
1.Example usage: modprobe nx_nic tx_desc=1024 or insmod nx_nic tx_desc=1024
Sets the size of the command descriptor rings (Tx). 1024 256 - 4096 (must be power or two)
jumbo_desc2 Sets the size of the receive ring for jumbo frames. 1024 64 - 4096(must be power or two)
rdesc_1g3 Sets the size of the receive ring for normal packets (1 GbE ports).
8192 64 - 32,768
rdesc_10g4 Sets the size of the receive ring for normal packets (10 GbE port).
32,768 64 - 32,768
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 19
4. Detailed Installation and Configuration
4.3.3 Linux NIC Driver Installation (RPM format)The following subsections describe how to install the NetXen Linux driver source in RPM format, create the binary, and install the binary.
Installing the Linux Driver Source in RPM Format
To install the Linux driver source in RPM format, execute the following command.
rpm -iv nx_nic-[version].src.rpm
Create the Binary
To create the binary RPM for Redhat systems, execute the following command:
rpmbuild -bb usr/src/redhat/SPECS/nx_nic-[version].spec
To create the binary RPM for SuSE systems, execute the following command:
rpmbuild -bb usr/src/packages/SPECS/nx_nic-[version].spec
Install the Binary
To install the binary RPM for Redhat systems, execute the following command:
rpm -i usr/src/redhat/RPMS/x86_64/nx_nic-[version]-1.x86_64.rpm
To create the binary RPM for SuSE systems, execute the following command:
rpm -i usr/src/packages/RPMS/i586/nx_nic-[version]-1.i586.rpm
Note that the i586 designation in the path above could also be i686 or i386 or x86_64 depending on the system.
Install the Tools
To install the binary tools, execute the following command:
rpm -ivh nx_tools-[version].rpm
The tools can be found in the following directory:
cd /opt/netxen
4.3.4 Configuring the NIC on LinuxAssign an IP address to the interface from the command line. For example:
1 Identify the Ethernet interface (eth1, eth2, etc.) and the nx_nic driver:
ethtool -i eth[n]
ifconfig -eth[n] -a
2 Assign an IP Address and activate the interface:
2.Example usage: modprobe nx_nic jumbo_desc=1024 or insmod nx_nic jumbo_desc=1024
3.Example usage: modprobe nx_nic rdesc_1g=8192 or insmod nx_nic rdesc_1g=8192
4.Example usage: modprobe nx_nic rdesc_10g=32768 or insmod nx_nic rdesc_10g=32768
20 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
ifconfig eth[n] down
ifconfig eth[n] <IP Address> <netmask> up
3 Verify network services are running, and ping:
service network restart
ping <IP address>
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 21
4. Detailed Installation and Configuration
4.4 Uninstalling the SoftwareThis section describes how to uninstall the NetXen NIC software from Windows and Linux systems.
4.4.1 Uninstalling Windows Drivers for Both NIC and ChimneyTo uninstall the NetXen NIC adapter driver software in Windows, perform the following steps:
1 Right-click on My Computer and click on Manage. The Computer Management window is displayed.
2 Double click on Device Manager. The list of devices in the computer is displayed on the right side of the window.
3 Locate and right-click on NetXen 10Gb Ethernet Server Adapter.4 Click Uninstall to perform the operation.
4.4.2 Uninstalling the NetXen NIC Driver in LinuxNote that if the LSA driver is installed, it must be uninstalled before the NetXen Linux NIC driver is uninstalled. Refer to Section 4.4.3 below for information on how to uninstall the LSA driver.
TAR FormatIn the Linux system, the TAR archive may be removed as follows:
rm -rf [version]
rm -f [version].tar.gz
ifconfig eth[n] downrmmod nx_nic.ko
The driver may be removed from /lib/modules as follows:
rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_nic.ko
RPM FormatThe RPM package may be removed as follows:
rpm -e nx_nic-[version]
4.4.3 Uninstalling the LSA Driver in Linux
TAR FormatNote that the LSA driver must be removed before the Linux NIC driver can be uninstalled as described in Section 4.4.1 above. The removal procedure for the LSA driver is shown below.
rm -rf [version]-lsa
rm -f [version]-lsa.tar.gz
22 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
rmmod nx_lsa.koThe driver may be removed from /lib/modules as follows:
rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_lsa.ko
RPM FormatThe RPM package may be removed as follows:
rpm -e nx_lsa-[version]
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 23
5. Firmware Updates
5 Firmware UpdatesThe hardware shipped with the product contains the latest firmware. The firmware must be properly flashed in order for the adapter to function properly. Check with NetXen periodically for firmware updates.
5.1 Using the NetXen Flash UtilityThe NetXen Flash utility can be used to modify the contents of the Flash. When using the nxflash.exe utility, the following syntax should be used.nxflash.exe -i NX_NIC [command] (for Windows OS)1
./nxflash -i eth[n] [command] (for Linux OS)In this command line, the -i indicates the interface command and the NX_NIC specifies the actual interface for Windows. The [n] indicates NetXen’s Ethernet interface.Along with the interface, the available commands are listed in Table 10.
The following are some examples of how the nxflash tool may be used.
Update firmware image
Get information about the Flash contents
Each of these functions is described in detail below.
1 Update firmware image
The following files must be present in the directory from where nxflash is executed:
1. For Windows, note that the interface type MUST BE ALL CAPITALS to be recognized as a validinterface.
Table 10 NetXen Flash Utility Commands
Command Definition Description
-i--interface
Interface Specifies the interface type, either [NX_NIC, NX_NIC1, etc.] or [eth1, eth2, etc.].
--info Information Prints board information such as board type, chip revision, serial ID, firmware version, BIOS version, MAC address, etc.
-a--all
Firmware update Complete update of BIOS and firmware.
-r--restore
Restore Restores the existing firmware version on the card.
--opt-rom-on Enable ROM Enable expansion ROM.--opt-rom-off Disable ROM Disable expansion ROM.
-h--help
Help Displays the help menu.
-v--version
Version Displays the version information.
24 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
5. Firmware Updates
xfp_romimage - This is the firmware image for an XFP card.cx4_romimage - This is the firmware image for a CX4 card.These files are supplied as part of the software release. To update the firmware, change directory where the above files are located, and type the following: nxflash.exe -i NX_NIC --all (for Windows) ./nxflash -i eth[n] --all (for Linux)This above update command displays the current firmware version on the card and the new firmware version to which the card will be updated upon completion. The Flash update takes a few minutes to complete and MUST NOT BE INTERRUPTED.2 Get Flash related informationThe --info option may be used to get information about the Flash. e.g. nxflash.exe -i NX_NIC --info (for Windows) ./nxflash -i eth[n] --info (for Linux)The following is an example of a Windows nxflash output:
D:\3.4.216\utils>nxflash.exe -i NX_NIC --infoVerifying flash...Done.Board Type : XGb XFPBoard Chip Rev : C1Serial Id : MX6AMK0006Firmware version (on Flash) : 3.4.216Active Firmware version : 3.4.216BIOS version : 1.0.2MAC Addr 0 : 00:0E:1E:01:03:D4Subsystem ID : 0x00014040Expansion ROM : Disabled
5.2 Windows Firmware UpdateThe following steps are used for updating the Windows OS firmware.1 Open Command window.2 Go to the utils directory in the driver package provided.3 The following binaries are required to be in the same directory from where the Flash
command will be executed: nxflash.exe xfp_romimage cx4_romimage4 Run the following command:
nxflash.exe -i NX_NIC --all
Writing of the flash requires a few minutes. Please do not interrupt or reboot during this time.5 Reboot the system.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 25
5. Firmware Updates
Verify the firmware has been updated using the following command.
nxflash.exe -i NX_NIC --info
5.3 Linux Firmware UpdatesThe following steps are used for updating the Linux OS firmware.
1 Using the existing firmware and driver on the host, bring the Ethernet interface up using the following command:
ifconfig eth[n] <IP addr> netmask <netmask> up
2 Update to the new firmware from the bin directory:
cd <new release version directory>/bin
./nxflash -i eth[n] --all
3 Bring down the Ethernet interface.
ifconfig eth[n] down
4 Remove the existing or old driver module.
rmmod nx_nic
5 Build and install the new driver.
cd <new release version directory>
make
make install
6 Reboot the host system.7 Load the new driver.
cd <new release version directory>/driver
insmod ./nx_nic.ko
5.4 Updating the Flash on Multiple Standalone Cards in WindowsThe Flash can be updated on multiple cards on the same host using one of the following procedures.
1 If the updates are being done from the same directory tree, the following commands must be run sequentially after the Flash update has completed.
nxflash.exe -i NX_NIC0 --all
nxflash.exe -i NX_NIC1 --all
2 Updates can be done simultaneously from two different directory trees. Note that the following two commands are being executed from two separate directory trees.
c:\[version]-a\utils\ nxflash.exe -i NX_NIC0 --all
c:\[version]-b\utils\ nxflash.exe -i NX_NIC1 --all
26 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
6 Linux Socket Accelerator (LSA)The purpose of LSA is to allow the user to offload a significant amount of TCP network processing on to the Netxen NIC, significantly reducing host CPU utilization. Note that the LSA does not require the user to rebuild the Linux kernel, as simple loadable modules are used.
As described below, the user has very fine-grained control over what can be offloaded. Offload decision can be based on application name, local TCP port number, or remote TCP port number.
6.1 LSA Driver InstallationThis section describes the LSA installation instructions for the tar and RPM formats.
NOTE: The LSA Intercept driver must be running at all times for the LSA driver to work. Whenever the server is rebooted, check if the NIC and Intercept driver is running lsmod before using the LSA driver. The LSA intercept driver cannot be removed with the command rmmod; Remove the driver file from /lib/modules/<kernel>/drivers/net/, and reboot the host.
6.1.1 LSA Driver Installation PrerequisitesThe following prerequisites are required before installing the LSA driver.
At least 1 GB of memory.
At least 64 MB of free disk space.
Any of the following Linux distributions
- RHEL4, U3, U4- RHEL5- SLES9 SP3- SLES10
In all cases, Linux kernel 2.6.5 or later is required.
6.1.2 TAR FormatThis section describes the LSA driver installation from the tar package.
1 First Load the NIC driver (nx_nic.ko)
2 Build and load the Intercept and LSA driver
% tar xvzf <version>-lsa.tar.gz
% cd <version>-lsa
% make
% make install
% insmod./driver/nx_intercept.ko
% insmod./driver/nx_lsa.ko
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 27
6. Linux Socket Accelerator (LSA)
3 Use offload rules to offload the applications
4 Use rmmod nx_lsa to remove the LSA driver
6.2 LSA Offload ConfigurationLSA can be configured via the nxoffload utility. This utility works by maintaining a list of rules that dictate which applications to offload. Each rule can take one of the following parameters:
Application name
Local TCP port number
Local and Remote TCP port number
6.2.1 LSA Configuration OptionsA list of parameters to this utility can be obtained by typing ./nxoffload -help
usage: ./nxoffload <options>
The options are listed below.
6.2.2 LSA Configuration Examples
To offload the iperf application, type the following: ./nxoffload -a -n iperf
To offload the application that uses local TCP port 80, type the following: ./nxoffload -a -p80
To remove the application that uses local TCP port 80, type the following: ./nxoffload -r -p80
To list the current offload rules, type the following: ./nxoffload -s
Table 11 LSA Configuration Options
<option> Definition Comments
-p Port number to be offloaded.-n Application name to be offloaded.-t TCP tuple to be offloaded. Format: local IP, local port, remote
OP, remove port. Example: ./nxoffload -a -t'0, 0, 0, 5001'./nxoffload -a -t'0, 2000, 0, 5001'
-a Add an offload rule.-r Remove and offload rule.-z Enable zero copy.-s List all ports and applications offloaded.
28 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
To offload an application with zero copy enabled, type the following: ./nxoffload -a -z -n <application-name>.
Zero copy is a way to significantly reduce CPU utilization in the send path for some applications. Not all applications may benefit from using zero copy, and some reduction in throughput may be observed. Applications that do well with zero copy are those that have large write sizes and a large number of individual connections.
6.2.3 nxoffload Caveatsnxoffload has the following caveats:
To add or remove offload rules, nxoffload requires root access
Rules may be added or removed at any time. The addition or removal of a rule, however, will not affect connections that have already been established. For example, if the user starts a non-offloaded iperf application and then types ./nxoffload -a -n iperf, this instantiation of iperf will NOT be subsequently offloaded. Future instantiations, however, will be offloaded.
6.3 Running LSAOnce the appropriate rule has been entered into nxoffload, no further user intervention is required. The offload occurs automatically. The offload statistics can be monitored by reading the file /proc/net/nx_nic/lsa_x/stats, where 'x' is NetXen card number.
6.4 LSA StatisticsTable 12 lists the different types of LSA statistics, along with the current support status for each type.
Table 12 LSA Statistics
Name Description Status
TCP StatisticsTcpMaxConn Total number of connections since initial power up. Not supported.
TcpActiveOpens Total number of connections opened via connect. Supported.TcpPassiveOpens Total number of connections opened via accept. Not supported.TcpAttemptFails Number of unsuccessful connection attempts. Supported.TcpEstabResets Resets received by connections in the established
state.Supported.
TcpCurrEstab Number of connections currently in the established state.
Supported.
TcpInSegs Number of received segments. Supported.TcpOutSegs Number of transmitted segments. Supported.
TcpSlowOutSegs Number of transmit segments processed by the slow firmware path.
Supported.
TcpRetransSegs Number of retransmitted Supported.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 29
6. Linux Socket Accelerator (LSA)
TcpInErrs Number of illegal received segments. Supported.TcpOutRsts Number of transmitted resets. Supported.
TcpOutCollapsed Number of collapsed transmitted segments. A collapsed segment is generated when smaller segments are combined to form a single large segment
Supported.
IP StatisticsIpInReceives The total number of input datagrams received from
interfaces, including those received in error.Supported.
IpInHdrErrors The number of input datagrams discarded due to errors in their IP headers, including bad checksums, version number mismatch, other format errors, time-to-live exceeded, errors discovered in processing their IP options, etc.
Supported.
IpInAddrErrors The number of input datagrams discarded because the IP address in their IP header's destination field was not a valid address to be received at this entity. This count. includes invalid addresses (e.g., 0.0.0.0) and addresses of unsupported Classes (e.g., Class E). For entities which are not IP routers and therefore do not forward datagrams, this counter includes datagrams discarded because the destination address was not a local address.
Not supported.
IpInNoRoutes (Not in ip-mib)
Number of input datagrams discarded because there was a problem with either determination of inbound interface or a problem with determination of a route back to the source of the datagram.
Supported.
IpInDiscards The number of input IP datagrams for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space). Note that this counter does not include any datagrams discarded while awaiting re-assembly.
Not supported.
IpInDelivers The total number of input datagrams successfully delivered to IP user-protocols (including ICMP).
Supported.
IpOutRequests The total number of IP datagrams which local IP user-protocols (including ICMP) supplied to IP in requests for transmission. Note that this counter does not include any datagrams counted in ipForwDatagrams.
Supported.
IpOutDiscards The number of output IP datagrams for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space). Note that this counter would include datagrams counted in ipForwDatagrams if any such packets met this (discretionary) discard criterion.
Supported.
Table 12 LSA Statistics (Continued)
Name Description Status
30 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
6.4.1 Example Read OutThe following is a sample of the TCP and IP statistics displayed when the stats command is executed.
cat /proc/net/nx_nic/lsa_1/stats
IpOutNoRoutes The number of IP datagrams discarded because no route could be found to transmit them to their destination. Note that this counter includes any packets counted in ipForwDatagrams which meet this `no-route' criterion. Note that this includes any datagrams which a host cannot route because all of its default routers are down.
Supported.
IpReasmTimeout The maximum number of seconds which received fragments are held while they are awaiting reassembly at this entity.
Not Supported.
IpReasmReqds The number of IP fragments received which needed to be reassembled at this entity.
Not Supported.
IpReasmOKs The number of IP datagrams successfully re-assembled.
Not Supported.
IpReasmFails The number of failures detected by the IP re-assembly algorithm (for whatever reason: timed out, errors, etc). Note that this is not necessarily a count of discarded IP fragments since some algorithms (notably the algorithm in RFC 815) can lose track of the number of fragments by combining them as they are received.
Not Supported.
IpFragOKs The number of IP datagrams that have been successfully fragmented at this entity.
Not Supported.
IpFragFails The number of IP datagrams that have been discarded because they needed to be fragmented at this entity but could not be, e.g., because their Don't Fragment flag was set.
Not Supported.
IpFragCreates The number of IP datagram fragments that have been generated as a result of fragmentation at this entity.
Not Supported.
TCP Statistics Port 0 Port 1 Port 2 Port 3
TcpMaxConn 0 0 0 0
TcpActiveOpens 180 0 0 0
TcpPassiveOpens 0 0 0 0
TcpAttemptFails 0 0 0 0
Table 12 LSA Statistics (Continued)
Name Description Status
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 31
6. Linux Socket Accelerator (LSA)
TcpEstabResets 0 0 0 0
TcpCurrEstab 0 0 0 0
TcpInSegs 54593130 0 0 0
TcpOutSegs 173931132 0 0 0
TcpSlowOutSegs 2690 0 0 0
TcpRetransSegs 743 0 0 0
TcpInErrs 0 0 0 0
TcpOutRsts 0 0 0 0
TcpOutCollapsed 0 0 0 0
TcpTimeWaitConns 0 0 0 0
IP Statistic Value
IpInReceives 54593098
IpInHdrErrors 0
IpInAddrErrors 0
IpInNoRoutes 0
IpInDiscards 0
IpInDelivers 54593130
IpOutRequests 173933823
IpOutDiscards 0
IpOutNoRoutes 0
IpReasmTimeout 0
IpReasmReqds 0
IpReasmOKs 0
IpReasmFails 0
IpFragOKs 0
IpFragFails 0
IpFragCreates 0
TCP Statistics Port 0 Port 1 Port 2 Port 3
32 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Troubleshooting Guidelines
7 Troubleshooting Guidelines
7.1 Unable to Boot the Host System with the NIC Card InstalledRemove the NIC and reboot. Does the system boot without the NIC?
If so, power down the system, insert the NIC into a different PCIe slot and boot-up the system.
Is the recommended minimum memory installed on the host system?
Note any error message during the OS boot sequence.
Check for system error messages:
Dbgview.exe (Windows event viewer)
dmesg (Linux)
Use the NetXen diagnostics utility for further testing.
7.2 Unable to ping the NIC Installed on the HostType lspci -v and look for vendor ID ‘4040’.
- Is the NIC visible on the PCI bus?
- Is the NIC is placed in the PCIe x8 slot?
Check if the driver is loaded correctly and the card is initialized
- dmesg (display system messages on Linux)
- Windows Event Viewer tool
Run the NetXen user diagnostics
Check the NIC XFP module and optical cables; For CX4, check the IB cable
Check Network Configuration
- Check for valid IP Address and subnet
- Check for duplicate IP address and subnet numbers
- If connected via gateway, check gateway address and network path
- Check the NIC MAC address values
- Network link status as seen by the host OS
- Check for LED activity (using ping)
- Using network monitoring tools, analyze the Tx and Rx packets for the NIC
- Verify the ARP entry and response
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 33
7. Troubleshooting Guidelines
7.3 Gather System Information and Check for Errors on Linux HostNetXen Product Information
- Card type
- NIC firmware and driver version
Operating System Information
- Windows OS (32 / 64) and Service Pack #
- Linux Distribution and Kernel Release (uname –a)
Host System Information
- CPU, Cores, Chipset – (Run System Information tool)
- BIOS information
- Host memory
- PCIe x8, x4, x1 …
- System setup description
- How many servers, clients etc
- Model of the host system
- Network setup (back to back, router, switch, backplane etc)
Check for error messages from the kernel:
- Type dmesg
- Type cat /var/log/messages
Check routing table, gateway, and IP Address conflicts for all interfaces:
- Type route
Check network interface TX and RX counts during pings:
- Type ifconfig <network interface>
Use ethereal, tethereal, or tcpdump utilities to capture packets:
- Analyze the ping packets
7.4 CaveatsJumbo frames of maximum size 8000 are supported.
The current version of the Windows Win2k3 OS does not support MSI in the network drivers (NDIS 5.1). However, hardware support is provided for NDIS versions 5.2 and 6.0.
34 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Troubleshooting Guidelines
7.4.1 MSI Interrupts for LinuxThe Linux OS version supports MSI, and is used by the Intelligent NIC driver (if available).
On the Linux host, type cat /proc/interrupts to check that PCI-MSI is enabled for the Ethernet interface
Determine if the installed Linux kernel has the MSI configuration option enabled (earlier versions of the kernel may require an MSI patch).Type make menuconfig from the directory /usr/src/kernels/<kernel version> and verify that MSI configuration option is supported/enabled.NetXen device ID = 4040
$ lspci -n | grep 4040
7.4.2 PCI-Express SlotThe NetXen reference card can be located on the PCI-Express bus using the following Linux procedures.
Determine Vendor and Device IDTo check if the PCIe slot has an x8 link for optimal PCIe performance, first determine the PCI vendor ID and device ID codes using the following command:
% lspci -n
A message similar to the following should be displayed:
06:00.0 0200: 4040:0001 (rev 24)
The NetXen vendor ID is 0x4040. A list of NetXen device ID’s is shown below.
Determine Interface AttributesTo determine the attributes of the interface, execute the following command:
% lspci -n -d 4040:0002 -vvv | grep -i width
A message similar to the one below should be displayed.
Link: Supported Speed 2.5 Gb/s, Width x8, ASPM L0s, Port 0
Link: Speed 2.5Gb/s, Width x8
Determine Width ParametersCheck if the lspci output "negotiated width" (the second width in the output) reflects the actual "supported width". In an XFP system for example, execute the following command to determine if these two widths are the same:
Table 13 NetXen Device IDs
Product Device ID Description
NXB-10GXSR 0x0001 10 Gbit Ethernet NIC with XFP optical interface.
NXB-10GCX4 0x0002 10 Gbit Ethernet NIC with CX4 copper interface.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 35
7. Troubleshooting Guidelines
% lspci -xxx -vvv -d 4040:0002
where 4040 indicates the NetXen vendor ID and 0002 indicates a CX4 card device ID. The XFP card device ID is 0001.Examine the following hex output:
From the lspci hex output:
d0: 10 00 01 00 02 80 00 00 40 20 01 00 81 04 00 00 /\
||_________ capability. "81 04" = x8
e0: 01 00 81 00 00 00 00 00 00 00 00 00 00 00 00 00 /\
||_________ negotiated. "81 04" = x8. "41 04" = x4, "11 04" = x1, "01
05" = x16
Note that some PCI-e chip sets with a x16 link may not support a x8 PCI-Express link width. A link is only required to support its maximum width and x1. Check the motherboard specification to determine if its PCI-e x16 slot can autonegotiate to x8. On some motherboards, the PCI-e slots are all the same physical size required by a x16 slot, but electrically they are different widths. Check the motherboard specification for the supported width of each of the PCI-e slots.
7.5 Additional SupportIf problems are encountered with server applications when either Chimney or RSS are enabled under Microsoft Win2K3 with SP2 (in Offload Mode), please follow the link below to see if Microsoft has addressed the specific issue.
http://support.microsoft.com/kb/936594
...... 81 04 00 00 01 00 81
Supported link width
Negotiated link width
36 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
8. User Diagnostics
8 User DiagnosticsThe user diagnostics can be invoked in Windows and Linux using the following command:
nxudiag.exe (Windows)
[version]/bin/nxudiag -i eth[n] (Linux)
8.1 Windows Tests The Windows user diagnostics utility performs the following tests using the nxudiag.exe command.
Table 14 NetXen Windows Diagnostic Utility Commands
Command Description
-i--interface
Specifies the interface type [NX_NIC, NX_NIC1, etc.]
-a--all
Perform all test, regardless of default.
-n--nodefault
No defaults, all tests will be set implicitly.
-R--CRegs
Test all control registers.
-F-Flash
Test the Flash.
-I--IRQS
Test interrupt mechanism.
-L--IntLB
Internal loopback test
-S--LinkST
Link status test.
-I--LED
LED test.
-nR--noCRegs
No control registers test.
-nF--noFLASH
No Flash test.
-nl--noIRQS
No interrupt test.
-nL--noIntLB
No internal loopback test.
-nS--noLinkSt
No Link status test.
-nl--noLED
No LED status test.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 37
8. User Diagnostics
8.2 Linux Tests The Linux user diagnostics utility performs the following tests using the ./nxudiag.exe -i eth[n] command.
-E--ExtLB
External full duplex loop-back test.
-h--help
Prints the help message.
Table 15 NetXen Linux Diagnostic Utility Commands
Command Description
-i--interface
Specifies the interface type [eth1, eth2, etc]
-a--all
Perform all test, regardless of default.
-n--nodefault
No defaults, all tests will be set implicitly.
-s--silent
Silent mode. No output to console.
-v--verbose
Verbose mode. Output result to console.
-R--CRegs
Test all control registers.
-F-Flash
Test the Flash.
-I--IRQS
Test interrupt mechanism.
-L--IntLB
Internal loopback test
-H--Hw
Hardware test.
-S--LinkST
Link status test.
-nR--noCRegs
No control registers test.
-nF--noFLASH
No Flash test.
-nl--noIRQS
No interrupt test.
-nL--noIntLB
No internal loopback test.
Table 14 NetXen Windows Diagnostic Utility Commands
Command Description
38 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
8. User Diagnostics
-nH--noHw
No hardware test.
-nS--noLinkSt
No Link status test.
-E--ExtLB
External full duplex loop-back test.
-C--Cmdif
Command interface status test.
-h--help
Prints the help message.
Table 15 NetXen Linux Diagnostic Utility Commands
Command Description
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 39
Appendix A. Tuning Guidelines
Appendix A Tuning Guidelines
A.1 Windows Performance Tools
A.1.1 NTttcp (Windows)There are two switches on ntttcp that boost performance: -a and -rb. Running ntttcp without an optimal value for the -a switch and without the -rb switch is acceptable for the Gigabit NIC product.In the 10G NIC product, however, and in particular with fewer threads on Windows, performance is limited by the Receive window for the most part and by the outstanding I/O (-a) value, regardless of hardware limitations.For receive, an optimal value is 16 for -a and 512000 for -rb (or something similar). With these values, a single thread should perform closer to a multi-thread configuration.
ntttcpr -m 16,0,<rcv ipaddr> -n <# of tcp segs for snd/rcv> -a 16
For transmit, -a should be a value less than 8. The more threads, the lower the number should be.
Note: The send and receive side NTTTCP commands must use the receive side’s IP address.
Steps for NIC Performance Testing using NTTTCP on Windows
1 Install Win2k3 SP1
2 Enable Windows Scaling Set TCp1323Opts parameter value to 1, if the parameter does not exist, create it: "regedit" to HKLM\system\currentcontrolset\services\Tcpip\Parameters3 The drivers property sheet default is 1500. For Jumbo, set the frame size to 8000.
Table 16 ntttcpr Command Options
Command Optionsntttcpr Description
-l Message size in bytes. Default is 64K.
-m Number of threads.
0 CPU number.
IPAddress
Receive IP address.(Send side also uses receive side's IP Address).
-n Number of TCP segments.
-a Number of requests in parallel.(Send and receive side values must match).
-rb Receive window size on the receiver.
40 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix A. Tuning Guidelines
Unidirectional: One Server to One ClientReceiver:
ntttcpr.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -fr -n 200000
Sender:ntttcps.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -n 200000
The above commands can be used in both Chimney and non-Chimney modes. With Chimney enabled, performance is higher and CPU utilization is also reduced.
Unidirectional: One Server to One Client — Single Thread, Multiple ProcessorsIf the system has more than one CPU, the load can be distributed across two CPUs and port numbers.Send Sidentttcps.exe -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5001ntttcps.exe -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5002
Receive Sidentttcpr.exe -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5001ntttcpr.exe -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5002
A.1.2 nttcps and ntttcpr Command Line Options
C:\NTTTCP_NEW>ntttcps.exe <parameters>orC:\NTTTCP_NEW>ntttcpr.exe <parameters>
Version 2.6ntttcpr.exe: [-l|-n|-p|-a|-x|-rb|-sb|-i|-f|-u|-w|-d|-t|-v|-6|-fr|-mb] -m <mapping> [mapping]
-l <Length of buffer> [default: 64K] -n <Number of buffers> [default: 20K] -p <Port base> [default: 5001] -a [outstanding I/O] [default: 2] -x [PacketArray size] [default: 1] -rb <Receive buffer size> [default: 64K] -sb <Send buffer size> [default: 0] -i Infinite Loop [Only UDP mode] -f <File Name> [default: output.txt] -u UDP send/recv -w WSARecv/WSASend -d Verify Flag -t <Runtime> in seconds -v enable verbose mode -6 enable IPv6 mode
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 41
Appendix A. Tuning Guidelines
-fr Full buffers posted on reads -mb Multiple buffer post mode -m <mapping> [mapping] where a mapping is a session(s),processor,receiver IP set e.g. -m 4,0,1.2.3.4 sets up: 4 sessions on processor 0 to test a network on 1.2.3.4
A.2 Windows Optimization — Tcp1323OptsThis parameter determines whether TCP uses the time-stamping and window scaling features described in RFC 1323, TCP Extensions for High Performance.Window scaling permits TCP to negotiate a scaling factor for the TCP receive window size, allowing for a very large TCP receive window of up to 1 Gbyte. The TCP receive window is the amount of data the sending host can send at one time on a connection.To set the window scaling and timestamp parameters, click Start -> Run, then type regedit to start the registry editor. Open the following location:HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Window ScalingFor more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes.Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte.When the value for window size is added to the registry and its size is larger than the default value, Windows attempts to use a scale value that accommodates the new window size.
A.3 Linux Performance ToolsOnce connectivity between the two cards is established, common performance tools can be used for performance measurements.
Data Type Range Recommended Value
REG_DWORD 0 | 1 | 2 | 3 1
Hex Value Meaning
0 Timestamps and window scaling are disabled.
1 Window scaling is enabled.
2 Timestamps are enabled.
3 Timestamps and window scaling are enabled.
42 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix A. Tuning Guidelines
Note: Various factors including tools, kernel, stacks, and related parameters affect the performance of a system. A discussion on performance criteria is beyond the scope for the getting started section.
Add the following for performance tuning on Linux systems under --/etc/sysctl.conf
A.3.1 IPERFThe -P option determines the number of threads; default is one.Server Side:
iperf -s -w256k -l 56k
Client Side:
iperf -c <ipAddr of Server> -w256k -P12 -l 56k
A.4 Linux OptimizationWhile optimization of the Linux operating system is a topic beyond the scope of this guide, below are a few common guidelines.
PCIe x8 provides optimal performance.
MSI interrupts are preferred over legacy interrupts.
Linux Kernel versions 2.6.15 or later provide better performance optimization for the NIC.
Check CPU utilization on the host for CPU-bound bottlenecks.
The tunable performance parameters are system and application specific. Understanding the relationship between the parameter values, target workload, performance applications, OS and system configuration provides a better understanding of performance optimization and bottlenecks.
net.core.rmem_max = 16777216net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_timestamps = 0net.ipv4.tcp_sack = 1net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216net.ipv4.tcp_low_latency = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_moderate_rcvbuf = 1net.ipv4.route.flush = 1
net.core.optmem_max = 524287
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 43
Appendix A. Tuning Guidelines
44 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix B. Glossary
Appendix B Glossary
Name Description
cx4_romimage NetXen firmware image for a CX4 card
dbgview.exe Debug viewer for Windows
dmesg Linux system messages
ethereal Packet analyzer
ethtool Linux Ethernet tool for card settings. See man ethtool in Linux
iperf Linux performance tool
LKM Linux Kernel Module
Load balancing Allows the load to be balanced between 2 or more ports in a system.
LSA Linux Socket Accelerator
NIC Network Interface Card
nxflash NetXen Flash utility
nx_intercept.ko Linux LSA intercept driver
nx_lsa.ko Linux LSA driver
nx_nic.ko Linux NIC driver
nxp2nic.sys Windows NIC driver
nxudiag NetXen user diagnostic utility
xfp_romimage NetXen firmware image for an XFP card
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 45
Appendix B. Glossary
46 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Recommended