Chelsio driver

We have two T110 Chelsio NICs.

During a first attempt with the patching and compilation of the kernel we encountered some problems and issues that have been reported to Chelsio. The content of this mail can be found in ChelsioDriverQuestions.

Software installation

In this documentation there hase been focussed upon the installation of TOE version T. 1.2.0 (TCP Offload Engine). Specific issues of the installation of TOE version 2.1.1 can be found in ChelsioDriverTOE211.

To be able to use the Chelsio driver the Linux 2.6.6 kernel should be patched with a patch supplied by Chelsio. The process of patching, the kernel, rebuilding it, and the installation of related software will be described in the following sections. The order in these steps are directly taken from the Chelsio installation user guide. For the buillding and installation of the kernel the guidelines from the Web100KernelInstallation HowTo had been used.

As a test the patching and building of the kernel was first performed as a non-root user. When it could be succesfully build the these steps including some more installation steps have been repeated as root from a different directory tree.

Unpack the Linux 2.6.6. kernel source and move it to a unique name

The string -chelsio-1.0 will be used for the EXTRAVERSION variable in the Makefile that will be used to build the kernel in the steps listed below.

Unpack the Chelsio tarball into a staging area

Install Chelsio's TOE patch for the 2.6.6 kernel tree

Create a new .config configuration file for the kernel to include the Chelsio ULD for the TOE card

The Chelsio ULD will show up under 2.6.6 kernel menu options that is illustrated briefly for the make menuconfig command below. As shown the menuconfig options indicate that the Chelsio ULD driver is to be compiled statically into the kernel for both 1G Dumb NIC support and TOE support, in addition to 10G support.

Enable compile-time support for the toectl tool

If you do not want to always offload all TCP connections, you need to enable compile-time support for the toectl tool by choosing the following configuration options:


Please note that all other options below Network packet filtering (replaces ipchains) -->, that are not listed here, has been switched of. Otherwise the compiler errors that have been reported in ChelsioDriverQuestions are occurring. May be some options could be switched on without leading to problems but that has not been checked yet.

Give the kernel an unique name

The line starting with the EXTRAVERSION variable has been set to

Compile the Chelsio tools chtoetool and toectl

Build chtoetool:


The Makefile does not contain an install target. Therefore, install the executable and man page by hand:


Build and install toectl:


During the loading there are some innocent warnings that some system libraries could not be statically linked.

Compile the patched kernel

For the compilation of the patched kernel the steps will be followed that have been described in the Web100KernelInstallation HowTo. Please note that above the kernel has already been given the unique name -chelsio-1.0.

Build the kernel

Store also the output of the build process such that any diagnostic messages can be easily read.


The kernel should now exist in arch/x86_64/boot/bzImage.

Build and install the modules


Copy the kernel, its configuration file, and the system map file to the boot sector


Create an initial ramdisk and copy it to the boot sector


Configure the grub boot loader


The following lines had been add to /boot/grub/menu.lst:


To use linux-2.6.6-chelsio-1.0 as the default kernel, the default varable in /boot/grub/menu.lst should be set to the number of the


entry. Please note that default should start wih zero.

Install the tools scripts

Create an unique directory where the script tools of this version should be installed. The directory name will be connected to the kernel version which is printed by the command uname -r. Copy the scripts to this directory.

Adjust the starttoe script to the local host properties

This script configures both the PCI bus and the TOE device for maximum performance, and it also enables the TOE mode. Make a copy from the original and make it executable.


Also change the following paths in it:

• /sbin/setpci into /bin/setpci.
• /sbin/lspci into /bin/lspci.

Create a general starttoe script

To be able to let various TOE and/or kernel versions coexist it is convenient to create a general starttoe script that can be used in the configuration files. It calls the dependent starttoe script after the scripts directory has been put in front of the executable path.

Create /usr/local/sbin/starttoe with the following content:

Run the general starttoe script when the interface has been brought up

To be able to run this script after the interface has been brought up the following line have been add to the configuration of interface eth0 in /etc/network/interfaces below iface eth0 inet static:

Add the 3w-9xxx SATA loadable module to the kernel

The Linux 2.6.6 kernel does not have the 3w-9xxx SATA loadable module included in the kernel sources. Therefore, the module should be compiled from sources and loaded into the kernel. The driver source can be downloaded from http://www.3ware.com/support/download.asp∞.

After downloading unpack the sources and build the loadable module:


A kernel object file 3w-9xxx.ko should have been created at this point. It will be loaded into the kernel with the following commands:


where:

• depmod -a determines the interpendencies between the modules.
• modprobe 3w-9xxx inserts the module into the kernel, but only when this is possible.

Mount finally the RAID partition:


After a reboot the 3w-9xxx module is in available to be loaded into the kernel, when needed. No special commands should be given anymore.