When TRex Walked the Earth

How to reach 100Gbps with open-source traffic generator

5 min readJan 10, 2022

We all had to make changes and adopt new ways of working.
The latest year gave us some perspective about importance of having a stable and reliable connection, either from home or in the office, it seems that we never have each capacity. The demand for high speed connection is contently rising. “Lightspeed” is not enough, and we need more.
If your product related to network performance, routing or security you had to use traffic generator tool to test your environment and products.

There are very good tools in the market to generate network traffic and packets customization. Ixia, keysight and Sprient are the most common one.
But you will have to assign a significant budget for using them.
The alternative is the open source tool called TRex.

What is TRex?

Realistic Traffic Generator , AKA TRex is an open-source traffic generator, developed by Cisco.
The tool works on standard Intel DPDK-based processors and supports both stateful and stateless modes.
It is a relatively simple to use it and it can give a lot of flexibility to generate different types of traffic or replay a pre recorded .pcap file.

More information could be found in TRex official page.
The documents and product descriptions are really good and provides a lot of detailed information to quickly start generating traffic and build a basic testing setup.

The statefull setup, TRes will work as Client and Server and could stream Tx (transmit) traffic from one interface and Rx (receive) from another interface.
While in the stateless mode traffic would be stream on one direction to provide maximum load towards the DUT (Device Under Test).

HW needed

We have use the following equipment’s:

Mellanox cards were installed in the Dell Server (used for the TRex application), connected with the 100Gb cables to the UfiSpace router that was used as DUT.

SW versions

I can say it was not an easy task.
I had to install several different Linux OS, TRex versions and MLNX_OFED drivers to make this setup working.

At the end this is the “combination” that works for us:

CentOS 7.9.2009
with Kernel version: Linux 3.10.0–1160.el7.x86_64
Mellanox OFED driver version:
MLNX_OFED_LINUX-5.2–1.0.4.0 (OFED-5.2–1.0.4)
TRex v2.88

Important note:

Make sure Mellanox cards are identified by the server BIOS. In our case we had to upgrade the BIOS to support the cards.

Mellanox ConnectX-5 adapter family supports 100/56/40/25/10 Gb/s Ethernet speeds. Its DPDK support is a bit different from Intel DPDK support. Intel NICs do not require additional kernel drivers (except for igb_uio which is already supported in most distributions).
ConnectX-5 works on top of Infiniband API (verbs) and requires special kernel modules/user space libs. This means that it is required to install OFED package to be able to work with this NIC.
Installing the full OFED package is the simplest way to make it work.

Installation Process

It is important to follow the exact commands to make the setup working.
Missing any of the flags could have unrequired results, like:
links were not fully synchronized on 100G Full duplex,
TRex refuse to start and reports about “strange” errors:

EAL: so/x86_64/libmlx5-64.so: symbol mlx5dv_dr_action_create_aso, version MLX5_OFED not defined in file libmlx5.so.1 with link time reference
   EAL: FATAL: Cannot init plugins
   EAL: Cannot init plugins
   You might need to run ./trex-cfg  once
   EAL: Error - exiting with code: 1
   Cause: Invalid EAL arguments

Other many other issues was encounter. Different Linux OS or virtualization management (KVM, ESXI) did not worked out of the box.
At the end we decided to install the recommended CentOS on the box and not the latest version.
(too much time spent on trial and error)

So, here is when we have done:

Download and install CentOS 7.9 minimal ios file.
Add the following packages:

$ sudo yum install pciutils
$ sudo yum group install “Development Tools”
$ sudo yum install lsof tcl tk
$ sudo yum install -y python3

3. Download and install the MLX5_OFED driver:
MLNX_OFED_LINUX-5.2–1.0.4.0-rhel7.9-x86_64.tgz

4. Run the MLNX_OFED installation scripts.
Make sure to use the exact flags

$ sudo ./mlnxofedinstall — with-mft — with-mstflint — dpdk — upstream-libs

When OFED installation finished, run the following commands and reboot:

$ sudo /etc/init.d/openibd restart
 
$ sudo mlxfwreset -d mlx5_0 — yes r
$ sudo mlxfwreset -d mlx5_1 — yes r
 
$ sudo reboot

5. Download and install TRex application with version v2.88:
https://trex-tgn.cisco.com/trex/release/v2.88.tar.gz

You will have to untar the file to specific directory.
At this point you can run the script to define the relevant ports to be allocated for TRex.

Use the ./dpdk_setup_ports.py -i script to define the ports.
The result could be saved to the main configuration file under /etc/trex_config.yaml

the PCI address of each port will be used to identify the allocated port for TRex. In our case we set mlx5_core driver with 3b:00.0 and 3b:00.1

More reference about the installation process from Mellanox could be found here.

6. Run TRex as a server.
After this section you should be able to start loading traffic through the ports

$ sudo ./t-rex-64 --no-ofed-check -i -c 14Note: * -c indicate the amount of CPU and threads you will be able to use
* --no-ofed-check could be used incase there are errors from MLX driver check

it is possible to connect to TRex via the console or GUI (the GUI will work only on v2.87 for the moment, but the console provides enough tools to use.

run ./trex-console from another terminal and us the start command to execute the required test.

Note the ./stl directory contains different types of scripts to execute stateless traffic trough the ports. It is bases on Scrapy engine (a Python module) to create the packet structure.
The -m option set the desired bandwidth to be used.

trex>start -f stl/imix.py -m 100gbps

Whishing you a lot of success with generating traffic with this great tool.