vitalif
/
mtcp
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
mtcp/README.netmap

196 lines
7.8 KiB
Plaintext
Raw Permalink Blame History

- NETMAP VERSION -
------------------
-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-
-A- Tested with linux-3.13.0-121-generic & netmap commit: 8f3f79e4ca168c6d3220e7a5198a5778b030445e,
and,
-B- Tested with linux-3.13.0-121-generic & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a,
and,
-C- Tested with 4.11.12-200.fc25.x86_64+debug & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a
-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-
FOR -A-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-3.15.1/ driver
- please go through netmap's documentation for installation
instructions. We used the following command to set the compilation
scripts for netmap (for ixgbe-3.15.1 driver).
# ./configure --kernel-dir=/path/to/kernel/src --no-drivers=i40e,virtio_net.c
- To run mTCP clients correctly, you need to modify the RSS
seed in ixgbe_main.c:ixgbe_setup_mrqc() function. Our mTCP stack
uses a specific RSS seed (mentioned below).
- seed[10] should be reset to {
0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505
};
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./ixgbe/ixgbe.ko
FOR -B-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-5.3.7/ driver
- please go through netmap's documentaiton for installation
instructions. We used the following command(s) to compile the driver
# cd LINUX; ./configure; make
- To run mTCP clients correctly, you need to modify the RSS seed in
ixgbe_main.c:ixgbe_init_rss_key() function. Our mTCP stack uses a specific
RSS seed (mentioned below).
static inline int ixgbe_init_rss_key(struct ixgbe_adapter *adapter)
{
#if 0
u32 *rss_key;
if (!adapter->rss_key) {
rss_key = kzalloc(IXGBE_RSS_KEY_SIZE, GFP_KERNEL);
if (unlikely(!rss_key))
return -ENOMEM;
netdev_rss_key_fill(rss_key, IXGBE_RSS_KEY_SIZE);
adapter->rss_key = rss_key;
}
#endif
static uint32_t sym_rss_key[10] = {
0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505
};
if (!adapter->rss_key)
adapter->rss_key = sym_rss_key;
return 0;
}
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./ixgbe/ixgbe.ko
For -C-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding i40e-2.4.6/ driver
- please go through netmap's documentation for installation
instructions. We used the following command(s) to compile the driver
# cd LINIX; ./configure; make
- To run mTCP clients correctly, you need to modify the RSS seed in
i40e_main.c:i40e_config_rss_aq() & i40e_main.c:i40e_config_rss_reg()
functions. Our mTCP stack uses a specific RSS seed (mentioned below).
static unsigned char sym_seed[] = {
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05
};
static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
u8 *lut, u16 lut_size)
{
struct i40e_pf *pf = vsi->back;
struct i40e_hw *hw = &pf->hw;
int ret = 0;
seed = sym_seed;
...
}
static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
u8 *lut, u16 lut_size)
{
struct i40e_pf *pf = vsi->back;
struct i40e_hw *hw = &pf->hw;
int ret = 0;
seed = sym_seed;
...
}
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./i40e/i40e.ko
FOR BOTH -A- & -B- & -C- (CONTINUED)
--------------------------------------------------------------------------------------------------
2. For optimum performance you are suggested to bind NICS IRQs to arbitrary
CPUs. Please use affinity-netmap.py script for this purpose. The current
script is setup for the netmap ixgbe-5.3.7 driver. Please use a variant
of this file for other cases (igb, i40e etc.).
# ./config/affinity-netmap.py ${IFACE}
- Disable flow control in Ethernet layer
# sudo ethtool -A ${IFACE} rx off
# sudo ethtool -A ${IFACE} tx off
- Disable lro (large receive offload) in Ethernet device. mTCP
does not support large packet sizes (> 1514B) yet)
# sudo ethtool -K ${IFACE} lro off
# sudo ethtool -K ${IFACE} gro off
- We used example/pktgen to test netmap raw network I/O
performance. Netmap's pktgen can be used not only for
packet generation but also packet reception. Since mTCP
relies on RSS-based NIC hardware queues, we recommend using
the following command-line arguments to test pkt-gen as a
sink before testing mTCP for netmap.
-SINK- (assuming the machine has 4 cpus)
# sudo ./pkt-gen -i ${IFACE}-0 -f rx -c 1 -a 0 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-1 -f rx -c 1 -a 1 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-2 -f rx -c 1 -a 2 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-3 -f rx -c 1 -a 3 -b 64 &
where ${IFACE} is netmap-enabled interface.
The netmap README file shows a concise description on how to
use the driver. We reiterate some points that are essential in
understanding the command line arguments above. An interface
name post-appended with a number means that the process will
read traffic from the specified NIC hardware queue. `-a` argument
lets the program bind to a specific core.
3. Setup mtcp library:
# ./configure --enable-netmap
# make
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
# ./configure --enable-netmap CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-2 (configure again):
# autoreconf -ivf
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
4. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
5. Run the applications. *If you run the application with one thread,
mTCP core will assume that the multi-queues option is disabled. This
assumption is only valid for netmap version.*
6. Netmap module (mtcp/src/netmap_module.c) by default uses blocking
I/O by default. Most microbenchmarking applications (epserver/epwget)
shows best performance with this setup in our testbed. In case the
performance is sub-optimal in yours, we recommend that you try polling
mode (by enabling CONST_POLLING in line 24). You can also try tweaking
IDLE_POLL_WAIT/IDLE_POLL_COUNT macros while testing blocking mode I/O.
Reference in New Issue View Git Blame Copy Permalink