Updated readme about dpdk submodule setup and changed permissions for setup scripts

master
EunYoung Jeong 2018-11-12 14:31:26 +09:00
parent 8fa46be977
commit e929e4ee14
5 changed files with 212 additions and 172 deletions

14
.gitignore vendored
View File

@ -1,3 +1,15 @@
# mtcp-specific files
Makefile
autom4te.cache/
config.h
config.log
config.status
dpdk-iface-kmod/dpdk_iface_main
io_engine/lib/Makefile
mtcp/src/Makefile
util/Makefile
stamp-h1
# Prerequisites
*.d
@ -49,4 +61,4 @@
modules.order
Module.symvers
Mkfile.old
dkms.conf
dkms.conf

366
README.md
View File

@ -1,24 +1,24 @@
[![Build Status](https://travis-ci.org/eunyoung14/mtcp.svg?branch=master)](https://travis-ci.org/eunyoung14/mtcp)
[![Build Status](https://scan.coverity.com/projects/11896/badge.svg)](https://scan.coverity.com/projects/eunyoung14-mtcp)
# README #
# README
mTCP is a highly scalable user-level TCP stack for multicore systems.
mTCP source code is distributed under the Modified BSD License. For
more detail, please refer to the LICENSE. The license term of io_engine
driver and ported applications may differ from the mTCPs.
### PREREQUISITE ###
## Prerequisites
We require the following libraries to run mTCP.
- ``libps`` (PacketShader I/O engine library) OR ``libdpdk`` (Intel's DPDK package*) or ``netmap`` driver
- ``libnuma``
- ``libpthread``
- ``librt``
- ``libgmp`` (for DPDK/ONVM driver)
- `libdpdk` (Intel's DPDK package*) or `libps` (PacketShader I/O engine library) or `netmap` driver
- `libnuma`
- `libpthread`
- `librt`
- `libgmp` (for DPDK/ONVM driver)
Compling PSIO/DPDK/NETMAP/ONVM driver requires kernel headers.
- For Debian/Ubuntu, try ``apt-get install linux-headers-$(uname -r)``
- For Debian/Ubuntu, try ``apt-get install linux-headers-$(uname -r)``
We have modified the dpdk package to export net_device stat data
(for Intel-based Ethernet adapters only) to the OS. To achieve this, we have
@ -27,7 +27,7 @@ created a new LKM dpdk-iface-kmow. We also modified
process of mTCP applications. We recommend using our package for DPDK
installation.
### INCLUDED DIRECTORIES ###
## Included directories
mtcp: mtcp source code directory
- mtcp/src: source code
@ -54,140 +54,154 @@ util: useful source code for applications
config: sample mTCP configuration files (may not be necessary)
### INSTALL GUIDES ###
## Install guides
mTCP can be prepared in three ways.
mTCP can be prepared in four ways.
***PSIO VERSION***
### ***DPDK VERSION***
1. make in io_engine/driver:
```# make```
- check ps_ixgbe.ko
- please note that psio only runs on linux-2.6.x kernels
(linux-2.6.32 ~ linux-2.6.38)
2. install the driver:
```# ./install.py <# cores> <# cores>```
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
```bash
# ./configure --with-psio-lib=<$path_to_ioengine>
## e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
# make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 8-core system, by using the following command:
```bash
# ./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-3 (configure again):
1. Download DPDK submodule.
```bash
# autoreconf -ivf
git submodule init
git submodule update
```
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
2. Setup DPDK.
4. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
```bash
./setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
```
5. Run the applications!
- Press [14] to compile x86_64-native-linuxapp-gcc version
- Press [17] to install the driver
- Press [21] to setup 2048 2MB hugepages
- Press [23] to register the Ethernet ports
- Press [34] to quit the tool
- Only those devices will work with DPDK drivers that are listed
on this page: http://dpdk.org/doc/nics. Please make sure that your
NIC is compatible before moving on to the next step.
***DPDK VERSION***
- We use `dpdk/` submodule as our DPDK driver. FYI, you can pass a different
dpdk source directory as command line argument.
1. Set up DPDK first.
```bash
# bash setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
```
Press [14] to compile x86_64-native-linuxapp-gcc version
Press [17] to install the driver
Press [21] to setup 2048 2MB hugepages
Press [23] to register the Ethernet ports
Press [34] to quit the tool
- Only those devices will work with DPDK drivers that are listed
on this page: http://dpdk.org/doc/nics. Please make sure that your
NIC is compatible before moving on to the next step.
- We use dpdk-18.02/ as our DPDK driver. FYI, you can pass a different
dpdk source directory as command line argument.
2. Next bring the dpdk-registered interfaces up, and then set RTE_SDK
3. Next bring the dpdk-registered interfaces up, and then set RTE_SDK
and RTE_TARGET environment variables.
```bash
# sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
# export RTE_SDK=`echo $PWD`/dpdk
# export RTE_TARGET=x86_64-native-linuxapp-gcc
sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
export RTE_SDK=`echo $PWD`/dpdk
export RTE_TARGET=x86_64-native-linuxapp-gcc
```
3. Setup mtcp library:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET
# make
4. Setup mtcp library:
```bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET
make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=32"
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
```bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=32"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-4 (configure again):
```# autoreconf -ivf```
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum``` to disable checksum offloading.
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
- In case `./configure` script prints an error, run the
following command; and then re-do step-4 (configure again):
```bash
autoreconf -ivf
```
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum``` to disable checksum offloading.
- check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
4. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
5. Check the configurations in `apps/example`
- `epserver.conf` for server-side configuration
- `epwget.conf` for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
7. You can revert back all your changes by running the following script.
```bash
./setup_linux_env.sh [<path to $RTE_SDK>]
```
- Press [29] to unbind the Ethernet ports
- Press [30] to remove igb_uio.ko driver
- Press [33] to remove hugepage mappings
- Press [34] to quit the tool
### ***PSIO VERSION***
1. make in io_engine/driver:
```bash
make
```
- check ps_ixgbe.ko
- please note that psio only runs on linux-2.6.x kernels
(linux-2.6.32 ~ linux-2.6.38)
2. install the driver:
```bash
./install.py <# cores> <# cores>
```
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
```bash
./configure --with-psio-lib=<$path_to_ioengine>
# e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 8-core system, by using the following command:
```bash
./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure` script prints an error, run the
following command; and then re-do step-3 (configure again):
```bash
autoreconf -ivf
```
- check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
4. Check the configurations in `apps/example`
- `epserver.conf` for server-side configuration
- `epwget.conf` for client-side configuration
- you may write your own configuration file for your application
5. Run the applications!
6. You can revert back all your changes by running the following script.
```bash
# bash setup_linux_env.sh [<path to $RTE_SDK]]
```
Press [29] to unbind the Ethernet ports
Press [30] to remove igb_uio.ko driver
Press [33] to remove hugepage mappings
Press [34] to quit the tool
***ONVM VERSION***
### ***ONVM VERSION***
***NEW***: Now you can run mTCP applications (server + client) locally.
A local setup is useful when only 1 machine is available for the experiment.
@ -201,88 +215,93 @@ ONVM basics are explained in https://github.com/sdnfv/openNetVM.
2. Next bring the dpdk-registered interfaces up. This can be setup using:
```# sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up```
```bash
sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
```
3. Setup mtcp library
```bash
# ./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib>
# e.g. ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=`echo $ONVM_HOME`/onvm
# make
./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib>
# e.g. ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=`echo $ONVM_HOME`/onvm
make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32"
```
```bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
- In case `./configure` script prints an error, run the
following command; and then re-do step-4 (configure again):
```# autoreconf -ivf```
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> --disable-hwcsum``` to disable checksum offloading.
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
```bash
autoreconf -ivf
```
5. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> --disable-hwcsum``` to disable checksum offloading.
- check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
5. Check the configurations in `apps/example`
- `epserver.conf` for server-side configuration
- `epwget.conf` for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
7. You can revert back all your changes by running the following script.
```bash
# bash setup_linux_env.sh
```
```bash
./setup_linux_env.sh
```
Press [29] to unbind the Ethernet ports
Press [30] to remove igb_uio.ko driver
Press [33] to remove hugepage mappings
Press [34] to quit the tool
- Press [29] to unbind the Ethernet ports
- Press [30] to remove igb_uio.ko driver
- Press [33] to remove hugepage mappings
- Press [34] to quit the tool
**Notes**
Once you have started onvm_mgr, sometimes an mTCP application may fail to get launched due
to an error resembling the one mentioned below:
(```EAL: FATAL: Cannot init memory```, or
``` Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option```, or
```EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000```)
- ```EAL: FATAL: Cannot init memory```
- ``` Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option```
- ```EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000```
To prevent this, use the base virtual address parameter to run the ONVM manager, e.g.:
```
```bash
cd openNetVM/onvm
./go.sh 1,2,3 1 -s stdout -v 0x7f000000000
```
***NETMAP VERSION***
### ***NETMAP VERSION***
See README.netmap for details.
***TESTED ENVIRONMENTS***
## Tested environments
mTCP runs on Linux-based operating systems (2.6.x for PSIO) with generic
x86_64 CPUs, but to help evaluation, we provide our tested environments
as follows.
Intel Xeon E5-2690 octacore CPU @ 2.90 GHz 32 GB of RAM (4 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Debian 6.0.7 (Linux 2.6.32-5-amd64)
Intel Xeon E5-2690 octacore CPU @ 2.90 GHz 32 GB of RAM (4 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Debian 6.0.7 (Linux 2.6.32-5-amd64)
Intel Core i7-3770 quadcore CPU @ 3.40 GHz 16 GB of RAM (2 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Ubuntu 10.04 (Linux 2.6.32-47)
Intel Core i7-3770 quadcore CPU @ 3.40 GHz 16 GB of RAM (2 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Ubuntu 10.04 (Linux 2.6.32-47)
Event-driven PacketShader I/O engine (extended io_engine-0.2)
@ -290,7 +309,7 @@ Event-driven PacketShader I/O engine (extended io_engine-0.2)
We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
***NOTES***
## Notes
1. mTCP currently runs with fixed memory pools. That means, the size of
TCP receive and send buffers are fixed at the startup and does not
@ -310,30 +329,30 @@ We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
5. mTCP has been tested with the following Ethernet adapters:
1. Intel-82598 ixgbe (Max-queue-limit: 16)
2. Intel-82599 ixgbe (Max-queue-limit: 16)
3. Intel-I350 igb (Max-queue-limit: 08)
4. Intel-X710 i40e (Max-queue-limit: ~)
5. Intel-X722 i40e (Max-queue-limit: ~)
1. Intel-82598 ixgbe (Max-queue-limit: 16)
2. Intel-82599 ixgbe (Max-queue-limit: 16)
3. Intel-I350 igb (Max-queue-limit: 08)
4. Intel-X710 i40e (Max-queue-limit: ~)
5. Intel-X722 i40e (Max-queue-limit: ~)
***FREQUENTLY ASKED QUESTIONS***
## Frequently asked questions
1. How can I quit the application?
- Use ^C to gracefully shutdown the application. Two consecutive
- Use ^C to gracefully shutdown the application. Two consecutive
^C (separated by 1 sec) will force quit.
2. My application keeps printing "No route to 0.0.0.0"
- Try to turn off your network-manager for xge*. The network manager
- Try to turn off your network-manager for xge*. The network manager
can override the IP configuration set by install.py in PSIO driver.
3. Can I statically set the routing or arp table?
- Yes, mTCP allows static route and arp configuration. Go to the
- Yes, mTCP allows static route and arp configuration. Go to the
config directory and see sample_route.conf or sample_arp.conf.
Copy and adapt it to your condition and link (ln -s) the config
directory to the application directory. mTCP will find
config/route.conf and config/arp.conf for static configuration.
***CAUTION***
## Caution
1. Do not remove I/O driver (```ps_ixgbe/igb_uio```) while running mTCP
applications. The application will panic!
@ -341,7 +360,12 @@ We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
2. Use the ps_ixgbe/dpdk driver contained in this package, not the one
from some other place (e.g., from io_engine github).
Contact: mtcp-user at list.ndsl.kaist.edu
April 2, 2015.
EunYoung Jeong <notav at ndsl.kaist.edu>
M. Asim Jamshed <ajamshed at ndsl.kaist.edu>
## Contacts
GitHub issue board is the preferred way to report bugs and ask questions about mTCP.
***CONTACTS FOR THE AUTHORS***
User mailing list <mtcp-user at list.ndsl.kaist.edu>
EunYoung Jeong <notav at ndsl.kaist.edu>
M. Asim Jamshed <ajamshed at ndsl.kaist.edu>

4
apps/example/.gitignore vendored Normal file
View File

@ -0,0 +1,4 @@
Makefile
epserver
epwget
log_*

0
setup_linux_env.sh Normal file → Executable file
View File

0
setup_mtcp_dpdk_env.sh Normal file → Executable file
View File