vitalif
/
mtcp
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Updated readme about dpdk submodule setup and changed permissions for setup scripts

master
EunYoung Jeong 2018-11-12 14:31:26 +09:00
parent 8fa46be977
commit e929e4ee14
5 changed files with 212 additions and 172 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
.gitignore vendored
View File

@ -1,3 +1,15 @@
# mtcp-specific files
Makefile
autom4te.cache/
config.h
config.log
config.status
dpdk-iface-kmod/dpdk_iface_main
io_engine/lib/Makefile
mtcp/src/Makefile
util/Makefile
stamp-h1
# Prerequisites # Prerequisites
*.d *.d
@ -49,4 +61,4 @@
modules.order modules.order
Module.symvers Module.symvers
Mkfile.old Mkfile.old
dkms.conf dkms.conf

366
README.md
View File

@ -1,24 +1,24 @@
[![Build Status](https://travis-ci.org/eunyoung14/mtcp.svg?branch=master)](https://travis-ci.org/eunyoung14/mtcp) [![Build Status](https://travis-ci.org/eunyoung14/mtcp.svg?branch=master)](https://travis-ci.org/eunyoung14/mtcp)
[![Build Status](https://scan.coverity.com/projects/11896/badge.svg)](https://scan.coverity.com/projects/eunyoung14-mtcp) [![Build Status](https://scan.coverity.com/projects/11896/badge.svg)](https://scan.coverity.com/projects/eunyoung14-mtcp)
# README # # README
mTCP is a highly scalable user-level TCP stack for multicore systems. mTCP is a highly scalable user-level TCP stack for multicore systems.
mTCP source code is distributed under the Modified BSD License. For mTCP source code is distributed under the Modified BSD License. For
more detail, please refer to the LICENSE. The license term of io_engine more detail, please refer to the LICENSE. The license term of io_engine
driver and ported applications may differ from the mTCP’s. driver and ported applications may differ from the mTCP’s.
### PREREQUISITE ### ## Prerequisites
We require the following libraries to run mTCP. We require the following libraries to run mTCP.
- ``libps`` (PacketShader I/O engine library) OR ``libdpdk`` (Intel's DPDK package*) or ``netmap`` driver - `libdpdk` (Intel's DPDK package*) or `libps` (PacketShader I/O engine library) or `netmap` driver
- ``libnuma`` - `libnuma`
- ``libpthread`` - `libpthread`
- ``librt`` - `librt`
- ``libgmp`` (for DPDK/ONVM driver) - `libgmp` (for DPDK/ONVM driver)
Compling PSIO/DPDK/NETMAP/ONVM driver requires kernel headers. Compling PSIO/DPDK/NETMAP/ONVM driver requires kernel headers.
- For Debian/Ubuntu, try ``apt-get install linux-headers-$(uname -r)`` - For Debian/Ubuntu, try ``apt-get install linux-headers-$(uname -r)``
We have modified the dpdk package to export net_device stat data We have modified the dpdk package to export net_device stat data
(for Intel-based Ethernet adapters only) to the OS. To achieve this, we have (for Intel-based Ethernet adapters only) to the OS. To achieve this, we have
@ -27,7 +27,7 @@ created a new LKM dpdk-iface-kmow. We also modified
process of mTCP applications. We recommend using our package for DPDK process of mTCP applications. We recommend using our package for DPDK
installation. installation.
### INCLUDED DIRECTORIES ### ## Included directories
mtcp: mtcp source code directory mtcp: mtcp source code directory
- mtcp/src: source code - mtcp/src: source code
@ -54,140 +54,154 @@ util: useful source code for applications
config: sample mTCP configuration files (may not be necessary) config: sample mTCP configuration files (may not be necessary)
### INSTALL GUIDES ### ## Install guides
mTCP can be prepared in three ways. mTCP can be prepared in four ways.
***PSIO VERSION*** ### ***DPDK VERSION***
1. make in io_engine/driver: 1. Download DPDK submodule.
```# make```
- check ps_ixgbe.ko
- please note that psio only runs on linux-2.6.x kernels
(linux-2.6.32 ~ linux-2.6.38)
2. install the driver:
```# ./install.py <# cores> <# cores>```
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
```bash
# ./configure --with-psio-lib=<$path_to_ioengine>
## e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
# make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 8-core system, by using the following command:
```bash
# ./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-3 (configure again):
```bash ```bash
# autoreconf -ivf git submodule init
git submodule update
``` ```
- check libmtcp.a in mtcp/lib 2. Setup DPDK.
- check header files in mtcp/include
- check example binary files in apps/example
4. Check the configurations in apps/example ```bash
- epserver.conf for server-side configuration ./setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
- epwget.conf for client-side configuration ```
- you may write your own configuration file for your application
5. Run the applications! - Press [14] to compile x86_64-native-linuxapp-gcc version
- Press [17] to install the driver
- Press [21] to setup 2048 2MB hugepages
- Press [23] to register the Ethernet ports
- Press [34] to quit the tool
- Only those devices will work with DPDK drivers that are listed
on this page: http://dpdk.org/doc/nics. Please make sure that your
NIC is compatible before moving on to the next step.
***DPDK VERSION*** - We use `dpdk/` submodule as our DPDK driver. FYI, you can pass a different
dpdk source directory as command line argument.
1. Set up DPDK first. 3. Next bring the dpdk-registered interfaces up, and then set RTE_SDK
```bash
# bash setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
```
Press [14] to compile x86_64-native-linuxapp-gcc version
Press [17] to install the driver
Press [21] to setup 2048 2MB hugepages
Press [23] to register the Ethernet ports
Press [34] to quit the tool
- Only those devices will work with DPDK drivers that are listed
on this page: http://dpdk.org/doc/nics. Please make sure that your
NIC is compatible before moving on to the next step.
- We use dpdk-18.02/ as our DPDK driver. FYI, you can pass a different
dpdk source directory as command line argument.
2. Next bring the dpdk-registered interfaces up, and then set RTE_SDK
and RTE_TARGET environment variables. and RTE_TARGET environment variables.
```bash ```bash
# sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
# export RTE_SDK=`echo $PWD`/dpdk export RTE_SDK=`echo $PWD`/dpdk
# export RTE_TARGET=x86_64-native-linuxapp-gcc export RTE_TARGET=x86_64-native-linuxapp-gcc
``` ```
3. Setup mtcp library: 4. Setup mtcp library:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET ```bash
# make ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET
make
``` ```
- By default, mTCP assumes that there are 16 CPUs in your system. - By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command: You can set the CPU limit, e.g. on a 32-core system, by using the following command:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=32" ```bash
``` ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=32"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding). (since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the - In case `./configure` script prints an error, run the
following command; and then re-do step-4 (configure again): following command; and then re-do step-4 (configure again):
```bash
```# autoreconf -ivf``` autoreconf -ivf
- checksum offloading in the NIC is now ENABLED (by default)!!! ```
- this only works for dpdk at the moment
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum``` to disable checksum offloading. - checksum offloading in the NIC is now ENABLED (by default)!!!
- check libmtcp.a in mtcp/lib - this only works for dpdk at the moment
- check header files in mtcp/include - use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum``` to disable checksum offloading.
- check example binary files in apps/example - check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
4. Check the configurations in apps/example 5. Check the configurations in `apps/example`
- epserver.conf for server-side configuration - `epserver.conf` for server-side configuration
- epwget.conf for client-side configuration - `epwget.conf` for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
7. You can revert back all your changes by running the following script.
```bash
./setup_linux_env.sh [<path to $RTE_SDK>]
```
- Press [29] to unbind the Ethernet ports
- Press [30] to remove igb_uio.ko driver
- Press [33] to remove hugepage mappings
- Press [34] to quit the tool
### ***PSIO VERSION***
1. make in io_engine/driver:
```bash
make
```
- check ps_ixgbe.ko
- please note that psio only runs on linux-2.6.x kernels
(linux-2.6.32 ~ linux-2.6.38)
2. install the driver:
```bash
./install.py <# cores> <# cores>
```
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
```bash
./configure --with-psio-lib=<$path_to_ioengine>
# e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
make
```
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 8-core system, by using the following command:
```bash
./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure` script prints an error, run the
following command; and then re-do step-3 (configure again):
```bash
autoreconf -ivf
```
- check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
4. Check the configurations in `apps/example`
- `epserver.conf` for server-side configuration
- `epwget.conf` for client-side configuration
- you may write your own configuration file for your application - you may write your own configuration file for your application
5. Run the applications! 5. Run the applications!
6. You can revert back all your changes by running the following script.
```bash ### ***ONVM VERSION***
# bash setup_linux_env.sh [<path to $RTE_SDK]]
```
Press [29] to unbind the Ethernet ports
Press [30] to remove igb_uio.ko driver
Press [33] to remove hugepage mappings
Press [34] to quit the tool
***ONVM VERSION***
***NEW***: Now you can run mTCP applications (server + client) locally. ***NEW***: Now you can run mTCP applications (server + client) locally.
A local setup is useful when only 1 machine is available for the experiment. A local setup is useful when only 1 machine is available for the experiment.
@ -201,88 +215,93 @@ ONVM basics are explained in https://github.com/sdnfv/openNetVM.
2. Next bring the dpdk-registered interfaces up. This can be setup using: 2. Next bring the dpdk-registered interfaces up. This can be setup using:
```# sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up``` ```bash
sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
```
3. Setup mtcp library 3. Setup mtcp library
```bash ```bash
# ./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib> ./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib>
# e.g. ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=`echo $ONVM_HOME`/onvm # e.g. ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=`echo $ONVM_HOME`/onvm
# make make
``` ```
- By default, mTCP assumes that there are 16 CPUs in your system. - By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command: You can set the CPU limit, e.g. on a 32-core system, by using the following command:
```bash
# ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32" ```bash
``` ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32"
```
Please note that your NIC should support RSS queues equal to the MAX_CPUS value Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding). (since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the - In case `./configure` script prints an error, run the
following command; and then re-do step-4 (configure again): following command; and then re-do step-4 (configure again):
```# autoreconf -ivf``` ```bash
- checksum offloading in the NIC is now ENABLED (by default)!!! autoreconf -ivf
- this only works for dpdk at the moment ```
- use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> --disable-hwcsum``` to disable checksum offloading.
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
5. Check the configurations in apps/example - checksum offloading in the NIC is now ENABLED (by default)!!!
- epserver.conf for server-side configuration - this only works for dpdk at the moment
- epwget.conf for client-side configuration - use ```./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> --disable-hwcsum``` to disable checksum offloading.
- check `libmtcp.a` in `mtcp/lib`
- check header files in `mtcp/include`
- check example binary files in `apps/example`
5. Check the configurations in `apps/example`
- `epserver.conf` for server-side configuration
- `epwget.conf` for client-side configuration
- you may write your own configuration file for your application - you may write your own configuration file for your application
6. Run the applications! 6. Run the applications!
7. You can revert back all your changes by running the following script. 7. You can revert back all your changes by running the following script.
```bash ```bash
# bash setup_linux_env.sh ./setup_linux_env.sh
``` ```
Press [29] to unbind the Ethernet ports - Press [29] to unbind the Ethernet ports
- Press [30] to remove igb_uio.ko driver
Press [30] to remove igb_uio.ko driver - Press [33] to remove hugepage mappings
- Press [34] to quit the tool
Press [33] to remove hugepage mappings
Press [34] to quit the tool
**Notes** **Notes**
Once you have started onvm_mgr, sometimes an mTCP application may fail to get launched due Once you have started onvm_mgr, sometimes an mTCP application may fail to get launched due
to an error resembling the one mentioned below: to an error resembling the one mentioned below:
(```EAL: FATAL: Cannot init memory```, or - ```EAL: FATAL: Cannot init memory```
``` Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option```, or - ``` Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option```
```EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000```) - ```EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000```
To prevent this, use the base virtual address parameter to run the ONVM manager, e.g.: To prevent this, use the base virtual address parameter to run the ONVM manager, e.g.:
``` ```bash
cd openNetVM/onvm cd openNetVM/onvm
./go.sh 1,2,3 1 -s stdout -v 0x7f000000000 ./go.sh 1,2,3 1 -s stdout -v 0x7f000000000
``` ```
***NETMAP VERSION*** ### ***NETMAP VERSION***
See README.netmap for details. See README.netmap for details.
***TESTED ENVIRONMENTS***
## Tested environments
mTCP runs on Linux-based operating systems (2.6.x for PSIO) with generic mTCP runs on Linux-based operating systems (2.6.x for PSIO) with generic
x86_64 CPUs, but to help evaluation, we provide our tested environments x86_64 CPUs, but to help evaluation, we provide our tested environments
as follows. as follows.
Intel Xeon E5-2690 octacore CPU @ 2.90 GHz 32 GB of RAM (4 memory channels) Intel Xeon E5-2690 octacore CPU @ 2.90 GHz 32 GB of RAM (4 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2) 10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Debian 6.0.7 (Linux 2.6.32-5-amd64) Debian 6.0.7 (Linux 2.6.32-5-amd64)
Intel Core i7-3770 quadcore CPU @ 3.40 GHz 16 GB of RAM (2 memory channels) Intel Core i7-3770 quadcore CPU @ 3.40 GHz 16 GB of RAM (2 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2) 10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Ubuntu 10.04 (Linux 2.6.32-47) Ubuntu 10.04 (Linux 2.6.32-47)
Event-driven PacketShader I/O engine (extended io_engine-0.2) Event-driven PacketShader I/O engine (extended io_engine-0.2)
@ -290,7 +309,7 @@ Event-driven PacketShader I/O engine (extended io_engine-0.2)
We tested the DPDK version (polling driver) with Linux-3.13.0 kernel. We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
***NOTES*** ## Notes
1. mTCP currently runs with fixed memory pools. That means, the size of 1. mTCP currently runs with fixed memory pools. That means, the size of
TCP receive and send buffers are fixed at the startup and does not TCP receive and send buffers are fixed at the startup and does not
@ -310,30 +329,30 @@ We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
5. mTCP has been tested with the following Ethernet adapters: 5. mTCP has been tested with the following Ethernet adapters:
1. Intel-82598 ixgbe (Max-queue-limit: 16) 1. Intel-82598 ixgbe (Max-queue-limit: 16)
2. Intel-82599 ixgbe (Max-queue-limit: 16) 2. Intel-82599 ixgbe (Max-queue-limit: 16)
3. Intel-I350 igb (Max-queue-limit: 08) 3. Intel-I350 igb (Max-queue-limit: 08)
4. Intel-X710 i40e (Max-queue-limit: ~) 4. Intel-X710 i40e (Max-queue-limit: ~)
5. Intel-X722 i40e (Max-queue-limit: ~) 5. Intel-X722 i40e (Max-queue-limit: ~)
***FREQUENTLY ASKED QUESTIONS*** ## Frequently asked questions
1. How can I quit the application? 1. How can I quit the application?
- Use ^C to gracefully shutdown the application. Two consecutive - Use ^C to gracefully shutdown the application. Two consecutive
^C (separated by 1 sec) will force quit. ^C (separated by 1 sec) will force quit.
2. My application keeps printing "No route to 0.0.0.0" 2. My application keeps printing "No route to 0.0.0.0"
- Try to turn off your network-manager for xge*. The network manager - Try to turn off your network-manager for xge*. The network manager
can override the IP configuration set by install.py in PSIO driver. can override the IP configuration set by install.py in PSIO driver.
3. Can I statically set the routing or arp table? 3. Can I statically set the routing or arp table?
- Yes, mTCP allows static route and arp configuration. Go to the - Yes, mTCP allows static route and arp configuration. Go to the
config directory and see sample_route.conf or sample_arp.conf. config directory and see sample_route.conf or sample_arp.conf.
Copy and adapt it to your condition and link (ln -s) the config Copy and adapt it to your condition and link (ln -s) the config
directory to the application directory. mTCP will find directory to the application directory. mTCP will find
config/route.conf and config/arp.conf for static configuration. config/route.conf and config/arp.conf for static configuration.
***CAUTION*** ## Caution
1. Do not remove I/O driver (```ps_ixgbe/igb_uio```) while running mTCP 1. Do not remove I/O driver (```ps_ixgbe/igb_uio```) while running mTCP
applications. The application will panic! applications. The application will panic!
@ -341,7 +360,12 @@ We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
2. Use the ps_ixgbe/dpdk driver contained in this package, not the one 2. Use the ps_ixgbe/dpdk driver contained in this package, not the one
from some other place (e.g., from io_engine github). from some other place (e.g., from io_engine github).
Contact: mtcp-user at list.ndsl.kaist.edu ## Contacts
April 2, 2015.
EunYoung Jeong <notav at ndsl.kaist.edu> GitHub issue board is the preferred way to report bugs and ask questions about mTCP.
M. Asim Jamshed <ajamshed at ndsl.kaist.edu>
***CONTACTS FOR THE AUTHORS***
User mailing list <mtcp-user at list.ndsl.kaist.edu>
EunYoung Jeong <notav at ndsl.kaist.edu>
M. Asim Jamshed <ajamshed at ndsl.kaist.edu>

4
apps/example/.gitignore vendored Normal file
View File

@ -0,0 +1,4 @@
Makefile
epserver
epwget
log_*

0
setup_linux_env.sh Normal file → Executable file
View File

0
setup_mtcp_dpdk_env.sh Normal file → Executable file
View File