Merge pull request #1 from markhpc/wip-readme

README: Update readme
master
Mark Nelson 2019-08-15 17:54:43 -05:00 committed by GitHub
commit 3649f24ef8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 141 additions and 81 deletions

222
README.md
View File

@ -1,107 +1,167 @@
# Introduction
# Hotsauce S3 Benchmark Version 0.1
## This is a fork version of [wasabi-tech/se-benchmark](https://github.com/wasabi-tech/s3-benchmark), to make it support minio and ceph.
## Introduction
Example output:
hsbench is an S3 compatable benchmark originally based on [wasabi-tech/s3-benchmark](https://github.com/wasabi-tech/s3-benchmark).
While there are already several S3 compatable benchmark suites available, each has various tradeoffs. What seemed to be missing
was a simple yet extremely fast benchmark that could easily be scripted into orchestration frameworks to run tests at scale.
hsbench tries to fill that niche. The current release of hsbench is considered to be alpha level software and may contain bugs.
```
[wen@t s3-benchmark]$ . ceph.env
[wen@t s3-benchmark]$ s3-benchmark
Wasabi benchmark program v2.0
Parameters: url=http://s3.t.haodai.net, bucket=loadgen, region=us-east-1, duration=60, threads=1, loops=1, size=1M
Loop 1: PUT time 60.1 secs, objects = 297, speed = 4.9MB/sec, 4.9 operations/sec. Slowdowns = 0
Loop 1: GET time 7.8 secs, objects = 297, speed = 38.2MB/sec, 38.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 24.2 secs, 12.3 deletes/sec. Slowdowns = 0
[wen@t s3-benchmark]$ . miniostest.env
[wen@t s3-benchmark]$ s3-benchmark
Wasabi benchmark program v2.0
Parameters: url=http://minio-cluster.tt.haodai.net, bucket=loadgen, region=us-east-1, duration=60, threads=1, loops=1, size=1M
Loop 1: PUT time 60.0 secs, objects = 576, speed = 9.6MB/sec, 9.6 operations/sec. Slowdowns = 0
Loop 1: GET time 9.4 secs, objects = 576, speed = 61.3MB/sec, 61.3 operations/sec. Slowdowns = 0
Loop 1: DELETE time 4.2 secs, 138.4 deletes/sec. Slowdowns = 0
[wen@t s3-benchmark]$
[wen@t s3-benchmark]$ . ceph.env
[wen@t s3-benchmark]$ s3-benchmark -t 10
Wasabi benchmark program v2.0
Parameters: url=http://s3.t.haodai.net, bucket=loadgen, region=us-east-1, duration=60, threads=10, loops=1, size=1M
Loop 1: PUT time 60.9 secs, objects = 513, speed = 8.4MB/sec, 8.4 operations/sec. Slowdowns = 0
Loop 1: GET time 45.6 secs, objects = 5130, speed = 112.5MB/sec, 112.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 38.4 secs, 13.4 deletes/sec. Slowdowns = 0
[wen@t s3-benchmark]$
[wen@t s3-benchmark]$ . miniostest.env
[wen@t s3-benchmark]$ s3-benchmark -t 10
Wasabi benchmark program v2.0
Parameters: url=http://minio-cluster.tt.haodai.net, bucket=loadgen, region=us-east-1, duration=60, threads=10, loops=1, size=1M
Loop 1: PUT time 62.4 secs, objects = 832, speed = 13.3MB/sec, 13.3 operations/sec. Slowdowns = 0
Loop 1: GET time 60.1 secs, objects = 6774, speed = 112.8MB/sec, 112.8 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.4 secs, 578.1 deletes/sec. Slowdowns = 0
[wen@t s3-benchmark]$
```
s3-benchmark is a performance testing tool provided by Wasabi for performing S3 operations (PUT, GET, and DELETE) for objects. Besides the bucket configuration, the object size and number of threads varied be given for different tests.
## Features
The testing tool is loosely based on the Nasuni (http://www6.nasuni.com/rs/nasuni/images/Nasuni-2015-State-of-Cloud-Storage-Report.pdf) performance benchmarking methodologies used to test the performance of different cloud storage providers
hsbench tries to improve on the original Wasabi s3-benchmark in the following ways:
* Threads can distribute IOS across an arbitrary number of buckets.
* Tests can be run individually and externally coordinated across multiple clients.
* Intermediate results are logged periodically at user-defined intervals.
* Min/avg/max/percentile latency results are included.
* Test length can be limited either by duration or maximum number of objects.
* Object prefixes can be set to test large object names (12 bytes reserved for uniqueness)
* Bucket/Object prefixes can be used to allow multiple clients to target the same buckets
# Prerequisites
## Limitations
* hsbench does not yet log log/csv/json output (will be added pending community feedback)
* hsbench has no built-in provisions for making graphs
* hsbench does not yet support mixed IO workloads
* hsbench is still in alpha and options/output may change at any moment
## Prerequisites
To leverage this tool, the following prerequisites apply:
* Git development environment
* Ubuntu Linux shell programming skills
* Access to a Go 1.7 development system (only if the OS is not Ubuntu Linux 16.04)
* Access to the appropriate AWS EC2 (or equivalent) compute resource (optimal performance is realized using m4.10xlarge EC2 Ubuntu with 10 GB ENA)
* Go version 1.7 or newer
* Access to an S3 compatable storage service (AWS, Ceph, etc)
## Install
# Building the Program
Obtain a local copy of the repository using the following git command with any directory that is convenient:
The easiest way to install hsbench is to use go's built in github support:
```
git clone https://github.com/wasabi-tech/s3-benchmark.git
go get github.com/markhpc/hsbench
```
You should see the following files in the s3-benchmark directory.
LICENSE README.md s3-benchmark.go s3-benchmark.ubuntu
If the test is being run on Ubuntu version 16.04 LTS (the current long term release), the binary
executable s3-benchmark.ubuntu will run the benchmark testing without having to build the executable.
Otherwise, to build the s3-benchmark executable, you must issue this following command:
/usr/bin/go build s3-bechmark.go
# Command Line Arguments
Below are the command line arguments to the program (which can be displayed using -help):
Then in the hsbench src directory run 'go build':
```
$ pwd
/home/perf/go/src/github.com/markhpc/hsbench
$ go build
```
## Usage
```
$ ./hsbench --help
USAGE: ./hsbench [OPTIONS]
OPTIONS:
-a string
Access key
-b string
Bucket for testing (default "wasabi-benchmark-bucket")
Access key
-b int
Number of buckets to distribute IOs across (default 1)
-bp string
Prefix for buckets (default "hotsauce_bench")
-d int
Duration of each test in seconds (default 60)
Maximum test duration in seconds <-1 for unlimited> (default 60)
-l int
Number of times to repeat test (default 1)
Number of times to repeat test (default 1)
-m string
Run modes in order. See NOTES for more info (default "cxipgdx")
-n int
Maximum number of objects <-1 for unlimited> (default -1)
-op string
Prefix for objects
-r string
Region for testing (default "us-east-1")
-ri float
Number of seconds between report intervals (default 1)
-s string
Secret key
Secret key
-t int
Number of threads to run (default 1)
Number of threads to run (default 1)
-u string
URL for host with method prefix (default "http://s3.wasabisys.com")
URL for host with method prefix
-z string
Size of objects in bytes with postfix K, M, and G (default "1M")
```
Size of objects in bytes with postfix K, M, and G (default "1M")
# Example Benchmark
Below is an example run of the benchmark for 10 threads with the default 1MB object size. The benchmark reports
for each operation PUT, GET and DELETE the results in terms of data speed and operations per second. The program
writes all results to the log file benchmark.log.
NOTES:
- Valid mode types for the -m mode string are:
c: clear all existing objects from buckets (requires lookups)
x: delete buckets
i: initialize buckets
p: put objects in buckets
g: get objects from buckets
d: delete objects from buckets
```
ubuntu:~/s3-benchmark$ ./s3-benchmark.ubuntu -a MY-ACCESS-KEY -b jeff-s3-benchmark -s MY-SECRET-KEY -t 10
Wasabi benchmark program v2.0
Parameters: url=http://s3.wasabisys.com, bucket=jeff-s3-benchmark, duration=60, threads=10, loops=1, size=1M
Loop 1: PUT time 60.1 secs, objects = 5484, speed = 91.3MB/sec, 91.3 operations/sec.
Loop 1: GET time 60.1 secs, objects = 5483, speed = 91.3MB/sec, 91.3 operations/sec.
Loop 1: DELETE time 1.9 secs, 2923.4 deletes/sec.
Benchmark completed.
These modes are processed in-order and can be repeated, ie "ippgd" will
initialize the buckets, put the objects, reput the objects, get the
objects, and then delete the objects. The repeat flag will repeat this
whole process the specified number of times.
```
## Example Benchmark
Below is an example run of the benchmark using a 10s test duration, 10 threads,
10 buckets, and a 4K object size against a Ceph RadosGW backed by a single Ceph OSD
running on an Intel P3700 NVMe device.
```
$ ./hsbench -a <access key> -s <secret key> -u http://127.0.0.1:7480 -z 4K -d 10 -t 10 -b 10
2019/08/15 18:32:36 Hotsauce S3 Benchmark Version 0.1
2019/08/15 18:32:36 Parameters:
2019/08/15 18:32:36 url=http://127.0.0.1:7480
2019/08/15 18:32:36 object_prefix=
2019/08/15 18:32:36 bucket_prefix=hotsauce_bench
2019/08/15 18:32:36 region=us-east-1
2019/08/15 18:32:36 modes=cxipgdx
2019/08/15 18:32:36 object_count=-1
2019/08/15 18:32:36 bucket_count=10
2019/08/15 18:32:36 duration=10
2019/08/15 18:32:36 threads=10
2019/08/15 18:32:36 loops=1
2019/08/15 18:32:36 size=4K
2019/08/15 18:32:36 interval=1.000000
2019/08/15 18:32:36 Running Loop 0 BUCKET CLEAR TEST
2019/08/15 18:32:36 Loop: 0, Int: ALL, Dur(s): 0.0, Mode: BCLR, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0
2019/08/15 18:32:36 Running Loop 0 BUCKET DELETE TEST
2019/08/15 18:32:36 Loop: 0, Int: ALL, Dur(s): 0.0, Mode: BDEL, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0
2019/08/15 18:32:36 Running Loop 0 BUCKET INIT TEST
2019/08/15 18:32:36 Loop: 0, Int: ALL, Dur(s): 0.0, Mode: BINIT, Ops: 10, MB/s: 0.00, IO/s: 796, Lat(ms): [ min: 10.1, avg: 11.3, 99%: 12.4, max: 12.4 ], Slowdowns: 0
2019/08/15 18:32:36 Running Loop 0 OBJECT PUT TEST
2019/08/15 18:32:37 Loop: 0, Int: 1, Dur(s): 1.0, Mode: PUT, Ops: 5201, MB/s: 20.32, IO/s: 5201, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.7, max: 6.1 ], Slowdowns: 0
2019/08/15 18:32:38 Loop: 0, Int: 2, Dur(s): 1.0, Mode: PUT, Ops: 5372, MB/s: 20.98, IO/s: 5372, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.3, max: 4.3 ], Slowdowns: 0
2019/08/15 18:32:39 Loop: 0, Int: 3, Dur(s): 1.0, Mode: PUT, Ops: 5270, MB/s: 20.59, IO/s: 5270, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.4, max: 10.3 ], Slowdowns: 0
2019/08/15 18:32:40 Loop: 0, Int: 4, Dur(s): 1.0, Mode: PUT, Ops: 5280, MB/s: 20.62, IO/s: 5280, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.4, max: 9.4 ], Slowdowns: 0
2019/08/15 18:32:41 Loop: 0, Int: 5, Dur(s): 1.0, Mode: PUT, Ops: 5356, MB/s: 20.92, IO/s: 5356, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.2, max: 4.3 ], Slowdowns: 0
2019/08/15 18:32:42 Loop: 0, Int: 6, Dur(s): 1.0, Mode: PUT, Ops: 5279, MB/s: 20.62, IO/s: 5279, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.0, max: 11.8 ], Slowdowns: 0
2019/08/15 18:32:43 Loop: 0, Int: 7, Dur(s): 1.0, Mode: PUT, Ops: 5296, MB/s: 20.69, IO/s: 5296, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.0, max: 10.2 ], Slowdowns: 0
2019/08/15 18:32:44 Loop: 0, Int: 8, Dur(s): 1.0, Mode: PUT, Ops: 5319, MB/s: 20.78, IO/s: 5319, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 2.9, max: 3.7 ], Slowdowns: 0
2019/08/15 18:32:45 Loop: 0, Int: 9, Dur(s): 1.0, Mode: PUT, Ops: 5161, MB/s: 20.16, IO/s: 5161, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.1, max: 9.8 ], Slowdowns: 0
2019/08/15 18:32:46 Loop: 0, Int: 10, Dur(s): 1.0, Mode: PUT, Ops: 4768, MB/s: 18.62, IO/s: 4768, Lat(ms): [ min: 1.2, avg: 2.1, 99%: 3.0, max: 89.3 ], Slowdowns: 0
2019/08/15 18:32:46 Loop: 0, Int: ALL, Dur(s): 10.0, Mode: PUT, Ops: 52312, MB/s: 20.43, IO/s: 5230, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.2, max: 89.3 ], Slowdowns: 0
2019/08/15 18:32:46 Running Loop 0 OBJECT GET TEST
2019/08/15 18:32:47 Loop: 0, Int: 1, Dur(s): 1.0, Mode: GET, Ops: 1190, MB/s: 4.65, IO/s: 1190, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.3, max: 1.6 ], Slowdowns: 0
2019/08/15 18:32:48 Loop: 0, Int: 2, Dur(s): 1.0, Mode: GET, Ops: 1142, MB/s: 4.46, IO/s: 1142, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.7 ], Slowdowns: 0
2019/08/15 18:32:49 Loop: 0, Int: 3, Dur(s): 1.0, Mode: GET, Ops: 1111, MB/s: 4.34, IO/s: 1111, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.2 ], Slowdowns: 0
2019/08/15 18:32:50 Loop: 0, Int: 4, Dur(s): 1.0, Mode: GET, Ops: 1113, MB/s: 4.35, IO/s: 1113, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 3.7 ], Slowdowns: 0
2019/08/15 18:32:51 Loop: 0, Int: 5, Dur(s): 1.0, Mode: GET, Ops: 1083, MB/s: 4.23, IO/s: 1083, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.4 ], Slowdowns: 0
2019/08/15 18:32:52 Loop: 0, Int: 6, Dur(s): 1.0, Mode: GET, Ops: 1089, MB/s: 4.25, IO/s: 1089, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.4 ], Slowdowns: 0
2019/08/15 18:32:53 Loop: 0, Int: 7, Dur(s): 1.0, Mode: GET, Ops: 1122, MB/s: 4.38, IO/s: 1122, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.5 ], Slowdowns: 0
2019/08/15 18:32:54 Loop: 0, Int: 8, Dur(s): 1.0, Mode: GET, Ops: 1089, MB/s: 4.25, IO/s: 1089, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.4 ], Slowdowns: 0
2019/08/15 18:32:55 Loop: 0, Int: 9, Dur(s): 1.0, Mode: GET, Ops: 1059, MB/s: 4.14, IO/s: 1059, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 5.7 ], Slowdowns: 0
2019/08/15 18:32:56 Loop: 0, Int: ALL, Dur(s): 10.0, Mode: GET, Ops: 11086, MB/s: 4.31, IO/s: 1104, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 5.7 ], Slowdowns: 0
2019/08/15 18:32:56 Running Loop 0 OBJECT DELETE TEST
2019/08/15 18:32:57 Loop: 0, Int: 1, Dur(s): 1.0, Mode: DEL, Ops: 5716, MB/s: 22.33, IO/s: 5716, Lat(ms): [ min: 1.1, avg: 1.7, 99%: 2.7, max: 3.8 ], Slowdowns: 0
2019/08/15 18:32:58 Loop: 0, Int: 2, Dur(s): 1.0, Mode: DEL, Ops: 5661, MB/s: 22.11, IO/s: 5661, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.8, max: 9.6 ], Slowdowns: 0
2019/08/15 18:32:59 Loop: 0, Int: 3, Dur(s): 1.0, Mode: DEL, Ops: 5616, MB/s: 21.94, IO/s: 5616, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.9, max: 10.1 ], Slowdowns: 0
2019/08/15 18:33:00 Loop: 0, Int: 4, Dur(s): 1.0, Mode: DEL, Ops: 5608, MB/s: 21.91, IO/s: 5608, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 7.7 ], Slowdowns: 0
2019/08/15 18:33:01 Loop: 0, Int: 5, Dur(s): 1.0, Mode: DEL, Ops: 5529, MB/s: 21.60, IO/s: 5529, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 3.0, max: 9.3 ], Slowdowns: 0
2019/08/15 18:33:02 Loop: 0, Int: 6, Dur(s): 1.0, Mode: DEL, Ops: 5438, MB/s: 21.24, IO/s: 5438, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 3.0, max: 10.4 ], Slowdowns: 0
2019/08/15 18:33:03 Loop: 0, Int: 7, Dur(s): 1.0, Mode: DEL, Ops: 5515, MB/s: 21.54, IO/s: 5515, Lat(ms): [ min: 1.3, avg: 1.8, 99%: 2.9, max: 7.5 ], Slowdowns: 0
2019/08/15 18:33:04 Loop: 0, Int: 8, Dur(s): 1.0, Mode: DEL, Ops: 5481, MB/s: 21.41, IO/s: 5481, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 3.0, max: 9.7 ], Slowdowns: 0
2019/08/15 18:33:05 Loop: 0, Int: 9, Dur(s): 1.0, Mode: DEL, Ops: 5614, MB/s: 21.93, IO/s: 5614, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 3.1, max: 8.0 ], Slowdowns: 0
2019/08/15 18:33:05 Loop: 0, Int: ALL, Dur(s): 9.4, Mode: DEL, Ops: 52312, MB/s: 21.76, IO/s: 5571, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.9, max: 10.4 ], Slowdowns: 0
2019/08/15 18:33:05 Running Loop 0 BUCKET DELETE TEST
2019/08/15 18:33:05 Loop: 0, Int: ALL, Dur(s): 0.1, Mode: BDEL, Ops: 10, MB/s: 0.00, IO/s: 133, Lat(ms): [ min: 54.1, avg: 68.9, 99%: 75.2, max: 75.2 ], Slowdowns: 0
```
# Note
Your performance testing benchmark results may vary most often because of limitations of your network connection to the cloud storage provider. Wasabi performance claims are tested under conditions that remove any latency (which can be shown using the ping command) and bandwidth bottlenecks that restrict how fast data can be moved. For more information,
contact Wasabi technical support (support@wasabi.com).