Ext4 vs XFS

Feature difference

• Ext4 supports big cluster sizes (up to 256Mb) with -O bigalloc, while XFS supports only 512b-4Kb cluster size
• XFS supports fully dynamic inode allocation, i.e. you’ll never run out of inodes, and at the same time you don’t need to waste disk space by reserving it for inodes
• Ext4 does NOT support changing inode count without reformatting the filesystem, even with resize2fs; by default, 1/64 of disk space is reserved for inodes (!!!)

  It's not hard to change inode count in theory: (1) move data blocks out of the way if we need to reserve them for inodes (2) change inode numbers in all directory entries (3) overwrite/move inode bitmaps and tables. But it's not implemented :-(

• XFS does NOT support shrinking of a filesystem at all (you can only grow it)

Benchmarks

Operations with kernel 3.10 source tree

• HDD: WD Scorpio Black 2.5" 750GB 7200rpm
• Kernel: 3.12.3 (Debian 3.12.3-1~exp1)

Copy kernel source from SSD to tested FS and then sync, with warm page cache (i.e. not read-bound):

• xfs 1 parallel copy: 12.348s
• xfs 4 parallel copies: 65.883s
• ext4 1 parallel copy: 7.662s
• ext4 4 parallel copies: 33.876s

tar 3 kernel source copies from tested FS to /dev/null (basically just read and discard) after 'echo 3 > /proc/sys/vm/drop_caches':

• xfs: real 26.815s, user 0.936s, sys 1.556s
• ext4: real 5.509s, user 0.584s, sys 0.872s (almost 5 times faster!)

rm 3 kernel source copies and sync after 'echo 3 > /proc/sys/vm/drop_caches':

• xfs: real 7.244s, user 0.148s, sys 2.748s
• ext4: real 8.993s, user 0.108s, sys 2.664s

(oh, xfs is in fact faster in this test!)

FS-Mark 3.3, creating 1M files

• HDD: WD Scorpio Black 2.5" 750GB 7200rpm
• Kernel: 3.12.3
• fs_mark is a write-only test and it does fsync(), so there should be no skew caused by page cache

sysbench random read/write 16K in 8M files

• HDD: WD Scorpio Black 2.5" 750GB 7200rpm
• Kernel: 3.12.3
• sysbench was ran with O_DIRECT, so the page cache should also have no impact.
• It’s not a filesystem benchmark at all! It tests disk performance because it holds ALL prepared files open during the test. It only shows us that neither ext4 nor XFS aren’t slowing down the direct access to underlying device (which is also good, of course)…
• Probably because of the above note, the filesystems don’t differ, and the results are totally same for 1x 1GB file and 128x 8MB files… and very similar for 3072x 16KB files (next test below).

sysbench random read/write 16K in 16K files

• HDD: WD Scorpio Black 2.5" 750GB 7200rpm
• Kernel: 3.12.3

filebench fileserver, dirty_ratio=1%

• HDD: WD VelociRaptor WD6000HLHX, 10000rpm
• Kernel: 3.10.11 (Debian 3.10-3-amd64)
• fileserver test is read whole file + append + write whole file test ran on 10000 files in X threads
• filebench fails to run fileserver test with O_DIRECT, so I tried to "disable" page cache using dirty_ratio=1% and ran tests like this:

echo 1 > /proc/sys/vm/dirty_ratio
echo 0 > /proc/sys/vm/dirty_bytes
echo 0 > /proc/sys/kernel/randomize_va_space
for i in 1 2 4 8 16 32 50 64; do
    echo
    echo "== $i threads =="
    echo
    echo 1 > /proc/sys/vm/drop_caches
    sync
    filebench <<EOF
load fileserver
set \$dir=/media/sdd
set \$nthreads=$i
run 30
EOF
done
echo 20 > /proc/sys/vm/dirty_ratio

filebench fileserver, dirty_ratio=20%

• HDD: WD VelociRaptor WD6000HLHX, 10000rpm
• Kernel: 3.10.11 (Debian 3.10-3-amd64)
• Same test but ran with default 20% dirty_ratio setting. It's clearly seen that the system was using page cache extensively - ext4 was permanently gaining an unreal result in the single-threaded test...