Do the different compression tools have speed differences? (i7 + NVME)

Utini · 2019-04-07 08:42:10

Hello everyone,

I am currently on step further at setting up my new system which includes a lot of fast components:
CPU: i7-8850H
RAM: 32GB 2666MHz CL15
2x 1TB Samsung 970 Pro NVME
1x 2TB Corsair P510 NVME (as external SSD with 1000MB/s read/write on USB-C)

I do a lot of compression / decompression and so far have always used p7zip with p7zip-gui from the AUR.
Mainly because I am used to 7-zip from Windows and there is has the biggest feature set (e.g. multicore support).

I now tried compressing several 5GB files into one .zip file and noticed that the speed was only at ~30MB/s with 1 CPU at 100% usage.
At the beginning of the compression it was at 60MB/s with all 12 CPUs at ~90% usage.
To me this seems a little slow when looking at my hardware components. Atleast my CPU is not at the maximum load/usage.
So what is the bottle neck and are different archive compression tools faster?

@edit: I now installed lbzip2. p7zip now seems to use this one as default instead of the lbzip2 (no-multithread). Because now all my 12 CPUs are at 100% but the compression speed is still at only 30MB/s.

Last edited by Utini (2019-04-07 08:57:36)

Lone_Wolf · 2019-04-07 12:35:04

You may find http://archlinux.2023198.n4.nabble.com/ … 22484.html interesting.

graysky · 2019-04-07 13:06:37

@op - with 32 G of memory, recommend that you do whatever testing in tmpfs to take disk access time out of the equation:

From /etc/fstab

tmpfs /scratch  tmpfs size=27G,nodev,lazytime  0 0

FYI - I do most things that are I/O intensive in /scratch (building packaging, transcoding media, benchmarking, data transfers, etc.) Just know that if the machine reboots, you lose the data there.

Trilby · 2019-04-07 13:25:06

Graysky, compression isn't I/O intensive in the way another process that has to read/write the same data many times is. Compression is a read-once write-once process. So while that single read and write might be faster from/to tmpfs, one still has to move the source into tmpfs and the result out.

So now you have just as much read/write to the NVME, but you also have two sets of intermediate write/read cycles to the tmpfs. Even though these are fast, they can at best add no detectable time to the full process, they can't possibly make it faster.

I suppose you did say just for testing - but then what is it that one would be testing? Finding the compression tool that works best for a situation other than how you intend to use it is not particularly helpful.

Last edited by Trilby (2019-04-07 13:26:52)

seth · 2019-04-07 14:14:01

I assume "testing whether the disk/FS I/O is the limiting factor" (eg. possible if ntfs, lvm or dmcrypt are part of this) - there's no point in investigating the best compression algorithm if your problem is somewhere else.

Utini · 2019-04-07 15:21:40

I am not sure if disk/FS I/O is the limiting factor tbh.

Ropid · 2019-04-07 20:43:23

Does the exact format for the archive files not matter to you? Can it be anything?

That .bz2 format you wanted to experiment with by using the lbzip2 tool is not a good choice. There's tools and formats that can do better compression at similar speeds, or can do same compression at faster speeds.

The 'xz' tool that creates .xz files is by default slow, but can be made faster when enabling its multi-threading feature and choosing a less aggressive compression level. You choose the level with command line options "-0" to "-9", and you enable multi-threading with "-T0". The .xz format is internally similar to what p7zip uses for its .xz files.

For very fast compression, there's an interesting tool 'zstd' that you could check out. It creates .zst files. With its default level 3 it does 480MB/s on a single core here for me, and going down to its lowest level 1 it does 650MB/s. You can enable multi-threading for it with "-T0", the level you choose with "-1" to "-19". Its compression is good. At its default level 3 it beats .zip files and .gz files for size.

The zstd tool can scale to many different speeds. The higher levels will do great compression and will beat that bzip2 format you wanted to use earlier in both size and speed. The highest levels can get annoyingly slow, same as 7z and xz tools, but will not be able to beat those in size.

There's other tools that could be interesting: lzop and lz4. Those create .lzo and .lz4 files. They are interesting if you want very fast compression. They are not good if you want strong compression.

Arch Linux

#1 2019-04-07 08:42:10

Do the different compression tools have speed differences? (i7 + NVME)

#2 2019-04-07 12:35:04

Re: Do the different compression tools have speed differences? (i7 + NVME)

#3 2019-04-07 13:06:37

Re: Do the different compression tools have speed differences? (i7 + NVME)

#4 2019-04-07 13:25:06

Re: Do the different compression tools have speed differences? (i7 + NVME)

#5 2019-04-07 14:14:01

Re: Do the different compression tools have speed differences? (i7 + NVME)

#6 2019-04-07 15:21:40

Re: Do the different compression tools have speed differences? (i7 + NVME)

#7 2019-04-07 20:43:23

Re: Do the different compression tools have speed differences? (i7 + NVME)

Board footer