Zstd for compressing package index databases

sncix · 2025-02-24 11:27:30

To my knowledge, zstd has been the default format for Arch Linux packages since a few years ago, while package index databases are still compressed using gzip.

Currently, the size of extra.db.tar.gz is about 7.7 MiB. If zstd is used instead of gzip, the compressed [extra] database will be about 7% (level 3, default) or 12% (level 9) smaller.

This proposed change would benefit people with slow connections. The benefit would be small (1~2 seconds for 500 KiB/s) and infrequent, but it's a second where people would likely be paying some attention to the screen at least, so it might arguably improve the experience for some.

On the other hand, this change would require at least two things:
- Pacman to support zstd'd package index (via pacman.conf, for example).
- Official mirrors to offer zstd'd package index.

Question: In your opinion, is this worth considering, or is it an over-optimisation?

mpan · 2025-02-24 13:20:06

Welcome to the forum.

You may consider dropping a feature request at the pacman GitLab or pacman-dev mailing list. I believe interaction there will be much more productive than on a commoners’ forum. I’m not going to comment on the benefits of reducing size, but IIRC the reasoning behind the move to Zstd for packages was speed, not compression ratio.

There is a thing to consider while opening a FR. Likely you’ll be told that at some point, so it’s worth addressing that in the request. Pacman isn’t used only by Arch Linux and the Zstd support isn’t built in. Zstd is handled by libarchive (pkg). Arch ships libarchive with Zstd support enabled, but in general it’s optional.

Last edited by mpan (2025-02-24 16:18:15)

gromit · 2025-02-24 14:04:58

Hey this sounds like a nice addition to our ecosystem! I think this is worth discussing!

Allan · 2025-02-24 21:15:10

pacman has support zstd compressed databases for a long long time - so both those suggestions of where to open bug reports are wrong.

As an aside, wewould save ~50% of the size by removing the %PGPSIG% field in the db (which pacman also supports).

NuSkool · 2025-02-24 22:04:29

This?
I copied autoconf-2.72-1 desc file from /var/lib/pacman/local/a52dec-0.8.0-2/mtree. Then removed the %PGPSIG% entry in the desc-wo....
Saving a tiny amount from less text, but it amounts to a nearly 50% reduction multiplied by all the pkgs in the repos?
I think I'm finally making sense of a conversation way back about this.

$ ls -lh
total 8.0K
-rw-r--r-- 1 jeff jeff 960 Feb 24 13:56 desc
-rw-r--r-- 1 jeff jeff 533 Feb 24 13:57 desc-wo

Last edited by NuSkool (2025-02-24 22:05:23)

gromit · 2025-02-24 22:18:25

I didn't want OP to propose a feature request to pacman but instead discuss this idea with people within Arch Linux so that eventually the related changes could be made in dbscripts

Allan · 2025-02-24 23:50:53

NuSkool wrote:

This?
I copied autoconf-2.72-1 desc file from /var/lib/pacman/local/a52dec-0.8.0-2/mtree. Then removed the %PGPSIG% entry in the desc-wo....
Saving a tiny amount from less text, but it amounts to a nearly 50% reduction multiplied by all the pkgs in the repos?
I think I'm finally making sense of a conversation way back about this.

That field is also the least compressible, so it makes a big difference.

Allan · 2025-02-24 23:51:47

gromit wrote:

I didn't want OP to propose a feature request to pacman but instead discuss this idea with people within Arch Linux so that eventually the related changes could be made in dbscripts

I was replying to the comment above yours that suggested taking it to pacman.

dbscripts is the right place.

gromit · 2025-02-25 07:59:02

Op has now posted the issue to dbscripts: https://gitlab.archlinux.org/archlinux/ … /issues/61

Arch Linux

#1 2025-02-24 11:27:30

Zstd for compressing package index databases

#2 2025-02-24 13:20:06

Re: Zstd for compressing package index databases

#3 2025-02-24 14:04:58

Re: Zstd for compressing package index databases

#4 2025-02-24 21:15:10

Re: Zstd for compressing package index databases

#5 2025-02-24 22:04:29

Re: Zstd for compressing package index databases

#6 2025-02-24 22:18:25

Re: Zstd for compressing package index databases

#7 2025-02-24 23:50:53

Re: Zstd for compressing package index databases

#8 2025-02-24 23:51:47

Re: Zstd for compressing package index databases

#9 2025-02-25 07:59:02

Re: Zstd for compressing package index databases

Board footer