[SOLVED] does makepkg compress too much?

masenko · 2024-05-10 20:07:34

Hi, I've noticed that the default zstd compression level in makepkg.conf is 20. It usually takes almost 2 minutes to compress some bigger packages on a slower system, and even on a faster system this can take up to 1 minute. I tested compression level 15 and the resulting file only grew by 10MB, while the time went down to 6 seconds!

Why are we still using such a high compression level, when nowadays most installs have 250GB of free space or more? And if people agree that this default is too much, where would we propose this change?

Last edited by masenko (2024-05-11 20:47:20)

ayekat · 2024-05-10 21:04:52

masenko wrote:

Why are we still using such a high compression level, when nowadays most installs have 250GB of free space or more?

The package compression level has nothing to do with how much space the files of a package use when installed to the system, but how much space the package file uses.
So I can imagine it's probably less a matter of "How much space do most users' filesystems have?", and more a matter of "How much space do we occupy on our infrastructure and the package mirrors for hosting all the repository package files?".

Also:

I've noticed that the default zstd compression level in makepkg.conf is 20

You are free to change it to something that suits your needs better.

masenko · 2024-05-10 21:17:15

ayekat wrote:

"How much space do we occupy on our infrastructure and the package mirrors for hosting all the repository package files?".

Would changing the default makepkg.conf values cause a change to the compression levels that mirrors use?
I am aware that i can change the file for my own system, the whole point of this post is to change the default value for everyone.
When using an AUR helper like yay, it uses the values stated in makepkg.conf and the resulting package files generated by yay take very long to compress.

ayekat · 2024-05-10 21:40:02

masenko wrote:

Would changing the default makepkg.conf values cause a change to the compression levels that mirrors use?

Right, I guess it's now a question of perspective.

You could argue that the default should cater to users using Yaourt-style helpers like Yay, and everyone else (Arch Linux packagers and everyone else managing repositories using e.g. aurutils) will have to explicitly adapt their makepkg.conf.

And as someone who belongs to the group of repo maintainers and Yaourt-non-users, I would oppose that.
But following my own words above, I can explicitly set a value for my needs if I don't like the default value set by the Pacman packagers for Arch.

masenko · 2024-05-10 21:52:51

ayekat wrote:

Right, I guess it's now a question of perspective.

Why? From my understanding, mirrors have compressed packages that users will decompress and install with pacman. But if you use yay you first download the sources, then compress into a package, and then install on your own system. I was simply asking if changing the makepkg.conf for every arch user, would also change it for mirrors.

ayekat wrote:

But following my own words above, I can explicitly set a value for my needs if I don't like the default value set by the Pacman packagers for Arch.

I might be wrong, but i feel like there are more people using the AUR than maintainers putting out new versions on the AUR. Wouldn't it then benefit more people to lower the default compression value, thus speeding up yay, and have the few people that maintain packages and mirrors use a different compression level for the packages that they put on said mirrors?

Scimmia · 2024-05-10 22:27:31

If changing a config file is so difficult, this is the wrong distro.

Allan · 2024-05-10 22:45:14

This is an Arch decision to use the same makepkg.conf for building Arch packages and distributing to users. I'm not sure keeping these the same is ideal (e.g. compression level, debug packages, ...), but that is the decision that was made.

masenko · 2024-05-10 22:53:25

Scimmia wrote:

If changing a config file is so difficult, this is the wrong distro.

I think you missed the point

Trilby · 2024-05-11 00:42:26

I also think I'm missing the point. Your concern is that compressing packages on your system takes to long for your tastes. This is a perfectly valid and reasonable concern, and we can help with that: it's trivial, change your makepkg.conf.

But then you talk about changing the default for everyone. Why do you care what compression other people use? On my system, compressing AUR packages (or other locally built packages) has not presented any problem at all for my tastes. So why should your conditions and your preferences dictate my defaults?

Chose what works best for you. Do not try to force everyone to use what works best for you ... why would you even care? This is what I feel I'm missing. You do not compress repo packages yourself - so why do you care what compression packagers / maintainers use for repo packages? Unless you are suggesting that decompressing pre-built packages from the repo is a problem on your hardware (this could be a fair point - but it hasn't really been mentioned).

Last edited by Trilby (2024-05-11 00:46:58)

seth · 2024-05-11 07:35:09

I don't compress, I don't enable debug builds (unless I actually wanted to debug a problem in front of me I cannot get rid of otherwise), and I strip. Also I use that flag in makepkg.conf
Is that an ideal solution for everyone? Do you want to see the random archer stripping? Who knows.

Would I use the same flags if I was distributing packages to others? Certainly not.
I'd also not strip by legal counsel advice. And court order.

Context matters and dictates the question to ask. If the context is

When using an AUR helper like yay, it uses the values stated in makepkg.conf and the resulting package files generated by yay take very long to compress.

the questions to ask are
1. wtf am I using yay?
2. why does yay not

export PKGEXT=".pkg.tar"
export SRCEXT=".src.tar"

?
3. is "yay" an appropriate response to anyone here stripping?

If we strip this from all the nonsense, file a bug against yay to set the compression levels to something it'll suit its userbase much better.

masenko · 2024-05-11 12:10:20

Trilby wrote:

I also think I'm missing the point.

Yes, you are. I told Scimmia that he is missing the point because i started this whole post to see if other people experience the same slow compression times as me, I now see that i am alone in my opinion and thus no further discussion is needed.

Trilby wrote:

why should your conditions and your preferences dictate my defaults?

That goes for this question too, i just made this post to see what other people think.

Trilby wrote:

why do you care what compression packagers / maintainers use for repo packages?

I don't.

seth wrote:

file a bug against yay to set the compression levels to something it'll suit its userbase much better.

AUR helpers use makepkg.conf to compress their downloaded sources, hence the post.

I feel like i am being misunderstood or i am not understanding what all of you are telling me. Never the less, I see most of you don't care about the compression level and i got my answer.

Last edited by masenko (2024-05-11 12:12:19)

Trilby · 2024-05-11 13:16:09

masenko wrote:

Trilby wrote:
why do you care what compression packagers / maintainers use for repo packages?
I don't.

You did:

masenko wrote:

I was simply asking if changing the makepkg.conf for every arch user, would also change it for mirrors.

But in anycase, if you have obtained the information you were looking for, you can mark the thread as solved.

seth · 2024-05-11 14:34:25

AUR helpers use makepkg.conf to compress their downloaded sources, hence the post.

No. AUR helpers use makepkg. makepkg will parse the package, makepkg.conf and the environment to decide what to do.
AUR helpers can manipulate the environment to control what makepkg does in their context - w/o affecting the behavior of makepkg in other contexts.

It's perfectly reasonable to disable compression for what are effectively transitional packages - whether it's "slow" or "slow-ish", it's certainly slowER for everyone and, in this context, a waste of time and resources.
Since the AUR helpers have advanced knowledge of the context, they're in pristine condition to decide what should happen - not some global default.

masenko · 2024-05-11 20:47:04

seth wrote:

AUR helpers can manipulate the environment to control what makepkg does in their context - w/o affecting the behavior of makepkg in other contexts.

Oh, i didn't know that. I'll see if it can get changed in yay then.

seth wrote:

It's perfectly reasonable to disable compression for what are effectively transitional packages - whether it's "slow" or "slow-ish", it's certainly slowER for everyone and, in this context, a waste of time and resources.
Since the AUR helpers have advanced knowledge of the context, they're in pristine condition to decide what should happen - not some global default.

Makes sense, i'll mark this discussion as solved.

Arch Linux

#1 2024-05-10 20:07:34

[SOLVED] does makepkg compress too much?

#2 2024-05-10 21:04:52

Re: [SOLVED] does makepkg compress too much?

#3 2024-05-10 21:17:15

Re: [SOLVED] does makepkg compress too much?

#4 2024-05-10 21:40:02

Re: [SOLVED] does makepkg compress too much?

#5 2024-05-10 21:52:51

Re: [SOLVED] does makepkg compress too much?

#6 2024-05-10 22:27:31

Re: [SOLVED] does makepkg compress too much?

#7 2024-05-10 22:45:14

Re: [SOLVED] does makepkg compress too much?

#8 2024-05-10 22:53:25

Re: [SOLVED] does makepkg compress too much?

#9 2024-05-11 00:42:26

Re: [SOLVED] does makepkg compress too much?

#10 2024-05-11 07:35:09

Re: [SOLVED] does makepkg compress too much?

#11 2024-05-11 12:10:20

Re: [SOLVED] does makepkg compress too much?

#12 2024-05-11 13:16:09

Re: [SOLVED] does makepkg compress too much?

#13 2024-05-11 14:34:25

Re: [SOLVED] does makepkg compress too much?

#14 2024-05-11 20:47:04

Re: [SOLVED] does makepkg compress too much?

Board footer