What is happening with glibc and gcc packages?

kjtsanaktsidis · 2022-02-04 00:40:44

08d09 wrote:

something else must be going on judging upon the current situation

Yes, this. It’s very frustrating and honestly making me want to remove my four day old arch installation on my new laptop and just plonk fedora on there. There is clearly something very sick inside the distribution that nobody wants to talk about; it seems most people are more interested in sarcasm and pot shots than actually doing anything about it.

Allan · 2022-02-04 00:52:20

Is there something very sick in the distro? Or is this just drama?

All that is known is that the previous toolchain maintainer resigned, a new maintainer was found, and there has been a big delay in recent updates. Many conclusions can be made from this information:

1) The new maintainer got busy
2) The new maintainer took on more than they could handle
3) The new maintainer is being paid to deliberately leave the packages outdated by a government wanting access to your computer.
4) The entire distribution is going to fall over any minute.

Some conclusions are likely more justified than others based on the available data...

cowlick · 2022-02-04 00:57:15

Allan wrote:

Is there something very sick in the distro? Or is this just drama?
All that is known is that the previous toolchain maintainer resigned, a new maintainer was found, and there has been a big delay in recent updates. Many conclusions can be made from this information:
1) The new maintainer got busy
2) The new maintainer took on more than they could handle
3) The new maintainer is being paid to deliberately leave the packages outdated by a government wanting access to your computer.
4) The entire distribution is going to fall over any minute.
Some conclusions are likely more justified than others based on the available data...

Option 4 please.

johnyan · 2022-02-04 03:21:56

Allan wrote:

Is there something very sick in the distro? Or is this just drama?
All that is known is that the previous toolchain maintainer resigned, a new maintainer was found, and there has been a big delay in recent updates. Many conclusions can be made from this information:
1) The new maintainer got busy
2) The new maintainer took on more than they could handle
3) The new maintainer is being paid to deliberately leave the packages outdated by a government wanting access to your computer.
4) The entire distribution is going to fall over any minute.
Some conclusions are likely more justified than others based on the available data...

By any chance, the new maintainer could give a status update on what's going on here?

ewaller · 2022-02-04 04:43:40

johnyan wrote:

[By any chance, the new maintainer could give a status update on what's going on here?

Probably not. This is not the venue where that would be discussed. That is more a mail list thing.

dogknowsnx · 2022-02-04 07:18:45

Allan wrote:

Is there something very sick in the distro? Or is this just drama?
All that is known is that the previous toolchain maintainer resigned, a new maintainer was found, and there has been a big delay in recent updates. Many conclusions can be made from this information:
1) The new maintainer got busy
2) The new maintainer took on more than they could handle
3) The new maintainer is being paid to deliberately leave the packages outdated by a government wanting access to your computer.
4) The entire distribution is going to fall over any minute.
Some conclusions are likely more justified than others based on the available data...

Read Razzolini's comment and jump to conclusion #2...

MS1 · 2022-02-04 11:49:23

Why can't Allan do the update if Razzolini is too busy?

V1del · 2022-02-04 11:51:17

See: https://bbs.archlinux.org/viewtopic.php … 0#p2019340

And pushing the actually updated "opinionated" packages as is would break general distribution assumptions/requirements (e.g. non possibility to build multilib) and others mentioned in his own readme.

Last edited by V1del (2022-02-04 11:55:25)

Trilby · 2022-02-04 13:34:55

So "problems of the distro" are being patched over and should instead be exposed. Yet later it is suggested that there could be no problem in the distro, but rather one maintainer took on more than they could handle. I've made very different inferences than the four proposed above, as none of those four actually fit the data. Suggesting those four are theoretically possible is valid, but why not say what is actually true? What problems have been being patched over, and why does there appear to be such a taboo on talking about them?

johnyan · 2022-02-04 13:39:58

See: https://bugs.archlinux.org/index/proj1? … &dev=20907

That's a list of 4 pages of bugs that have been assigned to Razzolini. If he is just one person, not a team of people, the how could he handle all these issue in a timely manner (even if he is full time working on these)?
Is the core team really short on hands?

Thanks.

Last edited by johnyan (2022-02-04 13:40:26)

Trilby · 2022-02-04 13:44:16

If a boat is sinking, I'd not focus too much on the guy who had all four limbs splayed out to have a finger plugging every whole they could - that is other than to praise him for his efforts. That person being stretched too far is a symptom, not the source of the problem.

Allan · 2022-02-04 14:02:02

Trilby wrote:

So "problems of the distro" are being patched over and should instead be exposed. Yet later it is suggested that there could be no problem in the distro, but rather one maintainer took on more than they could handle. I've made very different inferences than the four proposed above, as none of those four actually fit the data. Suggesting those four are theoretically possible is valid, but why not say what is actually true? What problems have been being patched over, and why does there appear to be such a taboo on talking about them?

There is a problem with the toolchain maintenence, which is a problem of the distro given a toolchain is quite important. I'd guess the problem is somewhere between #1 and #2 on my list. Me stepping in and doing a quick update just papers over the problem for now, but the distro is still left with no active toolchain maintainer. How do I know this? Guess who stepped in and did the glibc-2.33 toolchain update a year ago?! As pointed out by the list of bugs, active maintainers are not just package updaters.

Why is no-one talking about this? Mostly because nobody in this thread can fix the problem. So discussing it achieves nothing.

Psykorgasm · 2022-02-04 14:04:11

Allan wrote:

I'm hoping that the move of the repos to gitlab (at some future point...), will allow users to directly submit patches towards PKGBUILDs and will partially address this. But there also is a need to bring more people onto the team, and this is an area that could be targeted.

I don't use Arch much anymore but I would love to see this.
Years ago I used a source based distro and the users were pretty much maintainers but we had to submit patches via IRC and it was such a pain in the ass, things improved a lot once we had a gerrit instance (that crashed a lot) set up where we could more easily collab on patches.

I used to have some AUR packages and never needed to submit patches for the Arch repos outside of one bug report I did put the diff needed for the bump, the process wasn't really ideal. Having a gitlab instance would be much better than this thread for making progress and testing package updates. Plenty of people want to help and test, having easy tools makes it much easier to pull them in to "the team".

johnyan · 2022-02-04 14:11:32

Allan wrote:

Trilby wrote:
So "problems of the distro" are being patched over and should instead be exposed. Yet later it is suggested that there could be no problem in the distro, but rather one maintainer took on more than they could handle. I've made very different inferences than the four proposed above, as none of those four actually fit the data. Suggesting those four are theoretically possible is valid, but why not say what is actually true? What problems have been being patched over, and why does there appear to be such a taboo on talking about them?
There is a problem with the toolchain maintenence, which is a problem of the distro given a toolchain is quite important. I'd guess the problem is somewhere between #1 and #2 on my list. Me stepping in and doing a quick update just papers over the problem for now, but the distro is still left with no active toolchain maintainer. How do I know this? Guess who stepped in and did the glibc-2.33 toolchain update a year ago?! As pointed out by the list of bugs, active maintainers are not just package updaters.
Why is no-one talking about this? Mostly because nobody in this thread can fix the problem. So discussing it achieves nothing.

Hey Allan, if I understand correctly, the real problem is that nobody wants to maintain the toolchain?

What's the process of updating the toolchain? Is there a code repo and CI job? Is it documented somewhere people could take a look?

Thanks.

Allan · 2022-02-04 14:25:20

johnyan wrote:

What's the process of updating the toolchain? Is there a code repo and CI job? Is it documented somewhere people could take a look?

There maybe some very sparse documentation in the wiki. No separate code repo - just the package repo.

The process is fairly simple (if nothing goes wrong...). Just update the PKGBUILD for a new release, and build the package. If it is a major version bump, bootstrap the toolchain. Then look at the testsuite and make sure you understand every testsuite failure... because there are always some!

08d09 · 2022-02-04 14:38:56

For those who are considering using Allan's updated toolchain it does seem to be working perfectly fine for desktop use. based on my limited experience.

If you need lib32-gcc for wine like I do pacman will try to delete it because the package has glibc2.33 as a dependency, force installing glibc first seems to be a fine workaround for this. Wine still works fine on my end after force installing glibc and then upgrading the rest of the toolchain.

The second issue is that DKMS doesn't work with the newest GCC as you need to compile modules with the same version of GCC that was used to compile the kernel. It's a really ugly workaround but you can downgrade binutils and gcc before kernel updates, have pacman rebuild DKMS modules and upgrade gcc and binutils again. It sounds really bad and I don't know if it's safe but from what I can tell it does work as I can load & use modules compiled this way. So long story short you can unofficially have the latest toolchain if you are willing to put up with these two issues, thanks Allan!

M-Reimer · 2022-02-04 15:26:31

Allan wrote:

There is a problem with the toolchain maintenence, which is a problem of the distro given a toolchain is quite important. I'd guess the problem is somewhere between #1 and #2 on my list. Me stepping in and doing a quick update just papers over the problem for now, but the distro is still left with no active toolchain maintainer.

That may be true but at least doing a "minimal update", meaning keeping 2.33 and applying at least the needed security updates, would at least bring uninvolved users out of that equation. With the current state every Arch user, who did not bother to patch glibc manually somehow, is at risk.

So is there someone with the needed skills and permissions who can at least make 2.33 "less critical"? IMHO then the issue is still "exposed" as glibc is still outdated after that.

Toolybird · 2022-02-04 23:50:34

Allan wrote:

Given a user posted a complete set of working packages for the toolchain (which included updates and a lot of improvements) about 5 months ago on the bugtracker, a pessimist would conclude there is nothing to be done in this particular case.

Yeah, that user would have been me. FWIW, I intend getting back into toolchain stuff shortly. My attention lately has been on the virt stack. Finally got my act together and put all my crap on Gitlab [1]. Keep an eye on that repo as I plan to add my toolchain bits there too. Unlike your excellent work Allan, I'll try and stick to the current Arch multilib setup, etc. coz I like pain!

[1] https://gitlab.com/Toolybird/pkgbuilds

Allan · 2022-02-05 00:14:17

Toolybird wrote:

Keep an eye on that repo as I plan to add my toolchain bits there too. Unlike your excellent work Allan, I'll try and stick to the current Arch multilib setup, etc. coz I like pain!
[1] https://gitlab.com/Toolybird/pkgbuilds

That sounds great! We should definitely share ideas. I'll be in touch!

BTW, I'm down to zero gcc testsuite errors! Although not sure if one of my patches will be acceptable upstream...

afader · 2022-02-05 02:32:32

I love a happy ending!

Toolybird · 2022-02-05 02:33:20

Allan wrote:

BTW, I'm down to zero gcc testsuite errors!

Really? That's amazing. When I last worked on it, there was a very loooong list that I wasn't looking forward to investigating. Awesome stuff

johnyan · 2022-02-05 02:55:58

Toolybird wrote:

Allan wrote:
Given a user posted a complete set of working packages for the toolchain (which included updates and a lot of improvements) about 5 months ago on the bugtracker, a pessimist would conclude there is nothing to be done in this particular case.
Yeah, that user would have been me. FWIW, I intend getting back into toolchain stuff shortly. My attention lately has been on the virt stack. Finally got my act together and put all my crap on Gitlab [1]. Keep an eye on that repo as I plan to add my toolchain bits there too. Unlike your excellent work Allan, I'll try and stick to the current Arch multilib setup, etc. coz I like pain!
[1] https://gitlab.com/Toolybird/pkgbuilds

Hey, thank you for your awesome contribution to the community. Would you or Allen share more of your expertise on this matter? I have been searching around for a day or two. I could find very little info regarding to how the core/toolchain/kernel is maintained in Arch. If you don't know, would you talk about these questions:

1. If you need to bootstrap a major version, what are the bootstrap sequence/process for Arch? I have done LFS, but Arch seems different. Is there a repo for arch bootstraping from source?
2. Other than the default test suit, what other considerations/tests needs to run?
3. You probably won't upgrade one package at a time. You would probably upgrade many packages altogether like gcc/glibc/etc. Do you have a list of those?
4. If everything is clean and ready to go, what's the process for submitting code for review? Is there a CI job for automation?

Thanks

Allan · 2022-02-05 03:02:36

johnyan wrote:

1. If you need to bootstrap a major version, what are the bootstrap sequence/process for Arch? I have done LFS, but Arch seems different. Is there a repo for arch bootstraping from source?

Build order is:
linux-api-headers
glibc (no testsuite)
binutils (no testsuite)
gcc (no testsuite)
glibc
binutils
gcc

johnyan wrote:

2. Other than the default test suit, what other considerations/tests needs to run?

I just check my favourite piece of software (pacman!) builds when I install the packages as a final sanity check

johnyan wrote:

3. You probably won't upgrade one package at a time. You would probably upgrade many packages altogether like gcc/glibc/etc. Do you have a list of those?

Minor updates can happen one package at a time. Major updates require the whole toolchain to be rebootstrapped. Generally, linux gets rebuilt with every new gcc package (not sure about if this is needed for minor changes to gcc). libtool gets rebuild for gcc version bumps. Valgrind gets rebuilt for glibc version bumps.

johnyan wrote:

4. If everything is clean and ready to go, what's the process for submitting code for review? Is there a CI job for automation?

There is none.

johnyan · 2022-02-05 03:17:16

Allan wrote:

johnyan wrote:
1. If you need to bootstrap a major version, what are the bootstrap sequence/process for Arch? I have done LFS, but Arch seems different. Is there a repo for arch bootstraping from source?
Build order is:
linux-api-headers
glibc (no testsuite)
binutils (no testsuite)
gcc (no testsuite)
glibc
binutils
gcc

This is probably not the whole list of everything, isn't it? If you upgrade glibc/gcc, don't you also need to recompile the linux kernel? As you memtion earlier, maybe a list of kernel modules also need to be recompiled?
Also the process isn't just `makepkg -si` everything. There might be subtile difference in compilation between stages.

kjtsanaktsidis · 2022-02-05 03:22:56

johnyan wrote:

I have been searching around for a day or two. I could find very little info regarding to how the core/toolchain/kernel is maintained in Arch. If you don't know, would you talk about these questions:

I'm sure this is a bit of a case of the blind leading the blind, but I started having a swing at bootstrapping the toolchain this morning. I've been using the current PKGBUILDs from the svn packages trunk, with the minimal changes needed to bump the versions to the latest upstream ones. So, _not_ fixing any of the issues that Allan has outlined r.e. the bootstrap order being wrong, libiberty being installed from gcc instead of binutils, etc.

I think the specific thing that "bootstrapping a toolchain" refers to, AFAICT, is:
* Making a chroot using mkarchroot like normal,
* For each package in the sequence linux-api-headers -> glibc -> binutils -> gcc -> binutils -> glibc (yes - Allan said this order does not lead to a reproducible build, because some references to the stage1 glibc build get burned into gcc's startfiles IIRC - but this is historically the order used).
* Build the package from the PKGBUILD with makechrootpkg,
* Install that package into the chroot with "pacman --root ~/chroot/root -U ./built-package.pkg.tar.zst"
* Carry on building the next package in the sequence
* For the final gcc/binutils/glibc packages, (not the intermediate builds of them), check the test failures and try and grok them.

Then, the final gcc/binutils/glibc packages are the ones that would get uploaded to the repos. The intermediate ones are discarded AFAICT and end-users would never see them.

Again, AFAICT, I don't think there's any CI system for stitching this together. This is both an issue in its own right (because it makes building the toolchain a right PITA for the maintainer), but also makes it really important that the build is reproducible since the maintainer is just doing the package builds on their own machine; reproducibility means others can actually verify this.

Arch Linux

#76 2022-02-04 00:40:44

Re: What is happening with glibc and gcc packages?

#77 2022-02-04 00:52:20

Re: What is happening with glibc and gcc packages?

#78 2022-02-04 00:57:15

Re: What is happening with glibc and gcc packages?

#79 2022-02-04 03:21:56

Re: What is happening with glibc and gcc packages?

#80 2022-02-04 04:43:40

Re: What is happening with glibc and gcc packages?

#81 2022-02-04 07:18:45

Re: What is happening with glibc and gcc packages?

#82 2022-02-04 11:49:23

Re: What is happening with glibc and gcc packages?

#83 2022-02-04 11:51:17

Re: What is happening with glibc and gcc packages?

#84 2022-02-04 13:34:55

Re: What is happening with glibc and gcc packages?

#85 2022-02-04 13:39:58

Re: What is happening with glibc and gcc packages?

#86 2022-02-04 13:44:16

Re: What is happening with glibc and gcc packages?

#87 2022-02-04 14:02:02

Re: What is happening with glibc and gcc packages?

#88 2022-02-04 14:04:11

Re: What is happening with glibc and gcc packages?

#89 2022-02-04 14:11:32

Re: What is happening with glibc and gcc packages?

#90 2022-02-04 14:25:20

Re: What is happening with glibc and gcc packages?

#91 2022-02-04 14:38:56

Re: What is happening with glibc and gcc packages?

#92 2022-02-04 15:26:31

Re: What is happening with glibc and gcc packages?

#93 2022-02-04 23:50:34

Re: What is happening with glibc and gcc packages?

#94 2022-02-05 00:14:17

Re: What is happening with glibc and gcc packages?

#95 2022-02-05 02:32:32

Re: What is happening with glibc and gcc packages?

#96 2022-02-05 02:33:20

Re: What is happening with glibc and gcc packages?

#97 2022-02-05 02:55:58

Re: What is happening with glibc and gcc packages?

#98 2022-02-05 03:02:36

Re: What is happening with glibc and gcc packages?

#99 2022-02-05 03:17:16

Re: What is happening with glibc and gcc packages?

#100 2022-02-05 03:22:56

Re: What is happening with glibc and gcc packages?

Board footer