Linux 6.7 kernel creating very large initramfs.

cirrus9 · 2024-01-15 02:19:10

Getting "boot partition full" errors with 6.7 kernel. Initramfs 230Mb+ overloading 512Mb boot partition. I googled this, and found a discussion of the problem on an Arch derivative forum, with a link to Gitlab Archlinux
where they are discussing it as well. It seems that the KMS hook is the cause. Some say to disable the writing of the fall back image. I removed the KMS hook in mkinitcpio config, and the ran mkinitcpio -P linux again, and the initramfs were back to normal size. System is working fine without KMS hook. I'm using Nvidia, do I really even need KMS? I'm surprised that there are no posts about this yet on here.

V1del · 2024-01-15 03:05:51

The main reason you want "KMS" in the initramfs (what you actually want is the modules of your graphics driver which are quite big on nvidia, the kms hook just transparently takes care of that) that if you're on fast storage like an SSD chances are your graphical environment tries to start before the graphics driver has finished initializing which can lead to various visual bugs and potential complete UI failures between any two boots due to the unresolved race condition.

It would be nice if the kernel had a more surefire way of waiting for a graphics device to be finished before continuing starting a GUI but afaik that doesn't really exist.

FWIW as a general summary to everyone that gets linked to this thread: the quick fix if you don't want to test the one provided by nl6720 https://bbs.archlinux.org/viewtopic.php … 3#p2143733 is to change your /etc/mkinitcpio.conf and removing the kms hook from hooks and adding the kernel module of the graphics driver you're using to the MODULES=() array: https://wiki.archlinux.org/title/Kernel … _KMS_start

Last edited by V1del (2024-01-15 15:11:46)

cirrus9 · 2024-01-15 03:37:57

I do have the Nvidia modules listed on the "modules line" in mkinitcipo. Maybe that's why it's working okay without KMS hook. Why did kernel 6.7 make those huge initramfs in the first palace?

Scimmia · 2024-01-15 03:59:51

If you're willing to do some testing, use lsinitcpio to see what's in the new image vs the old image. It's likely going to be an nvidia specific thing, I'm not seeing that on AMD.

madscience · 2024-01-15 04:08:18

I have an Intel IGP, no Nvidia, and my initramfs is 240 Mb with 6.7 - my lts kernel is only 75ish Mb - any ideas?

Scimmia · 2024-01-15 04:11:49

Wait, so only AMD is immune? Strange that it wasn't caught in Testing.

Same statement, if you willing to test, use lsinitcpio

Trilby · 2024-01-15 04:25:51

I'm getting this on intel hardware too - it appears *all* graphics modules are included in my fallback image (including nvida, amdgpu, and radeon) but as are a huge number of other things. There are 2450 modules or bin.zst files included in my fallback image that are not in the regular image.

The contents of the fallback image:
http://0x0.st/HUGe.txt

And the contents of the fallback image not in the regular image:
http://0x0.st/HUG2.txt

And the contents of the regular image:
http://0x0.st/HUGL.txt

Last edited by Trilby (2024-01-15 04:30:34)

Scimmia · 2024-01-15 04:28:10

That's normal for the fallback, it doesn't use autodetect, so it includes everything. What about the non-fallback?

Last edited by Scimmia (2024-01-15 04:28:49)

Trilby · 2024-01-15 04:31:03

I've added that to my last post - though I don't see how that matters: the non-fallback image is of reasonable size.

Perhaps this will be more helpful. I downgraded and got the listing of all 4 initramfs's ("old" and current, base and fallback). I then used sed to trim the paths including the version numbers to allow a side-by-side comparison, and ...

$ comm -3 base.old base
	usr/lib/firmware/i915/mtl_gsc_1.bin.zst
	kernel/drivers/nvme/common/nvme-auth.ko
kernel/drivers/nvme/common/nvme-common.ko

$ comm -3 fallback.old fallback
	usr/lib/firmware/i915/mtl_gsc_1.bin.zst
	usr/lib/firmware/nvidia/ad102/
	usr/lib/firmware/nvidia/ad102/gsp/
	usr/lib/firmware/nvidia/ad102/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad102/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad102/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad102/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad103/
	usr/lib/firmware/nvidia/ad103/gsp/
	usr/lib/firmware/nvidia/ad103/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad103/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad103/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad103/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad104/
	usr/lib/firmware/nvidia/ad104/gsp/
	usr/lib/firmware/nvidia/ad104/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad104/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad104/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad104/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad106/
	usr/lib/firmware/nvidia/ad106/gsp/
	usr/lib/firmware/nvidia/ad106/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad106/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad106/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad106/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad107/
	usr/lib/firmware/nvidia/ad107/gsp/
	usr/lib/firmware/nvidia/ad107/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad107/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad107/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ad107/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga100/
	usr/lib/firmware/nvidia/ga100/gsp/
	usr/lib/firmware/nvidia/ga100/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga100/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga100/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga100/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga102/gsp/
	usr/lib/firmware/nvidia/ga102/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga102/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga102/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga102/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga103/gsp/
	usr/lib/firmware/nvidia/ga103/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga103/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga103/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga103/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga104/gsp/
	usr/lib/firmware/nvidia/ga104/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga104/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga104/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga104/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga106/gsp/
	usr/lib/firmware/nvidia/ga106/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga106/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga106/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga106/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga107/gsp/
	usr/lib/firmware/nvidia/ga107/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga107/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga107/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/ga107/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu102/gsp/
	usr/lib/firmware/nvidia/tu102/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu102/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu102/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu102/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu104/gsp/
	usr/lib/firmware/nvidia/tu104/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu104/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu104/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu104/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu106/gsp/
	usr/lib/firmware/nvidia/tu106/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu106/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu106/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu106/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu116/gsp/
	usr/lib/firmware/nvidia/tu116/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu116/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu116/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu116/gsp/gsp-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu117/gsp/
	usr/lib/firmware/nvidia/tu117/gsp/booter_load-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu117/gsp/booter_unload-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu117/gsp/bootloader-535.113.01.bin.zst
	usr/lib/firmware/nvidia/tu117/gsp/gsp-535.113.01.bin.zst
	kernel/arch/x86/crypto/chacha-x86_64.ko
	kernel/arch/x86/crypto/poly1305-x86_64.ko
	kernel/crypto/chacha_generic.ko
kernel/crypto/ecb.ko
	kernel/crypto/poly1305_generic.ko
	kernel/drivers/gpu/drm/drm_gpuvm.ko
	kernel/drivers/nvme/common/nvme-auth.ko
kernel/drivers/nvme/common/nvme-common.ko
	kernel/drivers/nvme/common/nvme-keyring.ko
	kernel/drivers/regulator/max77503-regulator.ko
	kernel/drivers/spi/spi-ljca.ko
	kernel/drivers/usb/misc/
	kernel/drivers/usb/misc/usb-ljca.ko
	kernel/fs/bcachefs/
	kernel/fs/bcachefs/bcachefs.ko
	kernel/lib/crypto/
	kernel/lib/crypto/libchacha.ko
	kernel/lib/crypto/libpoly1305.ko

Last edited by Trilby (2024-01-15 04:39:58)

Scimmia · 2024-01-15 04:33:21

ah, OK, that changes things, nobody had said that yet. nl6720 will hopefully be long at some point. I'll ping him on IRC

cirrus9 · 2024-01-15 04:44:48

Both of my images were much larger, not just the fall back. The image without KMS hook, has much less nvidia lines in it. I have to get some sleep, and can't fool with this anymore now.

If there is no resolution by tomorrow evening, I'll try to post the difference. I don't get off work until 17:00, so it will be late afternoon. Removing KMS cut it in half, and as per the posts
on other forums, that seems to be the difference. What is causing it is another matter, and is beyond my Linux technical ability .

Scimmia · 2024-01-15 04:54:41

Yeah, thinking about it, AMD and intel systems shouldn't be any different for the fallback. Something odd going on here for sure, but it's too late for me to dig into it too much, too.

kkoyung · 2024-01-15 05:24:08

When I upgrade to kernel 6.7.arch3-1, the image generation fails since my boot partition is full during upgrading. The incomplete images grow to >200MB. I then downgrade the kernel to 6.6.10.arch1-1, my image and fallback image go back to 100MB and 183MB respectively.

All images mentioned above contains nvidia KMS. So, I tried again with different combination, and got the following results. It seems that 6.7 is much larger than 6.6.10.

6.7.arch3-1 with nvidia KMS
image: 264MB
fallback: (disabled since my boot partition has not enough space)

6.6.10.arch1-1 with nvidia KMS
image: 100MB
fallback: 183MB

6.7.arch3-1 without nvidia KMS
image: 176MB
fallback: 261MB

6.6.10.arch1-1 without nvidia KMS
image: 12MB
fallback: 95MB

Edit: add extra test data

Last edited by kkoyung (2024-01-15 05:49:15)

LRitzdorf · 2024-01-15 05:33:08

Same issue here, upon upgrading to kernel 6.7. (I actually got the kernel bump and a very minor Nvidia upgrade simultaneously, so it wasn't initially clear whose fault it was.) I've expanded my ESP, and am currently just using the larger initramfs images.

On my system (Nvidia dGPU, Intel iGPU; using KMS), the standard image is 190MB, while the fallback is 248MB.
I don't have the originals for comparison, and would rather not roll back my kernel at the moment, so this may or may not actually be useful.

nl6720 · 2024-01-15 07:25:03

It's archlinux/mkinitcpio/mkinitcpio#238.

Since Linux 6.7, nouveau uses the GSP firmware so it's included in the initramfs and since the firmware files are huge, the initramfs gets bloated.
There's no simple solution for this. "firmware groups" would be a solution, but the patches are not yet merged in the kernel.

For now, the simplest workaround is to exclude the kms hook from the fallback initramfs. Edit /etc/mkinitcpio.d/linux.preset and change fallback_options to:

fallback_options="-S autodetect -S kms"

Alternatively disable fallback initramfs generation entirely:

PRESETS=('default')

If the size of the main initramfs image is the issue, then compress it more by using xz -9e.

jancici · 2024-01-15 07:34:32

I get warring message about low free space on boot partion during today update .... I did find this thread.

I dont have nvidia or amd graphics, using embeded intel one.
So my initramfs-linux.img has cca 16M and initramfs-linux-fallback.img has cca 248M. And my boot partion is 300M big, I am just ok for now ;-)

I think that we should update installation guide where you can find recommendation that boot partion should be at least 300M so new number should be 600M

nl6720 · 2024-01-15 07:39:16

Since most nvidia users presumably use the proprietary drivers, they shouldn't even use the kms hook. For everyone else, the issue is just with the fallback initramfs.

jancici wrote:

I think that we should update installation guide where you can find recommendation that boot partion should be at least 300M so new number should be 600M

4 GiB ought to be enough for anybody.

d.ALT · 2024-01-15 07:51:22

nl6720 wrote:

4 GiB ought to be enough for anybody.

1GiB for $ESP
+
2GiB for $BOOT (if \ when I need separate /boot)

M1CK431 · 2024-01-15 10:45:51

In my case, none of the workarounds suggested by @nl6720 works

My laptop is powered by hybrid Intel/Nvidia graphics and I discover that I don't really need the kms hook (only difference is a console resolution switch during boot => don't care).
In addition, since I'm using the proprietary Nvidia driver, the Nvidia wiki page state I should use the MODULES array instead of the kms hook.

After removing it from /etc/mkinitcpio.conf HOOKS array, my initramfs images:

-rwxr-xr-x 1 root root  38M 15 janv. 10:59  initramfs-linux-fallback.img
-rwxr-xr-x 1 root root 9,6M 15 janv. 10:59  initramfs-linux.img

nl6720 · 2024-01-15 11:46:45

This is actually a bug (or a missing feature, perhaps) in mkinitcpio.
I prepared a potential fix in archlinux/mkinitcpio/mkinitcpio!292 (untested!). Please test it if possible!

Last edited by nl6720 (2024-01-15 11:47:14)

Lone_Wolf · 2024-01-15 12:25:59

for reference my numbers :

mkinitcpio modules and hooks :

MODULES=(amdgpu ext4)
HOOKS=(base udev autodetect modconf block keyboard numlock fsck)

$ ls -l /boot
total 94156
-rw-r--r-- 1 root root    81920 11 dec 15:23 amd-ucode.img
-rw------- 1 root root 50209669 15 jan 10:59 initramfs-linux-fallback.img
-rw------- 1 root root 33106348 15 jan 10:59 initramfs-linux.img
-rw-r--r-- 1 root root      308  4 mrt  2023 refind_linux.conf
-rw-r--r-- 1 root root 13001216 15 jan 10:59 vmlinuz-linux
$

Scimmia · 2024-01-15 12:30:10

Yes, this is an nvidia-only issue (even if, for some reason, you have nvidia installed on an Intel system).

V1del · 2024-01-15 12:41:25

Didn't nouveau get support for loading GSP firmwares? Likely that will now lead to inclusion of all of those firmwares because nouveau can make use of them.

nl6720 · 2024-01-15 12:43:28

The firmware files are not that big of an issue. The issue is that mkinitcpio doesn't expect directory symlinks in the path to the firmware files, so it adds all the files individually thus bloating the image.

V1del · 2024-01-15 12:47:28

Ah yeah, didn't read the thread fully and only thought of that that they introduced GSP support so will likely lead to increasing size even when not using nvidia but just sticking with nouveau. But anyways, nice find. I'd have to dig up my laptop to test your draft, but that will have to wait until after work.

Arch Linux

#1 2024-01-15 02:19:10

Linux 6.7 kernel creating very large initramfs.

#2 2024-01-15 03:05:51

Re: Linux 6.7 kernel creating very large initramfs.

#3 2024-01-15 03:37:57

Re: Linux 6.7 kernel creating very large initramfs.

#4 2024-01-15 03:59:51

Re: Linux 6.7 kernel creating very large initramfs.

#5 2024-01-15 04:08:18

Re: Linux 6.7 kernel creating very large initramfs.

#6 2024-01-15 04:11:49

Re: Linux 6.7 kernel creating very large initramfs.

#7 2024-01-15 04:25:51

Re: Linux 6.7 kernel creating very large initramfs.

#8 2024-01-15 04:28:10

Re: Linux 6.7 kernel creating very large initramfs.

#9 2024-01-15 04:31:03

Re: Linux 6.7 kernel creating very large initramfs.

#10 2024-01-15 04:33:21

Re: Linux 6.7 kernel creating very large initramfs.

#11 2024-01-15 04:44:48

Re: Linux 6.7 kernel creating very large initramfs.

#12 2024-01-15 04:54:41

Re: Linux 6.7 kernel creating very large initramfs.

#13 2024-01-15 05:24:08

Re: Linux 6.7 kernel creating very large initramfs.

#14 2024-01-15 05:33:08

Re: Linux 6.7 kernel creating very large initramfs.

#15 2024-01-15 07:25:03

Re: Linux 6.7 kernel creating very large initramfs.

#16 2024-01-15 07:34:32

Re: Linux 6.7 kernel creating very large initramfs.

#17 2024-01-15 07:39:16

Re: Linux 6.7 kernel creating very large initramfs.

#18 2024-01-15 07:51:22

Re: Linux 6.7 kernel creating very large initramfs.

#19 2024-01-15 10:45:51

Re: Linux 6.7 kernel creating very large initramfs.

#20 2024-01-15 11:46:45

Re: Linux 6.7 kernel creating very large initramfs.

#21 2024-01-15 12:25:59

Re: Linux 6.7 kernel creating very large initramfs.

#22 2024-01-15 12:30:10

Re: Linux 6.7 kernel creating very large initramfs.

#23 2024-01-15 12:41:25

Re: Linux 6.7 kernel creating very large initramfs.

#24 2024-01-15 12:43:28

Re: Linux 6.7 kernel creating very large initramfs.

#25 2024-01-15 12:47:28

Re: Linux 6.7 kernel creating very large initramfs.

Board footer