Help with understanding the bisection process

Elizabeth · Yesterday 17:05:25

Hi! Recently ive been epxeriencing bugs with the kernel and as such wanna learn how to properly bisect it, however I keep getting confused on what exactly this process is supposed to look like.
Specifically, in this article https://docs.kernel.org/admin-guide/ver … lla-bissbs I see the following line

Remove all software that depends on externally developed kernel drivers or builds them automatically. That includes but is not limited to DKMS, openZFS, VirtualBox, and Nvidia’s graphics drivers (including the GPLed kernel module).

im confused on what exactly "externally developed" means, how im supposed to identify such software, and most importantly: how do I do this without compromising my actual setup?
As I understand it now, this would requite uninstalling major pieces of software from my machine, and I dont really want to do that, so im wondering whether ive misunderstood something, and I haven't then how people typically get around this?

cryptearth · Yesterday 17:21:56

"external" as in "not in the kernel", with examples listed
if you have an nvid gpu that means: remove anything nvid-proprietary but fallback to nouveou (or however that's spelled)
zfs? virtualbox? any other not-in-tree driver like for network?
it basically means: you need your system as clean as you end up after a fresh install before you go for anything else
if you don't want to (or can't) cripple your system down to such: use another drive, setup clean arch, and go on

Elizabeth · Yesterday 17:28:15

cryptearth wrote:

"external" as in "not in the kernel", with examples listed
if you have an nvid gpu that means: remove anything nvid-proprietary but fallback to nouveou (or however that's spelled)
zfs? virtualbox? any other not-in-tree driver like for network?
it basically means: you need your system as clean as you end up after a fresh install before you go for anything else
if you don't want to (or can't) cripple your system down to such: use another drive, setup clean arch, and go on

Is there some way I can check whether I have any software like that installed?

cryptearth · Yesterday 17:50:28

if you use out-of-tree kernel modules you should know
do you have an nvid gpu?
do you use zfs?
do you use virtualbox?
haver ever needed tinkering with your network to get it working?

Elizabeth · Yesterday 18:04:29

cryptearth wrote:

if you use out-of-tree kernel modules you should know
do you have an nvid gpu?
do you use zfs?
do you use virtualbox?
haver ever needed tinkering with your network to get it working?

None to all of those but why is DKMS among those examples?

dimich · Yesterday 18:10:26

Elizabeth wrote:

Is there some way I can check whether I have any software like that installed?

$ find /usr/lib/modules/$(uname -r) -type f -name '*.ko*' -print0 | xargs -r0 pacman -Qoq | sort -u

Bisection implies building the kernel from specific commit and testing whether it's "good" or "bad". Can you test the kernel in virtual environment instead of real hardware? It might turn out to be faster and easier.
If you want to bisect particular in-tree module, it might be not necessary to build entire kernel, just bisect that module standalone by building it for your current kernel.

Some time ago for bisecting entire kernel on real hardware with necessary dkms modules I took linux package sources, modified PKGBUILD to use git repository. Then downloaded kernel sources (makepkg's -o option) and initiated git bisect in fetched sources. Then on each bisect step built linux and linux-headers packages (without linux-docs) from existing sources (makepkg's -e option) and installed packages in a single transaction. All required out-of-tree dkms modules were built automatically on package upgrade.

cryptearth · Yesterday 18:48:14

Elizabeth wrote:

None to all of those but why is DKMS among those examples?

i question that kernel bissection is within your scope of skill

dkms - dynamic kernel module suppor
basically what it does: compiles the module each time you change the kernel from scratch
also: it's still an out of tree module which may could conflict with whatever goes wrong with the kernel itself
the idea is: to remove EVERYTHING EXCEPT from JUST THE MOST BASIC SYSTEM (sorry for caps but i feel the need to stress WHY dkms should also be removed: necause whatever module you build could be the reason of your issues instead of the kernel itself)
if you have dkms ot means you likely use at least some out of tree module

the entire point of bissecting the kernel is to figure if and which kernel commit is the reason for your issues - or some out of tree module not compatible with the current kernel version because it wasn't updated to support it

@dimich
OP unfortunate not explain what kind of issue they mean exactly by "expperiencing kernel bugs" - but its likely hardware or out-of-tree module related - running in a clean sterrile VM likely doesn't catch it

seth · Yesterday 20:11:31

https://bbs.archlinux.org/viewtopic.php?id=314837

@Elizabeth did you file a bug w/ AMD and did they ask you to bisect the problem?
As mentioned before, you can ping loqs at https://bbs.archlinux.org/viewtopic.php … 9#p2309389 and will likely receive assistance w/ the bisection in the form of compiled bisection kernels.

Otherwise, did you read the linked https://wiki.archlinux.org/title/Bisect … s_with_Git and the there linked https://wiki.archlinux.org/title/Kernel … egressions and https://wiki.gentoo.org/wiki/Kernel_git-bisect ?
A major pitfall when bisecting AUR packages is to NOT bisect the AUR git (ie. the package) but the nested kernel git.

The problem w/ dkms is the "builds them automatically" part as those builds stand a > 0 chance to fail against random kernel revisions.

@crypthearth

i question that kernel bissection is within your scope of skill

I assume changing that is why this thread was opened?

cryptearth · Yesterday 20:53:29

seth wrote:

@crypthearth
i question that kernel bissection is within your scope of skill
I assume changing that is why this thread was opened?

point taken and full ack
linux is free of charge because many people spending thier free time and money helping to improve it - and i happily invite everyone to take the challenge (gromit likely can be of great help here)
but on the other hand, let's dial back down to: I, that's my point if view, see Arch as at least targetting towards more ecperienced users (c'mon, even I skipped "bring along an editir and tools to setup your network" first time I tried after a close friend already used it for years and somehow was able to get EasyAntiCheat working for GhostRecon: Wildlands - something I still strughle with today) - so at least have an overview of your hardware and software and to understand "get all not-in-tree modules out of your way before you try to start this journey" is the bare minimum I just expect from someobe who's willing to go as deep as to bissect kernel commits
granted, once setup the actual bissection workflow, re-compile the kernel from source (which, depend on system resources and caching can take everything from a few minutes to over (maybe couple of) hours) and reboot about one to a couple of dozen times, isn't actually that hard - but if one fail to understand even the op quote it at least require quite some catch up to even get at the point where some usefull report ("i have this problem, after bissection it showed up after commit X") comes out
the initial "i encountered some kernel bugs" is as useless as many others "i have an issue (with software X)": it lacks (a lot of detailed) crucial information to even give smart brains like yourself of gromit ANY point to start with
unless we know what SPECIFIC issues OP encounters EXACTLY this entire topic of "uhm, I have some issues ... and [I think] it's related to some kernel update" is pretty much nonesense, as in "yea, ok, and how exactly do you think us to give you any helpfull input if you don't even tell us your EXACT issue(s)?"

don't get me wrong, but if any getting part of "the (salty) arch community - and what it turned me into" told me: "it doesn't work" is just not any helpful issue description AT ALL

and this gets back to about mid-2000s when i started my journey into programming: the vast majority of users just uses <google> to only search for the next best forum to drop thier question in - somewhat in the style of "yea, i'm the first and only who "invented" gravity and fire" - and although I can just ignore a lot of those "yea, but teacher not told us and my deadline is in less than 6h" as pretty much nonesense (when you take some course of advanced computing you're usually at an age your teacher can expect you to at least know how to use google) - but when someone goes as deep as bissecting the kernel on Arch - I just expect them to at least have some more advanced knowledge above "how can i detect if i use out of tree modules" - and as pointed out countless times: I'm just that crude a**home to write it down exactly as it comes into my head (because, yes, this forum has a great part in have me got this way over the past 4 years i use arch)
hence i'm not sorry for just writing some lines like: "@op: do you feel up for the task? as you seem to lack required basics!"

sidenote: sure, this doesn't really belong here, but honestly: if one can't even get a few more words out other than "i have an issue" to me they already failed the one important task of "write some USEFUL (to others) issue report!" - or fail to read this BBS's rules "[vanilla] arch ONLY!" (maybe the registration captcha should be extended to something that just fails on arch forks but only works on vanilla arch)

loqs · Yesterday 23:02:06

If you have anything external loaded the kernel will mark itself as tainted and you can check for that in dmesg. How reproducible is the issue you have; ideally for bisection it should be 100% reproducible to avoid false positives/negatives? Do you know which built kernel introduced the issue?

Last edited by loqs (Yesterday 23:02:43)

seth · Today 06:42:22

See https://bbs.archlinux.org/viewtopic.php?id=314837 - appared w/ 7.2rc1, from the description the wayland related symptom seems reliably reproducible on at least niri (maybe less so on sway)

@cryptearth, while it would certainly have been useful to link their original thread, the underlying issue is actually not immediately relevant to the question "how do you bisect the kernel"
Depending on ones experience level, any obstacle on your journey might appear either as a flight of stairs or a mountain.
But there're people who climbed the Everest and K9 and there're people who surrender to a flight of stairs. You never know before you give it a chance

Elizabeth · Today 09:08:57

seth wrote:

https://bbs.archlinux.org/viewtopic.php?id=314837
@Elizabeth did you file a bug w/ AMD and did they ask you to bisect the problem?
As mentioned before, you can ping loqs at https://bbs.archlinux.org/viewtopic.php … 9#p2309389 and will likely receive assistance w/ the bisection in the form of compiled bisection kernels.
Otherwise, did you read the linked https://wiki.archlinux.org/title/Bisect … s_with_Git and the there linked https://wiki.archlinux.org/title/Kernel … egressions and https://wiki.gentoo.org/wiki/Kernel_git-bisect ?
A major pitfall when bisecting AUR packages is to NOT bisect the AUR git (ie. the package) but the nested kernel git.
The problem w/ dkms is the "builds them automatically" part as those builds stand a > 0 chance to fail against random kernel revisions.
@crypthearth
i question that kernel bissection is within your scope of skill
I assume changing that is why this thread was opened?

Yep, made an issue and got told to bisect the kernel, so trying to figure that out right now. Also thank you for linking the two articles!
As a side question, how viable do you think making a USB flash drive dedicated to testing bisected kernels is? As in, I make a flash drive with some barebones software and package managment stuff and boot into that using the bisected kernels? Since the virtual machine option doesn't work and I'm still not quite confident enough to use my main machine to do this stuff.

Elizabeth · Today 10:30:09

loqs wrote:

If you have anything external loaded the kernel will mark itself as tainted and you can check for that in dmesg. How reproducible is the issue you have; ideally for bisection it should be 100% reproducible to avoid false positives/negatives? Do you know which built kernel introduced the issue?

Checking dmesg I dont see any mentions of the kernel being tainted but also I know I have the v4l2loopback package installed, but that its also a module that dinamically loaded (e.g. when I select an option in OBS)
As for reproducability, I dont know. Im not sure what a bisection being reproducible means either? And I know that 7.2.0-rc1-1-mainline introduced the issue in question, though im less interested in helping with my exact issue and more in figuring out the workflow of kernel bisection.

EDIT: if you are curious heres the forums thread with the exact issue im facing: https://bbs.archlinux.org/viewtopic.php?id=314837
And the issue I opened on amd's gitlab page: https://gitlab.freedesktop.org/drm/amd/ … items/5839

Last edited by Elizabeth (Today 10:31:15)

seth · Today 12:51:49

viable do you think making a USB flash drive dedicated to testing bisected kernels is

Not, because you're moving away from the known malfunction setup.
Avoid changing multiple variables at a time.
Once a commit is identified and AMD contests that it's a bug it might be necessary to check the commit against a different userspace stack - right now you're only compromising reproducibility.

I'm still not quite confident enough to use my main machine to do this stuff.

As long as you're only changing the kernel and have a reliable backup kernel (lts) installed in parallel, chances for this to go sideways are virtually zero.

Im not sure what a bisection being reproducible means either?

What loqs meant is "How reliable to do run into this w/ the affected kernels? Is if intermittent or immediate?"

figuring out the workflow of kernel bisection

Make sure to have seen the wiki links in #8, ask back when at any point you run into problems.
I'll stress that you want to bisect the git history of the kernel and NOT the one of the package itself.

loqs · Today 15:39:52

$ git clone https://aur.archlinux.org/linux-mainline.git
$ cd linux-mainline/
$ makepkg -Codd
$ cd src/linux-mainline/
$ git bisect start
status: waiting for both good and bad commits
$ git bisect bad v7.2-rc1
status: waiting for good commit(s), bad commit known
$ git bisect good v7.1
Bisecting: 6952 revisions left to test after this (roughly 13 steps)
[056e065a6b6e01ab54bb9770c0d5a15350e571e2] Merge tag 'sound-7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
$ cd ../..

At this point you want to make the following changes to the PKGBUILD to change the source to the commit above, add a pkgver function and remove building the documentation that doubles the build time:

diff --git a/PKGBUILD b/PKGBUILD
index 927b651..28cd719 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -7,7 +7,7 @@
 pkgbase=linux-mainline
 #pkgbase=linux-custom       # Build kernel with a different name
 _tag=v7.3-rc3
-pkgver=7.3rc3
+pkgver=7.1.r7442.g056e065a6b6e
 pkgrel=1
 pkgdesc="Linux Mainline"
 arch=(x86_64)
@@ -34,13 +34,6 @@ makedepends=(
   xz
   zlib
   zstd
-
-  # htmldocs
-  graphviz
-  imagemagick
-  python-sphinx
-  python-yaml
-  texlive-latexextra
 )
 options=(
   !debug
@@ -48,7 +41,7 @@ options=(
 )
 _srcname=linux-mainline
 source=(
-  "$_srcname::git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git#tag=$_tag"
+  "$_srcname::git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git#commit=056e065a6b6e01ab54bb9770c0d5a15350e571e2"
   config.x86_64         # the main kernel config file
 )
 validpgpkeys=(
@@ -56,7 +49,7 @@ validpgpkeys=(
   647F28654894E3BD457199BE38DBBDC86092693E  # Greg Kroah-Hartman
   83BC8889351B5DEBBB68416EB8AC08600F108CDF  # Jan Alexander Steffens (heftig)
 )
-sha256sums=('ba8e204ee407d3f9430580406ccab817b8293332f8e6581363a6617ad17e2c04'
+sha256sums=('SKIP'
             '9f6e1be50d2ebb69a23b14799ccf23462c80e29f284ce3b205186921c5c3bf3d')
 
 export KBUILD_BUILD_HOST=archlinux
@@ -89,15 +82,17 @@ prepare() {
   echo "Prepared $pkgbase version $(<version)"
 }
 
+pkgver() {
+  cd $_srcname
+
+  git describe | sed -E 's/^v//;s/([^-]*-g)/r\1/;s/-/./g;s/\.rc/rc/'
+}
+
 build() {
   cd $_srcname
 
-  make htmldocs SPHINXOPTS=-QT &
-  local pid_docs=$!
-
   make all
   make -C tools/bpf/bpftool vmlinux.h feature-clang-bpf-co-re=1
-  wait $pid_docs
 }
 
 _package() {
@@ -255,32 +250,9 @@ _package-headers() {
   ln -sr "$builddir" "$pkgdir/usr/src/$pkgbase"
 }
 
-_package-docs() {
-  pkgdesc="Documentation for the $pkgdesc kernel"
-
-  cd $_srcname
-  local builddir="$pkgdir/usr/lib/modules/$(<version)/build"
-
-  echo "Installing documentation..."
-  local src dst
-  while read -rd '' src; do
-    dst="${src#Documentation/}"
-    dst="$builddir/Documentation/${dst#output/}"
-    install -Dm644 "$src" "$dst"
-  done < <(
-    find Documentation \( -name '.*' -o -name __pycache__ \) -prune \
-      -o \! -type d -print0
-  )
-
-  echo "Adding symlink..."
-  mkdir -p "$pkgdir/usr/share/doc"
-  ln -sr "$builddir/Documentation" "$pkgdir/usr/share/doc/$pkgbase"
-}
-
 pkgname=(
   "$pkgbase"
   "$pkgbase-headers"
-  "$pkgbase-docs"
 )
 for _p in "${pkgname[@]}"; do
   eval "package_$_p() {

Then build then install the newly built linux-mainline package alongside your current kernel package, reboot and test then:

cd linux-mainline/src/linux-mainline
git bisect $result # where $result is the test result good or bad
cd ../..

Then edit the PKGBUILD to change the new commit git bisect will have given you. Then you repeat the build test report steps until the bisection is complete.

Elizabeth · Today 15:53:19

Okay so im just encountering this weird error when i try to build linux-git

==> Starting build()...
  SYNC    include/config/auto.conf
  DESCEND bpf/resolve_btfids
  DESCEND objtool
  GEN     usr/initramfs_data.cpio
  COPY    usr/initramfs_inc_data
  AS      usr/initramfs_data.o
  UPD     init/utsversion-tmp.h
  CC      init/version.o
  AR      usr/built-in.a
  AR      init/built-in.a
  GEN     kernel/kheaders_data.tar.xz
  CC [M]  drivers/net/ethernet/cadence/macb_main.o
drivers/net/ethernet/cadence/macb_main.c: In function ‘macb_probe’:
drivers/net/ethernet/cadence/macb_main.c:5951:15: error: implicit declaration of function ‘macb_alloc_tieoff’ [-Wimplicit-function-declaration]
 5951 |         err = macb_alloc_tieoff(bp);
      |               ^~~~~~~~~~~~~~~~~
drivers/net/ethernet/cadence/macb_main.c:5973:9: error: implicit declaration of function ‘macb_free_tieoff’ [-Wimplicit-function-declaration]
 5973 |         macb_free_tieoff(bp);
      |         ^~~~~~~~~~~~~~~~
make[6]: *** [scripts/Makefile.build:290: drivers/net/ethernet/cadence/macb_main.o] Error 1
make[5]: *** [scripts/Makefile.build:550: drivers/net/ethernet/cadence] Error 2
make[4]: *** [scripts/Makefile.build:550: drivers/net/ethernet] Error 2
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [scripts/Makefile.build:550: drivers/net] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [scripts/Makefile.build:550: drivers] Error 2
make[2]: *** Waiting for unfinished jobs....
  CC [M]  kernel/kheaders.o
make[1]: *** [/home/Elizabeth/build/bisect/linux-git/src/linux-torvalds/Makefile:2218: .] Error 2
make: *** [Makefile:248: __sub-make] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

Idk if this is a known issue or smth but I'm too busy rn to be dealing with this so ill come back to it later

Last edited by Elizabeth (Today 16:33:51)

loqs · Today 16:27:12

Fixed by https://git.kernel.org/pub/scm/linux/ke … 6d578e4791 however you should not be hitting that issue as the causal commit was only added during the 7.3 merge:

git tag --contains 5262eab9462adffd73a84409dee0cbf57fe31da3
v7.3-rc1
v7.3-rc2
v7.3-rc3

Elizabeth · Today 16:39:29

loqs wrote:

Fixed by https://git.kernel.org/pub/scm/linux/ke … 6d578e4791 however you should not be hitting that issue as the causal commit was only added during the 7.3 merge:
git tag --contains 5262eab9462adffd73a84409dee0cbf57fe31da3
v7.3-rc1
v7.3-rc2
v7.3-rc3

Okay wait can I ask like, what exactly is this doing? Is it compiling the kernel, and if yes then id just like to ask whether the files in the directory with the PKGBUILD are where I set up stuff like which modules to skip and what drivers to omit?

loqs · Today 16:45:30

The fix for the issue in post #16 appears to be commit https://git.kernel.org/pub/scm/linux/ke … 78e4791and in that commits message it states the issue was introduced by https://git.kernel.org/pub/scm/linux/ke … f57fe31da3. `git tag --contains 5262eab9462adffd73a84409dee0cbf57fe31da3` prints all tags containing that commit which shows only the 7.3 release candidates so I was wondering how you could have it that issue while bisecting between v7.1 and v7.2-rc1. It is not building anything just querying git.

Elizabeth · Today 16:49:23

Sorry this is my bad cause I quoted you when I shouldnt have. what I meant to ask is this:
what does building linux-git do? Am I just compiling the kernel? And if so are the files in PKGBUILD directory the ones I wanna modify to remove stuff like unneeded modules?

Last edited by Elizabeth (Today 16:49:45)

loqs · Today 17:00:42

Elizabeth wrote:

what does building linux-git do? Am I just compiling the kernel? And if so are the files in PKGBUILD directory the ones I wanna modify to remove stuff like unneeded modules?

The PKGBUILD automates the process of building the kernel. The config file in the same directory as the PKGBUILD controls options such as what modules will be built.

Elizabeth · Today 17:32:28

loqs wrote:

The PKGBUILD automates the process of building the kernel. The config file in the same directory as the PKGBUILD controls options such as what modules will be built.

Thank you for the answer!

Ive also tried recloning the git repo and the error is still there
So my only option now seems to be to manually build the kernel?

Last edited by Elizabeth (Today 17:41:27)

loqs · Today 17:42:58

What kernel version are you trying to build?

Elizabeth · Today 17:56:01

loqs wrote:

What kernel version are you trying to build?

7.2.1-rc1 i think? When i did a coarse bisection using the prebuilt kernels here https://pkgbuild.com/~gromit/linux-bisection-kernels/ that was the first version where my issue appeared
On the off chance youre volunteering to build it for me I should mention im investigating a bug in the amdgpu module in case it helps.

loqs · Today 18:35:02

Did you set the source to v7.1-rc1 in the PKGBUILD or did you change the git checkout in src/linux to v7.1-rc1 and use `makepkg -e` to use the existing src directory as that error to me indicates you are building a commit between 7.2 and 7.3 instead of 7.1 and 7.2.

Arch Linux

#1 Yesterday 17:05:25

Help with understanding the bisection process

#2 Yesterday 17:21:56

Re: Help with understanding the bisection process

#3 Yesterday 17:28:15

Re: Help with understanding the bisection process

#4 Yesterday 17:50:28

Re: Help with understanding the bisection process

#5 Yesterday 18:04:29

Re: Help with understanding the bisection process

#6 Yesterday 18:10:26

Re: Help with understanding the bisection process

#7 Yesterday 18:48:14

Re: Help with understanding the bisection process

#8 Yesterday 20:11:31

Re: Help with understanding the bisection process

#9 Yesterday 20:53:29

Re: Help with understanding the bisection process

#10 Yesterday 23:02:06

Re: Help with understanding the bisection process

#11 Today 06:42:22

Re: Help with understanding the bisection process

#12 Today 09:08:57

Re: Help with understanding the bisection process

#13 Today 10:30:09

Re: Help with understanding the bisection process

#14 Today 12:51:49

Re: Help with understanding the bisection process

#15 Today 15:39:52

Re: Help with understanding the bisection process

#16 Today 15:53:19

Re: Help with understanding the bisection process

#17 Today 16:27:12

Re: Help with understanding the bisection process

#18 Today 16:39:29

Re: Help with understanding the bisection process

#19 Today 16:45:30

Re: Help with understanding the bisection process

#20 Today 16:49:23

Re: Help with understanding the bisection process

#21 Today 17:00:42

Re: Help with understanding the bisection process

#22 Today 17:32:28

Re: Help with understanding the bisection process

#23 Today 17:42:58

Re: Help with understanding the bisection process

#24 Today 17:56:01

Re: Help with understanding the bisection process

#25 Today 18:35:02

Re: Help with understanding the bisection process

Board footer