You are not logged in.

#1 2023-02-25 18:30:12

mz
Member
Registered: 2015-07-11
Posts: 43

[SOLVED] Need help with lts git bisect

This is related to a previously reported issue:

https://bbs.archlinux.org/viewtopic.php?id=282125

I had abandoned my attempt at a git bisect at that time because it seemed that I needed to regress my whole system to get the older builds to work. Since the lts kernel worked fine I decided to stick with that.

The lts kernel now has the problem. Specifically, linux-lts-5.15.94-1 is good and linux-lts-6.1.12-1 is bad. Since these are adjacent versions, maybe I will have better luck with a bisect since the releases are separated by only a week.

I need some assistance setting up my git bisect to be specific to arch linux-lts; this is not obvious to me.

Last edited by mz (2023-03-05 18:38:49)

Offline

#2 2023-02-26 09:15:27

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,427

Re: [SOLVED] Need help with lts git bisect

These aren't adjacent versions by any definition of the word, you get the full development brunt of the last year you need to bisect against. You will have better luck picking the first bad 6.0.x version and bisecting it against the (apparently then) properly working 5.19.x (or even a stable break within 6.0)?

Loqs might pop in here on their own otherwise here's a gist of the process https://bbs.archlinux.org/viewtopic.php … 4#p1949234 - you will need to adjust the versions and commits to checkout appropriately.

Last edited by V1del (2023-02-26 09:21:26)

Offline

#3 2023-02-26 13:55:15

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

https://drive.google.com/drive/folders/ … share_link contains kernels I have built helping with a previous 6.0 to 6.1 bisection.  You can reuse them until your results differ.  Please do test both the 6.0 and 6.1 kernels to ensure you have one good and one bad otherwise the bisection will not work.

Offline

#4 2023-02-26 18:32:37

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

So far I have tried these, and all of them are bad. I will try the rest later today but I don't think I will find a good one unless I completely misunderstand the version numbering.

linux-6.0-1-x86_64.pkg.tar.zst
linux-6.0.r3239.ga47e60729d96-1-x86_64.pkg.tar.zst
linux-6.0.r7594.g33e591dee915-1-x86_64.pkg.tar.zst
linux-6.0rc3.r858.g6cf5e9066dd3-1-x86_64.pkg.tar.zst
linux-6.0rc4.r953.g2c119d9982b1-1-x86_64.pkg.tar.zst
linux-6.0rc6.r1412.g454b20e19322-1-x86_64.pkg.tar.zst
linux-6.0rc7.r1795.g915b96c52763-1-x86_64.pkg.tar.zst

Maybe it would be more fruitful to try the ALA to test 5.x kernels with x > 15.

Last edited by mz (2023-02-26 18:39:54)

Offline

#5 2023-02-26 18:38:35

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

No my mistake.  You want 5.19 to 6.0. https://drive.google.com/drive/folders/ … share_link
Edit:
You want linux-5.19.r8199.g78acd4ca4334-1-x86_64.pkg.tar first as that is the mid point commit.

Last edited by loqs (2023-02-26 18:43:45)

Offline

#6 2023-02-26 18:43:37

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Thanks. I will try 5.19 from the ALA first. If it's good I'll try your kernels. If not I will 'bisect' the ALA.

Edit: Well, it looks like 5.19.9-arch1-1 exhibits the problem so I will look back from there. I forgot to check if there were non-lts 5.x versions later than 5.15.

Last edited by mz (2023-02-26 19:00:12)

Offline

#7 2023-02-26 23:12:41

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

5.19

# pacman -U https://archive.archlinux.org/packages/l/linux/linux-5.19.arch1-1-x86_64.pkg.tar.zst https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.19.arch1-1-x86_64.pkg.tar.zst

You can replace 5.19 with any release back to 5.15,  use below for 5.14 back to 5.5

# pacman -U https://archive.archlinux.org/packages/l/linux/linux-5.14.1.arch1-1-x86_64.pkg.tar.zst https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.14.1.arch1-1-x86_64.pkg.tar.zst

Offline

#8 2023-02-27 03:39:07

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Finally making progress on this:

5.16.16-arch1-1 is good.
5.17.1-arch1-1 is bad.

Do you happen to have some kernels in between those two?

Offline

#9 2023-02-27 14:46:54

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

5.16 does not build with gcc 12:
  git cherry-pick -n 52a9dab6d892763b2a8334a568bd4e2c1a6fde66
  git cherry-pick -n 82880283d7fcd0a1d20964a56d6d1a5cc0df0713
5.16 does not build with pahole 1.24 -1 using pahole 1.22-1

Using config from 5.17.2 https://raw.githubusercontent.com/archl … unk/config

$ git bisect start
status: waiting for both good and bad commits
$ git bisect bad v5.17
status: waiting for good commit(s), bad commit known
$ git bisect good v5.16
Bisecting: 6995 revisions left to test after this (roughly 13 steps)
[22ef12195e13c5ec58320dbf99ef85059a2c0820] Merge tag 'staging-5.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging

https://drive.google.com/file/d/1V-ol65 … share_link linux-5.16-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/12yGaB_ … share_link linux-headers-5.16-1-x86_64.pkg.tar.zst

https://drive.google.com/file/d/1MWwKcy … share_link linux-5.17-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Whdi0w … share_link linux-headers-5.17-1-x86_64.pkg.tar.zst

https://drive.google.com/file/d/1HuGjbh … share_link linux-5.16.r7203.g22ef12195e13-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/11miLAN … share_link linux-headers-5.16.r7203.g22ef12195e13-1-x86_64.pkg.tar.zst

Offline

#10 2023-02-28 03:16:27

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Questions:

Do the two cherry-picks make it compile with gcc 12 or do I need to downgrade?

I'm not sure where to get the source. linux-git in AUR is flagged as out of date for more than two years and I tried this and failed:

$ git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/linux"
Cloning into 'packages'...
fatal: unable to connect to git.archlinux.org:
git.archlinux.org[0: 2a01:4f9:c010:2636::1]: errno=Permission denied
git.archlinux.org[1: 95.216.195.133]: errno=No route to host

Which I got from the link posted by V1del above.

Offline

#11 2023-02-28 13:39:20

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

Yes the cherry-picks allow the build to work with gcc 12,  pahole was the only package I had to downgrade.

$ git clone https://github.com/archlinux/svntogit-packages --single-branch --branch 'packages/linux' linux
$ cd linux/trunk/
$ curl -o PKGBUILD.diff http://ix.io/4ptx
$ git apply -v PKGBUILD.diff
$ makepkg -rsi

Edit:
You may have to remove the cherry-picks before `git bisect $result` at each step and reapply or track the bisection progress in a checkout but build in a clean chroot

extra-x86_64-build -- -I /var/cache/pacman/pkg/pahole-1.22-1-x86_64.pkg.tar.zst

Last edited by loqs (2023-02-28 13:56:30)

Offline

#12 2023-03-01 02:26:20

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

$ git cherry-pick -n 52a9dab6d892763b2a8334a568bd4e2c1a6fde66
fatal: bad object 52a9dab6d892763b2a8334a568bd4e2c1a6fde66
$ git cherry-pick -n 82880283d7fcd0a1d20964a56d6d1a5cc0df0713
fatal: bad object 82880283d7fcd0a1d20964a56d6d1a5cc0df0713

Same error while in linux and linux/trunk.

Edit: I went ahead and executed makepkg, expecting it to fail, but it successfully built linux-5.16.r7203.g22ef12195e13-1-x86_64.pkg.tar.zst, which booted, and proved to be a 'bad' kernel in the git bisect sense.

$ git bisect start
status: waiting for both good and bad commits
$ git bisect bad v5.17
error: Bad rev input: v5.17

'git tag' returns nothing, so I'm looking for how to specify the good and bad endpoints.

Another edit: Based on the archived releases I tried, I got these commits from git log:

$ git bisect bad cb8242a510d80d4e58215a639053fa61954e1b9d
status: waiting for good commit(s), bad commit known
$ git bisect good 3b39577d2ed83108724c83c5e91dfced35e68459
Bisecting: 2 revisions left to test after this (roughly 2 steps)
error: Your local changes to the following files would be overwritten by checkout:
	trunk/PKGBUILD
	trunk/config
Please commit your changes or stash them before you switch branches.
Aborting

So what should I do with those two files?

Last edited by mz (2023-03-01 04:20:55)

Offline

#13 2023-03-01 16:15:41

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

I think you are doing the bisect in the wrong directory.  From the directory with the PKGBUILD

$ cd src/linux/
$ git bisect start
$ git bisect bad v5.17
$ git bisect good v5.16
Bisecting: 6995 revisions left to test after this (roughly 13 steps)
[22ef12195e13c5ec58320dbf99ef85059a2c0820] Merge tag 'staging-5.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
$ git bisect bad
Bisecting: 4385 revisions left to test after this (roughly 12 steps)
error: Your local changes to the following files would be overwritten by checkout:
	tools/objtool/check.c
Please commit your changes or stash them before you switch branches.
Aborting
$ git reset --hard # Undo the cherry-picks
HEAD is now at 22ef12195e13 Merge tag 'staging-5.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
$ git bisect bad
Bisecting: 4385 revisions left to test after this (roughly 12 steps)
[9bcbf894b6872216ef61faf17248ec234e3db6bc] Merge tag 'media/v5.17-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
$ git cherry-pick -n 52a9dab6d892763b2a8334a568bd4e2c1a6fde66 #redo the cherry-picks
$ git cherry-pick -n 82880283d7fcd0a1d20964a56d6d1a5cc0df0713
Auto-merging tools/objtool/check.c
cd ../..
makepkg -ersi

Edit:
https://drive.google.com/file/d/1CEmDJp … share_link linux-5.16.r2817.g9bcbf894b687-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1QCWtaV … share_link linux-headers-5.16.r2817.g9bcbf894b687-1-x86_64.pkg.tar.zst

Last edited by loqs (2023-03-01 16:40:53)

Offline

#14 2023-03-02 03:45:07

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Thank you for your patience. I've got the process down now and it will take a few days to grind through it until the end.

Offline

#15 2023-03-02 15:14:15

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

I was hoping for smooth sailing, but of course there's always something... I have no idea what this means:

  ZSTD    /home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-rc5-1-07203-g22ef12195e13/kernel/sound/x86/snd-hdmi-lpe-audio.ko.zst
  ZSTD    /home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-rc5-1-07203-g22ef12195e13/kernel/sound/xen/snd_xen_front.ko.zst
  ZSTD    /home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-rc5-1-07203-g22ef12195e13/kernel/virt/lib/irqbypass.ko.zst
  DEPMOD  /home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-rc5-1-07203-g22ef12195e13
Warning: 'make modules_install' requires /doesnt/exist. Please install it.
This is probably in the kmod package.
rm: cannot remove '/home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-1-07203-g22ef12195e13/source': No such file or directory
rm: cannot remove '/home/mark/bisect/linux/trunk/pkg/linux/usr/lib/modules/5.16.0-1-07203-g22ef12195e13/build': No such file or directory
==> ERROR: A failure occurred in package_linux().
    Aborting...

Offline

#16 2023-03-02 15:23:40

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

https://bugs.archlinux.org/task/69869#comment199314

thx1138 wrote:

For git bisect, the simplest revision seems to be just repeating `make -s kernelrelease > version` at the beginning of the build() function, after `cd $_srcname`. There is no reason to remove this from the prepare() function.

Edit:
You may need to run  `make -s kernelrelease > version` after git bisect $result before git cherry-pick instead of at the start of build to avoid the cherry-picks being detected by `make -s kernelrelease`
I avoid this issue as I build each bisection step as a clean build,  tracking the bisection results in a separate git checkout.

Last edited by loqs (2023-03-02 21:34:29)

Offline

#17 2023-03-03 15:41:43

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

This is just plain weird, and I wonder if I just need to start over. Every kernel I build and boot self-identifies (uname -r) as 5.16.0-rc2-1-07203-g22ef12195e13.

I can see it being installed from a correctly named package file and I have no stray kernels in /boot. Is it possible for the version string in the kernel to be wrong?

Offline

#18 2023-03-03 16:00:20

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] Need help with lts git bisect

mz wrote:

Is it possible for the version string in the kernel to be wrong?

Yes.  In src/linux/ check the contents of the file 'version' and the outputs of:

$ git describe
$ make kernelrelease

Last edited by loqs (2023-03-03 16:59:55)

Offline

#19 2023-03-04 00:32:31

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

$ git describe
v5.16-rc2-633-g15bb79910fe7
$ make kernelrelease
5.16.0-rc2-1-07203-g22ef12195e13
$ cat version
5.16.0-rc2-1-07203-g22ef12195e13

If there is a way to make it do the right thing I would like to know it, but it's not really important since I finally got my first 'good' kernel so I now believe that the bisection is progressing.

Last edited by mz (2023-03-04 03:51:38)

Offline

#20 2023-03-04 04:05:52

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Latest fun problem:

make[4]: *** No rule to make target 'libbpf_version.h', needed by '/home/mark/bisect/linux/trunk/src/linux/tools/bpf/resolve_btfids/libbpf/staticobjs/libbpf.o'.  Stop.
make[3]: *** [Makefile:158: /home/mark/bisect/linux/trunk/src/linux/tools/bpf/resolve_btfids/libbpf/staticobjs/libbpf-in.o] Error 2
make[2]: *** [Makefile:44: /home/mark/bisect/linux/trunk/src/linux/tools/bpf/resolve_btfids//libbpf/libbpf.a] Error 2
make[1]: *** [Makefile:72: bpf/resolve_btfids] Error 2
make: *** [Makefile:1371: tools/bpf/resolve_btfids] Error 2
make: *** Waiting for unfinished jobs....

I have pahole 1.22-1, in case that's related.

Edit:
Tried pahole 1.22-2; no help; reverted to 1.22-1.
Edited .config:

CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_BTF is not set

hoping it will not invalidate my results. Building now...

Last edited by mz (2023-03-04 17:53:42)

Offline

#21 2023-03-05 18:37:20

mz
Member
Registered: 2015-07-11
Posts: 43

Re: [SOLVED] Need help with lts git bisect

Reached the end of the line:

$ git bisect good
527bab0473f28236e4587c7870586275c1ef5516 is the first bad commit
commit 527bab0473f28236e4587c7870586275c1ef5516
Author: Tilak Tangudu <tilak.tangudu@intel.com>
Date:   Tue Nov 16 21:22:38 2021 +0530

    drm/i915/rpm: Enable runtime pm autosuspend by default

    Let's enable runtime pm autosuspend by default everywhere.
    So, we can allow D3hot and bigger power savings on idle scenarios.

    But at this time let's not touch the autosuspend_delay time,
    what caused some regression on our previous attempt.

    Also, the latest identified issue on GuC PM has been fixed by
    commit 1a52faed3131 ("drm/i915/guc: Take GT PM ref when deregistering
    context")

    v1: Enable runtime pm autosuspend by default for Gen12
    and later versions.

    v2: Enable runtime pm autosuspend by default for all
    platforms(Syrjala Ville)

    v3: Change commit message(Nikula Jani)

    Signed-off-by: Tilak Tangudu <tilak.tangudu@intel.com>
    Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Link: https://patchwork.freedesktop.org/patch/msgid/20211116155238.3226516-1-tilak.tangudu@intel.com

 drivers/gpu/drm/i915/intel_runtime_pm.c | 3 +++
 1 file changed, 3 insertions(+)

Since this thread was all about getting the bisect to work I am calling it solved. Original thread updated with the bisect results.

Offline

Board footer

Powered by FluxBB