Can not boot after update to linux 5.16

loqs · 2022-01-22 05:59:05

git bisect good
Bisecting: 46 revisions left to test after this (roughly 5 steps)
[97eeb5fc14cc4b2091df8b841a07a1ac69f2d762] partitions/ibm: use bdev_nr_sectors instead of open coding it
https://drive.google.com/file/d/1pTibWM … sp=sharing linux-5.15rc6.r141.g97eeb5fc14cc-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1VlMsDy … sp=sharing linux-headers-5.15rc6.r141.g97eeb5fc14cc-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-22 15:23:34

loqs wrote:

git bisect good
Bisecting: 46 revisions left to test after this (roughly 5 steps)
[97eeb5fc14cc4b2091df8b841a07a1ac69f2d762] partitions/ibm: use bdev_nr_sectors instead of open coding it
https://drive.google.com/file/d/1pTibWM … sp=sharing linux-5.15rc6.r141.g97eeb5fc14cc-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1VlMsDy … sp=sharing linux-headers-5.15rc6.r141.g97eeb5fc14cc-1-x86_64.pkg.tar.zst

doesn't work.

btw, with a new wierd log: http://ix.io/3N8C

loqs · 2022-01-22 16:05:44

Interesting the error changed. May have to return to this point
git bisect bad
Bisecting: 16 revisions left to test after this (roughly 4 steps)
[74e157e6a499ef47edc39cff8c37f77d01c0d155] jfs: use bdev_nr_bytes instead of open coding it
https://drive.google.com/file/d/1t9pwwD … sp=sharing linux-5.15rc6.r124.g74e157e6a499-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1iXxB0p … sp=sharing linux-headers-5.15rc6.r124.g74e157e6a499-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-22 16:50:24

loqs wrote:

Interesting the error changed. May have to return to this point
git bisect bad
Bisecting: 16 revisions left to test after this (roughly 4 steps)
[74e157e6a499ef47edc39cff8c37f77d01c0d155] jfs: use bdev_nr_bytes instead of open coding it
https://drive.google.com/file/d/1t9pwwD … sp=sharing linux-5.15rc6.r124.g74e157e6a499-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1iXxB0p … sp=sharing linux-headers-5.15rc6.r124.g74e157e6a499-1-x86_64.pkg.tar.zst

doesn't work. same log as http://ix.io/3N8C

loqs · 2022-01-22 18:57:20

git bisect bad
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[64f0f42671b48ec30a3203818e26346d5b4ea5fa] target/iblock: use bdev_nr_bytes instead of open coding it
https://drive.google.com/file/d/10pcYve … sp=sharing linux-5.15rc6.r115.g64f0f42671b4-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1rFBOUq … sp=sharing linux-headers-5.15rc6.r115.g64f0f42671b4-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-23 05:06:11

loqs wrote:

git bisect bad
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[64f0f42671b48ec30a3203818e26346d5b4ea5fa] target/iblock: use bdev_nr_bytes instead of open coding it
https://drive.google.com/file/d/10pcYve … sp=sharing linux-5.15rc6.r115.g64f0f42671b4-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1rFBOUq … sp=sharing linux-headers-5.15rc6.r115.g64f0f42671b4-1-x86_64.pkg.tar.zst

doesn't work.

loqs · 2022-01-23 14:31:51

git bisect bad
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[da7b392467da82f8b8cbfe69360e3128688c9ddb] drbd: use bdev_nr_sectors instead of open coding it
https://drive.google.com/file/d/1MaRfKR … sp=sharing linux-5.15rc6.r111.gda7b392467da-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1q6vccn … sp=sharing linux-headers-5.15rc6.r111.gda7b392467da-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-23 15:01:45

loqs wrote:

git bisect bad
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[da7b392467da82f8b8cbfe69360e3128688c9ddb] drbd: use bdev_nr_sectors instead of open coding it
https://drive.google.com/file/d/1MaRfKR … sp=sharing linux-5.15rc6.r111.gda7b392467da-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1q6vccn … sp=sharing linux-headers-5.15rc6.r111.gda7b392467da-1-x86_64.pkg.tar.zst

doesn't work.

loqs · 2022-01-23 15:23:56

git bisect bad
Bisecting: 1 revision left to test after this (roughly 1 step)
[6436bd90f76e75d2c5786a50203b05a9b7f7100d] block: add a bdev_nr_bytes helper
https://drive.google.com/file/d/1IjbPF5 … sp=sharing linux-5.15rc6.r109.g6436bd90f76e-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EIWe98 … sp=sharing linux-headers-5.15rc6.r109.g6436bd90f76e-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-23 16:17:23

loqs wrote:

git bisect bad
Bisecting: 1 revision left to test after this (roughly 1 step)
[6436bd90f76e75d2c5786a50203b05a9b7f7100d] block: add a bdev_nr_bytes helper
https://drive.google.com/file/d/1IjbPF5 … sp=sharing linux-5.15rc6.r109.g6436bd90f76e-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EIWe98 … sp=sharing linux-headers-5.15rc6.r109.g6436bd90f76e-1-x86_64.pkg.tar.zst

doesn't work.

loqs · 2022-01-23 20:20:14

git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[99457db8b40c66941072a383a5ab4e36bc53fd3d] block: move the SECTOR_SIZE related definitions to blk_types.h
https://drive.google.com/file/d/1F3IU01 … sp=sharing linux-5.15rc6.r108.g99457db8b40c-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EBdp6L … sp=sharing linux-headers-5.15rc6.r108.g99457db8b40c-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-24 04:28:20

loqs wrote:

git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[99457db8b40c66941072a383a5ab4e36bc53fd3d] block: move the SECTOR_SIZE related definitions to blk_types.h
https://drive.google.com/file/d/1F3IU01 … sp=sharing linux-5.15rc6.r108.g99457db8b40c-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1EBdp6L … sp=sharing linux-headers-5.15rc6.r108.g99457db8b40c-1-x86_64.pkg.tar.zst

doesn't work.

loqs · 2022-01-24 05:08:14

git bisect bad
99457db8b40c66941072a383a5ab4e36bc53fd3d is the first bad commit
commit 99457db8b40c66941072a383a5ab4e36bc53fd3d
Author: Christoph Hellwig <hch@lst.de>
Date:   Mon Oct 18 12:11:01 2021 +0200

    block: move the SECTOR_SIZE related definitions to blk_types.h
    
    Ensure these are always available for inlines in the various block layer
    headers.
    
    Signed-off-by: Christoph Hellwig <hch@lst.de>
    Reviewed-by: Kees Cook <keescook@chromium.org>
    Reviewed-by: Jan Kara <jack@suse.cz>
    Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>
    Link: https://lore.kernel.org/r/20211018101130.1838532-2-hch@lst.de
    Signed-off-by: Jens Axboe <axboe@kernel.dk>

 include/linux/blk_types.h | 17 +++++++++++++++++
 include/linux/blkdev.h    | 17 -----------------
 2 files changed, 17 insertions(+), 17 deletions(-)

git bisect log
git bisect start
# bad: [df0cc57e057f18e44dac8e6c18aba47ab53202f9] Linux 5.16
git bisect bad df0cc57e057f18e44dac8e6c18aba47ab53202f9
# good: [8bb7eca972ad531c9b149c0a51ab43a417385813] Linux 5.15
git bisect good 8bb7eca972ad531c9b149c0a51ab43a417385813
# bad: [2219b0ceefe835b92a8a74a73fe964aa052742a2] Merge tag 'soc-5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 2219b0ceefe835b92a8a74a73fe964aa052742a2
# bad: [fc02cb2b37fe2cbf1d3334b9f0f0eab9431766c4] Merge tag 'net-next-for-5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect bad fc02cb2b37fe2cbf1d3334b9f0f0eab9431766c4
# good: [b7b98f868987cd3e86c9bd9a6db048614933d7a0] Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
git bisect good b7b98f868987cd3e86c9bd9a6db048614933d7a0
# bad: [6e5772c8d9cf0a77ba4d6fd34fd4126fb66c9983] Merge tag 'x86_cc_for_v5.16_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 6e5772c8d9cf0a77ba4d6fd34fd4126fb66c9983
# bad: [19901165d90fdca1e57c9baa0d5b4c63d15c476a] Merge tag 'for-5.16/inode-sync-2021-10-29' of git://git.kernel.dk/linux-block
git bisect bad 19901165d90fdca1e57c9baa0d5b4c63d15c476a
# good: [33c8846c814c1c27c6e33af005042d15061f948b] Merge tag 'for-5.16/block-2021-10-29' of git://git.kernel.dk/linux-block
git bisect good 33c8846c814c1c27c6e33af005042d15061f948b
# good: [643a7234e0960cf63f1a51a15cfc969fafcbabad] Merge tag 'for-5.16/drivers-2021-10-29' of git://git.kernel.dk/linux-block
git bisect good 643a7234e0960cf63f1a51a15cfc969fafcbabad
# good: [88459b50b42a4bd58e528006663afabd0b8652f2] io_uring: simplify io_file_supports_nowait()
git bisect good 88459b50b42a4bd58e528006663afabd0b8652f2
# bad: [97eeb5fc14cc4b2091df8b841a07a1ac69f2d762] partitions/ibm: use bdev_nr_sectors instead of open coding it
git bisect bad 97eeb5fc14cc4b2091df8b841a07a1ac69f2d762
# bad: [74e157e6a499ef47edc39cff8c37f77d01c0d155] jfs: use bdev_nr_bytes instead of open coding it
git bisect bad 74e157e6a499ef47edc39cff8c37f77d01c0d155
# bad: [64f0f42671b48ec30a3203818e26346d5b4ea5fa] target/iblock: use bdev_nr_bytes instead of open coding it
git bisect bad 64f0f42671b48ec30a3203818e26346d5b4ea5fa
# bad: [da7b392467da82f8b8cbfe69360e3128688c9ddb] drbd: use bdev_nr_sectors instead of open coding it
git bisect bad da7b392467da82f8b8cbfe69360e3128688c9ddb
# bad: [6436bd90f76e75d2c5786a50203b05a9b7f7100d] block: add a bdev_nr_bytes helper
git bisect bad 6436bd90f76e75d2c5786a50203b05a9b7f7100d
# bad: [99457db8b40c66941072a383a5ab4e36bc53fd3d] block: move the SECTOR_SIZE related definitions to blk_types.h
git bisect bad 99457db8b40c66941072a383a5ab4e36bc53fd3d
# first bad commit: [99457db8b40c66941072a383a5ab4e36bc53fd3d] block: move the SECTOR_SIZE related definitions to blk_types.h

Wonder if something went wrong when the error message changed. seth any thoughts?
Edit:
Which error do you get with the kernel below if any?
https://drive.google.com/file/d/1Gquqa0 … sp=sharing linux-5.15.r535.gfcaec17b3657-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/15kEAVq … sp=sharing linux-headers-5.15.r535.gfcaec17b3657-1-x86_64.pkg.tar.zst

Last edited by loqs (2022-01-24 05:23:37)

isaac_le · 2022-01-24 09:06:24

loqs wrote:

Wonder if something went wrong when the error message changed. seth any thoughts?

I have no idea. the log out puth the same with this: http://ix.io/3N8C

loqs wrote:

Which error do you get with the kernel below if any?
https://drive.google.com/file/d/1Gquqa0 … sp=sharing linux-5.15.r535.gfcaec17b3657-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/15kEAVq … sp=sharing linux-headers-5.15.r535.gfcaec17b3657-1-x86_64.pkg.tar.zst

doesn't work. the log: http://ix.io/3NlV

seth · 2022-01-24 09:31:51

One of those is a nullptr deref (first in #27) and the other one is an ATA timeout.
The isolated commit only shifts some preprocessor definitions, the only way this can go wrong is if "#ifndef" actually hits and the macro was previously defined as something entirely different.
I guess linux-5.15.r535.gfcaec17b3657-1-x86_64.pkg.tar.zst is also 5.15 w/ only that change reverted?

Afaiu #11

For TLS: I just notice that if I wait at the boot menu > 3s, it will boot successfully: w/ or w/o Microcode, w/o boot_dealy, w/o debug.
For 5.16 with Microcode: not working even waiting at the boot menu

You end up w/ non-deterministic behavior, driven by the boot delay, for the 5.15 series anyway and only on 5.16 the problem is deterministic?

isaac_le · 2022-01-24 12:40:46

You end up w/ non-deterministic behavior, driven by the boot delay, for the 5.15 series anyway and only on 5.16 the problem is deterministic?

I don't really understand the question but if you mean the 5.16 is not working in any situation : yes

seth · 2022-01-24 13:40:26

Less a question but to point out that the non-deterministic behavior of 5.15 might be a pitfall when testing those releases.
You do not necessarily know whether a release is good or bad.

loqs · 2022-01-24 14:54:14

git bisect bad
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[d6a644a795451d5fd063a5c08d6bb3a91d021887] io_uring: clean up timeout async_data allocation
https://drive.google.com/file/d/1FVnfWP … sp=sharing linux-5.15rc6.r200.gd6a644a79545-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Cok1Aw … sp=sharing linux-headers-5.15rc6.r200.gd6a644a79545-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-24 17:09:10

loqs wrote:

git bisect bad
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[d6a644a795451d5fd063a5c08d6bb3a91d021887] io_uring: clean up timeout async_data allocation
https://drive.google.com/file/d/1FVnfWP … sp=sharing linux-5.15rc6.r200.gd6a644a79545-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Cok1Aw … sp=sharing linux-headers-5.15rc6.r200.gd6a644a79545-1-x86_64.pkg.tar.zst

it works.

loqs · 2022-01-24 19:07:23

git bisect good
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[8d1f01775f8ead7ee313403158be95bffdbb3638] Merge tag 'for-5.16/io_uring-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/1kWnbLO … sp=sharing linux-5.15.r495.g8d1f01775f8e-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Qq_KBC … sp=sharing linux-headers-5.15.r495.g8d1f01775f8e-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-25 04:37:30

loqs wrote:

git bisect good
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[8d1f01775f8ead7ee313403158be95bffdbb3638] Merge tag 'for-5.16/io_uring-2021-10-29' of git://git.kernel.dk/linux-block
https://drive.google.com/file/d/1kWnbLO … sp=sharing linux-5.15.r495.g8d1f01775f8e-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Qq_KBC … sp=sharing linux-headers-5.15.r495.g8d1f01775f8e-1-x86_64.pkg.tar.zst

it works.

loqs · 2022-01-25 04:59:44

git bisect good
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[6b3bae2324d2ecaa404ceab869018011b7ef6a90] doc: document sysfs queue/independent_access_ranges attributes
https://drive.google.com/file/d/1Bb64_6 … sp=sharing linux-5.15rc6.r163.g6b3bae2324d2-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1zXk3ez … sp=sharing linux-headers-5.15rc6.r163.g6b3bae2324d2-1-x86_64.pkg.tar.zst

Looking to be an issue with multiple actuator support for hard drives. I suspect some storage device on your system has firmware claiming it supports the feature.

Last edited by loqs (2022-01-25 05:16:22)

isaac_le · 2022-01-25 06:16:25

loqs wrote:

git bisect good
Bisecting: 2 revisions left to test after this (roughly 2 steps)
[6b3bae2324d2ecaa404ceab869018011b7ef6a90] doc: document sysfs queue/independent_access_ranges attributes
https://drive.google.com/file/d/1Bb64_6 … sp=sharing linux-5.15rc6.r163.g6b3bae2324d2-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1zXk3ez … sp=sharing linux-headers-5.15rc6.r163.g6b3bae2324d2-1-x86_64.pkg.tar.zst
Looking to be an issue with multiple actuator support for hard drives. I suspect some storage device on your system has firmware claiming it supports the feature.

doesn't work. the log looks like this one http://ix.io/3NlV (ofc with new version in these lines: Not tainted 5.15...)

loqs · 2022-01-25 06:41:20

git bisect bad
Bisecting: 0 revisions left to test after this (roughly 1 step)
[fe22e1c2f705676a705d821301fc52eecc2fe055] libata: support concurrent positioning ranges log
https://drive.google.com/file/d/1Mub4Vs … sp=sharing linux-5.15rc6.r162.gfe22e1c2f705-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1F9Q5kL … sp=sharing linux-headers-5.15rc6.r162.gfe22e1c2f705-1-x86_64.pkg.tar.zst

isaac_le · 2022-01-25 07:20:09

loqs wrote:

git bisect bad
Bisecting: 0 revisions left to test after this (roughly 1 step)
[fe22e1c2f705676a705d821301fc52eecc2fe055] libata: support concurrent positioning ranges log
https://drive.google.com/file/d/1Mub4Vs … sp=sharing linux-5.15rc6.r162.gfe22e1c2f705-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1F9Q5kL … sp=sharing linux-headers-5.15rc6.r162.gfe22e1c2f705-1-x86_64.pkg.tar.zst

doesn't work.

Arch Linux

#26 2022-01-22 05:59:05

Re: Can not boot after update to linux 5.16

#27 2022-01-22 15:23:34

Re: Can not boot after update to linux 5.16

#28 2022-01-22 16:05:44

Re: Can not boot after update to linux 5.16

#29 2022-01-22 16:50:24

Re: Can not boot after update to linux 5.16

#30 2022-01-22 18:57:20

Re: Can not boot after update to linux 5.16

#31 2022-01-23 05:06:11

Re: Can not boot after update to linux 5.16

#32 2022-01-23 14:31:51

Re: Can not boot after update to linux 5.16

#33 2022-01-23 15:01:45

Re: Can not boot after update to linux 5.16

#34 2022-01-23 15:23:56

Re: Can not boot after update to linux 5.16

#35 2022-01-23 16:17:23

Re: Can not boot after update to linux 5.16

#36 2022-01-23 20:20:14

Re: Can not boot after update to linux 5.16

#37 2022-01-24 04:28:20

Re: Can not boot after update to linux 5.16

#38 2022-01-24 05:08:14

Re: Can not boot after update to linux 5.16

#39 2022-01-24 09:06:24

Re: Can not boot after update to linux 5.16

#40 2022-01-24 09:31:51

Re: Can not boot after update to linux 5.16

#41 2022-01-24 12:40:46

Re: Can not boot after update to linux 5.16

#42 2022-01-24 13:40:26

Re: Can not boot after update to linux 5.16

#43 2022-01-24 14:54:14

Re: Can not boot after update to linux 5.16

#44 2022-01-24 17:09:10

Re: Can not boot after update to linux 5.16

#45 2022-01-24 19:07:23

Re: Can not boot after update to linux 5.16

#46 2022-01-25 04:37:30

Re: Can not boot after update to linux 5.16

#47 2022-01-25 04:59:44

Re: Can not boot after update to linux 5.16

#48 2022-01-25 06:16:25

Re: Can not boot after update to linux 5.16

#49 2022-01-25 06:41:20

Re: Can not boot after update to linux 5.16

#50 2022-01-25 07:20:09

Re: Can not boot after update to linux 5.16

Board footer