[Solved] Not booting, no log messages whatsoever

loqs · 2020-02-20 20:13:36

Checking the keyboard at early boot would be if you needed it to enter a password to unlock an encrypted container before boot finished.
If you do not need the keyboard until boot has completed then that is when you would notice it is not working.
You can use the journal to find the kernel messages from old boots.
Filter it for just kernel messages and for a particular boot and near the start of the log will be the kernel version.

Might be worth checking linux-mainline 5.6-rc2 available from miffe's unofficial repository to see if the issue has been fixed in 5.6.
Otherwise you may have to bisect between 5.3 and 5.4.

Last edited by loqs (2020-02-20 20:16:50)

AnyTimeTraveller · 2020-02-21 00:09:16

I'll have to sleep now, but I'll continue tomorrow.
I have downgraded the lts kernel to __a__ working version and told pacman to ignore it for now.
I have downloaded all the kernels since 4.20 and tomorrow, I'll do a binary search through the list until I can identify the version where it starts breaking.

I have figured out how to add unofficial repos and installed miff's linux-mainline kernel, but it has the same problem.
So no progress forwards.

I always wanted to get into kernel development, maybe tracking down that bug will be my entry point

Thank you for all the help I have received so far!

AnyTimeTraveller · 2020-02-21 11:29:53

I tried a bunch of kernels today and found that the newest kernel that works is version
`linux-5.3.13.1-1-x86_64.pkg.tar.xz`
When installing version `linux-5.4.arch1-1-x86_64.pkg.tar.xz` the system doesn't boot anymore.

I looked at the the kernel changelogs and found other versions in between those, that I can't find packages for in the archive.
I assume that the problem is related to whatever change caused the version 5.3 to jump to 5.4.

Here is my dmesg for the latest working kernel: https://simonscode.org/dmesg-5.3.13.1-1-x86_64.log

I was reading the changelog overview and saw a pull request titled ACPI updates
I don't understand any of the code, but since future versions boot with the "acpi=off" parameter, maybe that makes sense?
I am on the very edge of my knowledge, so if I am off in a totally wrong direction here, just tell me.

I just installed an ssh server and booted the kernel version 5.5.4 that only worked with `acpi=off`.
I ssh'ed into the laptop from my phone and saved the dmesg log here: https://simonscode.org/dmesg-5.5.4-arch1-1.log

(the log files have color information in them as well, because I forgot that I had aliased that command with --color=always. I hope that isn't a problem)

EDIT: some more info
EDIT: Add additional dmesg log

Last edited by AnyTimeTraveller (2020-02-21 12:49:29)

loqs · 2020-02-21 12:42:44

The following assumes the base-devel group and git are installed and I recomend enabling Makepkg#Parallel_compilation to reduce build times

$ git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/linux"
$ cd packages/trunk
$ git checkout aa1b1a59ce1cb67d95021c26b24739d1e648a553 #5.3.13.arch1-1
$ cd ../..
$ cp -r packages/trunk linux-git
$ rm -rf packages
$ cd linux-git
# Edit replace the PKGBUILD with the one below
$ makepkg -rsi #This is to confirm 5.3 as built on your system does not have the issue.

There should be one config change for USB_RIO500 which was dropped in 5.3.7 select the default.
This build may take a very long time.
Update bootloader for new kernel if needed

$ cd linux-git/src/linux
$ git checkout v5.4
$ cd ../..
$ makepkg -ersi #This is to confirm 5.4 as built on your system does have the issue select the default option for all prompted options

Start bisection

$ cd linux-git/src/linux/
$ git bisect start
$ git bisect good v5.3
$ git bisect bad v5.4
$ cd ../..
$ makepkg -ersif

Bisection loop

$ cd linux-git/src/linux/
$ git bisect $result #Substitue good or bad here
$ cd ../..
$ makepkg -ersif #Repeat these four lines and test the generated kernel until git has found the bad commit

PKGBUILD

# Maintainer: Boohbah <boohbah at gmail.com>
# Contributor: Tobias Powalowski <tpowa@archlinux.org>
# Contributor: Thomas Baechler <thomas@archlinux.org>
# Contributor: Jonathan Chan <jyc@fastmail.fm>
# Contributor: misc <tastky@gmail.com>
# Contributor: NextHendrix <cjones12 at sheffield.ac.uk>

pkgbase=linux-git
_srcname=linux
pkgver=5.3.r0.g4d856f72c10e
pkgrel=1
arch=('x86_64')
url="https://www.kernel.org/"
license=('GPL2')
makedepends=('kmod' 'inetutils' 'bc' 'libelf' 'git')
options=('!strip')
source=('git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git#tag=v5.3'
        #'git+https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git#tag=vX.X.Y'
        'config'   # the main kernel config file
)

sha256sums=('SKIP'
            '10ee7800902b1d82f9c184b367c9d904f4dc48f6d9ce3277327e825d7ab690d1')

_kernelname=${pkgbase#linux}

pkgver() {
  cd $_srcname

  git describe --long | sed -E 's/^v//;s/([^-]*-g)/r\1/;s/-/./g;s/\.rc/rc/'
}

prepare() {
  cd ${_srcname}

  cp -Tf ../config .config

  # set localversion to git commit
  sed -i "s|CONFIG_LOCALVERSION=.*|CONFIG_LOCALVERSION=\"${_kernelname}\"|g" ./.config
  sed -i "s|^.*CONFIG_LOCALVERSION_AUTO.*|CONFIG_LOCALVERSION_AUTO=y|" ./.config


  # get kernel version
  make prepare

  # load configuration
  # Configure the kernel. Replace the line below with one of your choice.
  #make menuconfig # CLI menu for configuration
  #make nconfig # new CLI menu for configuration
  #make xconfig # X-based configuration
  #make oldconfig # using old config from previous kernel version
  #make olddefconfig # old config from previous kernel, defaults for new options
  # ... or manually edit .config
}

build() {
  cd $_srcname

  make LOCALVERSION= bzImage modules
}

_package() {
  pkgdesc="The Linux kernel and modules (git version)"
  depends=('coreutils' 'linux-firmware' 'kmod' 'initramfs')
  optdepends=('crda: to set the correct wireless channels of your country')

  cd ${_srcname}

  # get kernel version
  _kernver="$(make LOCALVERSION= kernelrelease)"
  _basekernel=${_kernver%%-*}
  _basekernel=${_basekernel%.*}
  local _modulesdir="$pkgdir/usr/lib/modules/$_kernver"

  install -Dm644 "$(make -s image_name)" "$_modulesdir/vmlinuz"

  # Used by mkinitcpio to name the kernel
  echo "$pkgbase" | install -Dm644 /dev/stdin "$_modulesdir/pkgbase"

  make LOCALVERSION= INSTALL_MOD_PATH="${pkgdir}/usr" modules_install

  # make room for external modules
  local _extramodules="extramodules-${_basekernel}${_kernelname:--ARCH}"
  ln -s "../${_extramodules}" "${pkgdir}/usr/lib/modules/${_kernver}/extramodules"

  # add real version for building modules and running depmod from hook
  echo "${_kernver}" |
    install -Dm644 /dev/stdin "${pkgdir}/usr/lib/modules/${_extramodules}/version"

  # remove build and source links
  rm "${pkgdir}"/usr/lib/modules/${_kernver}/{source,build}
}

_package-headers() {
  pkgdesc="Header files and scripts for building modules for Linux kernel (git version)"

  cd $_srcname
  local _builddir="${pkgdir}/usr/lib/modules/${_kernver}/build"

  install -Dt "$_builddir" -m644 .config Makefile Module.symvers
  install -Dt "$_builddir/kernel" -m644 kernel/Makefile
  install -Dt "$_builddir/arch/x86" -m644 arch/x86/Makefile
  cp -t "$_builddir" -a scripts

  # add objtool for external module building and enabled VALIDATION_STACK option
  install -Dt "$_builddir/tools/objtool" tools/objtool/objtool

  # add xfs and shmem for aufs building
  mkdir -p "$_builddir"/{fs/xfs,mm}

  cp -t "$_builddir" -a include
  cp -t "$_builddir/arch/x86" -a arch/x86/include
  install -Dt "$_builddir/arch/x86/kernel" -m644 arch/x86/kernel/asm-offsets.s


  install -Dt "$_builddir/drivers/md" -m644 drivers/md/*.h
  install -Dt "$_builddir/net/mac80211" -m644 net/mac80211/*.h

  # http://bugs.archlinux.org/task/13146
  install -Dt "$_builddir/drivers/media/i2c" -m644 drivers/media/i2c/msp3400-driver.h

  # http://bugs.archlinux.org/task/20402
  install -Dt "$_builddir/drivers/media/usb/dvb-usb" -m644 drivers/media/usb/dvb-usb/*.h
  install -Dt "$_builddir/drivers/media/dvb-frontends" -m644 drivers/media/dvb-frontends/*.h
  install -Dt "$_builddir/drivers/media/tuners" -m644 drivers/media/tuners/*.h

  # copy in Kconfig files
  find . -name 'Kconfig*' -exec install -Dm644 {} "$_builddir/{}" \;

  # remove unneeded architectures
  local _arch
  for _arch in "$_builddir"/arch/*/; do
    [[ ${_arch} == */x86/ ]] && continue
    echo "Removing $(basename "$_arch")"
    rm -r "$_arch"
  done

  # remove files already in linux-docs package
  rm -r "$_builddir/Documentation"

  echo "Removing broken symlinks..."
  find -L "$_builddir" -type l -printf 'Removing %P\n' -delete

  # strip scripts directory
  local _binary _strip
  while read -rd '' _binary; do
    case "$(file -bi "$_binary")" in
      *application/x-sharedlib*)  _strip="$STRIP_SHARED"   ;; # Libraries (.so)
      *application/x-archive*)    _strip="$STRIP_STATIC"   ;; # Libraries (.a)
      *application/x-executable*) _strip="$STRIP_BINARIES" ;; # Binaries
      *) continue ;;
    esac
    /usr/bin/strip $_strip "$_binary"
  done < <(find "$_builddir/scripts" -type f -perm -u+w -print0 2>/dev/null)

  # Fix permissions
  chmod -R u=rwX,go=rX "${_builddir}"
}

pkgname=("$pkgbase" "$pkgbase-headers")
for _p in ${pkgname[@]}; do
  eval "package_$_p() {
    $(declare -f "_package${_p#$pkgbase}")
    _package${_p#$pkgbase}
  }"
done

# vim:set ts=8 sts=2 sw=2 et:

AnyTimeTraveller · 2020-02-21 13:00:10

I'm gonna start that now.
My dual-core i5 is gonna have fun with that for a while...

AnyTimeTraveller · 2020-02-21 15:57:55

Everything worked beautifully so far.

# git checkout v5.4-arch1
# ...
# makepkg -ersi

This results in reinstalling of 5.3.13 without a rebuild.

Do I need to change the version in the pkgbuild or where does it need changes?

loqs · 2020-02-21 16:11:22

pacman -Q linux-git
cd linux-git
ls *.pkg.*
cd src/linux
git status

AnyTimeTraveller · 2020-02-21 16:50:26

# pacman -Q linux-git
error: package 'linux-git' was not found
# cd linux-git
# ls -1 *.pkg.*
linux-5.3.13.1-1-x86_64.pkg.tar.xz
linux-docs-5.3.13.1-1-x86_64.pkg.tar.xz
linux-headers-5.3.13.1-1-x86_64.pkg.tar.xz
# cd src/linux
cd: no such file or directory: src/linux
(there is a folder called archlinux-linux, though)
# git status
HEAD detached at v5.4-arch1
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	localversion.10-pkgrel
	localversion.20-pkgname
	version

nothing added to commit but untracked files present (use "git add" to track)

loqs · 2020-02-21 16:53:19

You did replace the PKGBUILD in linux-git with the one at the bottom of post #29 ?

AnyTimeTraveller · 2020-02-21 17:03:44

Oh, I misspelled the PKGBUILD, so it didn't get replaced.

AnyTimeTraveller · 2020-04-28 22:40:41

It appears that my problem has solved itself.
I couldn't make much time in the last few months for bisecting with the current situation, so I didn't make much progress until now.
Today, I saw that the version of the linux kernel had jumped to 5.6.7 and so I just tried to boot it again, which worked beautifully.

So unfortunately, due to a lack of time, I will have to stop my investigation here and leave unsatisfied, but happy to have a recent and working kernel again.

Arch Linux

#26 2020-02-20 20:13:36

Re: [Solved] Not booting, no log messages whatsoever

#27 2020-02-21 00:09:16

Re: [Solved] Not booting, no log messages whatsoever

#28 2020-02-21 11:29:53

Re: [Solved] Not booting, no log messages whatsoever

#29 2020-02-21 12:42:44

Re: [Solved] Not booting, no log messages whatsoever

#30 2020-02-21 13:00:10

Re: [Solved] Not booting, no log messages whatsoever

#31 2020-02-21 15:57:55

Re: [Solved] Not booting, no log messages whatsoever

#32 2020-02-21 16:11:22

Re: [Solved] Not booting, no log messages whatsoever

#33 2020-02-21 16:50:26

Re: [Solved] Not booting, no log messages whatsoever

#34 2020-02-21 16:53:19

Re: [Solved] Not booting, no log messages whatsoever

#35 2020-02-21 17:03:44

Re: [Solved] Not booting, no log messages whatsoever

#36 2020-04-28 22:40:41

Re: [Solved] Not booting, no log messages whatsoever

Board footer