You are not logged in.

#1 2020-02-25 02:43:24

terin7
Member
Registered: 2013-08-02
Posts: 5

Diagnosing suspend issue between two kernel versions?

Hey all,

I'm currently running Arch Linux on an Acer Chromebook c720. It's been going pretty well, except that the kernel update from 4.0 to 5.0 made it so that waking up my system from suspend causes it to restart. I've got a 4.19 package of linux-lts on my system still, so I can run off of that in the meantime, but I'd still like to figure this out.

So, given two versions of a kernel, where can I look to find what's causing an issue with suspend like this? What could I compare? I assume dmesg covers a lot, but it's still a massive flood of information that I'm not sure how to parse, or if it would contain this if it's immediately shutting down and restarting.

Offline

#2 2020-02-25 15:58:23

twelveeighty
Member
Registered: 2011-09-04
Posts: 1,456

Re: Diagnosing suspend issue between two kernel versions?

You can approach the problem from two angles: you can either find out exactly which kernel update/commit causes the problem by bisecting the kernel source code. Since this involves kernel compilation, I'm not sure if a Chromebook can handle that effectively. The other way is to investigate the failing kernel and go through the logs. Make sure journalctl is persistent between boots (the default) and walk through the logs.

What do you mean exactly by "causing it to restart" - are you sure that the suspend actually works? Maybe the suspend shuts down the system instead so "waking up" just does a cold boot?

Offline

#3 2020-02-28 22:03:28

terin7
Member
Registered: 2013-08-02
Posts: 5

Re: Diagnosing suspend issue between two kernel versions?

twelveeighty wrote:

You can approach the problem from two angles: you can either find out exactly which kernel update/commit causes the problem by bisecting the kernel source code. Since this involves kernel compilation, I'm not sure if a Chromebook can handle that effectively.

Yeah, and since this seemed to happen going from 4.x to 5.0, I imagine that would make searching for the exact commit a lot more work. Maybe not, though.

What do you mean exactly by "causing it to restart" - are you sure that the suspend actually works? Maybe the suspend shuts down the system instead so "waking up" just does a cold boot?

I'm not able to check the OS itself, but the suspend light on the laptop will be blinking like it should, but when I wake the machine, it'll immediately go to the BIOS.

I'll try comparing journals to see if I can get anything in particular.

Offline

#4 2020-02-28 22:23:17

loqs
Member
Registered: 2014-03-06
Posts: 19,089

Re: Diagnosing suspend issue between two kernel versions?

The issue was not present under 4.20.13.arch1-1 but was present under 5.0.arch1-1 ?

Offline

#5 2020-02-29 20:44:32

terin7
Member
Registered: 2013-08-02
Posts: 5

Re: Diagnosing suspend issue between two kernel versions?

loqs wrote:

The issue was not present under 4.20.13.arch1-1 but was present under 5.0.arch1-1 ?

Looking at it closer, there's no issue on kernels up to 5.0.13.arch1-1, but then 5.1.arch1-1 is where it starts happening. It looks like there was only a day or two between those kernel versions, but that still looks like 500 commits on May 5th and 6th in 2019, so that's a lot to sift through.

Looks like I'm going to have to do that, as well, since the journal just immediately cuts off with -- Reboot -- when I resume from suspend on a bad kernel. No useful information there.

Offline

#6 2020-02-29 20:54:11

loqs
Member
Registered: 2014-03-06
Posts: 19,089

Re: Diagnosing suspend issue between two kernel versions?

The following may help bisect between 5.0 and 5.1
The following assumes the base-devel group and git are installed and I recomend enabling Makepkg#Parallel_compilation to reduce build times

$ git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/linux"
$ cd packages/trunk
$ git checkout db9609721c9fc71343c17c9e460cc1f613ed1795 #5.0.13.arch1-1
$ cd ../..
$ cp -r packages/trunk linux-git
$ rm -rf packages
$ cd linux-git
# Edit replace the PKGBUILD and 90-linux.hook with the ones below
$ makepkg -rsi #this is to confirm 5.0 as built on your system does not have the issue update bootloader for new kernel if needed

$ cd linux-git/src/linux
$ git checkout v5.1
$ cd ../..
$ makepkg -ersi #this is to confirm 5.1 as built on your system does have the issue select the default option for all prompted options

$ cd linux-git/src/linux/
$ git bisect start
$ git bisect good v5.0
$ git bisect bad v5.1
$ cd ../..
$ makepkg -ersif

$ cd linux-git/src/linux/
$ git bisect $result #substitue good or bad here
$ cd ../..
$ makepkg -ersif #repeat these four lines and test the generated kernel until git has found the bad commit

PKGBUILD

# Maintainer: Boohbah <boohbah at gmail.com>
# Contributor: Tobias Powalowski <tpowa@archlinux.org>
# Contributor: Thomas Baechler <thomas@archlinux.org>
# Contributor: Jonathan Chan <jyc@fastmail.fm>
# Contributor: misc <tastky@gmail.com>
# Contributor: NextHendrix <cjones12 at sheffield.ac.uk>

pkgbase=linux-git
_srcname=linux
pkgver=4.20.r0.g8fe28cb58bcb
pkgrel=1
arch=('x86_64')
url="https://www.kernel.org/"
license=('GPL2')
makedepends=('kmod' 'inetutils' 'bc' 'libelf')
options=('!strip')
source=('git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git#tag=v5.0'
        #'git+https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git#tag=X.X.Y'
  'config'   # the main kernel config file
  '60-linux.hook'  # pacman hook for depmod
  '90-linux.hook'  # pacman hook for initramfs regeneration
  'linux.preset'   # standard config files for mkinitcpio ramdisk
)

sha256sums=('SKIP'
            '5eb7fb6aa90386735e5d8ee9528e82e12ba89875de058ffa4bf0a844ae055d58'
            'ae2e95db94ef7176207c690224169594d49445e04249d2499e9d2fbc117a0b21'
            '75f99f5239e03238f88d1a834c50043ec32b1dc568f2cc291b07d04718483919'
            'ad6344badc91ad0630caacde83f7f9b97276f80d26a20619a87952be65492c65')

_kernelname=${pkgbase#linux}
: ${_kernelname:=-ARCH}

pkgver() {
  cd "${_srcname}"

  git describe --long | sed -E 's/^v//;s/([^-]*-g)/r\1/;s/-/./g;s/\.rc/rc/'
}

prepare() {
  cd ${_srcname}

  cp -Tf ../config .config

  # set localversion to git commit
  sed -i "s|CONFIG_LOCALVERSION=.*|CONFIG_LOCALVERSION=\"${_kernelname}\"|g" ./.config
  sed -i "s|^.*CONFIG_LOCALVERSION_AUTO.*|CONFIG_LOCALVERSION_AUTO=y|" ./.config

  # don't run depmod on 'make install'. We'll do this ourselves in packaging
#  git tracks scripts/depmod.sh so do not change it when using the existing source dir for bisection
#  sed -i '2iexit 0' scripts/depmod.sh

  # get kernel version
  make prepare

  # load configuration
  # Configure the kernel. Replace the line below with one of your choice.
  #make menuconfig # CLI menu for configuration
  #make nconfig # new CLI menu for configuration
  #make xconfig # X-based configuration
  #make oldconfig # using old config from previous kernel version
  make olddefconfig # old config from previous kernel, defaults for new options
  # ... or manually edit .config
}

build() {
  cd ${_srcname}

  make bzImage modules
}

_package() {
  pkgdesc="The Linux kernel and modules (git version)"
  depends=('coreutils' 'linux-firmware' 'kmod' 'mkinitcpio>=0.7')
  optdepends=('crda: to set the correct wireless channels of your country')
  backup=("etc/mkinitcpio.d/${pkgbase}.preset")
  install=linux.install

  cd ${_srcname}

  # get kernel version
  _kernver="$(make kernelrelease)"
  _kernver=${_kernver%-dirty} #https://bbs.archlinux.org/viewtopic.php?id=236702
  _basekernel="$(make kernelversion)"
  _basekernel=${_basekernel%.*}

  mkdir -p "${pkgdir}"/{boot,usr/lib/modules}
  make INSTALL_MOD_PATH="${pkgdir}/usr" modules_install
  cp arch/x86/boot/bzImage "${pkgdir}/boot/vmlinuz-${pkgbase}"

  # make room for external modules
  local _extramodules="extramodules-${_basekernel}${_kernelname}"
  ln -s "../${_extramodules}" "${pkgdir}/usr/lib/modules/${_kernver}/extramodules"

  # add real version for building modules and running depmod from hook
  echo "${_kernver}" |
    install -Dm644 /dev/stdin "${pkgdir}/usr/lib/modules/${_extramodules}/version"

  # remove build and source links
  rm "${pkgdir}"/usr/lib/modules/${_kernver}/{source,build}

  # now we call depmod...
  depmod -b "${pkgdir}/usr" -F System.map "${_kernver}"

  # add vmlinux
  install -Dt "${pkgdir}/usr/lib/modules/${_kernver}/build" -m644 vmlinux

  # sed expression for following substitutions
  local _subst="
    s|%PKGBASE%|${pkgbase}|g
    s|%KERNVER%|${_kernver}|g
    s|%EXTRAMODULES%|${_extramodules}|g
  "

  # hack to allow specifying an initially nonexisting install file
  sed "${_subst}" "${startdir}/${install}" > "${startdir}/${install}.pkg"
  true && install=${install}.pkg

  # install mkinitcpio preset file
  sed "${_subst}" ../linux.preset |
    install -Dm644 /dev/stdin "${pkgdir}/etc/mkinitcpio.d/${pkgbase}.preset"

  # install pacman hooks
  sed "${_subst}" ../60-linux.hook |
    install -Dm644 /dev/stdin "${pkgdir}/usr/share/libalpm/hooks/60-${pkgbase}.hook"
  sed "${_subst}" ../90-linux.hook |
    install -Dm644 /dev/stdin "${pkgdir}/usr/share/libalpm/hooks/90-${pkgbase}.hook"
}

_package-headers() {
  pkgdesc="Header files and scripts for building modules for Linux kernel (git version)"

  cd ${_srcname}
  local _builddir="${pkgdir}/usr/lib/modules/${_kernver}/build"

  install -Dt "${_builddir}" -m644 Makefile .config Module.symvers
  install -Dt "${_builddir}/kernel" -m644 kernel/Makefile

  mkdir "${_builddir}/.tmp_versions"

  cp -t "${_builddir}" -a include scripts

  install -Dt "${_builddir}/arch/x86" -m644 arch/x86/Makefile
  install -Dt "${_builddir}/arch/x86/kernel" -m644 arch/x86/kernel/asm-offsets.s

  cp -t "${_builddir}/arch/x86" -a arch/x86/include

  install -Dt "${_builddir}/drivers/md" -m644 drivers/md/*.h
  install -Dt "${_builddir}/net/mac80211" -m644 net/mac80211/*.h

  # http://bugs.archlinux.org/task/13146
  install -Dt "${_builddir}/drivers/media/i2c" -m644 drivers/media/i2c/msp3400-driver.h

  # http://bugs.archlinux.org/task/20402
  install -Dt "${_builddir}/drivers/media/usb/dvb-usb" -m644 drivers/media/usb/dvb-usb/*.h
  install -Dt "${_builddir}/drivers/media/dvb-frontends" -m644 drivers/media/dvb-frontends/*.h
  install -Dt "${_builddir}/drivers/media/tuners" -m644 drivers/media/tuners/*.h

  # add xfs and shmem for aufs building
  mkdir -p "${_builddir}"/{fs/xfs,mm}

  # copy in Kconfig files
  find . -name Kconfig\* -exec install -Dm644 {} "${_builddir}/{}" \;

  # add objtool for external module building and enabled VALIDATION_STACK option
  install -Dt "${_builddir}/tools/objtool" tools/objtool/objtool

  # remove unneeded architectures
  local _arch
  for _arch in "${_builddir}"/arch/*/; do
    [[ ${_arch} == */x86/ ]] && continue
    rm -r "${_arch}"
  done

  # remove files already in linux-docs package
  rm -r "${_builddir}/Documentation"

  # remove now broken symlinks
  find -L "${_builddir}" -type l -printf 'Removing %P\n' -delete

  # Fix permissions
  chmod -R u=rwX,go=rX "${_builddir}"

  # strip scripts directory
  local _binary _strip
  while read -rd '' _binary; do
    case "$(file -bi "${_binary}")" in
      *application/x-sharedlib*)  _strip="${STRIP_SHARED}"   ;; # Libraries (.so)
      *application/x-archive*)    _strip="${STRIP_STATIC}"   ;; # Libraries (.a)
      *application/x-executable*) _strip="${STRIP_BINARIES}" ;; # Binaries
      *) continue ;;
    esac
    /usr/bin/strip ${_strip} "${_binary}"
  done < <(find "${_builddir}/scripts" -type f -perm -u+w -print0 2>/dev/null)
}

_package-docs() {
  pkgdesc="Kernel hackers manual - HTML documentation that comes with the Linux kernel (git version)"

  cd ${_srcname}
  local _builddir="${pkgdir}/usr/lib/modules/${_kernver}/build"

  mkdir -p "${_builddir}"
  cp -t "${_builddir}" -a Documentation

  # Fix permissions
  chmod -R u=rwX,go=rX "${_builddir}"
}

pkgname=("${pkgbase}" "${pkgbase}-headers" "${pkgbase}-docs")
for _p in ${pkgname[@]}; do
  eval "package_${_p}() {
    $(declare -f "_package${_p#${pkgbase}}")
    _package${_p#${pkgbase}}
  }"
done

# vim:set ts=8 sts=2 sw=2 et:

90-linux.hook

[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Target = boot/vmlinuz-%PKGBASE%
Target = usr/lib/initcpio/*

[Action]
Description = Updating %PKGBASE% initcpios...
When = PostTransaction
Exec = /usr/bin/mkinitcpio -p %PKGBASE%

Offline

#7 2020-03-01 03:34:18

terin7
Member
Registered: 2013-08-02
Posts: 5

Re: Diagnosing suspend issue between two kernel versions?

Ah, perfect! Was going to manually binary search through the commits, but that's great to know that git has it built in.

No go on any parallel compilation, since the c720's CPU is single core. Been at it for four hours, but I've got the time. I'll update the thread when I find the commit that breaks it.

Offline

Board footer

Powered by FluxBB