Kernel upgrade has broken suspend

Flapper · 2019-11-22 11:26:32

The upgrade from linux kernel 5.3.11 to 5.3.12 has broken suspend/hibernate.

The screen goes black, keyboard and mouse stop responding but power light on PC is still lit, CPU fan is still running, but no way of waking the system.
Reverting to kernel 5.3.11 has restored normal function.
Has anyone else had this problem?

Processor: AMD Athlon 64 3500+
Graphics: NVIDIA GT218 [GeForce 210]
Desktop: Xfce 4.14.1

I used to have this problem before, intermittently, but it resolved itself several years ago, with no clues as to the cause.

romstor · 2019-11-24 02:10:17

I have the same issue on a Dell Precision M4400 laptop that was always suspending and hibernating without issues. No errors in journalctl - it reaches the sleep target but the machine doesn't power off.

$ journalctl -b -1
Nov 23 16:06:31 m4400 polkitd[443]: Registered Authentication Agent for unix-process:2070:420640 (system bus name :1.342 [/usr/bin/pkttyagent --notify-fd 5 ->
Nov 23 16:06:31 m4400 NetworkManager[371]: <info>  [1574543191.3119] manager: sleep: sleep requested (sleeping: no  enabled: yes)
Nov 23 16:06:31 m4400 NetworkManager[371]: <info>  [1574543191.3120] manager: NetworkManager state is now ASLEEP
Nov 23 16:06:31 m4400 systemd[1]: Reached target Sleep.
Nov 23 16:06:31 m4400 systemd[1]: Starting Suspend...
Nov 23 16:06:31 m4400 polkitd[443]: Unregistered Authentication Agent for unix-process:2070:420640 (system bus name :1.342, object path /org/freedesktop/Poli>
Nov 23 16:06:31 m4400 systemd-sleep[2075]: Suspending system...
Nov 23 16:06:31 m4400 kernel: PM: suspend entry (deep)

$ uname -a
Linux m4400 5.3.12-arch1-1 #1 SMP PREEMPT Wed, 20 Nov 2019 19:45:16 +0000 x86_64 GNU/Linux

I didn't try downgrading the kernel yet.

Update: I have just tried booting from another disk where I have an older Arch install - suspends without issues. Looks like this version of the kernel 5.3.12-arch1-1 is indeed to blame.

Update 2: Downgrading to 5.3.11-arch1-1 restores suspend for me as well:

$ uname -a
Linux m4400 5.3.11-arch1-1 #1 SMP PREEMPT Tue, 12 Nov 2019 22:19:48 +0000 x86_64 GNU/Linux

Quick search on the Arch and Kernel bug trackers didn't yield any finds. I wander if more people experience this to warrant filing the issue. My laptop is pretty old, so this may just be an older hardware incompatibility.

Last edited by romstor (2019-11-24 02:44:39)

JanErik · 2019-11-26 06:52:56

Yes, same problem here. AsRock P67-PRO3-B3 and i7 2600k. Mitigations=off, otherwise running stock.
Updated on sunday (24.11) as I always do.

Last edited by JanErik (2019-11-26 06:55:56)

romstor · 2019-12-01 03:43:53

Tried two more things:

Changing suspend type to older s2idle (I have an old Dell Precision M4400 laptop) in hopes that would start working again:

$ sudo cat /sys/power/mem_sleep
s2idle [deep]
$ echo s2idle | sudo tee /sys/power/mem_sleep
s2idle
$ sudo cat /sys/power/mem_sleep
[s2idle] deep

It didn't help.

Turning on hibernate to a disk partition - also no success.

Flapper · 2019-12-02 23:20:01

I tried upgrading the kernels to the latest versions today.

The 4.19.87-1-lts kernel has no problem suspending.

However, I get the same problem of a half-suspended PC that won't wake up with the 5.4.1.arch1-1 kernel (which does not bode well as 5.4 will be the next lts kernel).

loqs · 2019-12-02 23:42:14

@Flapper I would suggest bisecting between 5.3.11 and 5.3.12 to try and find the cause.

Flapper · 2019-12-03 01:02:10

Errr... I've no idea what you mean.
Could you be more specific as to what I should do please?

loqs · 2019-12-03 13:12:03

The following assumes the base-devel group and git are installed and I recomend enabling Makepkg#Parallel_compilation to reduce build times

$ git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/linux"
$ cd packages/trunk
$ git checkout 5e1a24286849801b5f2723c16abf265140cd33eb #5.3.11.arch1-1
$ cd ../..
$ cp -r packages/trunk linux-git
$ rm -rf packages
$ cd linux-git
# Edit replace the PKGBUILD with the one below
$ makepkg -rsi #This is to confirm 5.3.11 as built on your system does not have the issue.  Update bootloader for new kernel if needed

$ cd linux-git/src/linux
$ git checkout v5.3.12
$ cd ../..
$ makepkg -ersi #This is to confirm 5.3.12 as built on your system does have the issue select the default option for all prompted options

$ cd linux-git/src/linux/
$ git bisect start
$ git bisect good v5.3.11
$ git bisect bad v5.3.12
$ cd ../..
$ makepkg -ersif

$ cd linux-git/src/linux/
$ git bisect $result #Substitue good or bad here
$ cd ../..
$ makepkg -ersif #Repeat these four lines and test the generated kernel until git has found the bad commit

PKGBUILD

# Maintainer: Boohbah <boohbah at gmail.com>
# Contributor: Tobias Powalowski <tpowa@archlinux.org>
# Contributor: Thomas Baechler <thomas@archlinux.org>
# Contributor: Jonathan Chan <jyc@fastmail.fm>
# Contributor: misc <tastky@gmail.com>
# Contributor: NextHendrix <cjones12 at sheffield.ac.uk>

pkgbase=linux-git
_srcname=linux
pkgver=5.3.11.r0.gdada86c5aaa8
pkgrel=1
arch=('x86_64')
url="https://www.kernel.org/"
license=('GPL2')
makedepends=('kmod' 'inetutils' 'bc' 'libelf' 'git')
options=('!strip')
source=(#'git+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git#tag=X.X'
        'git+https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git#tag=v5.3.11'
        'config'   # the main kernel config file
)

sha256sums=('SKIP'
            '54bd002b40030609bfb236c35d2a23d23e426edd1f84dd448afbadd1b0aacb55')

_kernelname=${pkgbase#linux}

pkgver() {
  cd $_srcname

  git describe --long | sed -E 's/^v//;s/([^-]*-g)/r\1/;s/-/./g;s/\.rc/rc/'
}

prepare() {
  cd ${_srcname}

  cp -Tf ../config .config

  # set localversion to git commit
  sed -i "s|CONFIG_LOCALVERSION=.*|CONFIG_LOCALVERSION=\"${_kernelname}\"|g" ./.config
  sed -i "s|^.*CONFIG_LOCALVERSION_AUTO.*|CONFIG_LOCALVERSION_AUTO=y|" ./.config


  # get kernel version
  make prepare

  # load configuration
  # Configure the kernel. Replace the line below with one of your choice.
  #make menuconfig # CLI menu for configuration
  #make nconfig # new CLI menu for configuration
  #make xconfig # X-based configuration
  #make oldconfig # using old config from previous kernel version
  #make olddefconfig # old config from previous kernel, defaults for new options
  # ... or manually edit .config
}

build() {
  cd $_srcname

  make LOCALVERSION= bzImage modules
}

_package() {
  pkgdesc="The Linux kernel and modules (git version)"
  depends=('coreutils' 'linux-firmware' 'kmod' 'initramfs')
  optdepends=('crda: to set the correct wireless channels of your country')

  cd ${_srcname}

  # get kernel version
  _kernver="$(make LOCALVERSION= kernelrelease)"
  _basekernel=${_kernver%%-*}
  _basekernel=${_basekernel%.*}
  local _modulesdir="$pkgdir/usr/lib/modules/$_kernver"

  install -Dm644 "$(make -s image_name)" "$_modulesdir/vmlinuz"

  # Used by mkinitcpio to name the kernel
  echo "$pkgbase" | install -Dm644 /dev/stdin "$_modulesdir/pkgbase"

  make LOCALVERSION= INSTALL_MOD_PATH="${pkgdir}/usr" modules_install

  # make room for external modules
  local _extramodules="extramodules-${_basekernel}${_kernelname:--ARCH}"
  ln -s "../${_extramodules}" "${pkgdir}/usr/lib/modules/${_kernver}/extramodules"

  # add real version for building modules and running depmod from hook
  echo "${_kernver}" |
    install -Dm644 /dev/stdin "${pkgdir}/usr/lib/modules/${_extramodules}/version"

  # remove build and source links
  rm "${pkgdir}"/usr/lib/modules/${_kernver}/{source,build}
}

_package-headers() {
  pkgdesc="Header files and scripts for building modules for Linux kernel (git version)"

  cd $_srcname
  local _builddir="${pkgdir}/usr/lib/modules/${_kernver}/build"

  install -Dt "$_builddir" -m644 .config Makefile Module.symvers
  install -Dt "$_builddir/kernel" -m644 kernel/Makefile
  install -Dt "$_builddir/arch/x86" -m644 arch/x86/Makefile
  cp -t "$_builddir" -a scripts

  # add objtool for external module building and enabled VALIDATION_STACK option
  install -Dt "$_builddir/tools/objtool" tools/objtool/objtool

  # add xfs and shmem for aufs building
  mkdir -p "$_builddir"/{fs/xfs,mm}

  cp -t "$_builddir" -a include
  cp -t "$_builddir/arch/x86" -a arch/x86/include
  install -Dt "$_builddir/arch/x86/kernel" -m644 arch/x86/kernel/asm-offsets.s


  install -Dt "$_builddir/drivers/md" -m644 drivers/md/*.h
  install -Dt "$_builddir/net/mac80211" -m644 net/mac80211/*.h

  # http://bugs.archlinux.org/task/13146
  install -Dt "$_builddir/drivers/media/i2c" -m644 drivers/media/i2c/msp3400-driver.h

  # http://bugs.archlinux.org/task/20402
  install -Dt "$_builddir/drivers/media/usb/dvb-usb" -m644 drivers/media/usb/dvb-usb/*.h
  install -Dt "$_builddir/drivers/media/dvb-frontends" -m644 drivers/media/dvb-frontends/*.h
  install -Dt "$_builddir/drivers/media/tuners" -m644 drivers/media/tuners/*.h

  # copy in Kconfig files
  find . -name 'Kconfig*' -exec install -Dm644 {} "$_builddir/{}" \;

  # remove unneeded architectures
  local _arch
  for _arch in "$_builddir"/arch/*/; do
    [[ ${_arch} == */x86/ ]] && continue
    echo "Removing $(basename "$_arch")"
    rm -r "$_arch"
  done

  # remove files already in linux-docs package
  rm -r "$_builddir/Documentation"

  echo "Removing broken symlinks..."
  find -L "$_builddir" -type l -printf 'Removing %P\n' -delete

  # strip scripts directory
  local _binary _strip
  while read -rd '' _binary; do
    case "$(file -bi "$_binary")" in
      *application/x-sharedlib*)  _strip="$STRIP_SHARED"   ;; # Libraries (.so)
      *application/x-archive*)    _strip="$STRIP_STATIC"   ;; # Libraries (.a)
      *application/x-executable*) _strip="$STRIP_BINARIES" ;; # Binaries
      *) continue ;;
    esac
    /usr/bin/strip $_strip "$_binary"
  done < <(find "$_builddir/scripts" -type f -perm -u+w -print0 2>/dev/null)

  # Fix permissions
  chmod -R u=rwX,go=rX "${_builddir}"
}

pkgname=("$pkgbase" "$pkgbase-headers")
for _p in ${pkgname[@]}; do
  eval "package_$_p() {
    $(declare -f "_package${_p#$pkgbase}")
    _package${_p#$pkgbase}
  }"
done

# vim:set ts=8 sts=2 sw=2 et:

JanErik · 2019-12-04 09:51:40

Kernel updated yesterday, did not help.

lschu · 2019-12-04 19:15:49

Same problem here - various kernel versions between 5.3.12 and latest (5.4.1) have all exhibited this problem. Downgrading to 5.3.11.1-1 fixes for me, for now, but I haven't been able to find mention of this anywhere else. I did notice in journalctl -b output that I had systemd-journal dumping core after resume with 5.4.1 alongside the suspend problems which is not present in 5.3.11 - not sure if related.

Last edited by lschu (2019-12-04 19:18:33)

twelveeighty · 2019-12-04 19:26:00

JanErik wrote:

Kernel updated yesterday, did not help.

Did anyone with this problem even attempt to roll up their sleeves and follow @loqs detailed instructions he posted yesterday on how to actually contribute to finding and fixing the issue? Because complaining / confirming that the next kernel release doesn't fix the problem isn't going to fix it, until someone makes some effort to bisect which actual commit introduced this issue. Arch (and the kernel to a large extend) is a community of volunteers, so you'll have to put some effort into this.

lschu · 2019-12-04 19:29:49

twelveeighty wrote:

JanErik wrote:
Kernel updated yesterday, did not help.
Did anyone with this problem even attempt to roll up their sleeves and follow @loqs detailed instructions he posted yesterday on how to actually contribute to finding and fixing the issue? Because complaining / confirming that the next kernel release doesn't fix the problem isn't going to fix it, until someone makes some effort to bisect which actual commit introduced this issue. Arch (and the kernel to a large extend) is a community of volunteers, so you'll have to put some effort into this.

I plan to give those a go this weekend if I have time to get around to it.

loqs · 2019-12-04 20:13:48

lschu wrote:

I plan to give those a go this weekend if I have time to get around to it.

I would suggest running the first two builds overnight as they can take a long time.

makepkg -s

Dropping the r from the makepkg call, makepkg will not uninstall make dependencies which will call sudo which will time out if you have left the system to run overnight.
Similar for i makepkg will not install the built package.
When you have time to test the package the next day install with:

pacman -U linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz linux-git-headers-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz

After the first two builds the build time should fall dramatically.
The first two builds are required to check the PKGBUILD and your build system can produce a kernel with and without the issue.

Flapper · 2019-12-04 20:55:59

I did start to compile the 5.3.11 kernel, but I cancelled it after 6 hours. I didn't realise it was meant to take this long...

loqs · 2019-12-04 21:12:42

Unfortunately it can take a very long time to build and I believe the AMD Athlon 64 3500+ is a single core CPU so can not benefit from parallelizing the build.
Edit:
You might be able to avoid the second full build by instead of doing

git checkout v5.3.12

You checkout that commits parent

git checkout v5.3.12~1

This avoids the version number change from 5.3.11 to 5.3.12 which triggers the full build.

Last edited by loqs (2019-12-04 22:47:29)

Flapper · 2019-12-06 14:19:46

@loqs,

I've followed your instructions, the first makepkg , then using

git checkout v5.3.12~1

and makepkg -es.

Now I've got 2 linux-git 5.3.11 versions:
linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
linux-git-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg.tar.xz
linux-git-headers-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
linux-git-headers-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg.tar.xz

I installed the first one using:

pacman -U linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz linux-git-headers-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz

How do I proceed from here? Can I have both versions installed simultaneously?

lschu · 2019-12-06 15:02:39

Flapper wrote:

@loqs,
I've followed your instructions, the first makepkg , then using
git checkout v5.3.12~1
and makepkg -es.
Now I've got 2 linux-git 5.3.11 versions:
linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
linux-git-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg.tar.xz
linux-git-headers-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
linux-git-headers-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg.tar.xz
I installed the first one using:
pacman -U linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz linux-git-headers-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
How do I proceed from here? Can I have both versions installed simultaneously?

My understanding is that the idea here is to install a version, check whether or not the problem exists, report it to git bisect, let it make another bisection, build and install, restart and check if the bug exists, etc. until the change that introduced the problem is identified. (loqs, correct me if I'm wrong here)

Flapper · 2019-12-06 15:52:46

OK, I've tried installing both kernels:
linux-git-5.3.11.r0.gdada86c5aaa8-1-x86_64.pkg.tar.xz
linux-git-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg.tar.xz

and suspend works fine with both of them!
Is it because the second isn't a 'real' 5.3.12 kernel?

> uname -r
5.3.11-git-00049-gb4f2692948fc

loqs · 2019-12-06 17:24:40

There was one config change between the arch packages for 5.3.11 and 5.3.12 [1]
If you add the kernel parameter init_on_alloc=1 to one of the working 5.3 kernels does it then fail?

[1] https://git.archlinux.org/svntogit/pack … c2c395f22f

Last edited by loqs (2019-12-06 18:38:52)

lschu · 2019-12-06 19:46:05

Oddly enough I tried building both the 5.3.11 and 5.3.12 versions before starting bisecting following your instructions and don't experience issues with either...will take a look at that kernel parameter.

lschu · 2019-12-06 19:53:32

Okay update - confirmed that 5.3.12 built from git has the problem with init_on_alloc=1 set in kernel parameters, and does not have the problem without that flag set. I'll try testing this on later kernel versions as well.

EDIT: confirmed that setting init_on_alloc=0 makes kernel 5.4.1 work fine with suspend as well, although I'm not sure how concerned security implications?

EDIT2: also confirmed that after booting 5.4.1 with init_on_alloc=0, suspending no longer makes systemd-journal dump core, which makes me think that might be related to the issue.

Last edited by lschu (2019-12-06 20:08:19)

Flapper · 2019-12-06 20:25:36

Just done the same - linux-git-5.3.11.r49.gb4f2692948fc-1-x86_64.pkg will not suspend with kernel parameter init_on_alloc=1.

Kernel 5.4.2-arch1-1 suspends fine for me too with init_on_alloc=0.

- just not quite quick enough!

NoSavvy · 2019-12-07 10:05:13

Great job chaps. I tried it on an older pc, another kernel and another OS. Suspend is now working as it was before. Thanks.

romstor · 2019-12-07 19:07:40

In my case adding init_on_alloc=0 didn't seem to help.

$ uname -a
Linux m4400 5.4.2-arch1-1 #1 SMP PREEMPT Thu, 05 Dec 2019 12:29:40 +0000 x86_64 GNU/Linux

Flapper · 2019-12-09 12:32:47

@loqs

Have we solved this (it's not solved for everyone by the looks of it)? Or do we need to execute the other statements you provided:

$ cd linux-git/src/linux/
$ git bisect start
$ git bisect good v5.3.11
$ git bisect bad v5.3.12
$ cd ../..
$ makepkg -ersif

$ cd linux-git/src/linux/
$ git bisect $result #Substitue good or bad here
$ cd ../..
$ makepkg -ersif #Repeat these four lines and test the generated kernel until git has found the bad commit

I have 2 v5.3.11's though (see above), and one needs a kernel parameter to show up the error, so how do the git bisect commands need changing from those above?

Arch Linux

#1 2019-11-22 11:26:32

Kernel upgrade has broken suspend

#2 2019-11-24 02:10:17

Re: Kernel upgrade has broken suspend

#3 2019-11-26 06:52:56

Re: Kernel upgrade has broken suspend

#4 2019-12-01 03:43:53

Re: Kernel upgrade has broken suspend

#5 2019-12-02 23:20:01

Re: Kernel upgrade has broken suspend

#6 2019-12-02 23:42:14

Re: Kernel upgrade has broken suspend

#7 2019-12-03 01:02:10

Re: Kernel upgrade has broken suspend

#8 2019-12-03 13:12:03

Re: Kernel upgrade has broken suspend

#9 2019-12-04 09:51:40

Re: Kernel upgrade has broken suspend

#10 2019-12-04 19:15:49

Re: Kernel upgrade has broken suspend

#11 2019-12-04 19:26:00

Re: Kernel upgrade has broken suspend

#12 2019-12-04 19:29:49

Re: Kernel upgrade has broken suspend

#13 2019-12-04 20:13:48

Re: Kernel upgrade has broken suspend

#14 2019-12-04 20:55:59

Re: Kernel upgrade has broken suspend

#15 2019-12-04 21:12:42

Re: Kernel upgrade has broken suspend

#16 2019-12-06 14:19:46

Re: Kernel upgrade has broken suspend

#17 2019-12-06 15:02:39

Re: Kernel upgrade has broken suspend

#18 2019-12-06 15:52:46

Re: Kernel upgrade has broken suspend

#19 2019-12-06 17:24:40

Re: Kernel upgrade has broken suspend

#20 2019-12-06 19:46:05

Re: Kernel upgrade has broken suspend

#21 2019-12-06 19:53:32

Re: Kernel upgrade has broken suspend

#22 2019-12-06 20:25:36

Re: Kernel upgrade has broken suspend

#23 2019-12-07 10:05:13

Re: Kernel upgrade has broken suspend

#24 2019-12-07 19:07:40

Re: Kernel upgrade has broken suspend

#25 2019-12-09 12:32:47

Re: Kernel upgrade has broken suspend

Board footer