You are not logged in.

#1 2017-10-19 21:10:38

snovik
Member
Registered: 2012-05-06
Posts: 140

Issues with swap - the system hangs while swap seems to be active

I have swap as file:

~ $ swapon --summary
Filename				Type		Size	Used	Priority
/dev/loop0                             	partition	8076736	0	1

However it does not seem to work. The system hangs on memory heavy operations. E.g. running this program kills the system when the RAM is filled in as in htop

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char** argv) {
    int max = -1;
    int mb = 0;
    char* buffer;

    if(argc > 1)
        max = atoi(argv[1]);

    while((buffer=malloc(1024*1024)) != NULL && mb != max) {
        memset(buffer, 0, 1024*1024);
        mb++;
        printf("Allocated %d MB\n", mb);
        sleep(1);
    }      
return 0;
}

I cannot figure out what is missing in the configuration.

~ $ systemctl status systemd-swap

● systemd-swap.service - Manage swap spaces on zram, files and partitions.
   Loaded: loaded (/usr/lib/systemd/system/systemd-swap.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2017-10-20 00:03:36 EEST; 5min ago
  Process: 424 ExecStart=/usr/bin/systemd-swap start (code=exited, status=0/SUCCESS)
 Main PID: 424 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
      CPU: 0
   CGroup: /system.slice/systemd-swap.service

Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: create empty file
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: mark file /var/lib/systemd-swap/swapfu nocow
Oct 20 00:03:36 sergej systemd-swap[424]: chattr: Operation not supported while setting flags on /var/lib/systemd-swap/swapfu
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: max file size: 0 -> 8076740K
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: attach /var/lib/systemd-swap/swapfu to free loop
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: using /dev/loop0
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: initialize swap device /dev/loop0
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: swapon - discard: enabled
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapFU: disable directio for /dev/loop0
Oct 20 00:03:36 sergej systemd-swap[424]: INFO: swapD: searching swap devices

/etc/systemd/swap.conf

################################################################################
# Defaults are optimized for general usage
################################################################################

################################################################################
# Zswap
#
# Kernel >= 3.11
# Zswap create compress cache between swap and memory for reduce IO
# https://www.kernel.org/doc/Documentation/vm/zswap.txt

zswap_enabled=0
zswap_compressor=lz4      # lzo lz4
zswap_max_pool_percent=25 # 1-99
zswap_zpool=z3fold        # zbud z3fold

################################################################################
# ZRam
#
# Kernel >= 3.15
# Zram compression streams count for additional information see:
# https://www.kernel.org/doc/Documentation/blockdev/zram.txt

zram_enabled=0
zram_size=$(($ram_size/4))K     # This is 1/4 of ram size by default.
zram_streams=$cpu_count
zram_alg=lz4                    # lzo lz4 deflate lz4hc 842 - for Linux 4.8.4
zram_prio=32767                 # 1 - 32767

################################################################################
# Swap File Universal
# loop + swapfile = support any fs (also btrfs)
swapfu_enabled=1
# File is sparse and dynamically allocated.
swapfu_size=${ram_size}K # Size of swap file.
# But you may don't like it, ex. in a case when you don't have much free space
swapfu_preallocated=0
# Don't use cow (btrfs)
swapfu_nocow=1
# Set directio for loop device
# If enabled, can be more safe in case of memory OOM
swapfu_directio=0
# Test setup:
# files with size 2G (mounted by loop), fs: btrfs
# fio: test 4k, randrw, iodepth 64, libaio, directio 1
# HDD 1TB 5400k
# Loop device:
# PREALLOC | NoCoW | DirectIO | ~ IO/s | Comment
#    -     |   -   |    -     |   2400 | NOT stable speed
#    +     |   -   |    -     |    160 | stable speed, HORRIBLY SLOW
#    -     |   +   |    -     |   3300 | stable speed, can degraded by time
#    +     |   +   |    -     |   2000 | stable speed
#    -     |   -   |    +     |     50 | not stable speed
#    +     |   -   |    +     |     69 | not stable speed
#    -     |   +   |    +     |     63 | not stable speed
#    +     |   +   |    +     |     75 | not stable speed
# raw file (loop device are slow):
# PREALLOC | NoCoW | DirectIO | ~ IO/s | Comment
#    -     |   -   |    +     |   7600 | not stable speed
#    +     |   -   |    +     |     69 | not stable speed
#    -     |   +   |    +     |  13000 | not stable speed
#    +     |   +   |    +     |    174 | not stable speed

# File will not be available in fs after script start
# Make sure what script can access to this path during the boot process.
# Full path to swapfile
swapfu_path=/var/lib/systemd-swap/swapfu
swapfu_prio=1     # 1 - 32767

################################################################################
# Swap File Chunked
# Allocate swap files dynamically
# Min swap size 256M, Max 16*256M
swapfc_enabled=0
swapfc_frequency=1s         # How often check free swap space
swapfc_chunk_size=256M      # Allocate size of swap chunk
swapfc_max_count=16         # 0 - unlimited, note: 32 is a kernel maximum
swapfc_free_swap_perc=15    # Add new chunk if free < 15%
                            # Remove chunk if free > 15+40% & chunk count > 2
swapfc_path=/var/lib/systemd-swap/swapfc/

################################################################################
# Swap devices
# Find and auto swapon all available swap devices
swapd_auto_swapon=1
swapd_prio=1024

Last edited by snovik (2017-10-19 21:13:40)

Offline

#2 2017-10-20 06:40:29

mpan
Member
Registered: 2012-08-01
Posts: 494
Website

Re: Issues with swap - the system hangs while swap seems to be active

Unless you have a reason to believe that the slowdown should not occur, and you can’t have one with that source code, everything is fine. Swap does its job: it prevents OOM killer from kicking in on expense of temporarily slowing down your system, possibly to the point of locking it temporarily if you keep pushing more and more memory into swap (and the code does exactly that).

Swap is not RAM. It’s not meant to be fast. If you want your system to be always working without slow downs, disable swap.


Sometimes I seem a bit harsh — don’t get offended too easily! PGP: 7C848198AE93D3BB

Offline

#3 2017-10-20 09:16:16

snovik
Member
Registered: 2012-05-06
Posts: 140

Re: Issues with swap - the system hangs while swap seems to be active

The system hangs, not slows down. The mouse does not move and only hard rest helps.

I have ssd and the program does not generate that much of disk load as it is also has a 1 sec delay between increments

Offline

#4 2017-10-20 13:17:34

snovik
Member
Registered: 2012-05-06
Posts: 140

Re: Issues with swap - the system hangs while swap seems to be active

Here are all complaints from dmesg | grep -i "error\|warn\|fail"

[    0.529483] platform MSFT0101:00: failed to claim resource 1: [mem 0xfed40000-0xfed40fff]
[    0.529489] acpi MSFT0101:00: platform device creation failed: -16
[    0.607851] pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
[    0.731351] RAS: Correctable Errors collector initialized.
[    3.202319] bluetooth hci0: Direct firmware load for rtl_bt/rtl8821a_config.bin failed with error -2
[    4.985790] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985828] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985848] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985877] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985896] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985955] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    4.985974] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)
[    5.000962] ACPI Warning: \_SB.PCI0.RP09.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20170531/nsarguments-95)

pci 0000:03:00.0 is nvidia. I found this  https://wiki.archlinux.org/index.php/NV … i_hangs.29
It is kind of correlates with ACPI messages above but I dont have problems suspending and am using discrete card only.

I keep on being lost. And I have no clue what the heck MSFT0101:00 is supposed to mean

Last edited by snovik (2017-10-20 13:20:52)

Offline

#5 2017-10-20 15:46:40

mpan
Member
Registered: 2012-08-01
Posts: 494
Website

Re: Issues with swap - the system hangs while swap seems to be active

And I was clear that the slowdown may look like the system hangs. If OOM killer does its job, it can also last very long — in my case it’s 20–30 minutes. Anyway, what I have missed is that you do MB allocations, not GB ones. Indeed I can’t imagine how ~1MB/s could cause much problems even in the most pessimistic scenarios.

The logs show nothing suspicious. dmesg refers to the first seconds after the system is started. So the questions: does the issue happen when you use normal swap partition? Or, if you don’t have one, a fully pre-allocated swap file? With no helper scripts etc.

There is also a similar topic — while I truly doubt that, there still is a slight possibility of a new bug in the kernel.


Sometimes I seem a bit harsh — don’t get offended too easily! PGP: 7C848198AE93D3BB

Offline

#6 2017-12-30 03:52:37

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 13
Website

Re: Issues with swap - the system hangs while swap seems to be active

I have the same problem. 16GB ram, NVMe SSD. I'm using LVM swap partition (16gb), as soon as it starts swapping, everything freezes. Sometimes after a while it unfreezes, sometimes it's just few seconds of freezing, sometimes it locks up for good. When it's frozen you can't do anything (swap consoles, etc), nothing works.

Offline

#7 2017-12-31 18:45:46

seth
Member
Registered: 2012-09-03
Posts: 13,080

Re: Issues with swap - the system hangs while swap seems to be active

https://bbs.archlinux.org/viewtopic.php … 7#p1758127 suggests it's indeed an issue with an LVM backed swap partition.

Offline

#8 2018-01-03 21:03:14

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 13
Website

Re: Issues with swap - the system hangs while swap seems to be active

Any idea where we should report this or if it's already reported?

Offline

#9 2018-01-03 21:18:29

seth
Member
Registered: 2012-09-03
Posts: 13,080

Re: Issues with swap - the system hangs while swap seems to be active

https://bugzilla.kernel.org/

For the records: you're doing swap on LVM as well?

Offline

#10 2018-01-03 21:23:13

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 13
Website

Re: Issues with swap - the system hangs while swap seems to be active

Yes I have swap partition on LVM.

Offline

#11 2018-01-03 21:29:43

seth
Member
Registered: 2012-09-03
Posts: 13,080

Re: Issues with swap - the system hangs while swap seems to be active

Ah sh**, I mistook you for snovik (I knew your setup from comment #6 of course) - sorry.

Offline

#12 2018-01-07 22:05:27

m3thodic
Member
Registered: 2004-08-24
Posts: 66

Re: Issues with swap - the system hangs while swap seems to be active

@MaikuMori I'm having almost the same identical problem. While I'm trying to figure out systemd-swap, I just submitted a pull request for earlyoom to be more aggressive when oom killing by not considering swap space usage: https://github.com/rfjakob/earlyoom/pull/44

Offline

#13 2018-01-07 23:24:47

m3thodic
Member
Registered: 2004-08-24
Posts: 66

Re: Issues with swap - the system hangs while swap seems to be active

@MaikuMori I think I figured it out.... set swapfu_nocow=0 in /etc/systemd/swap.conf, then systemctl restart systemd-swap.service

Offline

#14 2018-02-18 18:47:58

catalin.hritcu
Member
From: Paris, France
Registered: 2014-04-27
Posts: 26
Website

Re: Issues with swap - the system hangs while swap seems to be active

@m3thodic  I tried setting swapfu_nocow=0 but it made no difference for me. I'm using a swap file on an encrypted LVM volume, and reading on this forum it seems it's LVM that's causing the problem. Any fix that doesn't involve re-partitioning?

Offline

#15 2018-03-20 11:55:51

m3thodic
Member
Registered: 2004-08-24
Posts: 66

Re: Issues with swap - the system hangs while swap seems to be active

@catalin.hritcu the maintainer had made some updates, I am currently running 4.0.1-1 with only zram_enabled=1 set and it's been running great on all of my systems.

Offline

#16 2018-03-20 15:56:44

catalin.hritcu
Member
From: Paris, France
Registered: 2014-04-27
Posts: 26
Website

Re: Issues with swap - the system hangs while swap seems to be active

m3thodic wrote:

@catalin.hritcu the maintainer had made some updates, I am currently running 4.0.1-1 with only zram_enabled=1 set and it's been running great on all of my systems.

I'm afraid I don't understand this. What package are you talking about here when you write "the maintainer" and "4.0.1-1"?

Offline

#17 2018-03-20 17:03:54

loqs
Member
Registered: 2014-03-06
Posts: 9,235

Re: Issues with swap - the system hangs while swap seems to be active

catalin.hritcu wrote:
m3thodic wrote:

@catalin.hritcu the maintainer had made some updates, I am currently running 4.0.1-1 with only zram_enabled=1 set and it's been running great on all of my systems.

I'm afraid I don't understand this. What package are you talking about here when you write "the maintainer" and "4.0.1-1"?

https://www.archlinux.org/packages/comm … temd-swap/

Offline

#18 2018-03-20 18:53:20

MaikuMori
Member
From: Latvia
Registered: 2013-05-29
Posts: 13
Website

Re: Issues with swap - the system hangs while swap seems to be active

I'll try it ... anything to stop this madness.

But in reality this is not solving the underlying problem. If you only use zram/zswap then you're bypassing the swap partition. The problem is that using swap on LVM partition is causing system hangs and it shouldn't.

Offline

Board footer

Powered by FluxBB