You are not logged in.

#1 2019-03-25 15:45:15

honeyplease
Member
Registered: 2013-09-07
Posts: 33

How exactly does a swap file work?

I'm performing some spatial joins on a few million polygons and I'm running out of RAM. I need some swap space and I'm going for a swap file instead of repartitioning. I'm using ext4, kernel 5.0.3, gpt, swappiness at default 60, 16Gb RAM. I am following the wiki using the automated method, so installed and enabled systemd-swap and changed the conf file accordingly. Rebooted, ran the job again and ran out of RAM.

The same operation works fine in a mac with half the RAM but I can see its swap getting filled heavily (~4Gb) and increased dynamically as needed while RAM is held in check. I'd like to reproduce this in my linux machine, and I thought by using the automated method I would be able to but it's not working. My swap on the linux machine is now at 512M, but it doesn't kick in until RAM hits about 95% (~15 ish Gb) and I'm still running out of RAM while swap isn't even half way used.

So I have a few questions:

1) Is a swap file the right tool if I want a dynamically allocated swap memory?

2) (In /etc/systemd/swap.conf) swapfc_chunk_size=512M: is this the size of increments every time the system needs more memory?

3) swapfc_max_count=8: is this the max number of new memory chunks the system will make available? So 512*8 ~ 4Gb max of swap?

4) swapfc_free_swap_perc=15 # Add new chunk if free < 15%: is this 15% of the 512M swap chunk or is it 15% of total RAM?

What would be the smart way to reproduce the intended behavior in my arch machine? Basically: keep RAM in check (at say, 75%) by using swap memory handsomely.

Thanks.

Offline

#2 2019-03-25 21:35:51

seth
Member
Registered: 2012-09-03
Posts: 49,981

Re: How exactly does a swap file work?

You could use a cgroup and limit the memory, but keep the swap qutoa untouched for the group (to contain the effect to the particular group) ie. "memory.limit_in_bytes", but NOT "memory.memsw.limit_in_bytes", -1 is unlimited.
https://wiki.archlinux.org/index.php/Cgroups

Offline

#3 2019-03-25 22:04:31

honeyplease
Member
Registered: 2013-09-07
Posts: 33

Re: How exactly does a swap file work?

Interesting!

Does this interact seamlessly with systemd-swap?

From what I understand, its like I'm dividing the problem into two:

1) The cgroup will keep RAM in check to the value specified in bytes in 'memory.limit_in_bytes'. Say, 12 out of 16 Gb.
2) But, this does't help if there is no swap space, and so to deal with swap space is what I need systemd-swap for?

Is this correct?

Offline

#4 2019-03-25 22:42:09

seth
Member
Registered: 2012-09-03
Posts: 49,981

Re: How exactly does a swap file work?

When you're running out of memory.limit_in_bytes, you're gonna need some swap space or the process will act up (typically segfault)
How the swap is made available to the system (static, dynamic, partition, file, voodoo…) doesn't matter, see https://wiki.archlinux.org/index.php/Swap on how to knock yourself.

There should btw, not be a requirement to "keep RAM in check" - the quotas exist to keep different users apart or eg. "protect" realtime processes.
Or is your issue that systemd-swap cannot keep up w/ your memory demands? (Which is why you're "still running out of RAM")
In that case I've frankly no idea but would suggest to just create a sufficiently big swap file for the process ahead of running it - otherwise you might OOM at 12GB RAM demanded instead of 16GB.

Offline

#5 2019-03-25 23:34:06

honeyplease
Member
Registered: 2013-09-07
Posts: 33

Re: How exactly does a swap file work?

Thanks this is useful, sounds like what I need to look into, thanks for your time!

Offline

Board footer

Powered by FluxBB