Arch Linux

kmod

Member

Registered: 2021-07-29

Posts: 12

[SOLVED] numpy killed

Hi,

I tried to do a simple experiment with numpy and it crashes

Python 3.9.6 (default, Jun 30 2021, 10:22:16) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> a = np.random.rand(10000, 10000)
>>> b = np.random.rand(10000, 10000)
>>> a@b
Killed

I tried different versions of numpy: numpy installed from the main repository, python-numpy-mkl from AUR and numpy installed from pip (of course I removed one version before installing another) all with same result. This happens in the wayland session, on xorg this works but np.test() was killed there (which works in wayland)

Please help.

Thanks.

Last edited by kmod (2021-08-08 03:18:42)

kmod

Member

Registered: 2021-07-29

Posts: 12

Re: [SOLVED] numpy killed

So it looks like it is a memory problem. If I do 5000x5000 arrays instead of 10000x10000 it works as expected. If I increase ram from 4G to 6G python no longer get killed for 10000x10000 random arrays.

But it seems that 4G of ram should be sufficient for this experiment. On my other installation with Ubuntu 21.04 the experiment above works fine (with 10000x10000 matrices) with only 4G of ram.

Maybe there is some settings or tweak to optimize memory usage in Arch that I am missing? Please advise, thanks.

kmod

Member

Registered: 2021-07-29

Posts: 12

Re: [SOLVED] numpy killed

OK, I set up vm.swappiness by creating

a file 99-sysctl.conf in

/etc/sysctl.d and put this line in it

vm.swappiness=50

Problem solved. numpy no longer get killed with 4 G of ram.

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 77,368

Re: [SOLVED] numpy killed

You either setup some SWAP itfp. or freed memory from different processes.
Chances that making it 5% more likely to drop file caches has an actual impact here are virtually zero (but for some weird kernel bug)

If you've temporary ultra-high memory demands, you can add a swapfile anytime: https://wiki.archlinux.org/title/Swap#Swap_file

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

kmod

Member

Registered: 2021-07-29

Posts: 12

Re: [SOLVED] numpy killed

You either setup some SWAP itfp. or freed memory from different processes.
Chances that making it 5% more likely to drop file caches has an actual impact here are virtually zero (but for some weird kernel bug)

If you've temporary ultra-high memory demands, you can add a swapfile anytime: https://wiki.archlinux.org/title/Swap#Swap_file

Yes I ended up adding a swapfile.