Increase process limit

Sourav · 2020-08-22 15:25:13

I have a code that forks and runs database seed for a Ruby on Rails application. Fork makes it faster because Ruby up to v2.7 has GIL. It also creates zombie processes. For example, here I am creating around 10K zombie processes with this code:

$ ruby -r timeout -e "10000.times { Timeout.timeout(5) { fork { exit! } } rescue break } ; sleep 1 ; puts Dir.glob('/proc/[0-9]*').count "

The total process count is always 9602 to 9604.

If the program sleeps for long, I can't create a new process as expected, I get:

bash: fork: retry: Resource temporarily unavailable

It sounds like I have reached the maximum process limit, but nowhere it's mentioned that I can only fork 9600ish processes.
I have edited /etc/security/limits.conf and appended the following lines:

@sourav soft nproc 2000000
@sourav hard nproc 2000000
@sourav soft core 2000000
@sourav hard core 2000000

Also ran

ulimit -u unlimited ; ulimit -s unlimited ; ulimit -r unlimited ; ulimit -q unlimited ; ulimit -f unlimited

$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14924
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14924
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ systemctl show -p DefaultTasksMax
DefaultTasksMax=4477

Kernel:

Linux version 5.8.0-arch1-2 (linux@archlinux) (gcc (GCC) 10.1.0, GNU ld (GNU Binutils) 2.34.0) #1 SMP PREEMPT Sun, 09 Aug 2020 00:57:20 +0000

Why am I still receiving fork error on creating more than 9604ish processes? I have a program that runs on fork and the parent process stays in memory, that means it creates zombies, so I want to create at least 50K processes on my system.

loqs · 2020-08-22 17:19:36

What is the output of

$ cat /proc/sys/kernel/threads-max

Sourav · 2020-08-22 17:41:42

loqs wrote:

What is the output of
$ cat /proc/sys/kernel/threads-max

Hello, it's

loqs · 2020-08-22 18:07:55

29848/2 = 14924 the default max processes
29848*0.15 = 4477 systemd DefaultTasksMax

If you raise /proc/sys/kernel/threads-max to 100000 can you then increase max user processes, you may need to use prlimit as root with the PID of the shell.

Sourav · 2020-08-22 19:51:59

loqs wrote:

29848/2 = 14924 the default max processes
29848*0.15 = 4477 systemd DefaultTasksMax
If you raise /proc/sys/kernel/threads-max to 100000 can you then increase max user processes, you may need to use prlimit as root with the PID of the shell.

That's great. So I might write there with echo or sysctl if I am not wrong, so I tried executing:

# sysctl kernel.threads-max=100000

But after running this command and checking with cat and sysctl command, I immediately ran the ruby command to create zombies, and the problem still occurs. I can't even create 10K processes on my system.

Weird enough:

$ cat /proc/sys/kernel/threads-max 
100000

$ ulimit -a
core file size          (blocks, -c) 2000000
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14924
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2000000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ systemctl show -p DefaultTasksMax
DefaultTasksMax=15000

$ ruby -r timeout -e "10000.times { Timeout.timeout(5) { fork { exit! } } rescue break } ; sleep 1 ; puts Dir.glob('/proc/[0-9]*').count "
9528

The ruby program exits when it waits 5 second to create the fork, in most cases it's mainly because the resource limit is reached.

Similarly, I can't open any other process (tried another terminal for example) after running this command:

10000.times { fork { exit! } }

Even when the threads-max is 100,000. I don't have a lot of other processes opened, just firefox, and 2 - 3 tilix windows.
So this test should pass, and the system should let me create at least 10K zombie processes...

Last edited by Sourav (2020-08-22 20:06:30)

loqs · 2020-08-22 20:34:28

Going back to the initial issue. Why is the parent process not reaping the children? If you do not handle SIGCHLD you could easily be running into the pending signals limit.
File handle exhaustion would be another candidate or are you certain the max processes is the cause?

pending signals                 (-i) 14924
....
open files                      (-n) 1024

Sourav · 2020-08-22 21:00:09

AFAIK, processes you fork can be killed, or they can exit gracefully, but either way, you create a zombie, which can't be killed, even if you give it SIGKILL, it won't exit unless the parent exits.

So the ideal way maybe is to make the parent exit. But that's not my main goal. My main goal is to seed a rails database. The main process counts and shows the number of entries inserted into the db, and also it tracks the current time and estimated time. Also, it gives 16 tasks at a time, no more than that. Here's the code:

https://github.com/Souravgoswami/messag … b/seeds.rb

Sure, it's a bit of a bad example. But it gets the job done 3x faster. The problem is I love benchmarking and I wanted to create > 10K sample data. Which is surely viable with the code, but it will create thousands of zombie processes, but they don't cause any harm, I can't keep track of them because the parent program can't be exited.

I am not sure if a simple fork that becomes zombie exhausts number of opened files. The code in the question, just creates fork which lasts for in nano-seconds. But they becomes zombies and the fork limit is reached. I am not sure why it just stops at 9K or so.

systemd-cgtop shows the total tasks to only 10K or so.

loqs · 2020-08-22 21:10:19

Can you not ignore SIGCHLD in the parent or install a signal handler for it? https://www.geeksforgeeks.org/zombie-pr … revention/

Sourav · 2020-08-23 08:00:38

I was actually looking for increasing the number for processes... But for my purpose, I can use Ruby's Process.wait, it eliminates the chance of creating zombies. Thanks a lot for the suggestion!

Last edited by Sourav (2020-08-23 08:00:57)

Arch Linux

#1 2020-08-22 15:25:13

Increase process limit

#2 2020-08-22 17:19:36

Re: Increase process limit

#3 2020-08-22 17:41:42

Re: Increase process limit

#4 2020-08-22 18:07:55

Re: Increase process limit

#5 2020-08-22 19:51:59

Re: Increase process limit

#6 2020-08-22 20:34:28

Re: Increase process limit

#7 2020-08-22 21:00:09

Re: Increase process limit

#8 2020-08-22 21:10:19

Re: Increase process limit

#9 2020-08-23 08:00:38

Re: Increase process limit

Board footer