"unable to create new native thread" or "fork failed" since linux 4.3

kaouete · 2016-01-07 09:26:11

Hi,

I came back from vacations and I started to get lots of errors when running many processes (in particular Java ones that creates many threads) such as:

OutOfMemoryError: unable to create new native thread from Java

but also

fork failed: resource temporarily unavailable

When executing commands in bash.

It seems that ulimit -a tells me that the number of maximum file descriptor I can open is 1024, that seems a bit low, and checking my coworker settings, his debian is using 65536 as a max value!
If I try to set it to this value with "ulimit -n 65536" I get the error: ulimit: value exceeds hard limit

Is there something that changed lately that I am not aware of in Archlinux that resulted to this situation?

EDIT: The problem appeared when the package linux was upgraded from 4.2.5-1 to 4.3.3-2

Thank you for your help :)

Last edited by kaouete (2016-01-07 11:59:23)

lucke · 2016-01-07 09:56:11

My Debian server has it set to 1024.

You should be able to set it in /etc/security/limits.conf.

codyzu · 2016-01-07 10:16:10

I'm having the same problem with a Java application. Which JVM are you using? I'm using jdk 8u66-1 from the AUR (I have to use Oracle java for my work). I recently switched to Arch on my development machine and have not been able to run my java application (works in ubuntu 14.04 with the same JVM).

Like you I played with the ulimits stuff and after a lot of trial and error, my limits are huge:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63706
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 500000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 500000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

But like you my java application always crashes with:
java.lang.OutOfMemoryError: unable to create new native thread

On my arch machine, while most of the limits set in /etc/security/limits.conf are respected, "nofile" is not, and must be set in /etc/systemd/system.conf "DefaultLimitNOFILE=500000" as noted here.

I'm posting because I have the same problem and have spent several hours playing with the limits, but java still crashes saying it can't create native threads. I'm thinking it's a java (or dependency) problem. Very curious if you are able to resolve your issue...

kaouete · 2016-01-07 10:24:03

Codyzu, exactly my problem! Also for me the limits are not respected for nofile.

I think that maybe we made an error of interpretation, the max number of opened file descriptor is PER process, so maybe this is not the reason of our problem.

It's not so clear to me what is the reason… I will rename the topic of the thread.

kaouete · 2016-01-07 10:26:54

By the way I am using Java 7 currently, the one from Oracle.

I don't think it is a Java problem because when the "unknown" limit is reached, other applications starts to fail!

codyzu · 2016-01-07 10:51:31

You may want to try if your problem goes away with the openJDK (if you can, Oracle is a requirement for me)... I've seen similar posts where people noted it was only a problem with Oracle java...

Anyways, the nofile limit in limits.conf is not respected and must be set in /etc/systemd/system.conf as noted in these 2 links:

I'm curious if your problem is resolved with openJDK... I didn't try changing the version of glib2 (as noted in the post I linked to) yet as some people reported it can cause other things to crash.

kaouete · 2016-01-07 11:00:58

No, using openjdk doesn't change anything, but the errors you refer to are not the one I'm getting.

I'm getting things like that when my Java process already runs:
[exec] Exception in thread "Thread-4" java.lang.OutOfMemoryError: unable to create new native thread
[exec] at java.lang.Thread.start0(Native Method)
[exec] at java.lang.Thread.start(Thread.java:714)
[exec] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
[exec] at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
[exec] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
[exec] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[exec] at java.lang.Thread.run(Thread.java:745)

And I also get similar errors in zsh when running commands at the same time the problem arises with Java.

It doesn't seem that the process has so many opened fd neither…

kaouete · 2016-01-07 11:09:13

And of course changing the nofile limit in systemd changes it in ulimit, but doesn't solve my problem…

kaouete · 2016-01-07 11:56:08

Ok, after many tests, I found that the problems appear as soon as I upgrade the linux package from 4.2.5 to 4.3.3!

codyzu · 2016-01-07 13:42:46

Wow!!! I tried the lts kernel after your last post and it works perfect!

I think we found a problem with the current kernel version and java. I prefer to not have to use the lts kernel, but for the moment that is a workaround if anyone else encounters the problem...

kaouete · 2016-01-07 13:52:15

I still don't think it is related to Java, but java programs often have a way of abusing threads that makes the problem apparent.

I think there was some changes in the kernel that are either a choice or a bug, but I haven't had the time to investigate it yet

cdauth · 2016-01-07 19:38:14

I am discussing the same issue here: http://unix.stackexchange.com/q/253903/59955

I have opened a bug report here: https://bugs.archlinux.org/task/47662

kaouete · 2016-01-08 08:27:31

Thanks for the information

So maybe it is a problem of config, is that it?

kmacleod · 2016-01-08 14:57:48

I am seeing this exact problem.

I did some testing using the C code contained in:
http://stackoverflow.com/questions/3521 … s-at-65528

Using the current 4.3.3 kernel it maxes out at around 4000 threads for me. In 4.2/4.1 LTS it can get to over 31000.

I've also played around with ulimit settings in /etc/security/limits.conf (increasing for all users/root, etc) and also setting DefaultLimitNOFILE in /etc/systemd/system.conf to a high number but that has no affect.

The only sort of reference to the issue I can find is this entry in the mailing list, but there is no followup:
https://lkml.org/lkml/2015/11/24/203

I've switched to the linux-lts kernel in order to avoid the 4.3 kernel for now.

Last edited by kmacleod (2016-01-08 15:03:30)

kmacleod · 2016-01-11 19:26:44

Does anyone by chance know of any potential kernel compile flags that might fix this?

cdauth · 2016-01-15 19:23:49

In my case the problem came up with docker, and I have solved it now by setting "ThreadsMax=infinity" in my docker.service systemd file. More information here: http://unix.stackexchange.com/a/255603/59955

kmacleod · 2016-01-15 19:39:12

Hmm, thanks for that. This might explain my issue. I'm running into this issue mainly when connecting to my server via xrdp. The xrdp service is started via systemd.

I'm hitting the limit very early when connected via xrdp vs. a regular desktop session - it's very likely I'm hitting the task limit of 512. I'll try setting "ThreadsMax=infinity" in my xrdp services and see if that fixes it for me.

kaouete · 2016-01-16 16:01:28

I opened another bug report for when the problem is not with a systemd's started application but directly happening to the user: https://bugs.archlinux.org/task/47787

dead10ck · 2016-01-21 19:55:02

cdauth wrote:

In my case the problem came up with docker, and I have solved it now by setting "ThreadsMax=infinity" in my docker.service systemd file. More information here: http://unix.stackexchange.com/a/255603/59955

Thank you! That solved the problem for me.

mich41 · 2016-01-21 20:21:42

kaouete wrote:

I opened another bug report for when the problem is not with a systemd's started application but directly happening to the user: https://bugs.archlinux.org/task/47787

Probably you need to apply the same configuration change to your login manager's service.

kaouete · 2016-01-22 08:14:38

mich41 wrote:

kaouete wrote:
I opened another bug report for when the problem is not with a systemd's started application but directly happening to the user: https://bugs.archlinux.org/task/47787
Probably you need to apply the same configuration change to your login manager's service.

No, as described in the first comment of the bug report, the problem comes from systemd actually

mich41 · 2016-01-22 10:30:08

I looked at this bug and the linked systemd commit (awful commit message, btw) and it looks like they just added one line to some service file, exactly as others in this thread did.

kaouete · 2016-01-22 11:09:17

mich41 wrote:

I looked at this bug and the linked systemd commit (awful commit message, btw) and it looks like they just added one line to some service file, exactly as others in this thread did.

It's not any systemd file, it is the one used for the user session: it's not as if the user would decide to use this unit file, it is the one EVERY user will be using for its session.
Not saying I can't modify it myself, just that by default, any user will be limited to 512 processes, which is clearly not enough nowadays

It's a bug in systemd in the sense that this unit file is not really meant to be modified, and that the systemd team's desire was to limit the number of processes to 4096, not 512.

oconnor663 · 2016-02-15 16:08:27

Based on all the discussion above, I worked around this by creating the file /etc/systemd/system.conf.d/limits.conf (the exact file name doesn't matter) with the contents

[Manager]
DefaultLimitNOFILE=65536
DefaultTasksMax=65536

That's a pretty big hammer, but it seems to fix my "fork failed" errors. Maybe others can clarify what versions of linux/systemd will make this hack no longer necessary?

kaouete · 2016-02-15 17:37:01

oconnor663 wrote:

Based on all the discussion above, I worked around this by creating the file /etc/systemd/system.conf.d/limits.conf (the exact file name doesn't matter) with the contents
[Manager]
DefaultLimitNOFILE=65536
DefaultTasksMax=65536
That's a pretty big hammer, but it seems to fix my "fork failed" errors. Maybe others can clarify what versions of linux/systemd will make this hack no longer necessary?

Actually it depends on your situation, if you read the various bug report, you will notice for example that being on Gnome makes a difference when running thread intensive applications in a terminal.

Anyway the main fixes for this problem is coming with systemd 229 that is currently in testing

Arch Linux

#1 2016-01-07 09:26:11

"unable to create new native thread" or "fork failed" since linux 4.3

#2 2016-01-07 09:56:11

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#3 2016-01-07 10:16:10

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#4 2016-01-07 10:24:03

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#5 2016-01-07 10:26:54

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#6 2016-01-07 10:51:31

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#7 2016-01-07 11:00:58

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#8 2016-01-07 11:09:13

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#9 2016-01-07 11:56:08

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#10 2016-01-07 13:42:46

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#11 2016-01-07 13:52:15

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#12 2016-01-07 19:38:14

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#13 2016-01-08 08:27:31

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#14 2016-01-08 14:57:48

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#15 2016-01-11 19:26:44

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#16 2016-01-15 19:23:49

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#17 2016-01-15 19:39:12

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#18 2016-01-16 16:01:28

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#19 2016-01-21 19:55:02

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#20 2016-01-21 20:21:42

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#21 2016-01-22 08:14:38

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#22 2016-01-22 10:30:08

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#23 2016-01-22 11:09:17

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#24 2016-02-15 16:08:27

Re: "unable to create new native thread" or "fork failed" since linux 4.3

#25 2016-02-15 17:37:01

Re: "unable to create new native thread" or "fork failed" since linux 4.3

Board footer