Cannot increase nofile limits (ulimit)

tsdh · 2015-09-21 07:20:25

Today I wanted to compile a latex file which includes thousands of other files and got a "Too many open files" error. On Friday, I could still compile the file without problems.

Ok, so apparently the nofile limits have changed somehow and indeed, I get:

% ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       63409
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  1024
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 63409
-q: bytes in POSIX msg queues       819200
-e: max nice                        20
-r: max rt priority                 0
-N 15:                              unlimited

% ulimit -Hn
4096

% ulimit -Sn
1024

But /etc/security/limits.conf contains the lines

horn      soft    nofile      500000
horn      hard    nofile      900000

where horn is my user name. But apparently, these lines seem to have become ineffective, most probably by some very recent package upgrade.

Ok, so these were the packages that have been upgraded since Friday:

75  [2015-09-18 06:50] upgraded ncurses (5.9-7 -> 6.0-3)
74  [2015-09-18 06:50] upgraded readline (6.3.008-1 -> 6.3.008-2)
73  [2015-09-18 06:50] upgraded bash (4.3.042-1 -> 4.3.042-2)
72  [2015-09-18 06:50] upgraded gpm (1.20.7-4 -> 1.20.7-5)
71  [2015-09-18 06:50] upgraded aalib (1.4rc5-10 -> 1.4rc5-11)
70  [2015-09-18 06:50] upgraded psmisc (22.21-2 -> 22.21-3)
69  [2015-09-18 06:50] upgraded alsa-utils (1.0.29-1 -> 1.0.29-2)
68  [2015-09-18 06:50] upgraded aspell (0.60.6.1-2 -> 0.60.6.1-3)
67  [2015-09-18 06:50] upgraded libutil-linux (2.26.2-1 -> 2.27-2)
66  [2015-09-18 06:50] upgraded bind-tools (9.10.2.P4-1 -> 9.10.3-1)
65  [2015-09-18 06:50] upgraded less (481-1 -> 481-2)
64  [2015-09-18 06:50] upgraded texinfo (5.2-3 -> 5.2-4)
63  [2015-09-18 06:50] upgraded cmake (3.3.1-2 -> 3.3.1-3)
62  [2015-09-18 06:50] upgraded dialog (1:1.2_20150528-1 -> 1:1.2_20150528-2)
61  [2015-09-18 06:50] upgraded util-linux (2.26.2-1 -> 2.27-2)
60  [2015-09-18 06:50] upgraded libedit (20150325_3.1-1 -> 20150325_3.1-2)
59  [2015-09-18 06:50] upgraded llvm-libs (3.6.2-3 -> 3.6.2-4)
58  [2015-09-18 06:50] upgraded emacs (24.5-1 -> 24.5-2)
57  [2015-09-18 06:50] upgraded python (3.4.3-2 -> 3.4.3-3)
56  [2015-09-18 06:50] upgraded guile (2.0.11-2 -> 2.0.11-3)
55  [2015-09-18 06:50] upgraded gdb (7.10-1 -> 7.10-3)
54  [2015-09-18 06:50] upgraded libgdm (3.16.2-1 -> 3.16.3-1)
53  [2015-09-18 06:50] upgraded gdm (3.16.2-1 -> 3.16.3-1)
52  [2015-09-18 06:50] upgraded gettext (0.19.5.1-1 -> 0.19.6-1)
51  [2015-09-18 06:51] upgraded ghc (7.10.1-1 -> 7.10.1-2)
50  [2015-09-18 06:51] upgraded hunspell (1.3.3-1 -> 1.3.3-2)
49  [2015-09-18 06:51] upgraded gnome-online-accounts (3.16.3-1 -> 3.16.4.1-1)
48  [2015-09-18 06:51] upgraded gnugo (3.8-4 -> 3.8-5)
47  [2015-09-18 06:51] upgraded gptfdisk (1.0.0-1 -> 1.0.0-2)
46  [2015-09-18 06:51] upgraded htop (1.0.3-2 -> 1.0.3-3)
45  [2015-09-18 06:51] upgraded inetutils (1.9.4-1 -> 1.9.4-2)
44  [2015-09-18 06:51] upgraded lame (3.99.5-2 -> 3.99.5-3)
43  [2015-09-18 06:51] upgraded libcaca (0.99.beta18-2 -> 0.99.beta18-3)
42  [2015-09-18 06:51] upgraded libcdio (0.93-1 -> 0.93-2)
41  [2015-09-18 06:51] upgraded libwbclient (4.3.0-1 -> 4.3.0-2)
40  [2015-09-18 06:51] upgraded mosh (1.2.5-1 -> 1.2.5-2)
39  [2015-09-18 06:51] upgraded python2 (2.7.10-1 -> 2.7.10-2)
38  [2015-09-18 06:51] upgraded smbclient (4.3.0-1 -> 4.3.0-2)
37  [2015-09-18 06:51] upgraded mplayer (37379-4 -> 37379-5)
36  [2015-09-18 06:51] upgraded nano (2.4.2-1 -> 2.4.2-2)
35  [2015-09-18 06:51] upgraded nodejs (4.0.0-1 -> 4.1.0-1)
34  [2015-09-18 06:51] upgraded ocaml (4.02.3-1 -> 4.02.3-2)
33  [2015-09-18 06:51] upgraded pinentry (0.9.5-1 -> 0.9.5-2)
32  [2015-09-18 06:51] upgraded procps-ng (3.3.11-1 -> 3.3.11-2)
31  [2015-09-18 06:51] upgraded vi (1:070224-1 -> 1:070224-2)
30  [2015-09-18 06:51] upgraded vim-runtime (7.4.854-1 -> 7.4.854-2)
29  [2015-09-18 06:51] upgraded vim (7.4.854-1 -> 7.4.854-2)
28  [2015-09-18 06:51] upgraded virtualbox-host-modules (5.0.2-1 -> 5.0.4-1)
27  [2015-09-18 06:51] upgraded virtualbox (5.0.2-1 -> 5.0.4-1)
26  [2015-09-18 06:51] upgraded virtualbox-guest-iso (5.0.2-1 -> 5.0.4-1)
25  [2015-09-18 06:51] upgraded vlc (2.2.1-6 -> 2.2.1-7)
24  [2015-09-18 06:51] upgraded wavemon (0.7.6-2 -> 0.7.6-3)
23  [2015-09-18 06:51] upgraded xterm (320-1 -> 320-2)
22  [2015-09-18 06:51] upgraded zsh (5.1.1-1 -> 5.1.1-2)
21  [2015-09-18 06:51] upgraded zsh-doc (5.1.1-1 -> 5.1.1-2)
20  [2015-09-18 06:53] upgraded google-chrome-beta (46.0.2490.22-1 -> 46.0.2490.33-1)
19  [2015-09-18 07:38] installed xemacs (21.5.33-6)
18  [2015-09-18 08:05] removed xemacs (21.5.33-6)
17  
16  [2015-09-19 20:27] upgraded readline (6.3.008-2 -> 6.3.008-3)
15  [2015-09-19 20:27] upgraded bash (4.3.042-2 -> 4.3.042-3)
14  [2015-09-19 20:27] upgraded cmake (3.3.1-3 -> 3.3.2-2)
13  [2015-09-19 20:27] upgraded dhclient (4.3.2-2 -> 4.3.3-2)
12  [2015-09-19 20:27] upgraded git (2.5.2-1 -> 2.5.3-1)
11  [2015-09-19 20:27] upgraded gtk-update-icon-cache (3.16.6-1 -> 3.16.7-1)
10  [2015-09-19 20:27] upgraded gtk3 (3.16.6-1 -> 3.16.7-1)
9   
8   [2015-09-21 07:50] upgraded cmake (3.3.2-2 -> 3.3.2-3)
7   [2015-09-21 07:50] upgraded libsystemd (225-1 -> 226-1)
6   [2015-09-21 07:50] upgraded libdbus (1.10.0-2 -> 1.10.0-3)
5   [2015-09-21 07:50] upgraded dbus (1.10.0-2 -> 1.10.0-3)
4   [2015-09-21 07:50] upgraded systemd (225-1 -> 226-1)
3   [2015-09-21 07:50] upgraded systemd-sysvcompat (225-1 -> 226-1)
2   
1   [2015-09-21 09:01] SYSTEM REBOOT (last)

This askubuntu question (http://askubuntu.com/questions/162229/h … -root-user) says that one might also need to edit some files in /etc/pam.d/ in order to get that work, however the files in question are Ubuntu-specific. But grepping the files in that directory seems to suggest that pam_limits.so should actually be required.

% grep pam_limits.so /etc/pam.d/*
/etc/pam.d/crond:session   required    pam_limits.so
/etc/pam.d/system-auth:session   required  pam_limits.so
/etc/pam.d/system-services:session   required    pam_limits.so

Can anyone confirm this problem? And how can I fix it?

tsdh · 2015-09-21 07:31:28

I can work around this issue by editing and uncommenting this entry in /etc/systemd/system.conf:

DefaultLimitNOFILE=500000

At least that allows me to do my work again but it is obviously not the right solution. I just want to change my own user's limits, not the limits of each and everything.

tsdh · 2015-09-21 08:36:06

Now I added the "debug" option to all the pam entries requiring pam_limits.so. The logging output is this:

Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): reading settings from '/etc/security/limits.conf'
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): process_limit: processing - rtprio 0 for DEFAULT
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): process_limit: processing - nice 0 for DEFAULT
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): process_limit: processing soft nofile 500000 for USER
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): process_limit: processing hard nofile 900000 for USER
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): reading settings from '/etc/security/limits.d/10-gcr.conf'
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group users
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): process_limit: processing - memlock 1024 for GROUP
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): reading settings from '/etc/security/limits.d/99-audio.conf'
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_limits(gdm-password:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p gdm-password][682]: pam_unix(gdm-password:session): session opened for user horn by (uid=0)
Sep 21 10:16:28 thinkpad-t440p systemd[1]: Created slice user-1000.slice.
Sep 21 10:16:28 thinkpad-t440p systemd[1]: Starting User Manager for UID 1000...
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): reading settings from '/etc/security/limits.conf'
Sep 21 10:16:28 thinkpad-t440p systemd-logind[343]: New session c2 of user horn.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): process_limit: processing - rtprio 0 for DEFAULT
Sep 21 10:16:28 thinkpad-t440p systemd[1]: Started Session c2 of user horn.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): process_limit: processing - nice 0 for DEFAULT
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): process_limit: processing soft nofile 500000 for USER
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): process_limit: processing hard nofile 900000 for USER
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): reading settings from '/etc/security/limits.d/10-gcr.conf'
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group users
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): process_limit: processing - memlock 1024 for GROUP
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): reading settings from '/etc/security/limits.d/99-audio.conf'
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_limits(systemd-user:session): checking if horn is in group audio
Sep 21 10:16:28 thinkpad-t440p systemd[687]: pam_unix(systemd-user:session): session opened for user horn by (uid=0)
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Listening on D-Bus User Message Bus Socket.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Reached target Sockets.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Reached target Timers.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Reached target Paths.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Reached target Basic System.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Reached target Default.
Sep 21 10:16:28 thinkpad-t440p systemd[687]: Startup finished in 12ms.

Well, this looks as if the configured nofile limits have actually been applied. So either pam is malfunctioning without indication or something else changes the limits again afterwards...

tsdh · 2015-09-25 19:43:44

Does really nobody else have this problem?

colorprint · 2015-09-29 14:38:34

It seems a known problem with systemd:
https://sskaje.me/systemd-ulimit/

tsdh · 2015-09-30 05:56:17

colorprint wrote:

It seems a known problem with systemd:
https://sskaje.me/systemd-ulimit/

I think this is a different thing. That post explains how to increase limits for some particular service, and that's the intended way to do it. For normal human users, /etc/security/limits.conf should still be applicable because PAM sets the limits. And as you can see from my last log, it actually does. But either it doesn't work or something else sets different limits again after PAM has done its thing.

tsdh · 2015-09-30 06:29:30

I've filed a bug report now in the Arch bug tracker: https://bugs.archlinux.org/task/46490

cmatheson · 2016-02-03 06:33:57

I'm having this same issue. Did you ever get anywhere with it?

espace.ariane · 2016-02-03 21:03:54

Same here. As mentioned the bug report loging through ssh allows the user to open as many files as specified by limits.conf, so I'm using local ssh login as a workaround for now. I'm using GDM with a gnome environment, are you too ?

oconnor663 · 2016-02-08 21:19:24

I had a similar problem at https://bbs.archlinux.org/viewtopic.php?pid=1602812 and ended up just using the system-wide setting. I tried /etc/systemd/user.conf and /etc/systemd/user.conf.d/*.conf, but it seems like those don't work if the value they set is higher than what's in system.conf.

audiomuze · 2016-03-16 07:15:29

I've an application that needs this set to 8192 for the user account that launches the app. Running ulimit -n 8192 returns a permissions error. Anyone know what additional user permissions the account needs I order to be able to change the value? I really don't want to have to launch the app as root.

audiomuze · 2016-03-17 21:43:50

Really, no-one has any idea?

WorMzy · 2016-03-17 23:05:50

Please don't 'bump'.

audiomuze · 2016-03-18 07:21:11

Here's more info:

Running this as a regular user I get:

ulimit -n

returns

ulimit -n 8192

returns

-bash: ulimit: open files: cannot modify limit: Operation not permitted

Running same as root it sets ulimit to 8192.

ulimit cannot be executed using sudo. I don't want to run the app as root, I've searched the forum and used my goofle fu and come up empty. Any ideas/tips?

Last edited by audiomuze (2016-03-18 07:24:28)

kokoko3k · 2016-03-18 10:30:47

try with: sudo sh -c "ulimit -n 8192"

Last edited by kokoko3k (2016-03-18 10:32:26)

audiomuze · 2016-03-18 20:46:03

kokoko3k wrote:

try with: sudo sh -c "ulimit -n 8192"

Thanks, but that didn't work:

$ ulimit -n
1024
[x@alibbtrfs ~]$ sudo sh -c "ulimit -n 8192"
[sudo] password for x: 
[x@alibbtrfs ~]$ ulimit -n
1024
[x@alibbtrfs ~]$

webdawg · 2016-03-26 00:47:22

Allright.

So there is this: https://bugs.archlinux.org/task/46490

Which is closed...which kinda sucks.

When I was looking into this I went as far as enabling<ref>http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/</ref>:

session required pam_limits.so

in: /etc/pam.d/login

Still nothing....

So then I read this from that bug report: logging in at the console or ssh *does* respect the limits set in limits.conf.

I sshed in to the box and the limits worked!

So, I do not use gdm or whatever the hell, this box opens up a VNC session on init (or systemd start). I had to edit the systemd unit file that starts my vnc session and add<ref>https://unix.stackexchange.com/questions/152186/mysql-max-open-files-more-than-1024</ref>:

LimitNOFILE=infinity
LimitMEMLOCK=infinity

I do not know if I need the MEMLOCK but who cares at this point. Restarted the service and now I have what I want.

If this still is not fixed with the gdm or whatever start it has to be the unit file or something that is limiting the proper setup in the gdm or window session.

You could just ssh into your own box.

audiomuze · 2016-03-26 20:38:29

I solved it by editing /etc/security/limits.conf and adding the following to the end of the file, substituting "username" with that you're using to run whatever software it is that needs that many files open concurrently:

@username           soft    nofile          8192
@username           hard    nofile          8192

Would be nice if the documentation was a bit clearer/ more accurate.

Last edited by audiomuze (2016-03-26 20:39:01)

Arch Linux

#1 2015-09-21 07:20:25

Cannot increase nofile limits (ulimit)

#2 2015-09-21 07:31:28

Re: Cannot increase nofile limits (ulimit)

#3 2015-09-21 08:36:06

Re: Cannot increase nofile limits (ulimit)

#4 2015-09-25 19:43:44

Re: Cannot increase nofile limits (ulimit)

#5 2015-09-29 14:38:34

Re: Cannot increase nofile limits (ulimit)

#6 2015-09-30 05:56:17

Re: Cannot increase nofile limits (ulimit)

#7 2015-09-30 06:29:30

Re: Cannot increase nofile limits (ulimit)

#8 2016-02-03 06:33:57

Re: Cannot increase nofile limits (ulimit)

#9 2016-02-03 21:03:54

Re: Cannot increase nofile limits (ulimit)

#10 2016-02-08 21:19:24

Re: Cannot increase nofile limits (ulimit)

#11 2016-03-16 07:15:29

Re: Cannot increase nofile limits (ulimit)

#12 2016-03-17 21:43:50

Re: Cannot increase nofile limits (ulimit)

#13 2016-03-17 23:05:50

Re: Cannot increase nofile limits (ulimit)

#14 2016-03-18 07:21:11

Re: Cannot increase nofile limits (ulimit)

#15 2016-03-18 10:30:47

Re: Cannot increase nofile limits (ulimit)

#16 2016-03-18 20:46:03

Re: Cannot increase nofile limits (ulimit)

#17 2016-03-26 00:47:22

Re: Cannot increase nofile limits (ulimit)

#18 2016-03-26 20:38:29

Re: Cannot increase nofile limits (ulimit)

Board footer