pacman -Sy(yu) is broken. "too many open files"

tehkane · 2011-08-26 07:25:50

[root@myhost ~]# pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'core.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'core.db' from ftp.iinet.net.au : Too many open files
error: failed retrieving file 'core.db' from mirror.internode.on.net : Too many open files
error: failed retrieving file 'core.db' from mirror.internode.on.net : Too many open files
error: failed to update core (Too many open files)
error: failed retrieving file 'extra.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'extra.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'extra.db' from ftp.iinet.net.au : Too many open files
error: failed retrieving file 'extra.db' from mirror.internode.on.net : Too many open files
error: failed retrieving file 'extra.db' from mirror.internode.on.net : Too many open files
error: failed to update extra (Too many open files)
error: failed retrieving file 'community.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'community.db' from mirror.aarnet.edu.au : Too many open files
error: failed retrieving file 'community.db' from ftp.iinet.net.au : Too many open files
error: failed retrieving file 'community.db' from mirror.internode.on.net : Too many open files
error: failed retrieving file 'community.db' from mirror.internode.on.net : Too many open files
error: failed to update community (Too many open files)
error: failed to synchronize any databases

[root@myhost ~]# pacman -Q pacman
pacman 3.5.3-1

[root@myhost ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 7937
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2048
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7937
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The closest I've come to fixing this was by doing:

ulimit -n 2048
pacman -Syy

Which results in pacman hanging on "Synchronizing package databases...". I let it set there for ~10-15 minutes before finally killing it.

If it makes any difference, the box doesn't have a direct connection to the internet. I'm using `http_proxy`. Which I've tested with `wget`. Works fine.

Any ideas?

serh · 2011-08-26 17:38:30

Did you try switching the mirror to a global one?

tehkane · 2011-08-29 00:42:41

Yes, I enabled the mirrors.kernel.org mirror and the same thing happens.

I should point out, I've also rebooted a few times and the error still occurs. Which seems odd, to me, that I can have "too many open files" after a reboot

lilsirecho · 2011-08-29 00:44:45

try running ...pacman-db-upgrade...and again when changing mirrors run....pacman -Syy.....

No spaces in the db command.........EDIT:

Last edited by lilsirecho (2011-08-29 00:45:42)

toofishes · 2011-08-29 01:30:48

If you can get it to hang, can you get lsof output from the pacman process for us? Find the correct pid using ps once it is hung, then something like `lsof -p <pid>`.

falconindy · 2011-08-29 01:35:00

I'd be curious to see a stack trace of where all these files are being opened... `strace -eopen,close pacman -Syy'

tehkane · 2011-08-29 03:19:39

try running ...pacman-db-upgrade...and again when changing mirrors run....pacman -Syy.....

`pacman-db-upgrade` ran without error but the same problem is occurring when I try to `pacman -Syy`

If you can get it to hang, can you get lsof output from the pacman process for us? Find the correct pid using ps once it is hung, then something like `lsof -p <pid>`.

I'd be curious to see a stack trace of where all these files are being opened... `strace -eopen,close pacman -Syy'

I don't have `lsof` or `strace` installed and pacman isn't really available to install them. I'll see if I can grab a .tar.gz of them and compile them manually.

falconindy · 2011-08-29 03:26:29

tehkane wrote:

I don't have `lsof` or `strace` installed and pacman isn't really available to install them. I'll see if I can grab a .tar.gz of them and compile them manually.

You'd be better served to download the package manually and install them. pacman -U shouldn't pose the same problem here.

Assuming 64 bit...
http://mirrors.rit.edu/archlinux/extra/ … pkg.tar.xz
http://mirrors.rit.edu/archlinux/extra/ … pkg.tar.xz

tehkane · 2011-08-29 09:22:55

The problem seems to have been resolved. The only change made was removing my `http_proxy` setting and setting up a direct internet connection. pacman is now updating flawlessly. Seems strange that this would be the cause of the problem

Kinda wish I had strace'd and lsof'd before I fixed the issue... Though, hopefully, no one else encounters this problem.

bernarcher · 2011-08-29 11:58:51

tehkane,
weird, I wouldn't have thought of that cause.
But now that it works please mark this thread as [SOLVED] by editing the title of your first post.

toofishes · 2011-08-29 13:26:26

This isn't solved at all. Can you set up the http_proxy again now that you can have strace and lsof installed, and not mark a problem solved that you've worked around and not really fixed? We'd appreciate it.

@bernarcher: I appreciate your forum mod instincts kicking in, but when two pacman developers are looking for answers, we don't like our reproducible test case to disappear on us... We've both suspected it was http_proxy so this isn't unexpected whatsoever.

bernarcher · 2011-08-29 21:13:07

@toofishes
This was only moderating as some kind of side effect (because tehkane claimed it resolved).

In fact the bernarcher person tries to understand what happens. It sonded weird on first thought. But I did rather skim the topic.
On second thought, however, you are certainly right. It remains somewhat mysterious (to me at least). I still remain curious.

Keep up your research. I am waiting.

tehkane · 2011-09-05 04:58:13

Sorry for the late reply.

This isn't solved at all. Can you set up the http_proxy again now that you can have strace and lsof installed, and not mark a problem solved that you've worked around and not really fixed? We'd appreciate it.

I'll see if I can do this later today and post back the output.

neena · 2012-01-28 10:58:06

Hello

I'm having the same problem, and, I can only use the internet via the proxy, so, I can't really use the workaround either.
Here's the output of strace -eopen,close: http://pastebin.com/GW6xgXSs
The /etc/services file seems to be getting close'd twice somewhere, and that's making open increment the fd the next time?

falconindy · 2012-01-28 13:19:29

I wouldn't expect that this is affecting the latest version of pacman -- you're using (presumably) 3.5.x

Last edited by falconindy (2012-01-28 13:21:11)

neena · 2012-02-21 22:36:08

Yes, I was using 3.5.something, it worked fine after I installed with the latest.

I'm sorry, I had put off for later and then completely forgotten about it.
I suppose this thread could be marked as solved now.

Arch Linux

#1 2011-08-26 07:25:50

pacman -Sy(yu) is broken. "too many open files"

#2 2011-08-26 17:38:30

Re: pacman -Sy(yu) is broken. "too many open files"

#3 2011-08-29 00:42:41

Re: pacman -Sy(yu) is broken. "too many open files"

#4 2011-08-29 00:44:45

Re: pacman -Sy(yu) is broken. "too many open files"

#5 2011-08-29 01:30:48

Re: pacman -Sy(yu) is broken. "too many open files"

#6 2011-08-29 01:35:00

Re: pacman -Sy(yu) is broken. "too many open files"

#7 2011-08-29 03:19:39

Re: pacman -Sy(yu) is broken. "too many open files"

#8 2011-08-29 03:26:29

Re: pacman -Sy(yu) is broken. "too many open files"

#9 2011-08-29 09:22:55

Re: pacman -Sy(yu) is broken. "too many open files"

#10 2011-08-29 11:58:51

Re: pacman -Sy(yu) is broken. "too many open files"

#11 2011-08-29 13:26:26

Re: pacman -Sy(yu) is broken. "too many open files"

#12 2011-08-29 21:13:07

Re: pacman -Sy(yu) is broken. "too many open files"

#13 2011-09-05 04:58:13

Re: pacman -Sy(yu) is broken. "too many open files"

#14 2012-01-28 10:58:06

Re: pacman -Sy(yu) is broken. "too many open files"

#15 2012-01-28 13:19:29

Re: pacman -Sy(yu) is broken. "too many open files"

#16 2012-02-21 22:36:08

Re: pacman -Sy(yu) is broken. "too many open files"

Board footer