pacman -U sometimes hangs

jernst · 2014-08-05 19:11:48

I have a custom package created with makepkg and installed with pacman -U

Sometimes, when re-installing the package, pacman -U hangs.

ps shows pacmac -U being the parent process of a "[bash] <defunct>" process. This defunct process is apparently what's left-over from the post_upgrade invocation of the 'install' script in my package. However, my script finishes -- I put echo statements at the end of all defined subroutines, and they print. The last printout is "post_upgrade done".

Some further strangeness:
* Re-installing the same package with the same version and content also sometimes works without hanging. I might invoke the same pacman -U three times in a row, and it hangs once and finishes twices.
* I have not been able to correlate this with anything else going on

Also, reading up on what a <defunct> process is, it seems that it must be pacman somehow not noticing that my install script is done? Is there a way for me to help it figure that out?

Any help with debugging this would be greatly appreciated.

Addition: just for kicks, I added "exit 0" to the post_upgrade subroutine. No change.

Last edited by jernst (2014-08-05 19:53:17)

falconindy · 2014-08-05 19:15:24

strace will tell you where it's hanging.

jasonwryan · 2014-08-05 19:16:20

Please post your PKGBUILD and .install script.

You can also run pacman with the -debug option for more verbose ouptut.

jernst · 2014-08-05 19:26:30

Here's the trailing end of strace. It seems (disclaimer: I'm no strace expert) that my install script is read first (5) and then copied into some tmp dir as .INSTALL (6)

The "indiebox-admin post_upgrade 1638" is my echo at the end of post_upgrade.

The last few lines seem to repeat. (This is terminal-related and doesn't matter here)

access("/var/lib/pacman/local/indiebox-admin-0.57-1/install", R_OK) = 0
open("/var/lib/pacman/local/indiebox-admin-0.57-1/install", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=1942, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa7a7fd6000
read(5, "#\n# Installation scripts for ind"..., 4096) = 1942
close(5) = 0
munmap(0x7fa7a7fd6000, 4096) = 0
access("/tmp/", F_OK) = 0
mkdir("/tmp/alpm_0pwGwK", 0700) = 0
open("/var/lib/pacman/local/indiebox-admin-0.57-1/install", O_RDONLY) = 5
open("/tmp/alpm_0pwGwK/.INSTALL", O_WRONLY|O_CREAT, 0) = 6
fstat(5, {st_mode=S_IFREG|0644, st_size=1942, ...}) = 0
fchmod(6, 0100644) = 0
read(5, "#\n# Installation scripts for ind"..., 8192) = 1942
write(6, "#\n# Installation scripts for ind"..., 1942) = 1942
read(5, "", 8192) = 0
close(5) = 0
close(6) = 0
open(".", O_RDONLY) = 5
chdir("/") = 0
pipe([6, 7]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fa7a7fa2a10) = 16382
close(7) = 0
fcntl(6, F_GETFL) = 0 (flags O_RDONLY)
fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa7a7fd6000
lseek(6, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
read(6, "indiebox-admin post_upgrade 1638"..., 4096) = 34
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
write(3, "[2014-08-05 19:18] [ALPM-SCRIPTL"..., 70) = 70
write(1, "indiebox-admin post_upgrade 1638"..., 34indiebox-admin post_upgrade 16382
) = 34
read(6, "indiebox-admin post_upgrade done"..., 4096) = 39
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=16382, si_status=0, si_utime=0, si_stime=0} ---
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
write(3, "[2014-08-05 19:18] [ALPM-SCRIPTL"..., 75) = 75
write(1, "indiebox-admin post_upgrade done"..., 39indiebox-admin post_upgrade done 16382
) = 39
read(6, 0x7fa7a7fd6000, 4096) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
read(6, 0x7fa7a7fd6000, 4096) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
read(6, 0x7fa7a7fd6000, 4096) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
read(6,

Last edited by jernst (2014-08-05 19:45:50)

jernst · 2014-08-05 19:27:03

Here's the trailing end of pacman -U --debug:

debug: updating database
debug: adding database entry 'indiebox-admin'
debug: writing indiebox-admin-0.57-1 DESC information back to db
debug: writing indiebox-admin-0.57-1 FILES information back to db
debug: adding entry 'indiebox-admin' in 'local' cache
debug: executing ". /tmp/alpm_El6R0y/.INSTALL; post_upgrade 0.57-1 0.57-1"
debug: executing "/usr/bin/bash" under chroot "/"
indiebox-admin post_upgrade 25474
indiebox-admin post_upgrade done 25474

jernst · 2014-08-05 19:28:50

Here's the package: https://github.com/indiebox/indiebox-ad … ebox-admin
(minus the echo statements at the end of the methods in install)

falconindy · 2014-08-05 20:23:48

If I had to guess, I'd say something perl-ish is hanging (you can pass -f to strace to make it trace child processes). Better question -- what does that perl post_install stuff do that can't be done in the PKGBUILD itself?

Last edited by falconindy (2014-08-05 20:24:06)

jernst · 2014-08-05 21:33:24

@falconindy: even if we assume that the Perl'ish thing does something awful, the thread of control does return to the bash subroutine -- we know because it prints the echo statement after the perl invocation in the install bash script. Given this, I fail to see how anything in the Perl could stop pacman from continuing.

Also note that there aren't any further child processes in ps, nor is there any that has perl in it.

falconindy · 2014-08-05 23:07:26

jernst wrote:

Also note that there aren't any further child processes in ps, nor is there any that has perl in it.

That's rather strange, since the read that pacman is stuck on occurs before reaping the child process. I'd expect it to be a zombie at this point. What does the kernel show for /proc/$pid/fdinfo/6? Can you find the other end of the pipe via the inode listed for /proc/$pid/fd/6?

jernst · 2014-08-06 16:44:00

I decided this whole thing is too strange, and likely cost a lot of time to figure out. So I'm refactoring, putting a lot of code from the install script into a systemd script, thereby bypassing the problem. (@falconindy: Thank you!)

falconindy · 2014-08-06 17:19:29

Oh well. There's clearly bugs in the install scriptlet handling which I really would have liked to have fleshed out here. We've seen them manifest as SIGPIPE errors and also as the hangs you're seeing.

apg · 2014-08-06 19:17:32

Before you give up, can you at least tell me what version of pacman you're running?

EDIT: Looking at the perl script you were running, I suspect that one of the processes you were running in the background was hanging around keeping the pipe open.

Last edited by apg (2014-08-06 20:31:22)

jernst · 2014-08-07 18:43:14

I didn't realize that this thread might help shed some light on some other issues; didn't want to waste people's time answering my questions if I could circumvent the problem on my end. Hang on ...

jernst · 2014-08-07 18:47:31

apg wrote:

Before you give up, can you at least tell me what version of pacman you're running?

Pacman v4.1.2 - libalpm v8.0.2

apg wrote:

EDIT: Looking at the perl script you were running, I suspect that one of the processes you were running in the background was hanging around keeping the pipe open.

What pipe would that be? From stdout of the install script and whatever processes it started to pacman? If so, should I try redirecting all of their output to /dev/null?

apg · 2014-08-07 19:12:30

jernst wrote:

apg wrote:
EDIT: Looking at the perl script you were running, I suspect that one of the processes you were running in the background was hanging around keeping the pipe open.
What pipe would that be? From stdout of the install script and whatever processes it started to pacman? If so, should I try redirecting all of their output to /dev/null?

Yes, that pipe, but it includes both stdout and stderr from the script. I think redirecting stdout and stderr would probably solve the problem, although I think it would be better to run commands in the foreground and make sure they exit properly.

jernst · 2014-08-07 22:10:43

falconindy wrote:

jernst wrote:
Also note that there aren't any further child processes in ps, nor is there any that has perl in it.
That's rather strange, since the read that pacman is stuck on occurs before reaping the child process. I'd expect it to be a zombie at this point. What does the kernel show for /proc/$pid/fdinfo/6? Can you find the other end of the pipe via the inode listed for /proc/$pid/fd/6?

> sudo cat /proc/25028/fdinfo/6
pos:    0
flags:  00
mnt_id: 8

I don't quite know what that means, but perhaps you do :-)

jernst · 2014-08-07 22:29:14

Progress. In the hanging version:

pacman -U indiebox-admin
# which calls install
# which calls some perl
# which calls bash
# which calls
systemctl start mysqld &

and finally hangs, it turns out that

systemctl start mysqld &

apparently also hangs. It should be a brief operation, but it keeps hanging around the process table. I had missed that earlier.

If the invocation is

systemctl start mysqld > /tmp/foo 2>&1 &

instead, it does not seem to hang. It will create a zero-byte file. If I only redirect one of stdout and stderr, it still hangs.

It seems to me that pacman is waiting for systemctl, e.g. because systemctl still has a hold on the pipe that pacman reads from (does that sound right?), but systemctl does not seem to have to say anything (zero byte file). But what is systemctl waiting for? I redirected stdin from /dev/null, but no change in behavior.

falconindy · 2014-08-15 00:15:38

If you want a non-blocking systemctl call, then you should use --no-block, not background the process!

Arch Linux

#1 2014-08-05 19:11:48

pacman -U sometimes hangs

#2 2014-08-05 19:15:24

Re: pacman -U sometimes hangs

#3 2014-08-05 19:16:20

Re: pacman -U sometimes hangs

#4 2014-08-05 19:26:30

Re: pacman -U sometimes hangs

#5 2014-08-05 19:27:03

Re: pacman -U sometimes hangs

#6 2014-08-05 19:28:50

Re: pacman -U sometimes hangs

#7 2014-08-05 20:23:48

Re: pacman -U sometimes hangs

#8 2014-08-05 21:33:24

Re: pacman -U sometimes hangs

#9 2014-08-05 23:07:26

Re: pacman -U sometimes hangs

#10 2014-08-06 16:44:00

Re: pacman -U sometimes hangs

#11 2014-08-06 17:19:29

Re: pacman -U sometimes hangs

#12 2014-08-06 19:17:32

Re: pacman -U sometimes hangs

#13 2014-08-07 18:43:14

Re: pacman -U sometimes hangs

#14 2014-08-07 18:47:31

Re: pacman -U sometimes hangs

#15 2014-08-07 19:12:30

Re: pacman -U sometimes hangs

#16 2014-08-07 22:10:43

Re: pacman -U sometimes hangs

#17 2014-08-07 22:29:14

Re: pacman -U sometimes hangs

#18 2014-08-15 00:15:38

Re: pacman -U sometimes hangs

Board footer