RabbitMQ won't stop gracefully

r-darwish · 2017-09-30 13:36:40

I installed RabbitMQ with

pacman -Sy rabbitmq

and enabled it using

sudo systemctl enable --now rabbitmq

However, RabbitMQ seems to refuse shutting down gracefully, whether due to a system reboot or by using systemctl stop. After 90 seconds systemd decides to kill it. I didn't alter any configuration, nor do I know RabbitMQ enough to diagnose it. What am I doing wrong?

Last edited by r-darwish (2017-09-30 13:37:02)

WorMzy · 2017-09-30 14:10:01

For a start, you are doing partial updates. Update your system completely and see if you still have trouble. Post your journal if you do.

https://wiki.archlinux.org/index.php/Sy … nsupported

Mod note: moving to NC.

r-darwish · 2017-10-01 06:24:25

The problem persists even when the system is fully updated

ledbettj · 2017-10-02 13:01:49

I am also seeing this issue. It looks like rabbitmq believes it's shut down successfully, but the erlang VM never stops. I don't have much of a clue on how to debug further into the erlang VM. (PS "newbie corner" doesn't quite seem like the right place for this anymore )

The offending process that hangs around is:

rabbitmq 11820  0.0  0.0   7700    92 ?        S    09:06   0:00 /usr/lib/erlang/erts-9.0.1/bin/epmd -daemon

journalctl -u rabbitmq -r shows:

Sep 29 16:42:03 oberon systemd[1]: rabbitmq.service: Failed with result 'timeout'.
Sep 29 16:42:03 oberon systemd[1]: rabbitmq.service: Unit entered failed state.
Sep 29 16:42:03 oberon systemd[1]: Stopped RabbitMQ Messaging Server.
Sep 29 16:42:03 oberon systemd[1]: rabbitmq.service: Killing process 29693 (epmd) with signal SIGKILL.
Sep 29 16:42:03 oberon systemd[1]: rabbitmq.service: State 'stop-final-sigterm' timed out. Killing.
Sep 29 16:40:33 oberon rabbitmq[29371]: Stopping and halting node rabbit@oberon
Sep 29 16:40:33 oberon systemd[1]: Stopping RabbitMQ Messaging Server...
Sep 29 08:45:31 oberon systemd[1]: Started RabbitMQ Messaging Server.

/var/log/rabbitmq/rabbit@oberon.log shows:

=INFO REPORT==== 29-Sep-2017::16:40:33 ===
RabbitMQ is asked to stop...
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping RabbitMQ applications and their dependencies in the following order:
rabbitmq_management
amqp_client
rabbitmq_web_dispatch
cowboy
cowlib
rabbitmq_management_agent
rabbit
mnesia
rabbit_common
os_mon
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'rabbitmq_management'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'amqp_client'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'rabbitmq_web_dispatch'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'cowboy'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'cowlib'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'rabbitmq_management_agent'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'rabbit'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
stopped TCP Listener on 0.0.0.0:5672
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@oberon/msg_store_persistent'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@oberon/msg_store_persistent' is stopped
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@oberon/msg_store_transient'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@oberon/msg_store_transient' is stopped
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'mnesia'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'rabbit_common'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Stopping application 'os_mon'
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Successfully stopped RabbitMQ and its dependencies
=INFO REPORT==== 29-Sep-2017::16:40:33 ===
Halting Erlang VM with the following applications:
ranch
ssl
public_key
asn1
compiler
inets
crypto
xmerl
syntax_tools
sasl
stdlib
kernel

Last edited by ledbettj (2017-10-02 13:07:36)

ledbettj · 2017-10-02 13:13:18

I think I actually managed to figure this out.

epmd needs to be started for erlang processes, and if it's not running, rabbitmq will start it. However, it will not shut down correctly later when rabbit shuts down.

I was able to achieve successful shutdown by first starting the epmd service, and then starting rabbit:

# systemctl start epmd
# systemctl start rabbitmq
# systemctl stop rabbitmq

I suspect that rabbitmq.service should maybe require empd.service to be started? I was able to configure it thusly:

john@oberon:~$ cat /etc/systemd/system/rabbitmq.service.d/override.conf 
[Unit]
Requires=epmd.service
After=epmd.service

john@oberon:~$ cat /etc/systemd/system/epmd.service.d/override.conf 
[Unit]
Before=rabbitmq.service

john@oberon:~$ systemctl enable rabbitmq
john@oberon:~$ systemctl enable epmd

I am not a unit file expert but this seems to have the desired effect.

Last edited by ledbettj (2017-10-02 13:23:19)

r-darwish · 2017-10-02 13:49:57

That did it. Thanks!

I'm not sure that we need to alter epmd. I placed your override file just for rabbitmq.service and did not enable empd manually, and now it works as expected.

Perhaps the RabbitMQ package maintainer should be notified about this issue?

ledbettj · 2017-10-02 14:55:45

Looks like you're right, the change to the epmd.service file is not needed.

I opened a issue with the rabbitmq package.

Arch Linux

#1 2017-09-30 13:36:40

RabbitMQ won't stop gracefully

#2 2017-09-30 14:10:01

Re: RabbitMQ won't stop gracefully

#3 2017-10-01 06:24:25

Re: RabbitMQ won't stop gracefully

#4 2017-10-02 13:01:49

Re: RabbitMQ won't stop gracefully

#5 2017-10-02 13:13:18

Re: RabbitMQ won't stop gracefully

#6 2017-10-02 13:49:57

Re: RabbitMQ won't stop gracefully

#7 2017-10-02 14:55:45

Re: RabbitMQ won't stop gracefully

Board footer