Arch Linux

eomanis

Member

Registered: 2013-04-17

Posts: 50

systemd instantiated service with tmux: Can only start 1 instance

Edit: Solution
Edit: Solution did not work after all, changed requirements to get rid of tmux.

I have a home server running Arch on which I am running a few Minecraft servers.

Up until now I have been running those Minecraft servers in command line windows, which is a nuisance when I reboot the server, and to smoothen out the experience I want to run said Minecraft servers as systemd service instances.
Currently the Minecraft servers are running as their own user "minecraft", this being done by entering a sudo interactive shell as user "minecraft" and running them in there.

Minecraft servers have command line interaction. One can type in commands such as "save-all" or "stop" or "say XYZ" to save the map, shut down the server or chat to the world console, respectively.
I wish that this feature will continue to be available to me when the Minecraft servers run as systemd service instances, and because of this I want to run the servers in tmux sessions, to which I can attach using tmux as I please.

I have everything prepared and working when I run this stuff manually (e.g. start the Minecraft servers in tmux sessions as user "minecraft" and then attach to them using tmux, also as user "minecraft").

Now I need a systemd instantiated service unit file, which I also already figured out, using the example from the rtorrent ArchWiki site:
File "/etc/systemd/system/minecraft-server@.service":

[Unit]
Description=Minecraft server %I
Requires=network.target local-fs.target
After=network.target local-fs.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
KillMode=none
User=minecraft
ExecStart=/usr/bin/tmux new-session -d -c /home/minecraft/%I -s minecraft-server-%I -n minecraft-server-%I "java -Xmx2048M -jar minecraft_server.jar nogui"
ExecStop=/usr/bin/tmux send-keys -t minecraft-server-%I s t o p C-m
WorkingDirectory=/home/minecraft/%I
Restart=on-failure

I can start a single instance of this service and it runs like a charm, I can attach to the tmux session, and when I stop the service instance I can see in the tmux session the interactive server command "stop" being written, and the server shuts down gracefully and everything looks great.

However this only works for a single instance. When already an instance is running, and I want to start one of the other instances as well, starting that other instance fails.

The good thing here is that a Minecraft server takes about 2-3 seconds to get going, and during this time I am able to watch in the attached tmux session what is happening.
What is happening is that I see the line "stop" being written as the very first thing that is visible in the tmux session. Then I see the server start, and instantly after startup it recognizes the "stop" command, and shuts down again.
Therefore I conclude that systemd performs the "ExecStop=" command immediately after the service instance has been started.

I am at a loss to see why it would do this.
Running the "ExecStart=" command manually in a "minecraft" user terminal, while the single successful systemd servce instance is running, works, and it does not return a bad exit value ( -d is "detach", hence the "Type=forking").
Using "Restart=no" does not change this behavior either.

Also, "systemctl status minecraft-server@instance.service" does not show anything, and neither does "journalctl -u minecraft-server@instance.service".

Does anyone have an idea what this could be about?

Last edited by eomanis (2015-03-07 17:43:12)

eomanis

Member

Registered: 2013-04-17

Posts: 50

Re: systemd instantiated service with tmux: Can only start 1 instance

I have now an example systemd instantiated service unit file that I used to replicate this issue with on another Arch system. Instead of the Minecraft server this example uses "cat -n", which just echoes everything you type in back at you while prepending line numbers:
File "/etc/systemd/system/tmux-test@.service":

[Unit]
Description=systemd tmux test %I
Requires=network.target local-fs.target
After=network.target local-fs.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
KillMode=none
User=YOUR_USER
ExecStart=/usr/bin/tmux new-session -d -c /home/YOUR_USER -s tmux-test-%I -n tmux-test-%I "cat -n"
ExecStop=/usr/bin/tmux send-keys -t tmux-test-%I s t o p Enter ^c
WorkingDirectory=/home/YOUR_USER
Restart=on-failure

Just replace YOUR_USER with your user name (assuming your home directory is at /home/YOUR_USER).

Get a root terminal window.
You can now start an instance "one" of this service:

# systemctl start tmux-test@one

Then, open a new regular terminal window and attach to this tmux session:

$ tmux attach -t tmux-test-one

You can type there and it will be echoed back at you, prepended with line numbers.
You can then attempt to start another instance "two":

# systemctl start tmux-test@two

Now, attaching to this second instance's tmux session fails, because the tmux session "tmux-test-two" is not running

Since starting the second service instance does not work, get another regular terminal window and attempt to run the "ExecStart=" command for the second tmux session manually:

$ /usr/bin/tmux new-session -d -c /home/YOUR_USER -s tmux-test-two -n tmux-test-two "cat -n"

The command will spawn a new tmux session in the background and return immediately, and it returns with an exit value of 0.
You can now attach to this new non-systemd-managed tmux session (while the first successful service instance "tmux-test-one" is still running):

$ tmux attach -t tmux-test-two

*confusion intensifies*

I have attempted to coax out of systemd what is actually happening by setting LogLevel=debug in /etc/systemd/system.conf, to no avail. It does not appear to be willing to disclose its internal reasoning.

Can anybody replicate this issue using the steps above?

Last edited by eomanis (2015-03-04 20:45:03)

eomanis

Member

Registered: 2013-04-17

Posts: 50

Re: systemd instantiated service with tmux: Can only start 1 instance

According to this posting this is something related to tmux not playing well with Type=forking.

Guess I'll educate myself about the tmux concepts of sessions and windows and then try again.

nullified

Member

From: Massachusetts, USA

Registered: 2013-12-09

Posts: 468

Re: systemd instantiated service with tmux: Can only start 1 instance

I was able to replicate the problem. Changing the Type to oneshot, adding RemainAfterExit=yes, and removing the Restart option seems to produce the desired result.

[Unit]
Description=systemd tmux test %I
Requires=network.target local-fs.target
After=network.target local-fs.target

[Install]
WantedBy=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
KillMode=none
User=YOUR_USER
ExecStart=/usr/bin/tmux new-session -d -c /home/YOUR_USER -s tmux-test-%I -n tmux-test-%I "cat -n"
ExecStop=/usr/bin/tmux send-keys -t tmux-test-%I s t o p Enter ^c
WorkingDirectory=/home/YOUR_USER

---

"We may say most aptly, that the Analytical Engine weaves algebraical patterns just as the Jacquard-loom weaves flowers and leaves." - Ada Lovelace

eomanis

Member

Registered: 2013-04-17

Posts: 50

Re: systemd instantiated service with tmux: Can only start 1 instance

I was able to replicate the problem. Changing the Type to oneshot, adding RemainAfterExit=yes, and removing the Restart option seems to produce the desired result.

This.
This did the trick. I was messing around with "Type=oneshot" and it did not work. With "RemainAfterExit=yes" things are working as expected now.

Thank you a ton, I don't know how long I would have been fiddling until I would have arrived at "RemainAfterExit=yes".
Presumably, without "RemainAfterExit=yes", systemd eventually kills off child processes of "Type=oneshot" services.

To sum it up, here is the now working Minecraft server systemd instantiated service unit file (the "tmux send-keys" command has been tweaked a bit to have an exact "session:window" target):

File "/etc/systemd/system/minecraft-server@.service":

[Unit]
Description=Minecraft server %I
Requires=network.target local-fs.target
After=network.target local-fs.target

[Install]
WantedBy=multi-user.target

[Service]
Type=oneshot
KillMode=none
User=minecraft
ExecStart=/usr/bin/tmux new-session -d -c /home/minecraft/%I -s minecraft-server-%I -n minecraft-server-%I "java -Xmx2048M -jar minecraft_server.jar nogui"
ExecStop=/usr/bin/tmux send-keys -t minecraft-server-%I:minecraft-server-%I s t o p Enter
WorkingDirectory=/home/minecraft/%I
RemainAfterExit=yes

I will mark this thread as [SOLVED].
nullified, you're da real MVP

eomanis

Member

Registered: 2013-04-17

Posts: 50

Re: systemd instantiated service with tmux: Can only start 1 instance

So, it turns out, after an actual reboot with the Minecraft server service instances enabled, there are now other problems.

While the Minecraft servers are running inside tmux sessions that have been spawned from PID 1, neither can I attach to the tmux sessions, nor has stopping the service instances using systemd any effect.
For "tmux ls" as user "minecraft",  no sessions are listed, despite the Minecraft servers and their tmux processes being there in the process listings.

It seems that tmux does not find its sessions socket.
Issuing a "killall -10 tmux" seemed to make one of the Minecraft servers attacheable (this signal is supposed to cause tmx to re-setup its sockets or something like that).
Attaching to the others, or stopping any of them, still did not work.

As far automatic system service management goes this is not good enough.
Therefore I have decided to cut back on requirements - no more live service terminal interaction - to get rid of this tmux complexity that I can obviously not handle.

The service unit file now has "Type=simple", no more tmux, and the Minecraft servers will have to deal with being stopped using SIGTERM:

File "/etc/systemd/system/minecraft-server@.service":

[Unit]
Description=Minecraft server for root "%I"
Requires=network.target local-fs.target
After=network.target local-fs.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=minecraft
ExecStart=/usr/bin/java -Xmx2048M -jar minecraft_server.jar nogui
WorkingDirectory=%I

(The instance is now the full path to the Minecraft server root directory instead of a directory in "/home/minecraft".)

I'll just mentally note down for me, "systemd service + tmux = bad idea".
On the plus side, now systemd is able to capture and log the Minecraft servers' command line output into its journal, which was not possible when using tmux.

progandy

Member

Registered: 2012-05-17

Posts: 5,190

Re: systemd instantiated service with tmux: Can only start 1 instance

If you want to add remote control capability, I suggest a small c application that intercepts stdio for the server and makes it available with a unix socket, something like a screen-lite with foobard/foobarctl. Keep the service-type as simple, then add an ExecStop command that sends the proper shutdown commands to the server.

Last edited by progandy (2015-03-07 18:17:53)

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

tinyprog

Member

Registered: 2015-03-12

Posts: 1

Re: systemd instantiated service with tmux: Can only start 1 instance

I noticed that when I start a detached tmux session as a foreground service, the service exits, staying in a failed state.
So, I changed the service type to "forking".

Here are my files. User is craftbukkit, home is /srv/craftbukkit.

/usr/lib/systemd/system/spigot.service

[Unit]
Description=Spigot Minecraft Server
After=local-fs.target network.target

[Service]
Type=forking
User=craftbukkit
WorkingDirectory=/srv/craftbukkit
ExecStart=/srv/craftbukkit/spigot start
ExecStop=/srv/craftbukkit/spigot stop
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

/srv/craftbukkit/spigot

#!/bin/bash
case $1 in
	start)
		tmux new -d -s spigot 'java -Xmx1024m -Xms1024m -XX:MaxPermSize=128m -jar spigot.jar nogui'
	;;
	stop)
		tmux send -t spigot 'broadcast NOTICE: server restarting in 5s!' C-m
		sleep 5
		tmux send -t spigot stop C-m
	;;
	*)
		echo usage $0 {start|stop}
esac
exit 0

Enable and start the service:

# systemctl daemon-reload
# systemctl enable spigot
# systemctl start spigot

Then I can get a management console with:

$ sudo -u craftbukkit tmux attach-session -t spigot

That's it.

Last edited by tinyprog (2015-03-12 14:08:30)