daemontools vs rc.d scripts

Ishpeck · 2012-05-19 18:45:33

I've been tinkering around with DJB's daemontools (http://cr.yp.to/daemontools.html) in Arch for a while. I've started pushing things out of my /etc/rc.conf file's DAEMONS line and am creating run scripts for them under the /service directory.

So far, things appear to be going quite well. Boot time appears faster (haven't taken metrics or anything) and those daemons with dependencies will just have a call to svstat in their run script to check to see if the required daemon is up. This obviates being picky or even conscious about when a daemon starts relative to any others. Services that do not depend on each other can start in parallel. Services with dependencies only need to wait for the daemons they rely upon.

This could make it easier to have daemons start at boot time without having to edit or parse any config files or whatever. A package would only need to include a /service/daemonname/run file in order to get the daemon going on next boot. Seems like a decent blend of Arch-style simplicity with some ease-of-use.

I'm now exploring the potential to supervise more than just daemons as per these:
http://code.dogmap.org/svscan-1/
http://thedjbway.b0llix.net/svscanboot.html

Y'all are far smarter than I am so I wonder what good ideas I'm missing when I think daemontools is a better way to go. Why do we favor keeping PID files and making scripts in rc.d over an alternative like supervise?

Trilby · 2012-05-19 18:59:42

Are you familiar with systemd? I'm only vaguely familiar with it, but it sounds like what you are describing, and it seems like it may become the default init system eventually.

Bellum · 2012-05-19 19:11:56

Wha? We wont be getting rid of the DAEMONS array, will we? I don't like that idea at all. I don't want services starting up without me knowing about it or configuring it. Just because I have openssh installed doesn't mean I want to run automatically run sshd, for instance.

Ishpeck · 2012-05-19 19:23:52

Bellum wrote:

Wha? We wont be getting rid of the DAEMONS array, will we? I don't like that idea at all. I don't want services starting up without me knowing about it or configuring it. Just because I have openssh installed doesn't mean I want to run automatically run sshd, for instance.

You could have the package put the run file somewhere else and then have pacman give instructions in how to symlink the run script in to start the daemon up at boot time.

You'd still have the advantage of no PID files, no parsing, and no editing and supervise.

Last edited by Ishpeck (2012-05-19 19:30:44)

Bellum · 2012-05-19 20:19:45

So instead of editing a single line in a config file, we'll have to ln scripts we want or rm scripts we don't? That doesn't sound simpler to me. Secondly, what if I want to start a daemon after boot for whatever reason? Will I still be able to do sudo /etc/rc.d/$script start or will I have to reboot?

Ishpeck · 2012-05-20 01:59:00

Bellum wrote:

So instead of editing a single line in a config file, we'll have to ln scripts we want or rm scripts we don't? That doesn't sound simpler to me. Secondly, what if I want to start a daemon after boot for whatever reason? Will I still be able to do sudo /etc/rc.d/$script start or will I have to reboot?

If you're installing the package (not building or maintaining it) you will have to add a symlink. I'm not sure how much harder that is supposed to be than editing a file. Once the symlink is there, svcscan will spot the daemon and start it automatically.

You'd still be able to start the daemon however you like. Just execute the run script or you could supervise it after the fashion of daemontools. You should never have to reboot.

If you're building the package, you have to maintain a script either way. The script found in /etc/rc.d can be quite hairy.

Consider:

[root@mako ~]# cat /etc/rc.d/crond 
#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions
. /etc/conf.d/crond

PID=$(pidof -o %PPID /usr/sbin/crond)
case $1 in
start)
	stat_busy "Starting Cron Daemon"

	# defaults to using syslog, and sendmail-ing cron output to local user
	# to mail output to remote address instead, add "-m user@host"
	# to CROND_ARGS in /etc/conf.d/crond
	if [[ -z $PID ]] && env -i PATH="/sbin:/usr/sbin:/bin:/usr/bin" /usr/sbin/crond $CROND_ARGS; then

		PID=$(pidof -o %PPID /usr/sbin/crond)
		echo "$PID" > /var/run/crond.pid
		add_daemon crond
		stat_done
	else
		stat_fail
		exit 1
	fi
	;;

stop)
	stat_busy "Stopping Cron Daemon"
	if [[ ! -z $PID ]]  && kill "$PID" &>/dev/null; then
		rm_daemon crond
		stat_done
	else
		stat_fail
		exit 1
	fi
	;;

restart)
	$0 stop
	$0 start
	;;

*)
	echo "Usage: $0 {start|stop|restart}" >&2
	exit 1

esac

I get the same functionality with the following:

[root@mako ~]# cat /service/crond/run
#!/bin/sh
/usr/local/bin/svstat /service/syslog-ng | grep 'up (pid' >/dev/null && (/usr/sbin/crond -f 2>&1 | logger -t crond -i) || (logger -t '/service/crond/run' -- 'Failure starting crond' && sleep 3)

I don't have to create the pid file, I don't have to write anything to handle startup or restart cases (svc does that for me). If I'm just symlinking that into /service then I don't have to know which daemons need to come first. That can be handled in the run script.

Last edited by Ishpeck (2012-05-20 02:00:21)

Bellum · 2012-05-20 14:14:31

Hum. Still looks like one step forward, two steps back to me. But I can take it or leave it.

Ishpeck · 2012-05-20 17:15:02

Bellum wrote:

Hum. Still looks like one step forward, two steps back to me. But I can take it or leave it.

I have conceded that Arch is incompatible with the idea of automatically starting daemons after reboots. It seems now that your dissent is entirely whether a symlink is better than editing rc.conf. While I'm assuming that's a matter of opinion, I certainly do want to know why editing rc.conf is superior.

My proposition is much larger in scope, however, than whether we start daemons at boot time or not. As explained here...

http://cr.yp.to/daemontools/faq/create.html#why

... a single software package obviates all the script volume in /etc/rc.d

This means far less rewriting the same old massive case/esac blocks for every daemon you'd care to build. Supervise will automatically restart crashed daemons -- giving you a more reliable service. Restarts of the daemon are less likely to ruin the daemon if relevant state is contained in supervise rather than in files scattered about the hard disk (and accessible to any other process run by the super user).

I argue that, when considering the whole system, using daemontools in lieu of the scripts found in /etc/rc.d as well as the DAEMONS line in /etc/rc.conf, where the process is not simpler, it is no more complex and that where there is a difference in simplicity, daemontools wins-out.

Zancarius · 2012-05-22 07:35:34

Disclaimer: I was a huge proponent of daemontools many years ago (circa 2002-2005(ish), possibly longer), but it fell out of my personal favor for a variety of reasons both at work and later when I migrated my home server away from FreeBSD to Gentoo. Thus, much of my current experience and knowledge related to daemontools is probably sorely outdated. Take it with a grain of salt, because these are just my own opinions and observations having used it both in the context of an ISP (mostly for e-mail services with qmail) and on a home file server.

First, I'll say that your proposal is interesting. You're right in that it's largely incompatible with the Arch Way, but I'm interested by your experiment to convert as many daemons as possible to supervise directories. Perhaps you ought to consider writing a utility script that manages daemontools automatically, or perhaps serves as a bridge between rc.conf and daemontools. If nothing else, it might be an interesting experience even if it's ultimately an experiment in frustration.

Second, I stopped using daemontools for many of the reasons--perhaps ironically--that supporters tout as its greatest benefits. I don't recall the precise version(s) that I used last, but I seem to recall that it was growing increasingly more frustrating to maintain essentially two vastly different init systems. Moreover, the "portability" benefit of daemontools (also cited on DJB's page) became laughable outside the context of supported platforms; sure, you can build daemontools on any POSIX-like system as long as the toolchain is sane, but on systems that you don't own or--for reasons beyond your control--you're not authorized to alter the system services or how they start up, the only "portability" that remains is to use the default rc.d/init.d system that ships with the platform in question. In a sense, it became easier for me to learn the oddities of disparate systems than to install daemontools on those I controlled and suffering through (re-)learning the init scripts on those I didn't. Add to that the requirement, at the time, to use special tools to convert the TAI64 timestamps to something human-readable (and there was, at the time, no easy way to simply grep through them without those tools--or at least one that I could remember consistently), and large logs with many entries became nightmarish. That was even with adding symlinks back to /var/log to try to maintain a sense of order, if I recall correctly! I suppose what I'm saying is that there's an advantage to keeping deployments of *nix-like systems mostly homogeneous without the addition of new and bizarre systems.

The advantage Arch has is that it shares a great deal of similarity with the BSDs in terms of its simplicity and elegance, even if there's some disagreement on whether a collection of scripts tucked away in /etc qualifies as "elegant." I'm reminded of Canonical's "upstart" initiative for some reason whenever I now think back to daemontools, though they share little in common.

The other thing that I vaguely recall causing me some grief but certainly not among the reasons that drove me away from daemontools was its behavior (again, at the time) with poorly disciplined daemons. I don't precisely remember which gave me the most trouble, but I recall that there was some mix of frustration when converting PostgreSQL, MySQL, and/or Apache. I'm sure daemontools has improved since then; unfortunately, I'm just afraid that some of my memories are less than stellar. For instance, killing misbehaving processes when supervise was unable (or unwilling) to do anything about it became extraordinarily painful if it failed to respond to svc -d; killing supervise then became necessary if all else were to fail, and since the entire premise of a supervise-like platform is to not have to kill the supervisory process, it defeated the purpose. Hence, the simplicity of using an rc.d/init.d system where you have ultimate control became a superior choice in my mind rather than delegating the authority of my services to black box software that fails rarely but spectacularly in a manner that's difficult to resolve. That isn't to say that a system based upon directories and/or symlinks is somehow more complicated than scripts (it isn't; daemontools, in terms of its administrative overhead, is fairly simplistic), but it does become a matter of whether or not you're willing to relinquish near-total control to a process that (ideally) never crashes and always behaves itself. I'd argue that perhaps the reason init scripts still largely behave the same way they have for decades is due in no small part to a significant amount of historical inertia--and for good reason.

As I mentioned, this is just a stream-of-consciousness collection of my memories related to daemontools. At the time, it worked very well for me when I was using qmail. There are reasons I would never consider using it outside curiosity today, and I suspect you'll find that it's philosophically counter to virtually all of the mainstream distributions in some manner. I think you'll also find a rather round-about answer to your question "why."

Ishpeck · 2012-05-23 03:21:41

Zancarius wrote:

Disclaimer: I was a huge proponent of daemontools many years ago (circa 2002-2005(ish), possibly longer), but it fell out of my personal favor for a variety of reasons both at work and later when I migrated my home server away from FreeBSD to Gentoo. Thus, much of my current experience and knowledge related to daemontools is probably sorely outdated.

It really hasn't changed a heck of a lot, in my experience. "Outdated" may not be a very useful word where DJB's software is concerned.

Zancarius wrote:

I think you'll also find a rather round-about answer to your question "why."

I think I've explained the "why" in previous posts.

Your input has been valuable -- and was precisely the sort of thing I was looking for. Your remarks on undisciplined daemons are especially worth considering. Since I tend to avoid Apache and the relational database servers, I have not yet explored them well enough to know whether that sort of thing will be problematic for me.

It is true that most daemons are designed with the assumption that they won't be supervised and may not perform all that well when they are.

Ishpeck · 2012-05-24 03:14:58

Zancarius wrote:

I'm sure daemontools has improved since then

According to the changelog, the daemontools of today is identical to the one you used. Last update was July 12th 2001.

Zancarius wrote:

For instance, killing misbehaving processes when supervise was unable (or unwilling) to do anything about it became extraordinarily painful if it failed to respond to svc -d; killing supervise then became necessary if all else were to fail, and since the entire premise of a supervise-like platform is to not have to kill the supervisory process, it defeated the purpose.

I wrote the following python script:

#!/usr/bin/python
import signal
from time import sleep

def handler(sig, frm):
    print "Neener neener."
    return 0

def main():
    signal.signal(signal.SIGABRT, handler)
    signal.signal(signal.SIGALRM, handler)
    signal.signal(signal.SIGBUS, handler)
    signal.signal(signal.SIGCHLD, handler)
    signal.signal(signal.SIGCLD, handler)
    signal.signal(signal.SIGCONT, handler)
    signal.signal(signal.SIGFPE, handler)
    signal.signal(signal.SIGHUP, handler)
    signal.signal(signal.SIGILL, handler)
    signal.signal(signal.SIGINT, handler)
    signal.signal(signal.SIGIO, handler)
    signal.signal(signal.SIGIOT, handler)
    signal.signal(signal.SIGPIPE, handler)
    signal.signal(signal.SIGPOLL, handler)
    signal.signal(signal.SIGPROF, handler)
    signal.signal(signal.SIGPWR, handler)
    signal.signal(signal.SIGQUIT, handler)
    signal.signal(signal.SIGRTMAX, handler)
    signal.signal(signal.SIGRTMIN, handler)
    signal.signal(signal.SIGSEGV, handler)
    signal.signal(signal.SIGSYS, handler)
    signal.signal(signal.SIGTERM, handler)
    signal.signal(signal.SIGTRAP, handler)
    signal.signal(signal.SIGTSTP, handler)
    signal.signal(signal.SIGTTIN, handler)
    signal.signal(signal.SIGTTOU, handler)
    signal.signal(signal.SIGURG, handler)
    signal.signal(signal.SIGUSR1, handler)
    signal.signal(signal.SIGUSR2, handler)
    signal.signal(signal.SIGVTALRM, handler)
    signal.signal(signal.SIGWINCH, handler)
    signal.signal(signal.SIGXCPU, handler)
    signal.signal(signal.SIGXFSZ, handler)
    signal.signal(signal.SIG_IGN, handler) 
    while True:
        sleep(1)

if __name__=='__main__':
    main()

And the following run script:

#!/bin/sh
exec python siggy.py

Then went:

[ishpeck@kiyoshi siggy]$ supervise ./ &
[1] 11606
[ishpeck@kiyoshi siggy]$ ps -efw | grep s[i]ggy
ishpeck  11607 11606  0 21:03 pts/10   00:00:00 python siggy.py
[ishpeck@kiyoshi siggy]$ svstat ./
./: up (pid 11607) 29 seconds
[ishpeck@kiyoshi siggy]$ svc -d ./
Neener neener.
Neener neener.
[ishpeck@kiyoshi siggy]$ svstat ./
./: up (pid 11607) 58 seconds, want down
[ishpeck@kiyoshi siggy]$ svc -k ./
[ishpeck@kiyoshi siggy]$ svstat ./
./: down 3 seconds, normally up
[ishpeck@kiyoshi siggy]$ ps -efw | grep s[i]ggy
[ishpeck@kiyoshi siggy]$ echo $?
1

It appears as though a misbehaving process can indeed be put down with an svc -k. I never had to kill supervise in order to maintain order in this case.

I personally have seen Apache and Postgres supervised without any horrors. My curiosity compels me to ask what about your setup may have caused problems when you tried it (tho' I know that I try to forget most of the horrors I experienced with Apache and MySQL stuff not working).

Zancarius wrote:

In a sense, it became easier for me to learn the oddities of disparate systems than to install daemontools on those I controlled and suffering through (re-)learning the init scripts on those I didn't.

This is a valid assertion but irrelevant to my proposal. I'm not advising that we install daemontools on every _other_ platform. I am asking why we haven't adopted daemontools for ourselves because it seems like a pretty lucid tool that emphasizes simplicity, robustness, and code correctness (tho' we might have to clarify what "modernity" means with respect to software that has "just worked" for 11 years).

Last edited by Ishpeck (2012-05-24 03:19:14)

el mariachi · 2012-05-25 18:12:57

maybe it's a dumb question, but which is newer daemontools or runit? they seem very similar..

Zancarius · 2012-05-25 21:31:38

Ishpeck wrote:

My curiosity compels me to ask what about your setup may have caused problems when you tried it (tho' I know that I try to forget most of the horrors I experienced with Apache and MySQL stuff not working).

Then you'd essentially be asking me to recover information from my own (fallible) memory that is on the order of magnitude of a decade old and likely dates back to FreeBSD 4.4-4.6. This would have also likely coincided with the time when PostgreSQL 7.x was very new. Thus, I cannot specifically remember what the circumstances were beyond knowing that there's a time trade-off with critical services; if the service absolutely must be up and running, debugging and further analysis take something of a backseat to getting the service running (failing the needs of forensics, if there were a break in). I also didn't have the advantage of experience as I was much younger.

Since it seems you're inclined to try experimenting with signals (though I'm not sure whether that precisely matches my definition of "misbehaving daemons"), it might be worth emulating frozen or hung processes by listening for SIGINT/SIGTERM with a noop or doing other naughty things that you otherwise wouldn't do in a sane configuration. Though, all of this is probably a pointless exercise that cannot match real world use. Without documentation on what I encountered and going exclusively from memory, I cannot offer any further insight. Again, take my 2¢ earlier (and here) with a grain of salt!

Ishpeck wrote:

I am asking why we haven't adopted daemontools for ourselves because it seems like a pretty lucid tool that emphasizes simplicity, robustness, and code correctness (tho' we might have to clarify what "modernity" means with respect to software that has "just worked" for 11 years).

I think it might be more useful to demonstrate "how" rather than asking "why." This could be accomplished, for instance, by modifying the rc scripts as you've already illustrated and by providing a bridge between Daemontools and rc.conf that would read the daemons array and modify the svc directory accordingly. You've already done quite a bit of work, and I'd imagine that it wouldn't take a great deal more effort to complete something that would boot and function entirely with Daemontools. Additionally, you could then distribute a PKGBUILD that replaces initscripts with augmented copies that use Daemontools in place of rc.d scripts for others to test. I suppose you could say that I'm essentially suggesting that it's better to do the legwork and have a demonstrably functional, tangible utility than to ask why we're not already reaping the benefits of ${software}. Ultimately, though, change for the sake of change even if it may yield superior results isn't always worthwhile, because there will be unexpected hangups, incompatibilities, or outright weirdness. I point to the TAI64 timestamps and logging as just one example (possibly resolvable with this?).

I'm reminded that there were, once upon a time, historical reasons that played against DJB's software, namely that the original licenses under which qmail and (I assume) Daemontools were distributed may have introduced incompatibilities with the GPL and were not OSI-certified. I believe they've since been released under the public domain. Nevertheless, that most assuredly has had some measurable impact on their adoption early on, which is probably why no major distributions use it as part of their default init system as far as I know. There may be some systems out there that do.

It seems that you're trying to make a case to replace the current rc.d system for startup services. I would highly recommend reading this thread (FreeBSD mailing list); someone else has been working hard to argue the case for a Daemontools-replacement/-like system for FreeBSD and most likely for similar reasons. Reading the full conversation might give you a few more ideas, most especially with regards to the arguments against it and allow you to formulate an even better response to the critics.

Edit: There's also this article, which might aid in convincing skeptics like myself. And there's a few alternatives with examples that are worth examining in addition to or instead of daemontools.

Last edited by Zancarius (2012-05-25 21:38:31)

Ishpeck · 2012-05-26 02:36:26

Zancarius wrote:

Since it seems you're inclined to try experimenting with signals (though I'm not sure whether that precisely matches my definition of "misbehaving daemons"), it might be worth emulating frozen or hung processes by listening for SIGINT/SIGTERM with a noop or doing other naughty things that you otherwise wouldn't do in a sane configuration.

The Python script above does listen for both such signals. It handles them and continues running.

This "other naughty things" is certainly worth exploring. What naughty things do you suggest?

Zancarius wrote:

I think it might be more useful to demonstrate "how" rather than asking "why."

I just wanted to know if we already knew a glaring reason why it was a bad idea so I didn't waste my time and others trying to figure this out. Maybe it's something I'll have to learn the hard way.

Zancarius wrote:

Additionally, you could then distribute a PKGBUILD that replaces initscripts with augmented copies that use Daemontools in place of rc.d scripts for others to test.

This is certainly a good idea and part of my plan.

Zancarius wrote:

Ultimately, though, change for the sake of change even if it may yield superior results isn't always worthwhile...

I'm confused. If it yields superior results, then wouldn't the change be for the sake of those results?

Zancarius wrote:

....because there will be unexpected hangups, incompatibilities, or outright weirdness.

I do anticipate having to solve weird problem. S'part of the fun. I was also asking if such weirdness or hangups are already known to be irreconcilable. Thus, the original post.

The links are certainly worth looking at. Thanks for your feedback.

Last edited by Ishpeck (2012-05-26 02:37:39)

Ishpeck · 2012-05-26 14:31:41

el mariachi wrote:

maybe it's a dumb question, but which is newer daemontools or runit? they seem very similar..

runit is younger. And seems pretty good, too. I've never used it but it's worth looking into.

el mariachi · 2012-05-26 14:39:48

it seems one can use runit with inittab, thus not having to change a lot of stuff. I'm mainly interested in having a faster boot with parallel services. systemd is übercomplicated.
If there was a way of keeping the DAEMONS line to control what services are to be started, while using runit/daemontools' (DT) parallelization capabilities that would be cool.
Did you have a hard time setting up DT?

Ishpeck · 2012-06-01 02:58:41

el mariachi wrote:

it seems one can use runit with inittab, thus not having to change a lot of stuff.

You can use daemontools with inittab, too. The point is not minimal change but system-wide consistency.

el mariachi wrote:

Did you have a hard time setting up DT?

Not at all. The instructions on DJB's web site were brief and adequate to the task.

tomegun · 2012-06-01 11:08:17

Bellum wrote:

So instead of editing a single line in a config file, we'll have to ln scripts we want or rm scripts we don't? That doesn't sound simpler to me. Secondly, what if I want to start a daemon after boot for whatever reason? Will I still be able to do sudo /etc/rc.d/$script start or will I have to reboot?

Just to clear up some basic things.

Services will not be enabled by default with systemd, just like how it is now with initscripts.

To enable a systemd daemon you do "systemctl enable <your daemon>". This will create the correct symlink, but that's not something you need to care about.

To start a daemon you do "systemctl start <your daemon>".

Edit: nevermind, I got confused about who you were replying to, I see this was not related to systemd. igonre me :-)

Last edited by tomegun (2012-06-01 11:10:21)

hobarrera · 2012-06-21 06:31:44

Ishpeck wrote:

[..]Supervise will automatically restart crashed daemons -- giving you a more reliable service. Restarts of the daemon are less likely to ruin the daemon if relevant state is contained in supervise rather than in files scattered about the hard disk (and accessible to any other process run by the super user).

daemons shouldn't crash in the first place. If one crashes, then an administrator needs to look into it carefully, not just restart and ignore the issue.

Ishpeck · 2012-06-26 16:09:49

hobarrera wrote:

daemons shouldn't crash in the first place. If one crashes, then an administrator needs to look into it carefully, not just restart and ignore the issue.

It's true. Software should never have problems. Ever.

For a well-built but not perfect daemon, it's conceivable to have extreme, rare cases where crashes happen. Should the crash make the entire service go away for all the normal, non-crash-inducing uses? Or should the one extreme case get reported by your standard heartbeat monitoring to the admin and let everyone else continue using the service normally?

#!/bin/sh
echo "Notice sent from /service/foo/run" | mail -s "service started at `date`" root
exec daemon-in-question

Problem solved.

Last edited by Ishpeck (2012-06-26 16:24:03)

Arch Linux

#1 2012-05-19 18:45:33

daemontools vs rc.d scripts

#2 2012-05-19 18:59:42

Re: daemontools vs rc.d scripts

#3 2012-05-19 19:11:56

Re: daemontools vs rc.d scripts

#4 2012-05-19 19:23:52

Re: daemontools vs rc.d scripts

#5 2012-05-19 20:19:45

Re: daemontools vs rc.d scripts

#6 2012-05-20 01:59:00

Re: daemontools vs rc.d scripts

#7 2012-05-20 14:14:31

Re: daemontools vs rc.d scripts

#8 2012-05-20 17:15:02

Re: daemontools vs rc.d scripts

#9 2012-05-22 07:35:34

Re: daemontools vs rc.d scripts

#10 2012-05-23 03:21:41

Re: daemontools vs rc.d scripts

#11 2012-05-24 03:14:58

Re: daemontools vs rc.d scripts

#12 2012-05-25 18:12:57

Re: daemontools vs rc.d scripts

#13 2012-05-25 21:31:38

Re: daemontools vs rc.d scripts

#14 2012-05-26 02:36:26

Re: daemontools vs rc.d scripts

#15 2012-05-26 14:31:41

Re: daemontools vs rc.d scripts

#16 2012-05-26 14:39:48

Re: daemontools vs rc.d scripts

#17 2012-06-01 02:58:41

Re: daemontools vs rc.d scripts

#18 2012-06-01 11:08:17

Re: daemontools vs rc.d scripts

#19 2012-06-21 06:31:44

Re: daemontools vs rc.d scripts

#20 2012-06-26 16:09:49

Re: daemontools vs rc.d scripts

Board footer