Migrating Arch's initscripts to dash

djclue917 · 2007-11-30 03:39:36

I recently "discovered" dash and how fast it is as compared to bash. FYI, dash is "a POSIX-compliant shell that is much smaller than bash". That being said, here are some simple benchmarks which compare the speed of bash and dash using a for loop (btw, I'm ran the tests below in bash since 'time' is a bash built-in):

(test.sh w/ output redirection)

for x in `seq 50000`; do
        echo "" > /dev/null
done

results for bash:

$ time bash test.sh
real 0m2.143s
user 0m1.740s
sys 0m0.413s
$ time bash test.sh
real 0m2.180s
user 0m1.730s
sys 0m0.450s
$ time bash test.sh
real 0m2.189s
user 0m1.717s
sys 0m0.460s

ave: 2.171 seconds

results for dash:

$ time dash test.sh
real 0m0.741s
user 0m0.350s
sys 0m0.403s
$ time dash test.sh
real 0m0.729s
user 0m0.413s
sys 0m0.327s
$ time dash test.sh
real 0m0.732s
user 0m0.360s
sys 0m0.373s

ave: 0.734 seconds

* In this test, dash is 2.96 times faster than bash!

(test.sh w/o output redirection)

for x in `seq 50000`; do
        echo -n ""
done

results for bash:

$ time bash test.sh
real 0m0.845s
user 0m0.833s
sys 0m0.010s
$ time bash test.sh
real 0m0.867s
user 0m0.863s
sys 0m0.007s
$ time bash test.sh
real 0m0.871s
user 0m0.870s
sys 0m0.003s

ave: 0.861 seconds

results for dash:

$ time dash test.sh
real 0m0.167s
user 0m0.177s
sys 0m0.003s
$ time dash test.sh
real 0m0.165s
user 0m0.153s
sys 0m0.020s
$ time dash test.sh
real 0m0.174s
user 0m0.167s
sys 0m0.013s

ave: 0.169 seconds

* In this test, dash is 5.10 times faster than bash! Astonishing!

Now think about this... How much will Arch's boot speed increase if we use dash instead of bash for our initscripts? Of course, there are some caveats like some "bash-specific" or "bash-oriented" (non-POSIX-compliant) shell scripts that won't work in dash.

I don't know if there are already existing efforts to make this happen so I'm bringing this up now. Tell me what you think.

BTW, I also found a discussion about dash and bash in the ubuntu-devel mailing list.

Last edited by djclue917 (2007-11-30 03:40:40)

PJ · 2007-11-30 07:34:16

djclue917 wrote:

I don't know if there are already existing efforts to make this happen so I'm bringing this up now. Tell me what you think.

From reading the arch-dev-public mailing list it seems like this is going to happened.
http://archlinux.org/pipermail/arch-dev … 03392.html

It seems like some of the scripts have been migrated too. It doesn't seem like any daemons have been migrated yet.
http://archlinux.org/pipermail/arch-dev … 03430.html

Looking at does numbers it seems like a great idea.

Last edited by PJ (2007-11-30 07:35:36)

iphitus · 2007-11-30 09:13:41

PJ wrote:

djclue917 wrote:
I don't know if there are already existing efforts to make this happen so I'm bringing this up now. Tell me what you think.
From reading the arch-dev-public mailing list it seems like this is going to happened.
http://archlinux.org/pipermail/arch-dev … 03392.html
It seems like some of the scripts have been migrated too. It doesn't seem like any daemons have been migrated yet.
http://archlinux.org/pipermail/arch-dev … 03430.html
Looking at does numbers it seems like a great idea.

Not really, it's just making dash available as an 'unbreakable' useful fallback shell.

There's no plans to make initscripts work with any other shell. I don't see the need, initscripts don't do loops with 50000 iterations. As always though, if it retains compatibility with bash, doesnt complicate the scripts, and has some benchable benefit... patches welcome.

Last edited by iphitus (2007-11-30 09:14:25)

djclue917 · 2007-11-30 10:03:06

Actually, the 50,000 iterations are just a way of exaggerating the speed difference between dash and bash. I've just tried running the configure script of Linphone 2.0.0 in dash and bash and still, the speed difference is quite obvious and very significant. ~22 seconds for dash and ~28 seconds for bash.

dash is a POSIX-compliant shell and I don't think making the scripts compatible with dash will be a real trouble. BTW, the configure sript I mentioned above didn't need any modifications for it to run in dash.

Problems will arise with scripts using "[[ ]]" and [ -a $FILE ] instead of [ -e $FILE ].

PJ · 2007-11-30 10:09:09

iphitus wrote:

Not really, it's just making dash available as an 'unbreakable' useful fallback shell. There's no plans to make initscripts work with any other shell.

I looked at my second posted url and yes, I made an incorrect assumption that the initscripts where in one of those package. I am at work so it is not like I could use pacman to check which packages that provides initscripts. I used the web based search when I suspected that I had made a wrong assumption and it turned out that I done that. Sorry, my intention wasn't to spread some incorrect rumors.

VikM · 2007-11-30 10:53:32

Here http://archlinux.org/pipermail/arch-dev … 03392.html are some good reasons, but if you have scripts using arrays (or other bash specific goodies) it is not funny at all to try them on dash or any other bsd /bin/sh. Maybe bash is not POSIX-compliant as the bsd sh is, but things evolve, it has nice features. After all there is nothing you can do in bash you can't do in (da)sh, but cleaner/simpler code in bash may be more important than -0.0004 seconds to boot time.
Dash is a good experience if you have to write portable scripts, otherwise I'm not sure...

pelle.k · 2007-11-30 15:00:15

Ubuntu has sh symlinked to dash, and that can be quite confusing if you aren't prepared for it. I don't really see a good reason to change the target of /bin/sh so that it doesn't work like it used to all of a sudden.

shining · 2007-11-30 15:34:29

pelle.k wrote:

Ubuntu has sh symlinked to dash, and that can be quite confusing if you aren't prepared for it. I don't really see a good reason to change the target of /bin/sh so that it doesn't work like it used to all of a sudden.

Scripts using bash specific features should use bash, that's all.
That should make things less confusing, as it would be easier to see which scripts require bash and which one don't.

Cerebral · 2007-11-30 16:02:44

pelle.k wrote:

Ubuntu has sh symlinked to dash, and that can be quite confusing if you aren't prepared for it. I don't really see a good reason to change the target of /bin/sh so that it doesn't work like it used to all of a sudden.

Actually, the confusing behaviour is that /bin/sh supports bash-specific stuff. sh by itself should only be expected to support sh scripts, and dash is a fine candidate for that.

toofishes · 2007-11-30 16:16:36

Note that dash is now a package in core (although it is idling in testing right now) so that we do have an unbreakable fallback shell on our system. Not only is it faster, but the ML links above outline some of its benefits.

I'm not sure if we will ever fully convert our initscripts due to the usage of bash arrays in rc.conf, but it would be ideal to do so. However, backwards compatibility becomes quite tough, especially with something as touchy as initscripts which if broken can really hurt your system, which is something we want to be careful not to do. The single largest holdup in our boot process is probably load-modules.sh, which udev calls repeatedly during its initialization- if you watch your bootup, you'll note that the rest is quite fast while udev tends to grind the system to a halt.

Enough said here- if you really want to discuss this, bring it to the ML where devs normally respond to and read your comments.

lloeki · 2008-02-28 11:22:11

before bothering people notably devs, whose time is certainly scarce) on the mailing list, I need some background.

vaguely off topic:
is there a difference in bash behavior whenever it is called as /bin/bash or /bin/sh ?
if yes, is there a bash switch that could enable this behavior without resorting to calling it as /bin/sh?

more on the topic:
could initscripts (and anything else in Arch dependent on bash features) at least use #!/bin/bash instead of #!/bin/sh ?

PS: simply for reference, failure-case workarounds, and bashisms and their sh match as recensed by Ubuntu, in a condensed and commented manner.

EDIT: nevermind, I found this in the 'man bash' page:

.
If bash is invoked with the name sh, it tries to mimic the startup
behavior of historical versions of sh as closely as possible, while
conforming to the POSIX standard as well. When invoked as an interac‐
tive login shell, or a non-interactive shell with the --login option,
it first attempts to read and execute commands from /etc/profile and
~/.profile, in that order. The --noprofile option may be used to
inhibit this behavior. When invoked as an interactive shell with the
name sh, bash looks for the variable ENV, expands its value if it is
defined, and uses the expanded value as the name of a file to read and
execute. Since a shell invoked as sh does not attempt to read and exe‐
cute commands from any other startup files, the --rcfile option has no
effect. A non-interactive shell invoked with the name sh does not
attempt to read any other startup files. When invoked as sh, bash
enters posix mode after the startup files are read.
When bash is started in posix mode, as with the --posix command line
option, it follows the POSIX standard for startup files. In this mode,
interactive shells expand the ENV variable and commands are read and
executed from the file whose name is the expanded value. No other
startup files are read.

so maybe #!/bin/bash --posix would do.

Last edited by lloeki (2008-02-28 11:38:34)

chicha · 2008-02-28 11:53:14

Bash is supposed to work in POSIX sh compliant mode when invoked as pure sh (#!/bin/sh).
But it also handle pure bash code in "sh mode", where Dash will report an error.

This is the point here : bash do not considere bashism as sh errors. People who write /bin/sh scripts using bash to interpret them may believe their scripts are POSIX sh compliant but they may not in reality as long as they use bash only syntax and utilities inside their scripts !

So you are totally right : Devs should replace #!/bin/sh by #!/bin/bash in their init scripts because they are not sh compliant.

I think Dash is great when you have to write pure sh script and need a hight level of portability.
In the case of Arch : do we need such portability ? I do not think so !

So why use Dash in our init scripts ? One reason invoked is speed and low memory usage. It would be interesting to see if using Dash is really faster than using bash and how much we loose using sh instead of bash. In any case I think the migration cost would be quiet heavy, so Dash had better really rock to make our devs move to it

LTSmash · 2008-03-06 00:09:44

Anybody remembers the speedup when changing from Ubuntu Dapper to Edgy? It was because they changed bash for dash, with a high noticeable speed burst.

I'm willing to hel the porting of the scripts if the devs say they wish to change to dash.

Dusty · 2008-03-06 00:20:14

I'm guessing that if you port it they will use it, but discuss with them regarding these sticky issues (arrays, etc). Even if you can get a rudimentary work in progress working that they can test it would help.

Dusty

cactus · 2008-03-06 01:20:51

LTSmash wrote:

Anybody remembers the speedup when changing from Ubuntu Dapper to Edgy? It was because they changed bash for dash, with a high noticeable speed burst.

I would need some hard data to believe that.
More than just the shell for the initscripts changed with the dapper to edgy upgrade.

LTSmash · 2008-03-06 02:34:34

cactus wrote:

LTSmash wrote:
Anybody remembers the speedup when changing from Ubuntu Dapper to Edgy? It was because they changed bash for dash, with a high noticeable speed burst.
I would need some hard data to believe that.
More than just the shell for the initscripts changed with the dapper to edgy upgrade.

https://wiki.ubuntu.com/DashAsBinSh

gorn · 2008-03-06 04:24:27

Out of curiosity (Okay I'm procrastinating doing something else) I did a quick port of /lib/udev/load-modules.sh from bash to dash. This is a known bottleneck for boot time because it's called several times and is a bit slow.

The big problem is that /etc/rc.conf uses arrays and dash doesn't support them. In the case of load-modules.sh we don't need true arrays but just space-separated values. In a complete move to dash, /etc/rc.conf would be updated to reflect this. But to not break things I just use eval/grep/tr to parse rc.conf. Which is probably not very efficient. (And could lead to bugs)

I tested both with a module I actual use, and a blacklisted module. I saw speed improvements with dash in both cases. Because my testing was not very scientific, I'm posting my code instead of numbers:

load-module.dash

#! /bin/dash
# Implement blacklisting for udev-loaded modules
#   Includes module checking
# - Aaron Griffin & Tobias Powalowski for Archlinux

# /etc/rc.conf is only parsable by bash. Fake it for dash.
#. /etc/rc.conf

eval `grep MOD /etc/rc.conf | tr '()' '""'`

[ $# -ne 1 ] && exit 1

if [ -f /proc/cmdline ]; then
    for cmd in $(cat /proc/cmdline); do
            case $cmd in
                *=*) eval $cmd ;;
            esac
    done
fi

# blacklist framebuffer modules
for x in $(echo /lib/modules/$(uname -r)/kernel/drivers/video/*/*fb*); do
    BLACKLIST="$BLACKLIST $(basename $x .ko)"
done
for x in $(echo /lib/modules/$(uname -r)/kernel/drivers/video/*fb*); do
    BLACKLIST="$BLACKLIST $(basename $x .ko)"
done
# get the real names from modaliases
i="$(/sbin/modprobe -i --show-depends $1 | sed "s#^insmod /lib.*/\(.*\)\.ko.*#\1#g" | sed 's|-|_|g')"
# get blacklistet modules
k="$(echo $BLACKLIST ${MOD_BLACKLIST} | sed 's|-|_|g')"
j="$(echo $MODULES | sed 's|-|_|g')"
#add disabled MODULES (!) to blacklist - much requested feature
for m in ${j}; do
    [ "$m" != "${m#!}" ] && k="${k} ${m#!}"
done
# add disablemodules= from commandline to blacklist
k="${k} $(echo ${disablemodules} | sed 's|-|_|g' | sed 's|,| |g')"
# echo "k = [[ $k ]]" # Confirm that module parsing is working
# echo "m = [[ $m ]]" # Confirm that module parsing is working

if [ "$MOD_AUTOLOAD" = "yes" -o "$MOD_AUTOLOAD" = "YES" ] && [ "$load_modules" != "off" ]; then
    if [ "${k}" != "" ]; then
            for n in ${i}; do
                if echo ${k} | /bin/grep "\<$n\>" 2>&1 >/dev/null; then
                    /usr/bin/logger -p info \
                    "udev load-modules: $i is blacklisted"
                    exit 1
                fi
            done
    fi
    /sbin/modprobe $1
else
    /usr/bin/logger -p info \
        "udev load-modules: autoloading is disabled, not loading $i"
fi
# vim: set et ts=4:

testdash

#!/bin/dash

[ $# -ne 2 ] && exit 1

test_shell=$1
test_mod=$2

modprobe -r $test_mod
for i in `seq 1 100`; do
    if [ $test_shell = "dash" ]; then
        /bin/dash ./load-modules.dash $test_mod
    elif [ $test_shell = "sh" ]; then
        /bin/sh ./load-modules.sh $test_mod
    elif [ $test_shell = "bash" ]; then
        /bin/bash ./load-modules.sh $test_mod
    elif [ $test_shell = "shdash" ]; then
        /bin/sh ./load-modules.dash $test_mod
    elif [ $test_shell = "bashdash" ]; then
        /bin/bash ./load-modules.dash $test_mod
    fi
    modprobe -r slhc
done

Invoke with

time ./testdash dash slhc

Or

time ./testdash sh slhc

You also can test /bin/bash versus /bin/sh (Which i assume is a symlink to bash, but bash handles it differently) or you can test bash or sh with my dash script (Which is slower due to the eval/grep/tr hack)

Someone running bootchart before and after might be interesting.

toofishes · 2008-03-06 06:29:12

The fact that i got zero responses to this thread really killed my enthusiasm for this kind of change. I would still love for my system to boot with /bin/sh linked to dash instead of bash though, which should be easily doable. The only real changes would be to ensure any script requiring bash-specific features such as arrays used a #!/bin/bash line instead of #!/bin/sh.

Last edited by toofishes (2008-03-06 06:29:51)

lloeki · 2008-03-06 06:58:27

I would still love for my system to boot with /bin/sh linked to dash instead of bash

toofish, this is exactly what I'd like to see in the very near future (ability to boot with another sh than bash), my main gripe being that scripts relying on bashisms are started with /bin/sh at the very core of arch. this just seems wrong to me.

conversion of initscripts to be sh compliant is not mandadory, but is beneficial if it's faster, and is anyway a second step, at best.
I was not so much worried about performance loss starting them with bash as such rather than the sh mode of bash, than a difference of behavior that would break the scripts. it seems there is none, and as such, changing #!/bin/sh to #!/bin/bash in initscripts is both easy (if not trivial) and fixes the thing.

what would be interesting too is bash package not setting the symlink. my idea would be to have a bash package, and a bash-sh package with the symlink, the latter provides=('sh'), and having the same for dash.

lloeki · 2008-03-06 07:04:54

$ pacman -Ql initscripts
initscripts /etc/
initscripts /etc/conf.d/
initscripts /etc/inittab
initscripts /etc/rc.conf
initscripts /etc/rc.d/
initscripts /etc/rc.d/functions
initscripts /etc/rc.d/functions.d/
initscripts /etc/rc.d/netfs
initscripts /etc/rc.d/network
initscripts /etc/rc.local
initscripts /etc/rc.local.shutdown
initscripts /etc/rc.multi
initscripts /etc/rc.shutdown
initscripts /etc/rc.single
initscripts /etc/rc.sysinit
initscripts /sbin/
initscripts /sbin/makedevs
initscripts /sbin/minilogd

$ sudo grep -R '#!/bin/sh' /etc/*
/etc/NetworkManager/dispatcher.d/netfs:#!/bin/sh
/etc/NetworkManager/dispatcher.d/ntpdate:#!/bin/sh
/etc/X11/xinit/xinitrc:#!/bin/sh
/etc/acpi/handler.sh:#!/bin/sh
/etc/avahi/avahi-dnsconfd.action:#!/bin/sh
/etc/avahi/avahi-autoipd.action:#!/bin/sh
/etc/cron.daily/logrotate:#!/bin/sh
/etc/cron.daily/shadow:#!/bin/sh
/etc/cron.daily/whatis:#!/bin/sh
/etc/cron.daily/updatedb:#!/bin/sh
grep: /etc/fonts/conf.d/20-lohit-gujarati.conf: No such file or directory
grep: /etc/fonts/conf.d/40-generic.conf: No such file or directory
grep: /etc/fonts/conf.d/30-amt-aliases.conf: No such file or directory
/etc/ifplugd/ifplugd.action.old:#!/bin/sh
/etc/ifplugd/ifplugd.action:#!/bin/sh
/etc/ppp/ip-down:#!/bin/sh
/etc/ppp/ip-up:#!/bin/sh
/etc/rc.d/bluetooth:#!/bin/sh
/etc/security/namespace.init:#!/bin/sh -p
/etc/skel/.xinitrc:#!/bin/sh
/etc/skel/.xsession:#!/bin/sh
/etc/ssl/misc/c_info:#!/bin/sh
/etc/ssl/misc/CA.sh:#!/bin/sh
/etc/ssl/misc/c_hash:#!/bin/sh
/etc/ssl/misc/c_name:#!/bin/sh
/etc/ssl/misc/c_issuer:#!/bin/sh

it seems tare using #!/bin/bash already... nevermind.

it's in fact udev, not initscripts which refers to sh instead of bash...

Last edited by lloeki (2008-03-06 07:07:16)

lloeki · 2008-03-06 07:42:49

So, I decided to go for it.

$ sed -i 's:#!/bin/sh:#!/bin/bash:' /lib/udev/load-modules.sh
$ rm /bin/sh && ln -s /bin/dash /bin/sh
$ shutdown -r now
and... booted successfully.

the only thing seemingly wrong is that NetworkManager seems to complain:

/etc/rc.conf: 34: Syntax error: "(" unexpected

yet there's a #!/bin/bash in /etc/rc.d/networkmanager.
what's more it starts successfully.

lloeki · 2008-03-06 09:21:51

WARNING:

shift: 10: can't shift that many

this is what may happen if one uses pacman with /bin/sh as /bin/dash. end result: part of the system is b0rked.

gorn · 2008-03-06 15:41:57

lloeki wrote:

So, I decided to go for it.
$ sed -i 's:#!/bin/sh:#!/bin/bash:' /lib/udev/load-modules.sh
$ rm /bin/sh && ln -s /bin/dash /bin/sh
$ shutdown -r now
and... booted successfully.
the only thing seemingly wrong is that NetworkManager seems to complain:
/etc/rc.conf: 34: Syntax error: "(" unexpected
yet there's a #!/bin/bash in /etc/rc.d/networkmanager.
what's more it starts successfully.

Use my load-modules.dash posted a few up. That will handle the ()s in rc.conf.

edit: oh sorry didn't read your post clear enough. So it looks like some other script is loading /etc/rc.conf from /bin/sh.

Maybe rc.d/wlan ?

Last edited by gorn (2008-03-06 15:52:47)

Misfit138 · 2008-03-06 16:17:14

toofishes wrote:

The fact that i got zero responses to this thread really killed my enthusiasm for this kind of change. I would still love for my system to boot with /bin/sh linked to dash instead of bash though, which should be easily doable. The only real changes would be to ensure any script requiring bash-specific features such as arrays used a #!/bin/bash line instead of #!/bin/sh.

toofishes, I would like to say, do not let your enthusiasm diminish for such a project! Such an innovation will enhance the overall Arch experience for all users; the interest in this thread is a testimony to that. Since "patches welcome" is a perpetual theme among Arch development, I for one would like to encourage you to go ahead with your initial desire and enthusiasm to update to dash, regardless of whether other developers have responded or not.

ibendiben · 2008-03-06 23:15:50

+1

Arch Linux

#1 2007-11-30 03:39:36

Migrating Arch's initscripts to dash

#2 2007-11-30 07:34:16

Re: Migrating Arch's initscripts to dash

#3 2007-11-30 09:13:41

Re: Migrating Arch's initscripts to dash

#4 2007-11-30 10:03:06

Re: Migrating Arch's initscripts to dash

#5 2007-11-30 10:09:09

Re: Migrating Arch's initscripts to dash

#6 2007-11-30 10:53:32

Re: Migrating Arch's initscripts to dash

#7 2007-11-30 15:00:15

Re: Migrating Arch's initscripts to dash

#8 2007-11-30 15:34:29

Re: Migrating Arch's initscripts to dash

#9 2007-11-30 16:02:44

Re: Migrating Arch's initscripts to dash

#10 2007-11-30 16:16:36

Re: Migrating Arch's initscripts to dash

#11 2008-02-28 11:22:11

Re: Migrating Arch's initscripts to dash

#12 2008-02-28 11:53:14

Re: Migrating Arch's initscripts to dash

#13 2008-03-06 00:09:44

Re: Migrating Arch's initscripts to dash

#14 2008-03-06 00:20:14

Re: Migrating Arch's initscripts to dash

#15 2008-03-06 01:20:51

Re: Migrating Arch's initscripts to dash

#16 2008-03-06 02:34:34

Re: Migrating Arch's initscripts to dash

#17 2008-03-06 04:24:27

Re: Migrating Arch's initscripts to dash

#18 2008-03-06 06:29:12

Re: Migrating Arch's initscripts to dash

#19 2008-03-06 06:58:27

Re: Migrating Arch's initscripts to dash

#20 2008-03-06 07:04:54

Re: Migrating Arch's initscripts to dash

#21 2008-03-06 07:42:49

Re: Migrating Arch's initscripts to dash

#22 2008-03-06 09:21:51

Re: Migrating Arch's initscripts to dash

#23 2008-03-06 15:41:57

Re: Migrating Arch's initscripts to dash

#24 2008-03-06 16:17:14

Re: Migrating Arch's initscripts to dash

#25 2008-03-06 23:15:50

Re: Migrating Arch's initscripts to dash

Board footer