Looking for advice how to setup a few network booted systems

Isola · 2011-02-07 14:43:54

Hi!

I've manage to get PXE and gPXE working, aswell as booting Arch via NFS. This all works great. However, this is for a single system.

What is the common setup most people do for a bunch of network booted system?

Currently I have NFS mount a root directory, and that is all. But this gets pretty big as the amount of machines increases if each system would have their own root directory over NFS.

So how to Arch, or any GNU/Linux, distribute NFS shares?

I guess every host could share the same root directory. Even /etc since the /etc/passwd would be there. Or maybe there is another form of auth. method commonly used?

I guess I'm asking for a list of directories that can be global, and directories that should be local (system specific), and the options for them (readonly, etc).

/ - global - nfs - readonly?
/home - global - nfs - writeable
/tmp - local - tmpfs
/var/lock/ - local - tmpfs
/var/run/ - local - tmpfs

Maybe I'm missing something! Please let me know!

VitaminJ · 2011-03-04 00:07:46

Hi there,

I'm looking for similar functionality for a flock of computers that will use Chromium almost exclusively. An LTSP of sorts, except being fat clients that share a read-only root. My university does something like this, though they are doing it with Ubuntu.

The trick is to mount any areas where programs will write as tmpfs (which only stores in memory).
The uni's /etc/fstab on the booted NFS root clients looks like this:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/root       /               nfs     ro              0       0
proc            /proc           proc    defaults        0       0

Here's the partial output of the mount command on one of those running systems (the rest is home directory automounts that follow a similar format as the last line):

none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /dev type devtmpfs (rw,relatime,size=1014640k,nr_inodes=215155,mode=755)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
192.168.2.52.12:/ubuntu-lucid on / type nfs (ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,posix,nolock,proto=tcp,port=65535,timeo=7,retrans=3,sec=sys,mountport=65535,addr=192.168.2.52.12)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /home type tmpfs (rw,relatime,mode=755)
tmpfs on /local/etc type tmpfs (rw,relatime)
tmpfs on /local/var type tmpfs (rw,relatime)
tmpfs on /local/usr type tmpfs (rw,relatime,mode=755)
tmpfs on /local/rw type tmpfs (rw,relatime,mode=755)
tmpfs on /local/media type tmpfs (rw,relatime,mode=755)
none on /etc type aufs (rw,relatime,si=b51e62a9)
none on /var type aufs (rw,relatime,si=b51e63a9)
none on /usr type aufs (rw,relatime,si=b51e66a9)
none on /lib/init/rw type aufs (rw,relatime,si=b51e69a9)
none on /media type aufs (rw,relatime,si=b51e61a9)
/dev/sda2 on /diskless type ext3 (rw,noatime,errors=continue,data=ordered)
/dev/sda2 on /tmp type ext3 (rw,noatime,errors=continue,data=ordered)
/dev/sda2 on /home type ext3 (rw,noatime,errors=continue,data=ordered)
none on /sys/fs/fuse/connections type fusectl (rw,relatime)
none on /sys/kernel/debug type debugfs (rw,relatime)
none on /sys/kernel/security type securityfs (rw,relatime)
none on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /var/run type tmpfs (rw,nosuid,relatime,mode=755)
none on /var/lock type tmpfs (rw,nosuid,nodev,noexec,relatime)
/etc/autofs/auto.profs on /home/profs type autofs (rw,relatime,fd=150,pgrp=1353,timeout=60,minproto=5,maxproto=5,indirect)

They are using NIS to do authentication, so you might want to look at the ypbind homepage.

I am emailing the network admin myself to maybe invite him out for a beer to see about how it's done, and thinking out loud there are a couple of things that have to be done when using a setup like this.

Need central authentication
Need to make sure no one writes to the NFS root
Need to mount user's home directory on login
There has to be somewhere to write temp data (this is where tempfs comes in)
mkinitcpio needs to be given instructions to include network and to mount the nfs in the initrd (sounds like you already figured this out from PXE NFS boot, though)
Hostname has to be set and made unique (in the LTSP world this is done with lts.conf, or is assigned based on IP)
A way to add programs easily to the NFS root either by chroot, or a special machine that can mount the NFS root as read-write
If you have differing H/W, make sure the NFS root has the necessary tools for configuring stuff

Personally I think Arch would be beautiful for this. I have a nicely configured 1.0GB VM image of Arch with Chromium and XFCE4 that boots in 20 seconds, would be great for a networked fleet of little diskless Intel Atom machines.

Isola · 2011-03-05 16:16:18

Woah! That's a lot of mounts! I'm not sure what all of them do.

I recently found a page about netbooting in the gentoo handbook. It had most of the neccessary mounts listed. I'll look into it soon.

About your list:
- Need to mount user's home directory on login
I was thinking about this too. If /home would be mounted for all clients - would it slow down if someone made a find /home even if each directory in /home is rwx------?

- Hostname has to be set and made unique (in the LTSP world this is done with lts.conf, or is assigned based on IP)
I think this is one thing great with arch, just modify /etc/rc.conf and add some bash to figure out something unique based on H/W or something

- I A way to add programs easily to the NFS root either by chroot, or a special machine that can mount the NFS root as read-write
used chroot on the server-side. It worked well. I did also try becoming root on the client side and run pacman -Syu. This is bad though. It's not only insanely slow, but very risky since the network has to be running stable during this time.

It should be noted though, that NFS is very robust. When I was logged in and netbooted I issued a reboot on the server. I figured the client would freeze or get input/output error or something along those lines, but it didn't. It did freeze of course, but once the server had completed the reboot the client continued.

The diskless client I'm using is Arch too, but with full KDE. I'm currently looking through the best options for keeping a local repository for pacman for the different machines. Mirroring a whole repository each week is not an option.

Thank you for replying!

Isola · 2011-03-17 00:01:13

This is how far I've come now:

The server modifications:

/diskless contains a whole system of Archlinux, with minor modifications to fstab, mkinitcpio, and rc.conf. There is more to do here.
/diskless/boot contains:
pxelinux.0 <-- this is the first file transfered to PXE-clients
pxelinux.cfg <-- this is a folder, not a file (This is the defaults, but it can be overridden with dnsmasq)
pxelinux.cfg/default <-- this is the configuration read by pxelinux.0 (This is the defaults, but it can be overridden with dnsmasq)
/aufs/diskless <-- this is exported in rw with NFS. It contains the same content as /diskless, but since it's aufs:
/diskless is used by aufs in mode read-only
/mnt/tmpfs is used by aufs in mode read-write
So all writes made to /aufs/diskless seems to be there, but in reality they are written to /mnt/tmpfs, while /diskless stays untouched.

The client modifications in order that they are used while booting:

/boot/pxelinux.cfg/default

default linux
label linux
kernel vmlinuz26
append initrd=kernel26.img rootfstype=nfs root=/dev/nfs nfsroot=192.168.0.3:/aufs/diskless,v3 ip=::::::dhcp

/etc/mkinitcpio.conf
MODULES="nfs"
/etc/fstab
tmpfs for /tmp, /var/lock, /var/run
/etc/rc.conf
DAEMONS=(!network ....
NETWORK_PERSIST=yes

That is all I've done so far. My goal is to have a system which can, in theory, be booted and run in parallell with infinite number of clients. This is why a separate /etc for each client is a no-no!

At the moment I'm looking at a way to remount /etc at the client with aufs when running through the different runlevels as early as possible when trying to avoid to change any initscripts.. actually, the whole / should probably be remouted with aufs to prevent writes being written to the server's /mnt/tmpfs.

I've also seen alternatives to NFS, like NBD. Both have good and bad features.

All in all, I think this is pretty simple so far, don't you think? I don't think there's any need for complex setups like the one used in your uni!

BTW, this is the content of /mnt/tmpfs of the server after a client boot. It can boot with a read-only root, but then a lot of errors are thrown at you. That's why I use aufs.

/mnt/tmpfs/
/mnt/tmpfs/etc
/mnt/tmpfs/etc/profile.d
/mnt/tmpfs/etc/profile.d/locale.sh
/mnt/tmpfs/etc/localtime
/mnt/tmpfs/etc/mtab
/mnt/tmpfs/var
/mnt/tmpfs/var/lib
/mnt/tmpfs/var/lib/misc
/mnt/tmpfs/var/lib/misc/random-seed
/mnt/tmpfs/var/log
/mnt/tmpfs/var/log/dmesg.log
/mnt/tmpfs/var/log/lastlog
/mnt/tmpfs/var/log/faillog
/mnt/tmpfs/var/log/wtmp
/mnt/tmpfs/var/run
/mnt/tmpfs/var/run/utmp
/mnt/tmpfs/.wh..wh.orph
/mnt/tmpfs/.wh..wh.plnk
/mnt/tmpfs/.wh..wh.aufs

Arch Linux

#1 2011-02-07 14:43:54

Looking for advice how to setup a few network booted systems

#2 2011-03-04 00:07:46

Re: Looking for advice how to setup a few network booted systems

#3 2011-03-05 16:16:18

Re: Looking for advice how to setup a few network booted systems

#4 2011-03-17 00:01:13

Re: Looking for advice how to setup a few network booted systems

Board footer