Backup advice

cfr · 2012-01-29 22:30:50

I have read the wiki page which provides an overview of different backup options (https://wiki.archlinux.org/index.php/Backup) and also several of the more detailed wiki pages and external links covering the different possibilities in detail. It is nice to have so much choice. It is also a bit overwhelming and I'm finding it hard to map my previous backup system onto something I might implement on Arch.

I'm therefore thinking that either (1) I'm not understanding what I'm reading; or (2) the kind of backup system I'm used to using is either inappropriate or just unavailable on Linux. I'm hoping somebody can clarify things a bit so that I can make a sensible decision (or at least one which seems sensible at the time!).

So in OS X, I used a combination of Carbon Copy Cloner and DropBox. Although I haven't tackled the setting up yet, DropBox can be used here too, along with several possibilities not open to me before. So that bit should hopefully be fine. The issue is finding something to replace CCC. This uses rsync under the hood, along with additional tools, to allow block copy, file copy, archiving of deleted/changed items, incremental and non-incremental, scheduled and immediate backups. All this seems straightforward to replicate under Linux even if I'm not sure of the best way. But the key feature of CCC is its ability to create a bootable clone of the hard drive on an external (or remote) disk. This makes it possible to switch boot volume and continue working in the event of disk failure.

My standard set up involved three partitions on an external firewire drive. Two of these were used for backup on alternate days (MWF or TThSa) with the third being used once per week (Su). The weekly backup also archived deleted items and I had a post-install script which moved these to a smaller firewire drive to keep space usage down. Each of the three backups was a possible boot volume.

I don't have firewire now but the main external drive can be used over usb. However, it is now smaller rather than larger than my internal drive. So I could probably make one, possibly two, backup copies on it.

Initially, I thought that Clonezilla might fit the bill. But that does not permit live backup (the source volumes must be unmounted) or incremental backup. Also, I'd assume that to get a bootable clone, you'd need to copy the disk rather than the partition and that would require a destination at least as large as the source which I don't have available right now. One thing confusing me, I think, is that in OS X I had the typical single-volume setup so backing up a single partition was backing up the system. So long as the partition map etc. was correct on the external drive, CCC could automatically make this bootable. But now I've got multiple partitions being part of a single system and they do not all have the same type of filesystem on them.

I'm not even certain how the destination disk should be prepared. Should the type of partition map on source and destination match, for example? Should I mirror the partition structure from my internal drive? What about swap? (I know I don't need to back it up but should the external drive have one?) What precautions are recommended in preparing a backup disk for use with Linux?

So am I looking for a solution which does not exist? Or am I just misunderstanding how to go about it?

A pointer to some documentation would be great. Everything I've read seems to assume that I already have a sense of the overall picture and just need to fill in the details. But actually I'm semi-familiar with rsync etc. and really need to know all the stuff that comes before you get to that!

thisoldman · 2012-01-30 04:00:34

cfr wrote:

... But the key feature of CCC is its ability to create a bootable clone of the hard drive on an external (or remote) disk. This makes it possible to switch boot volume and continue working in the event of disk failure.
...
I'm not even certain how the destination disk should be prepared. Should the type of partition map on source and destination match, for example? Should I mirror the partition structure from my internal drive? What about swap?

There's a very good guide in the wiki for creating a bootable backup of your Archlinux installation with rsync: https://wiki.archlinux.org/index.php/Fu … with_rsync. The original contribution and discussion thread is https://bbs.archlinux.org/viewtopic.php?id=83071.

This is not a solution that clones disks. It creates a bootable backup. You do have to create a '/boot/grub/menu.lst' and an '/etc/fstab' by manually editing for the new bootable backup to work correctly. You can create the backup on a single partition and if your machine has adequate RAM, you don't need to have a swap partition or swapfile.

--------
There's an article, which I've done no more than skim, "Time Machine for every Unix out there", at http://blog.interlinked.org/tutorials/r … chine.html. The "classic" howto for rsync-based time-stamped incremental backups is http://www.mikerubel.org/computers/rsync_snapshots/. A good update of the Mike Rubel article is at http://webgnuru.com/linux/rsync_incremental.php.

graysky · 2012-01-30 10:10:30

@op - tl;dr

Does something like backintime fit the bill? Find it in the AUR.

cfr · 2012-01-31 02:13:34

Thanks for the suggestions. I will get reading later this week. I'd actually looked at the wiki page on full system backup with rsync but I hadn't realised that it meant what CCC means by "clone". I thought it just meant backing up the lot but not necessarily in a bootable package. I think the terminology has thrown me - what CCC counts as a "clone" wouldn't qualify in the sense it is used in Linux, I don't think. (CCC *can* clone in this sense but most of its "clones" are not clones...)

As I understand it backintime is a Linux version of Apple's Time Machine. I've never used Time Machine but I know enough to know that's not what I'm after here. You can't boot from a TM backup and I would like my primary backup to produce a bootable backup.

I'm using grub2 on my machine. Presumably I could install that on the external drive instead of grub.

I really appreciate somebody helping me narrow down the options a bit and figure out where to look. I was starting to think that the more I read, the more confused I kept getting. The 'clone' issue really threw me...

I'll let you know when I've figured it out (or hit a snag which is probably more likely).

David Batson · 2012-01-31 03:35:11

I believe with clonezilla you can clone and compress the drive or partition to an image. Takes some time though.

I recently cloned a 320GB drive to an identical drive (without compression) and it took about 3 hours more or less IIRC. This made a completely bootable hard drive identical to my source drive. I used dd for the job (which was on the Clonezilla CD). Got the how-to info from the following thread. Had to log in as root on the CD with "sudo -i", then I could run dd.
http://comments.gmane.org/gmane.comp.sy … a.live/749

EDIT: BTW, this is the dd command I ran: # dd if=/dev/sda of=/dev/sdb

Maybe this helps. If you can boot a Live CD, you can clone from there.
http://www.tuxradar.com/content/how-clo … clonezilla

Last edited by David Batson (2012-01-31 03:41:20)

thisoldman · 2012-01-31 08:49:45

A supplementary, non-bootable option is to archive partitions with fsarchiver.

I use 'fsarchiver', launched from a hard drive image of SystemRescueCD, to copy partitions. I use it when I rearrange partitions or when I wish to experiment with different filesystem types, say switching from ext4 to jfs.

Fsarchiver copies a partitition's files to a compressed archive, it does not copy the hard drive's blocks. It can restore to a smaller or larger partition than the original and keeps partition labels and UUIDs intact – no need to rewrite your fstab unless you change fs types.

http://www.fsarchiver.org/Main_Page
http://www.sysresccd.org/Main_Page

Last edited by thisoldman (2012-01-31 09:23:22)

cfr · 2012-02-04 16:53:24

Thanks for the further suggestions. Right now, I'm really looking for something I can run while booted from the source because for regular backups, they won't get done if I have to boot from a live usb stick to make them (I assume this would work - no optical drive here). But I can see that they would be good options for occasional use when swapping disks or changing filesystems, as mentioned.

Also, my destination is smaller than my source so I can't just clone the entire disk and I don't really need to copy all of the empty blocks to do what I want.

So I'm about to read the stuff posted earlier about rsync. In the meantime, I have got DropBox working, I think. (I'm not entirely certain about it right now but it seems to work using kfilebox.)

rwd · 2012-02-04 17:00:36

It is impossible to give any sensible advise without knowing which aspects of backup are essential to you. Each backup method has pros/cons.

- do you need a gui?
- do you need to recover every deleted file, or just from a daily backup?
- do you use a server to store backups
etc

I use rdiff-backup as a cron-job that makes a daily incremental backup to a home-server, and that works for me. I have restored an entire system from it a few times, although it takes some work.

cfr · 2012-02-04 19:00:51

Apologies for missing the Full system backup before. I really had looked at the page and it does say it makes it bootable right at the top but somehow I still managed to miss it.

I understand (I think) about how rsync works in this case. This isn't using incremental backup but just making a strict copy of the source, right? And following the first backup, at least, I need to modify fstab, install the bootloader and configure the bootloader to use the external drive. And I can do the latter by adding entries to my existing grub menu which will then get copied to the destination during the next rsync (or during a quick partial rsync on the first go). So if I then exclude fstab from what's copied after the first run, subsequent updates shouldn't, I think, require manual tweaking to make the destination bootable. (I might have missed something - will find out, I expect.)

I'm a bit confused about how to begin. For OS X, advice on backing up always seems to include elaborate instructions for preparing the destination drive and volumes prior to the first run but all of the stuff I've read about rsync seems to take this part for granted. So I'm guessing it is supposed to somehow be obvious...

I assume that I need to erase the destination drive as a first step and create a new partition map, new partitions and new filesystems. Should I write zeros to the drive before making the partition map or is it sufficient to create the map? I need to change from one sort of partition map to another because I don't think that Linux will boot my current machine with the current map. I don't know whether a simple erase is enough of if residues might remain if I don't also zero it. Obviously, if zeroing is definitely unnecessary, I'd rather not bother but if it might be a good idea, I'd rather make the extra effort to be sure. (And I will find out how to zero a drive using Linux tools.)

Current plan [comments welcome]:
[1. figure out how to zero drive if required and zero it]
2. create GPT map on drive
3. create partitions - I'm thinking it might be simplest to mirror those on the source as far as possible, excluding those which are empty, contain a broken, half-installed Ubuntu or provide swap and making the others smaller as necessary
4. use commands from the fully system backup with rsync wiki to produce initial backup of partitions [should this include the EFI partition? or should that be done as part of grub2 installation?]
5. modify fstab
6. install grub2
7. add entries to grub menu and copy configuration to desination
8. set up script and cron job to keep destination updated

This will hopefully give me a bootable 250G backup of my 320G drive (which doesn't contain 320G of data so it should all fit with room to spare).

cfr · 2012-02-06 01:57:20

rwd wrote:

It is impossible to give any sensible advise without knowing which aspects of backup are essential to you. Each backup method has pros/cons.
- do you need a gui?
- do you need to recover every deleted file, or just from a daily backup?
- do you use a server to store backups
etc
I use rdiff-backup as a cron-job that makes a daily incremental backup to a home-server, and that works for me. I have restored an entire system from it a few times, although it takes some work.

I realise this and I'm sorry if I was insufficiently clear. As I mentioned, I'm trying to backup to a USB external drive which is smaller than my internal hard disk (but not smaller than the data on that disk). So no server is involved. (Not right now, anyway. I'm trying to start with what I think is a simpler problem.)

I should have said: no, I don't need a gui. CCC does offer a gui but that isn't the bit I care about. Rather the things I highlighted: bootable backup, backup performed live, automated backup are key.

I don't expect my backup to my USB drive to enable me to recover every deleted file. I expect it to enable me to continue working with minimal data loss in case of catastrophe and, later, to restore to my internal drive.

For example, when the original hard drive in my previous laptop failed, I just worked from my backup drive until I got a new one and got somebody to install it, working as usual but from the external drive and ignoring the internal one. Once the new drive was installed, I prepared it by partitioning etc. and then reversed the process to get a working and updated system on the new drive. This new drive was bigger than both the old drive and the backup drive. The result not only restored everything lost when the original drive failed but also, crucially, included all the work I'd done while booted from backup in the meantime. I've also continued working in the absence of my own computer in this way by booting a similar machine from my backup and using all my usual applications with all my usual data from there. (I actually wiped my machine's drive before letting it be sent off for repair for privacy/security reasons and restored the updated versions of everything when I got the machine back.)

cfr · 2012-02-06 02:16:10

Anyway, this is my current experimental script (hasn't actually run yet):

#!/bin/bash -
# rsync backup script
# cfr: based on script given at https://wiki.archlinux.org/index.php/Full_System_Backup_with_rsync
#set -x


PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
export PATH

list="/etc/cfrbackup/backup.lst"

force=0

allan=0

usage="Usage: $0 [OPTION] [SOURCE DIRECTORY] [DESTINATION DIRECTORY]
Backup a directory (perhaps a partition) using rsync.
Options:
        -f --force              backup to a destination on the root volume
        -h --help               print this message and exit
        -l --list               specify list of files for inclusion/exclusion           [/etc/cfrbackup/backup.lst]"

error () {
        echo "$@" 1>&2
        ((allan++))
        usage_and_exit $allan
}
usage () {
        printf %b "$usage\n"
}
usage_and_exit () {
        usage
        exit $1
}

tempargs=$(getopt -o fhl: --long force,help,list: -- "$@")
if [ $? != 0 ];
then
        usage_and_exit
fi
eval set -- "$tempargs"

while true
do
        case "$1"
        in
                -f | --fo | --for | --forc | --force | --frce | --fce)
                        force=1;
                        shift;;
                -h | --help)
                        usage;
                        exit $allan;;
                -l | --li | --lis | --list | --lst | --ls)
                        if [ -f "$2" ]
                        then
                                list="$2"
                        else
                                error "$2" is not a regular file.
                        fi
                        shift;
                        shift;;
                --)
                        shift;
                        break;;
                *)
                        error Unrecognised option "$1".
        esac
done

args="$@"

if [ ! $# == 2 ]
then
        error "Exactly two arguments are required. Please adjust your requirements accordingly."
else
        destdir=$(echo $2 | sed 's+/$++')
        srcdir=$(echo $1 | sed 's+/$++')
fi
if [ ! -f "$list" ]
then
        error "Default backup list is not a regular file. A list must be specified in this case."
fi

/bin/df > /tmp/df.$$
grep "$destdir" /tmp/df.$$ > /dev/null
if [ $? != 0 ]
then
        tuuid=$(grep "$destdir" /etc/fstab | sed 's/\(UUID[^ ]*\) .*$/\1/' | sed 's/\"/\\\"/g')
        /sbin/blkid | grep $tuuid > /dev/null
        if [ $? != 0 ]
        then
                error Cannot find or mount "$destdir".
        else
                /bin/mount $destdir
                if [ $? != 0 ]
                then
                        error Cannot mount "$destdir".
                fi
        fi
fi
rootvol=$(grep " /$" /tmp/df.$$ | grep -v rootfs | sed 's/ .*$//' | sed 's+/dev/++' | sed 's+^\(sd.\)[0-9]*$+\1+')
grep " ${destdir}$" /tmp/df.$$ | grep "/dev/"  > /dev/null
if [ $? != 0 ]
then
        printf %b "$destdir/ is probably a temporary directory! (Or it could be /.)\n"
        ((allan++))
        exit $allan
fi
targvol=$(grep " ${destdir}$" /tmp/df.$$ | grep -v rootfs | sed 's/ .*$//' | sed 's+/dev/++' | sed 's+^\(sd.\)[0-9]*$+\1+')
if [ "$rootvol" == "$targvol" ]
then
        if [ $force != 1 ]
        then
                printf %b "$destdir is on the root volume. Will not proceed without --force.\n"
                ((allan++))
                exit $allan
        fi
fi
rm /tmp/df.$$

if [ ! -d "$srcdir/" ]
then
        error "$srcdir/" does not seem to be a directory.
fi

if [ ! -x "/usr/bin/rsync" ]
then
        error "Executable /usr/bin/rsync not found."
fi

# suggested invocation uses -av
# add -A for ACLs ; -X for extended attributes; -H for hard linkx
/usr/bin/rsync -aAXv --delete-excluded --exclude-from=$list "$srcdir/" "$destdir";
# not sure what the purpose of this line is?
touch "$destdir/BACKUP"

exit $allan

And then I can run a cron job via anacron with, for example:

#!/bin/bash -
#set -x
PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
export PATH

/etc/cfrbackupup.sh -l /etc/cfrbackup/root.lst / /mnt/backup/
/etc/cfrbackupup.sh -l /etc/cfrbackup/home.lst /home /mnt/backup/home
/etc/cfrbackupup.sh -l /etc/cfrbackup/var.lst /var /mnt/backup/var
/etc/cfrbackupup.sh -l /etc/cfrbackup/local.lst /usr/local /mnt/backup/usr/local
/etc/cfrbackupup.sh -l /etc/cfrbackup/somedir.lst /mnt/between /mnt/backup/mnt/somedir
/etc/cfrbackupup.sh -l /etc/cfrbackup/somepart.lst /mnt/finnix /mnt/backup/mnt/somepart
/etc/cfrbackupup.sh -l /etc/cfrbackup/efi.lst /boot/efi /mnt/backup/boot/efi

After the first run, I'll try to install grub2 to the backup drive, edit fstab and then edit root.lst to exclude fstab from future runs. I'll also change grub configuration to include the usb drive as an option and see if I can figure out how to copy that to the backup (but I might just wait for it to sort itself out in due course). I also may split the backup at some point as I don't really need all of these backed up equally frequently.

But first I'll obviously have to figure out why it isn't working as I currently expect!

cfr · 2012-02-06 23:45:01

Well, my anacron job failed due to a typo so I ended up running it by hand and left it overnight. Hard to check but it looks good.

I've edited the backed up version of fstab and added a P rule to the include/exclude list for rsync, together with a rule to exclude /etc/fstab. (I hope this means that it will not backup /etc/fstab but will not delete the existing (edited) backup copy. That's what the man page suggests to me...) I also fiddled a little with the list as I realised I'd backed up my backup mount point which seems merely potentially confusing. I've also added a mount point for the internal drive and added a P rule for it.

I now realise I'm a bit confused.

I went back to the grub2 wiki page and realised that it seems to have been significantly edited since I installed a couple of months ago so none of my directories seem to quite match... The real problem, though, is that I'm not entirely clear:

1) What, if anything, I need to do to install grub2 to the backup EFI partition which is currently a copy of my original EFI partition (and so includes the results of installing grub2 on my internal drive). I assumed I'd have to do something but now I get here, I'm not sure what, if anything. Does installing grub2 to a drive do anything beyond populating the relevant directories of the EFI partition? Would those need to be different for the external drive?

2) How best to add the external drive to my grub2 menu. I have managed to add entries to this before but in those cases I could more-or-less copy paste and adapt from the existing files /etc/grub.d/10_linux to create entries which would be just like those but specifying run level 3. But the problem now is that I need grub2's configuration to find the external drive and somehow label the entries for that drive so that I can distinguish them (The external drive is not usually attached so I don't want to boot from that by mistake!)

I know it is possible to edit grub.cfg manually but I'd rather not do that unless I really have to since I'm trying to stick to "recommended" methods for now, at least. (My installation now diverges from the wiki grub2 setup but at least it matches a layout which was recommended at one point!)

Or should I just run the configuration and will it figure everything out automagically? I had so much trouble getting this machine to boot, I'm wary of screwing it up...

David Batson · 2012-02-07 00:49:39

The naming of a second drive is different than the first, so that shouldn't be a problem with grub - especially if you use uuid partition identification. You can run: # blkid from terminal to get the uuid of a partition. # fdisk -l as well will show the partitions on your drive. You can put everything in /etc/grub.d/custom_40 and run # grub-mkconfig -o /boot/grub/grub.cfg to send your changes to /boot/grub/grub.cfg

Last edited by David Batson (2012-02-07 00:50:53)

cfr · 2012-02-07 01:07:07

So for the external drive I should specify the UUID in the custom rules rather than letting the configuration script find it automatically? (I say "let" but I'm not sure how I'd get it to do that.)

40_custom doesn't give any examples. The wiki page gives an example for manually configuring grub2. Based on the 'tail' bit at the top of this file, I guessing that I can maybe copy and adapt an example for the wiki for manual configuration e.g.

menuentry "Arch Linux" {
    set root=(hd0,1)
    linux /vmlinuz-linux root=/dev/sda3 ro
    initrd /initramfs-linux.img
}

or, better, from the grub.cfg I already generated for my internal drive, adapting as needed. If that's right, I can presumably name them as I wish and so avoid any confusion.

I definitely will be using UUIDs and finding them isn't an issue - I've already got them in fstab.

[The drive is GPT so gdisk -l is needed in place of fdisk -l in case anybody ever looks at this thread for information.]

Last edited by cfr (2012-02-07 01:07:57)

David Batson · 2012-02-07 01:23:28

I've got my /etc/grub.d/custom_40 file in the later part of the following post.

https://bbs.archlinux.org/viewtopic.php … 2#p1019742

cfr · 2012-02-07 01:56:08

Thanks! That answers several questions. One more: how do I know which disk number to specify? Right now, the disk is hd2 by grub's lights, I think (sdc). And if I choose the external on a normal boot, presumably that'd be right. But if I'm actually using grub from the backup, wouldn't that disk be sda and so hd0 according to grub? Or have I got this confused? I've got the search line with --set-root and the correct UUID and the UUID is in the linux command line as well. I'm not clear what the additional

set root='(hd0,gpt5)'

that precedes these does. I assume I should at least change the '5' to '3' because / on the backup is the third rather than fifth partition but should the 'hd0' also be 'hd2' or should it remain as it is?

I guess one way of putting this is that I'm not clear how using UUIDs avoids the problems of identifying disks and partitions if you still need to set root according to an unpredictable disk and partition number. So I think I'm missing something because I know that's the whole point!

Last edited by cfr (2012-02-07 01:57:41)

rwd · 2012-02-07 07:53:03

which drive becomes hd0 and which hd1 is set by/boot/grub/device.map in which /dev/disk/by-id/... is mapped to a number.

Last edited by rwd (2012-02-07 07:56:36)

cfr · 2012-02-07 22:19:09

Thanks. I attached the drive and created a new device.map. I later created a second one in the right place.

I used grub-mkconfig to generate the new config file, passing it GRUB_PREFIX as /boot/efi/efi/grub and setting /boot/efi/efi/grub/grub.cfg as output. I am not sure how convinced I am by the result.

Has grub2 changed a good deal in the last month or two? My auto-generated entries look quite different now! (And probably mean that the ones I just added as custom entries are rather out of date since I just copied and adapted the previous versions.)

In particular, there are lots of conditionals in both the setup stuff and the menu entries. For example:

menuentry 'Arch Linux, with Linux linux' --class archlinux --class gnu-linux --class gnu --class os {
        savedefault
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt5'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt5 --hint-efi=hd0,gpt5 --hint-baremetal=ahci0,gpt5 --hint='hd0,gpt5'  <uuid>
        else
          search --no-floppy --fs-uuid --set=root <uuid>
        fi
        echo    'Loading Linux linux ...'
        linux   /boot/vmlinuz-linux root=/dev/disk/by-uuid/<uuid> ro  quiet resume=/dev/disk/by-uuid/<uuid> pcie_aspm=force i915.i915_enable_rc6=1 i915.i915_enable_fbc=1 i915.lvds_downclock=1
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-linux.img
}

Is that right? It doesn't look right!

cfr · 2012-02-08 00:23:46

Well I can still boot the autogenerated default entry, anyway. But I'd like to know what the --hints do and whether they should be added (conditionally) to my custom menu entries (which I haven't actually tried yet)...

the.ridikulus.rat · 2012-02-08 08:17:02

Yes, grub2 bzr repo is being updated at a very fast pace, and arch uses bzr snapshots instead of 1.99 tarball, to take advantage of newer features. Next grub2 package will involve a change that moves platform specific files from GRUB_PREFIX to GRUB_PREFIX/platform wherein a user having both grub2-bios and grub2-uefi can install both to the same GRUB_PREFIX (actually bios, 64-bit uefi, 32-bit uefi, coreboot etc. any no. of platforms), and then carry the disk for portable booting in systems having different firmwares. Platform independent files like unicode.pf2, grub.cfg will be in GRUB_PREFIX itself. It still requires minor tweaks to scripts like /etc/grub.d/00_header etc. so that platform specific video modules can be properly loaded by a common grub.cfg (using if-else statements like in bash)

the.ridikulus.rat · 2012-02-08 08:32:38

cfr wrote:

Well, my anacron job failed due to a typo so I ended up running it by hand and left it overnight. Hard to check but it looks good.
I've edited the backed up version of fstab and added a P rule to the include/exclude list for rsync, together with a rule to exclude /etc/fstab. (I hope this means that it will not backup /etc/fstab but will not delete the existing (edited) backup copy. That's what the man page suggests to me...) I also fiddled a little with the list as I realised I'd backed up my backup mount point which seems merely potentially confusing. I've also added a mount point for the internal drive and added a P rule for it.
I now realise I'm a bit confused.
I went back to the grub2 wiki page and realised that it seems to have been significantly edited since I installed a couple of months ago so none of my directories seem to quite match... The real problem, though, is that I'm not entirely clear:
1) What, if anything, I need to do to install grub2 to the backup EFI partition which is currently a copy of my original EFI partition (and so includes the results of installing grub2 on my internal drive). I assumed I'd have to do something but now I get here, I'm not sure what, if anything. Does installing grub2 to a drive do anything beyond populating the relevant directories of the EFI partition? Would those need to be different for the external drive?

You need to use either --removable flag or use grub-mkstandalone to create a all-in-one grub.efi that includes the modules embedded inside a memdisk inside grub.efi .

2) How best to add the external drive to my grub2 menu. I have managed to add entries to this before but in those cases I could more-or-less copy paste and adapt from the existing files /etc/grub.d/10_linux to create entries which would be just like those but specifying run level 3. But the problem now is that I need grub2's configuration to find the external drive and somehow label the entries for that drive so that I can distinguish them (The external drive is not usually attached so I don't want to boot from that by mistake!)

You can use variables and search command to detect external drives and then setup appropriate menu entries

I know it is possible to edit grub.cfg manually but I'd rather not do that unless I really have to since I'm trying to stick to "recommended" methods for now, at least. (My installation now diverges from the wiki grub2 setup but at least it matches a layout which was recommended at one point!)
Or should I just run the configuration and will it figure everything out automagically? I had so much trouble getting this machine to boot, I'm wary of screwing it up...

Arch uses bzr snapshots for grub2 pkgs so the wiki is updated as and when things are changed. You can either try the grub2 pkgs and follow the wiki, or chakout grub2 bzr repo and use https://github.com/the-ridikulus-rat/My … b2_uefi.sh script which should (hopefully) setup everything automatically for use. But read the script once to understand the options (especially configure options) and make sure you do not overwrite any path that is managed by pacman as part of any package. In my system I use /_grub_/ as the prefix.

Also using grub-mkconfig is not the recommended method, it is just one of the methods, which help in updating the config in case of grub2 or kernel upgardes. My configs are

EDIT: grub2-uefi and grub2-bios merged config. Requires grub2 bzr r3849 or above for this config to work.

set _kernel_params="radeon.modeset=1 gpt loglevel=6 printk.time=y pcie_aspm=force elevator=noop"
set _systemd="init=/bin/systemd"
set _rootfstype="ext4"

## Linux Kernel Video Mode 1024x768x32 code = 824
set _video_mode="824"

insmod part_gpt
insmod fat
insmod ext2

set _root_part_id_1="ata-WDC_WD3200BEVT-75ZCT2_WD-WXEY08JK4324-part9"
set _root_part_id_2="scsi-SATA_WDC_WD3200BEVT-_WD-WXEY08JK4324-part9"
set _root_part_uuid="12a1a38d-6339-459e-977b-f1ab192ecdad"
set _root_fs_uuid="0cc6e472-7f98-42ff-b7f3-309b641377ad"

set _boot_part_id_1="ata-WDC_WD3200BEVT-75ZCT2_WD-WXEY08JK4324-part3"
set _boot_part_id_2="scsi-SATA_WDC_WD3200BEVT-_WD-WXEY08JK4324-part3"
set _boot_part_uuid="016f0c6e-d1aa-45cf-9423-5358770c18ee"
set _boot_fs_uuid="2ccd62fe-2204-44e8-bcfd-ce33996f9852"

set _uefi_sys_part_id_1="ata-WDC_WD3200BEVT-75ZCT2_WD-WXEY08JK4324-part1"
set _uefi_sys_part_id_2="scsi-SATA_WDC_WD3200BEVT-_WD-WXEY08JK4324-part1"
set _uefi_sys_part_uuid="ed5cd10c-939a-49c1-ae67-b7f2a76949dd"
set _uefi_sys_fs_uuid="1BF5-7E28"

search --fs-uuid --no-floppy --set=_arch64_boot --hint-bios=hd0,gpt3 --hint-efi=hd0,gpt3 --hint-baremetal=ahci0,gpt3 ${_boot_fs_uuid}
search --fs-uuid --no-floppy --set=_arch64_root --hint-bios=hd0,gpt9 --hint-efi=hd0,gpt9 --hint-baremetal=ahci0,gpt9 ${_root_fs_uuid}

search --fs-uuid --no-floppy --set=_uefisys_part --hint-bios=hd0,gpt1 --hint-efi=hd0,gpt1 --hint-baremetal=ahci0,gpt1 ${_uefi_sys_fs_uuid}

if [ "${grub_platform}" == "efi" ]; then
	set _UEFI_ARCH="${grub_cpu}"
	
	if [ "${grub_cpu}" == "x86_64" ]; then
		set _SPEC_UEFI_ARCH="x64"
	fi
	
	if [ "${grub_cpu}" == "i386" ]; then
		set _SPEC_UEFI_ARCH="ia32"
	fi
	
	set _kernel_params="${_kernel_params} add_efi_memmap none=UEFI_ARCH_${_UEFI_ARCH}"
	
	insmod efi_gop
	# insmod efi_uga
	# insmod video_bochs
	# insmod video_cirrus
fi

if [ "${grub_platform}" == "pc" ]; then
	insmod vbe
	# insmod vga
	# insmod video_bochs
	# insmod video_cirrus
fi

insmod font

if loadfont (${_uefisys_part})/efi/grub_uefi_x86_64/unicode.pf2 ; then
	insmod gfxterm
	
	set gfxmode="1024x768x32;auto"
	
	terminal_input console
	terminal_output gfxterm
	
	# set color_normal=light-blue/black
	# set color_highlight=light-cyan/blue
	
	# insmod png
	# insmod jpeg
	# insmod gfxmenu
	
	# background_image (${_arch64_boot})/images/archlinux.png
	
	set locale_dir=${prefix}/locale
	set lang="en_US"
	
	# insmod gettext
fi

set default="Arch Linux Testing Kernel"

set _hidden_timeout="5"

if sleep --verbose --interruptible ${_hidden_timeout} ; then
	set timeout="0"
fi

# menuentry "Arch Linux Testing Kernel" --users "keshav_pr,user" {

menuentry "Arch Linux Testing Kernel" {
	set gfxpayload="keep"
	set root=(${_arch64_boot})
	linux /vmlinuz-linux root=/dev/disk/by-id/${_root_part_id_1} ro rootfstype=${_rootfstype} ${_kernel_params} ${_systemd}
	initrd /initramfs-linux.img
}

menuentry "Arch Linux Testing Kernel Fallback" {
	set gfxpayload="keep"
	set root=(${_arch64_boot})
	linux /vmlinuz-linux root=/dev/disk/by-id/${_root_part_id_1} ro rootfstype=${_rootfstype} ${_kernel_params} ${_systemd}
	initrd /initramfs-linux-fallback.img
}

menuentry "Arch Linux Mainline Kernel" {
	set gfxpayload="keep"
	set root=(${_arch64_boot})
	linux /vmlinuz-linux-mainline root=/dev/disk/by-id/${_root_part_id_1} ro rootfstype=${_rootfstype} ${_kernel_params} ${_systemd}
	initrd /initramfs-linux-mainline.img
}

menuentry "Arch Linux Mainline Kernel Fallback" {
	set gfxpayload="keep"
	set root=(${_arch64_boot})
	linux /vmlinuz-linux-mainline root=/dev/disk/by-id/${_root_part_id_1} ro rootfstype=${_rootfstype} ${_kernel_params} ${_systemd}
	initrd /initramfs-linux-mainline-fallback.img
}

if [ "${grub_platform}" == "pc" ]; then
	
	menuentry "Boot Tianocore UDK DUET UEFI x86_64 - Partition" {
		search --fs-uuid --no-floppy --set=root 5FA3-2472
		chainloader +1
	}
	
	# menuentry "Boot Tianocore UDK DUET UEFI x86_64 - Memdisk" {
	# 	set root=(${_arch64_boot})
	# 	linux16 /memdisk_syslinux floppy ro nopass
	# 	initrd16 /Tianocore_UEFI_UDK_DUET_X86_64.img
	# }
	
fi

if [ "${grub_platform}" == "efi" ]; then
	
	menuentry "UEFI ${_UEFI_ARCH} Shell 2.0" {
		set root=(${_uefisys_part})
		chainloader /efi/shell/shell${_SPEC_UEFI_ARCH}.efi
	}
	
	menuentry "UEFI ${_UEFI_ARCH} Shell 1.0" {
		set root=(${_uefisys_part})
		chainloader /efi/shell/shell${_SPEC_UEFI_ARCH}_old.efi
	}
	
	menuentry "Microsoft Windows x86_64 UEFI-GPT" {
		set root=(${_uefisys_part})
		chainloader /efi/Microsoft/Boot/bootmgfw.efi
	}
	
fi

Last edited by the.ridikulus.rat (2012-02-08 09:21:42)

the.ridikulus.rat · 2012-02-08 08:36:53

cfr wrote:

Well I can still boot the autogenerated default entry, anyway. But I'd like to know what the --hints do and whether they should be added (conditionally) to my custom menu entries (which I haven't actually tried yet)...

Hints speeds up the search process by trying the hinted device first if required, but if the hinted (for example) does not match the given FS_UUID then it tries normal search.

search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt3 --hint-efi=hd0,gpt3 --hint-baremetal=ahci0,gpt3  2ccd62fe-2204-44e8-bcfd-ce33996f9852

Search first tries (hd0,gpt3) for 2ccd62fe-2204-44e8-bcfd-ce33996f9852, if it succeeds fine, otherwise search all other filesystems. Unlike "set root=(hd0,gpt3)", if that device does not exist, boot will fail, but in case hint device does not exist, it simply makes the boot slightly slower (but not noticable).

cfr · 2012-02-08 17:51:16

Wow! Thanks for the very thorough responses. I'm not sure I fully understand it but I hope I'll get there eventually.

By 'platform', you mean the firmware/boot stuff (e.g. bios or uefi or open firmware etc.) rather than an OS, right?

You need to use either --removable flag or use grub-mkstandalone to create a all-in-one grub.efi that includes the modules embedded inside a memdisk inside grub.efi .

Is this because grub works differently from a removable vs. an internal drive? So it can't use the stuff installed to the EFI partition in the usual way?

I think I will stick to the version of grub2 provided by Arch for now. I don't think I'm competent enough to take on the bzr repo version from upstream!

I mentioned grub-mkconfig being recommended because that's what the wiki said. I don't like the entries it produces as they are much too messy but perhaps I should stick with it if it will let me get updates since I'm certainly not likely to know what to add otherwise.

More (hopefully) later!

the.ridikulus.rat · 2012-02-08 18:22:05

cfr wrote:

Wow! Thanks for the very thorough responses. I'm not sure I fully understand it but I hope I'll get there eventually.
By 'platform', you mean the firmware/boot stuff (e.g. bios or uefi or open firmware etc.) rather than an OS, right?

Yes. In grub2's language cpu-firmware combination is called platform. i386-pc denotes PC BIOS (which is actually 16/32-bit hybrid, hence i386). x86_64-efi menas 64-bit (U)EFI (i386-efi for 32-bit), i386-coreboot for coreboot in 32-bit processor etc.

You need to use either --removable flag or use grub-mkstandalone to create a all-in-one grub.efi that includes the modules embedded inside a memdisk inside grub.efi .
Is this because grub works differently from a removable vs. an internal drive? So it can't use the stuff installed to the EFI partition in the usual way?

No. Using --removable flag makes grub2 copy grub.efi to (UEFISYS)/efi/boot/bootx64.efi instead of (UEFISYS)/efi/arch/grubx64.efi . This flag is present only for (u)efi systems, not for bios or other platforms.

I think I will stick to the version of grub2 provided by Arch for now. I don't think I'm competent enough to take on the bzr repo version from upstream!

Currently Arch grub2 pkgs itself are updated periodically with bzr snapshots as new features are added which are invariably requested by users in the bug tracker (which are legitimate requests, not just for the heck of it - like lvm and uefi fixes etc.).

I mentioned grub-mkconfig being recommended because that's what the wiki said. I don't like the entries it produces as they are much too messy but perhaps I should stick with it if it will let me get updates since I'm certainly not likely to know what to add otherwise.
More (hopefully) later!

Its your choice really. I use the config I posted because grub-mkconfig generated config do not allow variables. I have been using manually edited config for more than a year with no issues. The reason the mkconfig generated config has the "DO NOT EDIT" warning is because that file may be overwritten by scripts run during kernel upgrades or grub2 upgrade itself. In Arch grub2 pkg or kernel pkg updates doe not touch any config, consistent with its policy of having the configs manually managed by the user. But in Ubuntu, Debian, Fedora etc. kernels upgrades run grub-mkconfig and overwrite the old file.

If you want to update your config you have to manually run grub-mkconfig, no script will run it automatically for you. Even Archboot setup script does not run grub-mkconfig, it sets up a custom config on the lines of mine (which I coded actually so its obvious).

EDIT: There is nothing wrong with the mkconfig generated entry you mentioned at https://bbs.archlinux.org/viewtopic.php … 9#p1054299 .

Last edited by the.ridikulus.rat (2012-02-08 18:23:59)

cfr · 2012-02-09 00:57:40

It's not that running grub-mkconfig is a problem. It is more that if I didn't use that method, I wouldn't have a clue what changes might be good to make e.g. adding the --hint stuff. It would be kind of nice to know to do this - a sort of grub2 extension to pacmatic, maybe. But I'd really rather it didn't actually run it automatically.

EDIT: There is nothing wrong with the mkconfig generated entry you mentioned at https://bbs.archlinux.org/viewtopic.php … 9#p1054299 .

Thanks. It just looked very different so I wasn't sure. My previous one didn't seem to have the conditional statements.

No. Using --removable flag makes grub2 copy grub.efi to (UEFISYS)/efi/boot/bootx64.efi instead of (UEFISYS)/efi/arch/grubx64.efi . This flag is present only for (u)efi systems, not for bios or other platforms.

It's been a long day but I'm still really confused by this. Presumably, I didn't use --removable when installing to my internal drive but I have /boot/efi/efi/boot/bootx64.efi rather than grubx64.efi under arch/. This means that I also have /boot/efi/efi/boot/bootx64.efi on the backup (or I should do). So is the problem that this won't include the modules which are needed to boot from the external drive but not the internal one? I'm still not clear what exactly needs to be done differently on the external drive and why. (But I may just need some sleep.)

Arch Linux

#1 2012-01-29 22:30:50

Backup advice

#2 2012-01-30 04:00:34

Re: Backup advice

#3 2012-01-30 10:10:30

Re: Backup advice

#4 2012-01-31 02:13:34

Re: Backup advice

#5 2012-01-31 03:35:11

Re: Backup advice

#6 2012-01-31 08:49:45

Re: Backup advice

#7 2012-02-04 16:53:24

Re: Backup advice

#8 2012-02-04 17:00:36

Re: Backup advice

#9 2012-02-04 19:00:51

Re: Backup advice

#10 2012-02-06 01:57:20

Re: Backup advice

#11 2012-02-06 02:16:10

Re: Backup advice

#12 2012-02-06 23:45:01

Re: Backup advice

#13 2012-02-07 00:49:39

Re: Backup advice

#14 2012-02-07 01:07:07

Re: Backup advice

#15 2012-02-07 01:23:28

Re: Backup advice

#16 2012-02-07 01:56:08

Re: Backup advice

#17 2012-02-07 07:53:03

Re: Backup advice

#18 2012-02-07 22:19:09

Re: Backup advice

#19 2012-02-08 00:23:46

Re: Backup advice

#20 2012-02-08 08:17:02

Re: Backup advice

#21 2012-02-08 08:32:38

Re: Backup advice

#22 2012-02-08 08:36:53

Re: Backup advice

#23 2012-02-08 17:51:16

Re: Backup advice

#24 2012-02-08 18:22:05

Re: Backup advice

#25 2012-02-09 00:57:40

Re: Backup advice

Board footer