[Solved] Advice on scripted in-place install/upgrade

Jens Clasen · 2018-02-01 22:22:00

Good evening!

I'd like to request some advice from people with a bit more experience than myself. This text might get a little lengthy and English is not my native language, therefore & first and foremost: Thank you to everybody taking the time to read it and sorry for any spelling or grammar mistakes.

Here's the situation:

What I'm planning to do is an in-place upgrade of a pretty old Gentoo system. The Gentoo system was setup around 2009/2010 and was more or less kept up to date for quiet a few years. About two years ago, time got a problem and the keeping the system current stopped. So now it's time for a new start. Since time spent updating the system will keep being a problem in the future, a switch away from Gentoo is intended. I'd like to keep on a rolling release plan, but those compile time orgies have to stop.

The old system holds quite a bit of data, mostly inside a single XFS partition (I know...) and I want to keep the downtime as low as possible. Therefor my intend is to do an in-place upgrade and distro change.

The new arch install was already prepared inside a VirtualBox container. I did the setup and I tested what I could test. Afterwards, I transferred the whole system (without mnt, dev, proc, sys and the packman tar.gz files) into a directory inside the XFS partition /mnt/arch for the moment. My plan for the next steps is booting into a live CD, transfer everything from the old system (beside /home, /root and a couple of system directories) to a backup location and then move the arch directories over to / (or to be precise /mnt, since that'll be the name, while running the live CD).

After that, I'll have to transfer the fstab, chroot and install grub, once again. When that is done, I'll probably have to fix a couple of permission problems in the home directories, due to different UIDs and GIDs, and I should be ready to reboot.

The reboot will probably only work with the fallback initrd and I'll have to recreate the initrd image afterwards. That should be it.

Is there anything I'm missing here? Is there something I should check or read before continuing any further?

I created a script to do the setup:

#! /bin/bash

#### Manual steps ####
#1) Boot into liveCD
#2) prepare mount points & network (outside of script)
#mount /dev/sda3 /mnt/
#mount /dev/sda1 /mnt/boot
#if possible:
#mount -t cifs \
# -o username=***,password=***,uid=1000,gid=1000,file_mode=0660,dir_mode=0770 \
# //192.168.1.5/backup /mnt/mnt/gentoo.old
#if not, simply let it be and transfer later
#nmtui

#3) Automatic preparations

# Test for available sources, targets and work locations: 
echo "Testing if relevant locations can be accessed"
[[ ! -d /mnt ]] && echo "Missing mount point - even though I don't know how that would be possible" && return
[[ ! -d /mnt/mnt ]] && echo "Missing working location" && return
[[ ! -d /mnt/mnt/arch ]] && echo "Missing source data" && return
[[ ! -d /mnt/mnt/gentoo.old ]] && echo "Missing backup data" && return

# Test moving dirs and files from arch to future /
echo "Testing the install process..."
cd /mnt/mnt/arch
mv testdir /mnt/
[[ ! -d /mnt/mnt/arch/testdir ]] && echo "Failed moving testdir" && return
echo "Testdir moved. Continuing with further tests..."
[[ ! -f /mnt/mnt/arch/testdir/testfile ]] && echo "Failed moving testfile" && return
echo "Testfile found. Continuing..."
[[ ! -f /mnt/mnt/arch/testdir/.testfile ]] && echo "Failed moving hidden testfile" && return
echo "Hidden testfile found..."
echo "Install process test complete. Source can be read and target can be written"

# Test moving dirs and files from future / to backup
echo "Testing the backup process..."
cd /mnt
mv testdir /mnt/mnt/gentoo.old
[[ ! -d /mnt/mnt/gentoo.old/testdir ]] && echo "Failed moving testdir" && return
echo "Testdir moved. Continuing with further tests..."
[[ ! -f /mnt/mnt/gentoo.old/testdir/testfile ]] && echo "Failed moving testfile" && return
echo "Testfile found. continuing..."
[[ ! -f /mnt/mnt/gentoo.old/testdir/.testfile ]] && echo "Failed moving hidden testfile" && return
echo "Hidden testfile found..."
echo "Backup process test complete. We can remove data from target and backup location is writable."

#4) For each of these entries in /mnt do
echo "Moving old gentoo installation to backup location now..."
cd /mnt
# move regular directories to backup, starting with something unimportant
dirs = ( "tmp" "bin" "etc" "lib32" "lib64" "media" "opt" "sbin" "usr" "var" )
for dir in "${dirs[@]}"
do
    echo "Moving $dir to backup"
    [[ ! -d $dir ]] && echo "$dir not found!" && return
    mv $dir /mnt/mnt/gentoo.old/
    [[ ! -d /mnt/mnt/gentoo.old/$dir ]] && echo "Failed moving $dir" && return
    echo "$dir move complete"
done
echo "bin, etc, lib32, lib64, media, opt, sbin, usr and var should have been moved"

# special treatment
# move away the content of boot, but leave the directory in place
echo "Moving /boot/* to backup"
[[ ! -d boot ]] && echo "boot not found!" && return
mkdir /mnt/mnt/gentoo.old/boot
[[ ! -d /mnt/mnt/gentoo.old/boot ]] && echo "failed creating backup target for boot" && return
mv boot/* /mnt/mnt/gentoo.old/boot/
[[ ! -f /mnt/mnt/gentoo.old/boot/gentoo.igz ]] && echo "A file in boot backup is missing. Aborting here" && return
echo "Boot partition backup complete"

# copy content of root but leave the data in place
echo "Copy /root contents to backup"
[[ ! -d root ]] && echo "root not found!" && return
mkdir /mnt/mnt/gentoo.old/root
[[ ! -d /mnt/mnt/gentoo.old/root ]] && echo "failed creating backup target for /root" && return
cp -r root/. /mnt/mnt/gentoo.old/root
[[ ! -f /mnt/mnt/gentoo.old/boot/xorg.conf.new ]] && echo "A file in root backup is missing. Aborting here" && return
echo "Root directory backup complete"

# leave these untouched: proc,dev,mnt,sys,home

# symbolic links (just one)
echo "Moving symbolic link for lib"
[[ ! -L lib ]] && echo "lib not found!" && return
mv lib /mnt/mnt/gentoo.old/
[[ ! -L /mnt/mnt/gentoo.old/lib ]] && echo "lib symlink not found on backup!" && return
echo "Move to backup complete..."

#5) For each of these entries in /mnt/mnt/arch do: 
echo "Starting arch install..."
cd /mnt/mnt/arch

# move regular directories to target, starting with something unimportant
dirs = ( "tmp" "etc" "opt" "srv" "usr" "var" )
for dir in "${dirs[@]}"
do
    echo "Moving $dir"
    [[ ! -d $dir ]] && echo "$dir not found!" && return
    mv $dir /mnt/
    [[ ! -d /mnt/mnt/gentoo.old/$dir ]] && echo "Failed moving $dir" && return
    echo "$dir moved"
done
echo "tmp, etc, opt, srv, usr, and var should have been moved"

# special treatment
# copy content but leave directory
echo "Installing boot data"
[[ ! -d boot ]] && echo "boot not found!" && return
cp -r boot/. /mnt/boot
[[ ! -f /mnt/boot/vmlinuz-linux ]] && echo "A file in boot target is missing. Aborting here" && return
echo "Boot partition install complete"

# symbolic links
echo "Moving symbolic links..."
slinks = ( "bin" "lib" "lib64" "sbin" )
for slink in "${slinks[@]}"
do
    echo "Moving $slink"
    [[ ! -L $slink ]] && echo "$slink not found!" && return
    mv $slink /mnt/
    [[ ! -L /mnt/$slink ]] && echo "Failed moving $slink" && return
    echo "$slink moved"    
done
echo "bin, lib, lib64 and sbin should have been moved"

# These should still be available: proc,dev,mnt,sys,home
echo "Let's make sure, if the few untouched directories are still available..."
dirs = ( "proc" "dev" "mnt" "sys" "home" )
for dir in "${dirs[@]}"
do
    echo "Testing $dir"
    [[ ! -d $dir ]] && echo "$dir not found!" && return
    echo "$dir found"
done
echo "Paranoia test completed succesfully"

# 6) setup:
# copy fstab from old install
echo "Fetching fstab from backup..."
cp /mnt/mnt/gentoo.old/etc/fstab /mnt/etc/fstab
[[ ! -f /mnt/etc/fstab ]] && echo "Failed" && return
echo "Done"
echo "Fetching resolv.conf from live cd..."
cp /etc/resolv.conf /mnt/etc/
[[ ! -f /mnt/etc/resolv.conf ]] && echo "Failed" && return
echo "Done"


#### MANUAL STEPS ONCE AGAIN: ####
echo "Next steps / printing some reminders:"
echo "$ arch-chroot"
echo "$ grub-install --target=i386 /dev/sda"
echo "$ grub-mkconfig -o /boot/grub/grub.cfg"
echo "run chgrp -R for all directories in /home"
echo "run chown -R for all directories in /home"
echo "$ reboot"
echo "$ mkinitcpio -p linux

Can you provide any advice regarding the script, before I'm starting with a dry run?

Possible pitfalls - as far as I can see - are driver problems. But those should either happen while booting the live CD, while setting up network and mount points or when starting X after a reboot. As long as I get networking up and ready, those problems should all be fixable. And: Most networking problems should already appear when configuring the network from the live CD. The chance for that happening with the final system are pretty low.

Do you see any further pitfalls I should watch out for? Can you recommend any reading/articles/tutorials I should consider, before attempting such a move?

Once again thanks for your consideration.

Regards, Jens

Last edited by Jens Clasen (2018-02-16 20:35:31)

ooo · 2018-02-03 22:34:32

I think you might be overcoplicating things, unless I'm missing something.

In your situation, I would pretty much simply:
1. Backup the old Gentoo installation.
2. Install and configure Arch the regular way.
3. Copy necessary data, and fix permissions for data partitions if required.

Since your old installation haven't been updated for years, is a completely different distribution, and you have some undisclosed issues with it, I would avoid copying anything from the old gentoo installation to your arch system. That's including any configuration files. Configure everything by hand instead.

I would even avoid directly copying /home as a whole if possible, but just copy necessary files to newly created user directories.

I don't see how you could have driver issues if everything worked on old installation with in-kernel modules alone.

Jens Clasen wrote:

XFS partition (I know...)

There's nothing wrong with XFS. Lately it's been more actively developed than even ext4.

Jens Clasen · 2018-02-06 18:45:31

Hey ooo,

thanks for taking the time to comment. I will readily agree that I'm sometimes overcomplicating a problem. In this case, I don't think, I do, though. (Does the english language really need all the commas in the last sentence, btw?)

I refused the apparently easier approach for these reasons:

The major mistake I'm dealing with is an almost 1TB sized, single XFS partition, containing almost everything beside /boot.
/home alone has about 800GB of data. /var contains another 120GB of database data files. When I last switched disks and grew the XFS partition, the whole thing took quiet some time. I'm trying to avoid that.
Preparing an arch install, including the configuration for a couple of services and programs, takes some time. Especially including some testing. During that time, the target system would be offline, if I didn't come prepared. That's why I already did all that inside a virtual environment.

My major concern is downtime here. Even recreating the database after arch is running, will take quiet some time. That's about all the downtime, I'd like to accept. And: the database rebuild is necessary anyway, since I'd like to incorporate some improvements. What I'm trying to do here is minimizing the downtime for the base system and the relevant software.

As for copying configuration files: The only real copy work I'm doing is copying the old fstab file over to arch. The rest of the configuration was done by hand and by diff already inside the virtual test installation. Since a lot of software versions do not match between the old gentoo system and the new arch system, I didn't think it was viable to keep the old config, as well.

In the meantime I did a test run with a reduced data set inside another VirtualBox host. The whole thing worked reasonably well and the system was back up soon enough. I'll probably do the real setup at the next weekend. Let's see, what kind of problem reality will have in store for me.

As for your XFS point:

You're right. XFS isn't a bad FS. That's not what I'm saying at all.

I'm only having two problems with XFS: Someone stupid (namely me) decided to use a not so elegant partition scheme on a certain disk set, relevant to my current problem. AND: you still can't shrink an XFS partition. The latter would be needed to correct my partition scheme problem.

Regards, Jens

Jens Clasen · 2018-02-16 20:40:23

I just wanted to let you know, that my in-place upgrade was done successfully.

This script did the job in the end:

#! /bin/bash
#### Manual steps ####
#1) Boot into liveCD
#2) prepare mount points & network (outside of script)
#mount /dev/sda3 /mnt/
#mount /dev/sda1 /mnt/boot
## configure networking
#if possible:
#mount -t cifs \
# -o username=***,password=***,uid=1000,gid=1000,file_mode=0660,dir_mode=0770 \
# //192.168.1.5/backup /mnt/mnt/gentoo.old
#if not, simply let it be and transfer later
#

#3) Automatic preparations

# Test for available sources, targets and work locations: 
echo "Testing if relevant locations can be accessed"
[[ ! -d /mnt ]] && echo "Missing mount point - even though I don't know how that would be possible" && exit
[[ ! -d /mnt/mnt ]] && echo "Missing working location" && exit
[[ ! -d /mnt/mnt/arch ]] && echo "Missing source data" && exit
[[ ! -d /mnt/mnt/gentoo.old ]] && echo "Missing backup data" && exit

# Test moving dirs and files from arch to future /
echo "Testing the install process..."
cd /mnt/mnt/arch 
mv testdir /mnt/
[[ ! -d /mnt/testdir ]] && echo "Failed moving testdir" && exit
echo "Testdir moved. Continuing with further tests..."
[[ ! -f /mnt/testdir/testfile ]] && echo "Failed moving testfile" && exit
echo "Testfile found. Continuing..."
[[ ! -f /mnt/testdir/.testfile ]] && echo "Failed moving hidden testfile" && exit
echo "Hidden testfile found..."
echo "Install process test complete. Source can be read and target can be written"

# Test moving dirs and files from future / to backup
echo "Testing the backup process..."
cd /mnt
mv testdir /mnt/mnt/gentoo.old
[[ ! -d /mnt/mnt/gentoo.old/testdir ]] && echo "Failed moving testdir" && exit
echo "Testdir moved. Continuing with further tests..."
[[ ! -f /mnt/mnt/gentoo.old/testdir/testfile ]] && echo "Failed moving testfile" && exit
echo "Testfile found. continuing..."
[[ ! -f /mnt/mnt/gentoo.old/testdir/.testfile ]] && echo "Failed moving hidden testfile" && exit
echo "Hidden testfile found..."
echo "Backup process test complete. We can remove data from target and backup location is writable."

#4) For each of these entries in /mnt do
echo "Moving old gentoo installation to backup location now..."
cd /mnt
# move regular directories to backup, starting with something unimportant
dirs=( "tmp" "bin" "etc" "lib32" "lib64" "media" "opt" "sbin" "usr" "var" )
for dir in "${dirs[@]}"
do
    echo "Moving $dir to backup"
    [[ ! -d $dir ]] && echo "$dir not found!" && exit
    mv "$dir" /mnt/mnt/gentoo.old/
    [[ ! -d /mnt/mnt/gentoo.old/$dir ]] && echo "Failed moving $dir" && exit
    echo "$dir move complete"
done
echo "bin, etc, lib32, lib64, media, opt, sbin, usr and var should have been moved"

# special treatment
# move away the content of boot, but leave the directory in place
echo "Moving /boot/* to backup"
[[ ! -d boot ]] && echo "boot not found!" && exit
mkdir /mnt/mnt/gentoo.old/boot
[[ ! -d /mnt/mnt/gentoo.old/boot ]] && echo "failed creating backup target for boot" && exit
mv boot/* /mnt/mnt/gentoo.old/boot/
[[ ! -f /mnt/mnt/gentoo.old/boot/gentoo.igz ]] && echo "A file in boot backup is missing. Aborting here" && exit
echo "Boot partition backup complete"

# copy content of root but leave the data in place
echo "Copy /root contents to backup"
[[ ! -d root ]] && echo "root not found!" && exit
mkdir /mnt/mnt/gentoo.old/root
[[ ! -d /mnt/mnt/gentoo.old/root ]] && echo "failed creating backup target for /root" && exit
cp -r root/. /mnt/mnt/gentoo.old/root
[[ ! -f /mnt/mnt/gentoo.old/root/xorg.conf.new ]] && echo "A file in root backup is missing. Aborting here" && exit
echo "Root directory backup complete"

# leave these untouched: proc,dev,mnt,sys,home

# symbolic links (just one)
echo "Moving symbolic link for lib"
[[ ! -L lib ]] && echo "lib not found!" && exit
mv lib /mnt/mnt/gentoo.old/
[[ ! -L /mnt/mnt/gentoo.old/lib ]] && echo "lib symlink not found on backup!" && exit
echo "Move to backup complete..."

#5) For each of these entries in /mnt/mnt/arch do: 
echo "Starting arch install..."
cd /mnt/mnt/arch

# move regular directories to target, starting with something unimportant
dirs=( "tmp" "etc" "opt" "srv" "usr" "var" )
for dir in "${dirs[@]}"
do
    echo "Moving $dir"
    [[ ! -d $dir ]] && echo "$dir not found!" && exit
    mv "$dir" /mnt/
    [[ ! -d /mnt/$dir ]] && echo "Failed moving $dir" && exit
    echo "$dir moved"
done
echo "tmp, etc, opt, srv, usr, and var should have been moved"

# special treatment
# copy content but leave directory
echo "Installing boot data"
[[ ! -d boot ]] && echo "boot not found!" && exit
cp -r boot/. /mnt/boot
[[ ! -f /mnt/boot/vmlinuz-linux ]] && echo "A file in boot target is missing. Aborting here" && exit
echo "Boot partition install complete"

# symbolic links
echo "Moving symbolic links..."
slinks=( "bin" "lib" "lib64" "sbin" )
for slink in "${slinks[@]}"
do
    echo "Moving $slink"
    [[ ! -L $slink ]] && echo "$slink not found!" && exit
    mv "$slink" /mnt/
    [[ ! -L /mnt/$slink ]] && echo "Failed moving $slink" && exit
    echo "$slink moved"    
done
echo "bin, lib, lib64 and sbin should have been moved"

echo "Creating /run..."
mkdir /mnt/run
[[ ! -d /mnt/run ]] && echo "/run not created!" && exit
echo "done"

cd /mnt

# These should still available: proc,dev,mnt,sys,home
echo "Let's make sure, if the few untouched directories are still available..."
dirs=( "proc" "dev" "mnt" "sys" "home" "run" )
for dir in "${dirs[@]}"
do
    echo "Testing $dir"
    [[ ! -d $dir ]] && echo "$dir not found!" && exit
    echo "$dir found"
done
echo "Paranoia test completed succesfully"

# 6) setup:
# copy fstab from old install
echo "Fetching fstab from backup..."
cp /mnt/mnt/gentoo.old/etc/fstab /mnt/etc/fstab
[[ ! -f /mnt/etc/fstab ]] && echo "Failed" && exit
echo "Done"
echo "Fetching resolv.conf from live cd..."
cp /etc/resolv.conf /mnt/etc/
[[ ! -f /mnt/etc/resolv.conf ]] && echo "Failed" && exit
echo "Done"


#### MANUAL STEPS ONCE AGAIN: ####
echo "Next steps:"
echo "$ arch-chroot"
echo "$ grub-install --target=i386-pc /dev/sda"
echo "$ grub-mkconfig -o /boot/grub/grub.cfg"
echo "run chgrp -R for all directories in /home"
echo "run chown -R for all directories in /home"
echo "$ reboot"
echo "$ mkinitcpio -p linux"#! /bin/bash

I don't think, the script will be usable for anybody without exactly the same setup. But if somebody can use it for anything... Well, feel free to do so.

Regards, Jens

Last edited by Jens Clasen (2018-03-04 23:09:39)

Arch Linux

#1 2018-02-01 22:22:00

[Solved] Advice on scripted in-place install/upgrade

#2 2018-02-03 22:34:32

Re: [Solved] Advice on scripted in-place install/upgrade

#3 2018-02-06 18:45:31

Re: [Solved] Advice on scripted in-place install/upgrade

#4 2018-02-16 20:40:23

Re: [Solved] Advice on scripted in-place install/upgrade

Board footer