Arch Linux

nicky7

Member

Registered: 2010-02-22

Posts: 3

Accidentally overwrote /etc from another server, tips on proceeding?

I done goofed

Over a decade administering linux servers and this is by far the worst mistake I've made. I have a virtualization server (VT1) with 2 "production" VMs: a file server (NFS1) and another (LNS1) for various local network services and dev'ing environments. The latter broke hard after a system update and I eventually decided to rebuild it. To ensure I had the freshest backups of the two main volumes of LNS1, I quickly ran a couple rsyncs from within the backup directory:

# rsync -ax lns1:/home/ home/

and then for the / dir:

# rsync -ax lns1:/ /

... FFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUU-, I forgot a period! By the time I realized my mistake, /etc was already copied over. No biggie, I have everything in /etc/ under a git repo... which is now copied over! Well, I have regular backups... oops, that's just of /home FS, not the root FS

So multiple mistakes and I now have 2 servers to rebuild and an audit of my backup scripts.

VT1 wasn't running anything outside of qemu-kvm really, and those are on a separate FS and are backed up, so it shouldn't be too bad to rebuild. I also have to remember to bridge the nics, setup the mounts (some are passthrough to VMs)...

* /usr, /lib, /etc were all destroyed
* /var, /home, /boot, /tmp are all safe (different FSs)
* no vim swap files

Onto my questions

I doubt anything short of a fresh install is going to fix this, but is there anyway to pull any of the configs from memory, or some other output, rebuild them etc.? Any tips on recovering anything would be helpful in rebuilding the new one. Additionally, I'll be wiping the root FS clean, but I could just leave the others intact but can I force the -f flag on the pacman installer? Or should I try chrooting and installing everything manually (i.e. not use the arch installer).

My plan

Already done:

• full backups of current state.

• saved pacman -Q output saved

• blkid output saved

• ifconfig output saved

To do:

• arch boot

Fork in the road:
  If I can force the -f flag on the pacman installer, then I can just wipe the root FS and install (leaving some of my log files for example). Otherwise, I'll either mkfs or rm each parititon before using the installer

Continued:

• install packages from pacman output

• configure networking based on ifconfig output

• configure fstab based on df (mount -l produces errors currently)

• additional config, bootloader, ssh, etc.

• add users

• rsync /home (user dirs and VM dirs)

• diff on / and backup / to find unique files (for example, the VMs get started with their own /etc/rc.d/vm_nfs1 script.) and copy those over

• maybe copy over /var/cache and /var/logs

• tidy, test, debug and done.

Last edited by nicky7 (2012-03-25 17:45:25)

nomorewindows

Member

Registered: 2010-04-03

Posts: 3,362

Re: Accidentally overwrote /etc from another server, tips on proceeding?

/var is still there which means your package files and your pacman database is still intact leaving the ability to recover exactly what programs you had still leaving /etc out.  It will leave you with default configs.  pacman -Qk > some_file will tell you which programs were completely hosed (missing files).  However, this is if you can still run pacman.

---

I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

zero_one

Member

Registered: 2010-07-07

Posts: 104

Re: Accidentally overwrote /etc from another server, tips on proceeding?

I didnt see this mentioned; have you tried to recover anything with testdisk?