So I rm -rf /etc today, but managed to recover, here's how

bruenig · 2008-12-15 04:36:42

Today I was trying to rm -rf *.pacsave in /etc but of course put a space between the * and the .pacsave. Now before today if someone had said they deleted /etc, I would say "have fun reinstalling." Reinstalling is obnoxious and so I thought I would try to recover it all, the following is how to do that.

Now my circumstances were ideal, I was using screen, had a root terminal going and irssi going pointed to #archlinux on freenode. The first thing to know if you ever remove /etc, is DO NOT get out of your root terminal. If you get out of it, you are done. There is no switching back into root because there is no root user anymore. The basic strategy I decided to use was to just reinstall everything that ever had a file in /etc with pacman.

To do this, one needs a pacman.conf, this is where #archlinux came in handy, someone was able to upload a pacman.conf that I I could download via the ip address with wget and dump into /etc, some basic editing to it and it was good to go. But, pacman needs to be able to resolve the mirrors in pacman.conf, so I made an /etc/resolv.conf and added the opendns name servers to the file. I tested pacman and lo and behold it worked.

From here I used some bash+pacman cleverness to isolate only packages that had stuff in /etc and had pacman reinstall all of them (113 in total on my system). Now from messing up the first time, the advice I would give to anyone else here is to install filesystem and initscripts first because some packages try to create groups and if you don't have the files in place the installations fail miserably. Also, it is a good idea to reinstall all of them twice so that any errors caused by missing /etc files are gone the second time around.

A fairly hackish one time way to get a list of packages with /etc files.

for file in $(pacman -Q 2>/dev/null | awk '{print $1}'); do pacman -Ql $file 2>/dev/null | grep -F '/etc/'; done | grep -v '^error' | awk '{print $1}' | sort -u | tr '\n' ' ' > thing.list
pacman -S $(<thing.list)

This downloading took considerable time which gave me a chance to create a checklist of what I needed to do before reboot so as to make sure the system still worked. I needed to create a root password, needed to create my user and its groups, I needed to edit rc.conf to add all of my settings, I needed to edit fstab to put in my drives, I needed to edit inittab, sudoers, basically everything that you have ever edited in etc needs to be done again. Prioritize important stuff first like fstab, but eventually you are going to have to get around to all of it. Remember the cardinal rule in all of this, do not get rid of the root terminal until you are certain it will work, if you do, it is game over.

At this point, just reboot and hope you didn't screw up. You probably did somewhere, hopefully it isn't critical and you can just correct it in tty.

Last edited by bruenig (2008-12-15 04:40:13)

fukawi2 · 2008-12-15 04:44:53

Nice effort... Good to see someone doens't give up too easily

bruenig wrote:

Remember the cardinal rule in all of this, do not get rid of the root terminal until you are certain it will work, if you do, it is game over.

Just to be absolutely correct... There would be nothing stopping you from booting from the install CD/USB and then chrooting in to your damaged installation.

Personally, I'd just go:

mount -t nfs server:/srv/BackupData/servers/ /mnt/backup
cd /
tar xvzf /mnt/backup/hostname-LastNightsDate.tar.gz /etc/*
*wait while we travel back in time*
umount /mnt/backup

Last edited by fukawi2 (2008-12-15 04:45:12)

Ranguvar · 2008-12-15 04:54:02

Personally, I would have STOPPED, powered off (power button, don't risk writing to the hard drive), and used SystemRescueCd to try and recover. A simple delete is very, very often 100% recoverable _if_ you don't write to the disk after. The space is only marked as empty, but it's not until some app needs a place to write and it comes across that space.

louipc · 2008-12-15 05:24:13

Hah hah. Nice adventure.
Personally I would just run `git reset --hard`

.:B:. · 2008-12-15 05:30:48

What is this world coming to bruenig? A helpful post from you?

It's nice to see you made it though, but I didn't doubt that you'd succeed .

fukawi2 wrote:

Nice effort... Good to see someone doens't give up too easily
Personally, I'd just go:
mount -t nfs server:/srv/BackupData/servers/ /mnt/backup
cd /
tar xvzf /mnt/backup/hostname-LastNightsDate.tar.gz /etc/*
*wait while we travel back in time*
umount /mnt/backup

The thing is, he had no backup . I'm not sure whether he learnt anything from this experience other than some more awk and grep magic .

fukawi2 · 2008-12-15 10:41:03

I think this is the way everyone learns about backups

bruenig · 2008-12-15 15:34:09

B is just a troll, can't imagine how he became a forum moderator, amirite or amirite?

pmw

sabooky · 2008-12-16 03:50:06

fukawi2 wrote:

I think this is the way everyone learns about backups

Sigh.. after losing a big chunk of /etc once. (wasn't my fault package manager blew things up).
And completely frying my ~ directory with a lot of work on there (that I never recovered.. not even sure what all was on there..)

You'd think that I'd learn to backup.. but nope.. here I am saying I'll set it up "tomorrow"... sigh

PS: I've though about making a git repo out of /etc and syncing that to my server.. maybe one day (read: "tomorrow" aka never) I'll do this..

.:B:. · 2008-12-16 05:24:06

I know some Arch users do use version control for their /etc and even ~ . Don't have any names though...

Svenstaro · 2008-12-16 09:01:30

Ranguvar wrote:

Personally, I would have STOPPED, powered off (power button, don't risk writing to the hard drive), and used SystemRescueCd to try and recover. A simple delete is very, very often 100% recoverable _if_ you don't write to the disk after. The space is only marked as empty, but it's not until some app needs a place to write and it comes across that space.

Assuming it was EXT3, how would you do that?

ninian · 2008-12-16 09:38:11

Svenstaro wrote:

Assuming it was EXT3, how would you do that?

+1, would you use TestDisk from the SystemRescue CD?

Dieter@be · 2008-12-16 15:47:42

B wrote:

I know some Arch users do use version control for their /etc and even ~ . Don't have any names though...

Me! my /etc is in svn, which works fine. My ~ is also in svn, I should migrate that go git once.

Ranguvar · 2008-12-17 01:26:03

Likely PhotoRec (misnomer) or TestDisc, yeah. Note that I've never used them.

sokuban · 2008-12-17 02:38:28

Nice recovery. I always find reading these stories interesting.

Though I wish I saw that hackish way to install all the packages in /etc. The other day I did a rm -r /usr/share (I don't have -i aliased so I don't need to put -f to use the deadly command.) which isn't nearly as bad as /etc, but I still went and reinstalled every package. Would have shaved a couple of min (and have been cooler) if I had only done so for the packages in /usr/share.

That was the first accidental rm -rf in my life. I learned to "cd before you rm -rf" the hard way :P

Last edited by sokuban (2008-12-17 02:45:39)

lldmer · 2008-12-17 02:43:12

Dieter@be wrote:

B wrote:
I know some Arch users do use version control for their /etc and even ~ . Don't have any names though...
Me! my /etc is in svn, which works fine. My ~ is also in svn, I should migrate that go git once.

I have my config files in git as well. Copies of them though, because having your /etc/ in git wouldn't save you when you remove /etc/.git as well:P

vogt · 2008-12-18 05:47:49

lldmer wrote:

I have my config files in git as well. Copies of them though, because having your /etc/ in git wouldn't save you when you remove /etc/.git as well:P

You forgot that the etc repo is probably going to be mirrored somwhere, so you're probably only a `git clone' away from getting a recent backup of /etc.

Then again, I don't really see what advantages a revision control system has over regular backup systems (ex rdiff-backup) for plain old config files. To me it just seems like its one less interface to learn.

Maki · 2008-12-19 01:40:53

So the lesson here is why use alias when all the fun is repairing stuff

Arch Linux

#1 2008-12-15 04:36:42

So I rm -rf /etc today, but managed to recover, here's how

#2 2008-12-15 04:44:53

Re: So I rm -rf /etc today, but managed to recover, here's how

#3 2008-12-15 04:54:02

Re: So I rm -rf /etc today, but managed to recover, here's how

#4 2008-12-15 05:24:13

Re: So I rm -rf /etc today, but managed to recover, here's how

#5 2008-12-15 05:30:48

Re: So I rm -rf /etc today, but managed to recover, here's how

#6 2008-12-15 10:41:03

Re: So I rm -rf /etc today, but managed to recover, here's how

#7 2008-12-15 15:34:09

Re: So I rm -rf /etc today, but managed to recover, here's how

#8 2008-12-16 03:50:06

Re: So I rm -rf /etc today, but managed to recover, here's how

#9 2008-12-16 05:24:06

Re: So I rm -rf /etc today, but managed to recover, here's how

#10 2008-12-16 09:01:30

Re: So I rm -rf /etc today, but managed to recover, here's how

#11 2008-12-16 09:38:11

Re: So I rm -rf /etc today, but managed to recover, here's how

#12 2008-12-16 15:47:42

Re: So I rm -rf /etc today, but managed to recover, here's how

#13 2008-12-17 01:26:03

Re: So I rm -rf /etc today, but managed to recover, here's how

#14 2008-12-17 02:38:28

Re: So I rm -rf /etc today, but managed to recover, here's how

#15 2008-12-17 02:43:12

Re: So I rm -rf /etc today, but managed to recover, here's how

#16 2008-12-18 05:47:49

Re: So I rm -rf /etc today, but managed to recover, here's how

#17 2008-12-19 01:40:53

Re: So I rm -rf /etc today, but managed to recover, here's how

Board footer