Installing system with Btrfs, managing root snapshots

Liquid5 · 2013-10-28 12:55:41

Hi,

In the near future I would like to install Arch or an Arch based distro (the jury is still out on that one, since I did have some problems installing it, but that's another issue). I would like to be able to rollback from an upgrade-gone-bad (which happened to my current Xubuntu system, whee upgrading resulted in some glitches) or any of the other reasons that a person would have to require making snapshots and be able to restore the system to its previous state.

Now the problem for me isn't partitioning or installing, but the way that I should proceed in managin snapshots. I have looked at the btrf wiki, but everything is still not very clear to me, hence the reason I am posting this thread.

As I said, I want to be able to rollback my root system (/). I will probably have a separate home partition. As far as I understand, I should modify my fstab file in order to set the default subvolume. The example given on the btrfs wiki (Sysadmin Guide) is:

LABEL=btr_pool / btrfs defaults 0 0
LABEL=btr_pool /home btrfs defaults,subvol=home 0 0
LABEL=btr_pool /media/btrfs btrfs defaults,noauto,subvolid=0 0 0

If I understand correctly, the last Label creates a subvolume in the root filesystem? Does this mean, that the system will automatically mount the subvolume that is set to 0 at boot? At least that's what I can infer form the document on the Oracle website (http://docs.oracle.com/cd/E37670_01/...se3_btrfs.html), which says: "By default, the operating system mounts the parent btrfs volume, which has an ID of 0, unless you use set-default to change the default subvolume."

Okay, let's say I make a snapshot of the system before upgrading. The btrfs site suggests:

# mount /media/btrfs
# cd /media/btrfs
# btrfs subvolume snapshot root root_snapshot_2011_01_11
# cd ~
# umount /media/btrfs

As I can understand, mounting the /media/btrfs subvolume wihtout any further options automatically mounts the root filesystem? Also the "btrfs subvolume snapshot" command automatically recognizes "root" as / ?

Now suppose an upgrade went bad. How to rollback? The btrfs wiki suggests a solution, but instead of using the root filesystem as the example, it goes with the /home partition:

# mount /media/btrfs
# umount /home
# mount -o defaults,subvol=home_snapshot_A /dev/sda /home
# btrfs subvolume delete /media/btrfs/home # optional; this is so the
# mv /media/btrfs/home_snapshot_A /media/btrfs/home # /etc/fstab need not change.
# umount /media/btrfs

Now since I am still concerned with rolling back the root partition, I guess that I do not need to perform steps 2 (umounting home) and 3 (overriding the default subvolume mount) and just use the mv command to rename the root_snapshot to the name root?

Is that how this process is supposed to be done?

I tried to do it this way (minus the fstab entries) in one of my Vbox installations, but I foud out, that using the mv command just sent my PC into a 20 min high-cpu activity that ate up all most of my designated virtual hard drive space.

Also, since Btrfs uses copy-on-write, does this mean, that snapshots initially do not consume a significant amount of space, but that this increases with changes made to the system?

WonderWoofy · 2013-10-28 15:19:19

I think the part that you are missing is that the root of the filesystem in that example is actually on a subvolume itself.

So it sounds as though the root of the btrfs filesystem as a whole (subvolid=0) is mounted at /media/btrfs. So from there it seems as though that example is indicating that from that subvolid=0, you then have something like /media/btrfs/root and /media/btrfs/home. So if you wanted to mount the rootfs, you would have to mount with subvol=root in the options. If you wanted to mount /home you would need subvol=home.

In order to roll back the rootfs, set the fstab up similar to the example above. Then you can tell the filesystem which subvol you want mounted by default if you just mount it (disabling the need for subvol=<whatever>). You can do this with btrfs subvolume set-default <subvolid> <path>. So if /media/btrfs/root has a subvolid of 5, I would do btrfs sub set 5 /media/btrfs.

In the event that you want to boot with a snapshot, you would use the kernel command line to give it a specific subvol to be mounting. So way in that example above, you wanted to roll back to the root_2011_01_11 snapshot you made. In the btrfs filesystem you would essentially have /root, /root_2011_01_11, and /home. So in the bootloader, use whatever key allows you to edit the kernel command line, and then somewhere on there, you would add rootflags=subvol=root_2011_01_11. That would override the set default subvolume and mount the one you want instead.

The reason you want to avoid the fstab is beacuse the information in the fstab is not initially used for mounting the rootfs. To start the bootloader loaders the kernel and uses the kernel command line to also find and load the initramfs. The kernel (and initramfs) is then responsible for initializing all the hardware it can support, which includes the drivers/modules for both the drive interface as well as the filesystem that sits on them. It mounts the rootfs, does a switch_root, then runs /sbin/init. But it isn't until the rootfs is actually mounted, that it can actually get the info in the fstab. So at some point before full bootup, it will do a mount -o remount / which presumable would be really f*cked up if it suddenly told it to mount a different subvol.

Keep in mind that these threads are really for Arch Linux support only. I personally found this question generic enough that it didn't bother me, and is also information that is necessary in order to actually install the damn system. So presumably it is info that might be imprtant before a potential Arch user even becomes an Arch user. But just know you are walking a fine line here.

It sounds like you are a virtualbox user. Start up a VM with the archiso and have at it 'til it breaks. I learned Arch by installing on bare metal, and the first try was painful (I was trying to do fancy things, which wasn't a good idea). It makes me think that I should really have tried going through the install process in some kind of a sane environemnt first... meh, hindsight.

Edit: Regarding your last question about the space snapshots take, your idea there is pretty correct. Btrfs uses a b-tree filesystem, and each subvolume has its own kind of separately rooted b-tree. When you take a snapshot it just creates a new b-tree root that has pointers to the tree you just snapshotted. So it really takes almost no space whatsoever initially. As you start writing to it, it will write the differences and the pointers in the tree will reroute to reflect those changes. But the old data (in the other subvol/snapshot) will stay in tact.

Last edited by WonderWoofy (2013-10-28 15:23:10)

Liquid5 · 2013-10-28 16:28:17

Thanks a lot for your very detailed answer, I really apreciate it. I will look into editing the kernel command line, as I'm not familiar with it at all at the moment. I will look into it this evening or tomorrow, try it out and see what happenes.

In regards to walking a fine line, I was a little hesitant myself in publishing it in the Arch forums, especially since I am not at all sure if I will be installing Arch or Arch derivative. But I thought, as you mentioned, that the question is generic enough, that it would be of interest to the arch user, as it can be applied to any system and the information about this particular issue seems to be scarce on the web.I didn't want to break any forum etiquette, so my apologies if I inadvertendly had.

donniezazen · 2013-10-28 17:03:17

I don't personally use partitions and see no purpose in it. It is just waste of space. Modern personal computers don't really go bad that often. But in the end it is a personal choice.

I use a single btrfs partition with subvolumes root, home, opt, and /var/cache/pacman/pkg. You can add and remove as many subvolumes on-the-fly as you wish. I use OpenSUSE's snapper to take snapshots every hour. Older snapshots get's deleted automatically. Difference in size of two snapshots is just the amount of modified data. Home, opt, & pkg are subvolumes inside root subvolumes. I only snapshot root and recursive snapshotting is not allowed. So basically I have snapshots of essential root and not let's say home which changes a lot.

My installation method in a nutshell is as follows.

Partition 
2MB for Gurb
4GB for swap
rest for btrfs volume

mkfs.btrfs /dev/sdxY
mount /dev/sdxY /mnt
btrfs subvol create /mnt/root
btrfs subvol create /mnt/root/home
btrfs subvol create /mnt/root/opt
btrfs subvol create /mnt/root/var/cache/pacman/pkg

umount /mnt

mount -o defaults,compress=lzo,subvol=root /dev/sdxY /mnt

Installations as usual

Snapper snapshots everything in subvolumes' .snapshots folder. You just have to point to that subvolume to rollback to that time. Snapper package has instructions that will help you set it up. You can also manually create snapshots before running upgrade if that is preferable. I have been meaning to create wiki page for Snapper but it hasn't happen yet.

Here is a picture of Snapper.

WonderWoofy · 2013-10-28 18:33:32

@donniezazen,
Honestly, I would probably not just stick subvols everywhere in the filesystem itself. Rather, make a few subvols off the root (subvolid=0) and then mount them. So for example, I personally see no reason to separate /opt (there's hardly anything in there anyway), so I have /rootfs, /home, pacpkg (mounted to /var/cache/pacman/pkg), and /archbuild (which is used by devtools for build chroots and mounts to /var/lib/archbuild).

I think that creating true separation like that makes it not only incredibly easy to share subvols between systems (less path to type out/remember), but it also allows for easier replacement of those sections as well. So if for example, I wanted to replace /home with a snapshot. If I had it at /rootfs/home, then I would have to remove /rootfs/home and then snapshot (or move a snap) to that location. With it at /home, I just have to specify that I want to mount -o subvol=home-snapshot /dev/sdXY /home (of course it would be a modification to fstab).

BTW, from the OP's first post it rather sounds like he/she is fully embracing the idea of a single partition with subvols anyway, so I don't think there is a need to convince anyone of that advantage here.

@OP, I think that what really helped me with getting a grasp of the btrfs basics was reading the two articles by Margaret Bierman with Lenz Grimmer on the front page of the btrfs wiki (the real btrfs wiki, not the Arch page). It just walks you through some of the more "basic" advanced features and capabilities of the filesystem. Though there is a hell of a lot more than btrfs is capable of.

teekay · 2013-10-28 20:03:18

donniezazen wrote:

I have been meaning to create wiki page for Snapper but it hasn't happen yet.

I went ahead and started one: https://wiki.archlinux.org/index.php/Snapper
Please proof-read, fix and contribute

On topic: my rootfs setup looks pretty much like WonderWoofy's: subvolumes for rootfs, home, pkgs (beside the fact that home is split in some more subvolumes for things i definately do not want to get snapshotted like ~/.cache and ~/Downloads etc). On the data disks there's a lot more of course - but those are seperate BTRFS filesystems on other storage areas. With all of them I'm following that "true separation" pragma, but I think this is a matter of taste. There is nothing wrong with doing it like donniezazen did - the most important thing is that / should actually be a subvolume.

Last edited by teekay (2013-10-28 20:18:13)

donniezazen · 2013-10-29 02:32:18

WonderWoofy wrote:

@donniezazen,
Honestly, I would probably not just stick subvols everywhere in the filesystem itself. Rather, make a few subvols off the root (subvolid=0) and then mount them. So for example, I personally see no reason to separate /opt (there's hardly anything in there anyway), so I have /rootfs, /home, pacpkg (mounted to /var/cache/pacman/pkg), and /archbuild (which is used by devtools for build chroots and mounts to /var/lib/archbuild).

I am not sure what you mean by subvols off the root. How have you partitioned your hard drive? Where do you make /rootfs, /home, pacpkg and /archbuilds subvols? I haven't fully looked into fs to decide what should be snapshotted and what shouldn't. I make /opt subvolume because a lot of AUR packages are installed in there which have nothing to do with system itself. My btrfs knowledge is pretty lame though. It's mostly reading here and there off blogs/posts. Although btrfs IRC guys are very helpful.

WonderWoofy · 2013-10-29 02:46:14

@donniezazen, I have one btrfs filesystem that is raid1 over 119.2GB (one disk is a 128GB Samsung 830 and the other is a 250GB Samsung 840). So I have it set up linke this:

/  ← The root of the total btrfs filesystem (subvolid=0)
├── archbuild
├── home
├── pacpkg
├── rootfs
└── snaps

So then I have something like this in my fstab:

LABEL=archlinux   /                       btrfs   autodefrag,noatime                                             0   0
LABEL=archlinux   /home                   btrfs   autodefrag,noatime,subvol=home                                 0   0
LABEL=archlinux   /var/lib/archbuild      btrfs   autodefrag,noatime,noauto,x-systemd.automount,subvol=archbuild 0   0
LABEL=archlinux   /var/cache/pacman/pkg   btrfs   autodefrag,noatime,noauto,x-systemd.automount,subvol=pacpkg    0   0

Of course the rootfs has no subvol set so that I can put rootflags=subvol=rootfs on the kernel command line. So if for example I snapshot subvol=home to snaps/home@2013-10-28, then it is just a matter of changing the second line of that example fstab with something like:

# sed -i 's|\(subvol=\)home|\1snaps/home@2013-10-28|' /etc/fstab

So that way I don't have to (re)move the subvol from /home and replace it with a moved/snap of snaps/home@2013-10-28. Does that make a bit more sense?

Edit: Its funny because today I was just commenting to a friend and fellow btrfs user about how amazingly helpful the devs on #btrfs are compared to some other dev channels.

Last edited by WonderWoofy (2013-10-29 02:47:30)

donniezazen · 2013-10-29 03:01:40

@WonderWoofy That makes a lot of sense. Thanks.

In my setup, all snapshots are stored in .snapshots folder in the subvolume. If I want to boot any other snapshot I just have to point to it in grub kernel line. This should work well for fixing system. I currently am only taking snapshots of root, so, your setup might be cleaner in a multi-subvolume-snapshot-system scenario.

WonderWoofy · 2013-10-29 03:15:32

donnizazen wrote:

...so, your setup might be cleaner in a multi-subvolume-snapshot-system scenario.

Yeah, that is what I was trying to get at there... thanks for summarizing.

donniezazen · 2013-10-29 04:43:25

@teekay Thanks for doing that. It looks good.

Liquid5 · 2013-10-29 09:38:01

Thanks a lot to everyone for all the additional advices. I'm still taking it all in (my concentration is unfortunately split between this and another full time project that is completely non-linux and non-computer related, so it takes me a little bit longer to comprehend everything) so I probably still missed some details of how to do things, as illustrated with my next example.

First of all, after installing a system (i did do a derivative install, but I will try with arch net install next; I've been avoiding pure arch for just this moment, because I had repeated localization setting issues in the past) with / and /home btrfs partitions, I get thethe GRUB bug: "sparse file not allowed", which can supposedly be solved quite easily (http://askubuntu.com/questions/100329/m … filesystem).

Then it fails to mount the home partition and keeps putting an Unknown tag in fstab (I change it to btrfs, but then it doesn't mount it anyway).

Also I am not 100% clear on the Fstab edit: do I just put the Labels after the usual stuff, where mount points are specified with their UUID numbers and mount options, or do I have to completely replace the original with the Labels?

Also, I've read that, when installing a btrfs system, it is a good idea to make a separate boot partition, which is not formated with btrfs in order to make booting less of a problem (especially with UEFI; I still have BIOS, but nevertheless, might be worth considering in the future).
But I'm guessing since the boot partition then contains files that are changed during major upgrades (for example kernel modules) and it's probably necessary that they correspond to the actual state of the system in order to boot. If you have an UEFI based computer, how do you solve that problem? I guess that you could, for example, rebuild mkinitcpio and the corresponding initramfs images, but how would you do that before the boot process itself (is there an option in the boot menu for that?)

Last edited by Liquid5 (2013-10-29 09:39:41)

donniezazen · 2013-10-29 11:16:35

You are suppose to run genfstab once and then edit /etc/fstab as needed. I prefer UUIDs. You can get more info about mount points from man pages. You may comment out things that you see troublesome and try things make sense.

Btrfs supports booting root without a dedicated root partition. I don't have a dedicated boot partition on a legacy BIOS machine. UEFI requires a FAT32 partition. My machine supports both UEFI and BIOS, I use BIOS.

WonderWoofy · 2013-10-29 16:18:07

Yeah, it is a personal preference whether you use UUID's or Labels. But with btrfs, you should use one of those and not the kernel namespace (ie. /dev/sda1). This is because of how /dev/btrfs-control handles multi-device btrfs filesystems, as well as the always present possibility of your disks not being named persistently by the kernel.

If you have udev (or system) on your system, and in the initramfs, you can use /dev/disk/by-label/<label> or /dev/disk/by-uuid/<uuid>. But I always use LABEL=<label> or UUID=<uuid>, as it just seems a bit more universal to me.

I use UEFI, and on in my MBR, I also have syslinux installed as a backup solution. They can coexist without issue. In reality, I just wanted to have syslinux so that I could make use the the memdisk feature, which allows one to boot an iso as though it were the initramfs. Unfortunately, the memdisk functionality has not (yet?) been ported to UEFI. If you are going to use grub with GPT, you will need to make a 1-2MB partition with gdisk type code ef02. This is because grub has bloated to the point where it needs extra space to install its core.img to. So with MBR partitioning, it would use the space where the GPT would live (or some of it at least). On my other system that is legacy bios only, I just stuck a partition slightly larger than 1MB at the very end of the disk. That way, if I ever switch bootloaders, I can simply remove that partition, expand the btrfs partition and let the filesystem fill to the end.

Liquid5 · 2013-10-31 14:23:24

Ok, I'll try to do a VB install – where I'll definitely take into consideration all of the good advice given here – in a couple of days, when I'll finally have some time. Thanks again for your replies. Until then.

Liquid5 · 2013-11-15 23:11:16

Okay, I finally found the time to really try it out.*

Firstly, there is a bug in the grub, so-to-speak, since it gives me the "Sparse file not allowed" message and then requests a button press to continue (it will boot after approx. five seconds). This is an issue that has been reported on, mainly in the Ubuntu forums, but the workaround that they provide ( Proposed Solution ) doesn't work for me, because my 00_header file does not have the lines that they recommend to comment out. How did you guys deal with this issue?

Anyway, after booting in, I created a subvol "snapshots" in /mnt, then created a snapshot of the fresh install, called "fresh-system" and put it in the snaphots subvolume. I then updated the system and created another snapshot "updated-system". Rolling back to the fresh install and forward to the updated system is easily achieved with setting default subvol IDs. So hooray for the first success!

There is another quirk: btrfs command lists the subvolumes and snapshots, however, I cannot see them in their respective folders (they are empty). Should I change some settings in order to be able to see those subvolumes?

Also after trying to reboot or shutdown computer, I am presented with a tty1 console login (this console also shows briefly at startup), that sometimes doesn't do anything and I have to reset the VM. I don't know if it's related to btrfs, but it didn't show before the conversion ...

*To be transparent (as per the advisory from the post of WonderWoofy), this is still being done on a pre-existing fresh install of an arch-derivative in VB (the reason being time constraints, as I was not able to read the whole installation guide for Arch, which is hopefully the next thing on my list ); – I posted for the same reason as in the beginning, because, it being a question of handling a specific partition type, it can pretty much apply to any distro that uses an up-to-date kernel. If I'm wrong, I apologize.

Reason for edit: threw out stuff that wasn't directly related to arch

Last edited by Liquid5 (2013-11-15 23:28:55)

Liquid5 · 2013-11-16 18:02:34

I guess the least problematic way of installing a btrfs system would be to use a separate boot partition, formatted to ext4. Then write a script that would do the root snapshot and compress the boot partition in a tar archive. Then, when I would need to restore to a previous state, I would have to set a new default subvolume id, mount the boot partition and (after backing it up) overwrite it with the archived version?

teekay · 2013-11-16 19:00:04

Liquid5 wrote:

I guess the least problematic way of installing a btrfs system would be to use a separate boot partition, formatted to ext4. Then write a script that would do the root snapshot and compress the boot partition in a tar archive. Then, when I would need to restore to a previous state, I would have to set a new default subvolume id, mount the boot partition and (after backing it up) overwrite it with the archived version?

Be aware that if you're rolling back to a state that has had an older kernel version, you'll screw up things because the kernel image in /boot is the newer one, while all the modules in /usr/lib/modules get rolled back to the previous version. BAD idea, no modules will load, system won't boot perperly. You should keep /boot inside your /-subvolume. GRUB can handle /boot on btrfs, no problem here.
AFAIK Grub can also be used in UEFI as a standalone UEFI application, whatever that means (I have a dual-BIOS board, and use legacy BIOS).

The solution of tarring up /boot together with the snapshots, and overwriting it after a restore sounds complicated, and requires you to boot off a live cd.

Last edited by teekay (2013-11-16 19:10:22)

WonderWoofy · 2013-11-16 20:20:46

Agreed, that this sounds like a big PITA when simply keeping /boot on the same subvol is so easy. I have grub set up in UEFI, it doesn't need to be make into a standalone UEFI application, though it can be. Grub definitely works just fine with btrfs. Syslinux also apparently works, as in it is able to read off of it. But I have also read that syslinux depends on certain files being at a certain location on the disk (not a certain location in the filesystem), so doing a btrfs balance is said to break things since it actually relocates just about everything on the disk.

I have also set up grub in bios mode, which works perfectly. The grub-mkconfig command is able to auto-detect the true location of the the boot/grub directory and adjusts itself accordingly.

Of course, this all assumes that you are actually using Arch Linux and its packages. It is just that I cannot comment on whatever else might have been done by other distrolets to their scripts, and whatnot. So while I commend your honesty in indicating that you are not using Arch Linux, I have to say that if you want real help from these particular forums, you need to install proper Arch Linux. It is really quite simple once you get a grasp of what you are actually doing. That process also gives you a much better understanding of how things work and are laid out, which should also give insight into how to work with btrfs and why using a bootloader with it might not be entirely straight forward.

Liquid5 · 2013-11-16 21:29:00

teekay wrote:

Be aware that if you're rolling back to a state that has had an older kernel version, you'll screw up things because the kernel image in /boot is the newer one, while all the modules in /usr/lib/modules get rolled back to the previous version. BAD idea, no modules will load, system won't boot perperly.

That's exactly why I would replace the contents of boot with the archive that was made at the same time as the snapshot that I am rolling back to. Wouldn't that mean that the boot partition would then correspond to the system state of the snapshot (previous kernel version and all)? Or am I still missing something?

WonderWoofy wrote:

Agreed, that this sounds like a big PITA when simply keeping /boot on the same subvol is so easy. I have grub set up in UEFI, it doesn't need to be make into a standalone UEFI application, though it can be. Grub definitely works just fine with btrfs.

So if I understand you corectly, grub does not complain about "sparse file not allowed" and/or does not need manual editing?

WonderWoofy wrote:

Of course, this all assumes that you are actually using Arch Linux and its packages. It is just that I cannot comment on whatever else might have been done by other distrolets to their scripts, and whatnot. So while I commend your honesty in indicating that you are not using Arch Linux, I have to say that if you want real help from these particular forums, you need to install proper Arch Linux

I agree completely with your point and will get to reading the installation wikis next week (due to some other projects I'm forced to do things in slow steps). I did try to install arch proper a couple of times before (I think I even got through the beginning installation and set-up a basic X system), but something always didn't work as well as it should; no matter I'll make it work this time .

WonderWoofy · 2013-11-16 21:33:34

Liquid5 wrote:

So if I understand you corectly, grub does not complain about "sparse file not allowed" and/or does not need manual editing?

I have never had this message/error appear before. Though admittedly, grub it my last choice in bootloaders. I use it simply to learn about it, but also because it supports booting iso files in a loopback device. So I haven't done much more than test it a few times initially and typically once after I update it.

jasonwryan · 2013-11-16 22:40:10

You have had ample opportunity to use Arch (note Arch derivatives are not Arch), and have enjoyed considerable support from the community, but our policy is quite clear:
https://wiki.archlinux.org/index.php/Fo … pport_ONLY

Closing.

Arch Linux

#1 2013-10-28 12:55:41

Installing system with Btrfs, managing root snapshots

#2 2013-10-28 15:19:19

Re: Installing system with Btrfs, managing root snapshots

#3 2013-10-28 16:28:17

Re: Installing system with Btrfs, managing root snapshots

#4 2013-10-28 17:03:17

Re: Installing system with Btrfs, managing root snapshots

#5 2013-10-28 18:33:32

Re: Installing system with Btrfs, managing root snapshots

#6 2013-10-28 20:03:18

Re: Installing system with Btrfs, managing root snapshots

#7 2013-10-29 02:32:18

Re: Installing system with Btrfs, managing root snapshots

#8 2013-10-29 02:46:14

Re: Installing system with Btrfs, managing root snapshots

#9 2013-10-29 03:01:40

Re: Installing system with Btrfs, managing root snapshots

#10 2013-10-29 03:15:32

Re: Installing system with Btrfs, managing root snapshots

#11 2013-10-29 04:43:25

Re: Installing system with Btrfs, managing root snapshots

#12 2013-10-29 09:38:01

Re: Installing system with Btrfs, managing root snapshots

#13 2013-10-29 11:16:35

Re: Installing system with Btrfs, managing root snapshots

#14 2013-10-29 16:18:07

Re: Installing system with Btrfs, managing root snapshots

#15 2013-10-31 14:23:24

Re: Installing system with Btrfs, managing root snapshots

#16 2013-11-15 23:11:16

Re: Installing system with Btrfs, managing root snapshots

#17 2013-11-16 18:02:34

Re: Installing system with Btrfs, managing root snapshots

#18 2013-11-16 19:00:04

Re: Installing system with Btrfs, managing root snapshots

#19 2013-11-16 20:20:46

Re: Installing system with Btrfs, managing root snapshots

#20 2013-11-16 21:29:00

Re: Installing system with Btrfs, managing root snapshots

#21 2013-11-16 21:33:34

Re: Installing system with Btrfs, managing root snapshots

#22 2013-11-16 22:40:10

Re: Installing system with Btrfs, managing root snapshots

Board footer