My way of using an ssd to cache hdd reads... use NFS (!)

kokoko3k · 2017-01-04 15:25:12

I've just copied my arch system to a new ssd, the rest of the partitions still lives on the HDD.
ssd is 120GB, and the arch installation is just 20GB, so i've plenty of space to use, but my /home dir wouldn't fit, so i started to look around at different caching solutions (bcache,sd-cache,userspace fuse implementations), but really none of them fitted my needs, so i started to think outside of the box and saw that Linux indeed has a FS-CACHE that does caching using simple folders, but only for nfs and afs shares.
So, what stops me from running my own nfs server and client on the same machine?

What i want to cache is my /home partition on HDD, so what i did is:
* Install needed packages:

pacman -S nfs-utils
yaourt -S --noconfirm cachefilesd #<-- that should live in official repos, really...

* Made my /etc/exports file:

/home                           127.0.0.1(rw,fsid=1,no_subtree_check,no_root_squash)

* Modified /etc/sysconfig/nfs to only use protocol 4 (not really needed):

RPCNFSDARGS="-N 2 -N 3"

In the same file, i lowered NFSD_V4_GRACE and NFSD_V4_LEASE to 10 (seconds), or nfs4 will hangs for 90 seconds after every server restarts, i've to investigate that.

* Started relevant systemd services:

# systemctl restart nfs-config
# systemctl start nfs-server.service
# systemctl enable nfs-server.service

* Modified /etc/cachefilesd.conf so that "dir" directive points somewhere on the ssd, of course, and started the service

systemctl start cachefilesd

* Added the following to /etc/fstab

127.0.0.1:/ /mnt/cached nfs rsize=1048576,wsize=1048576,async,vers=4,fsc,x-systemd.requires=nfs-server.service       0 0

The rest will be a matter of playing with bind mounts.

I'm still on a test system now, so i tried to use an HDD to cache another HDD, it is working fine and the cache dir is filling up;
I noticed that when a file changes on the server and another copy is requested by the client, the old file is not deleted, but 2 (or multiple) copies of that file (current and outdated) are kept into the cache; that should not be a problem if the cache clean process is aware of that, i still don't know.

Anyway, i made quick benchmarks via dd to see how much overhead nfs would introduce.

Copying from /dev/zero to the real device reached about 113MBps, while using the nfs "wrapped" one ended up to 97MB, i think it is still good.

Some things remains to me to explore:
1) Are there any other optimizations i can do to the nfs mounts?
2) This setup doesn't need network at all, so i'm worried by what i read on the wiki:

The _netdev option tells the system to wait until the network is up before trying to mount the share. systemd assumes this for NFS, but anyway it is good practice to use it for all types of networked file systems

How can i tell systemd to not wait for network? I already tried no_netdev, but that doesn't seems to work in fstab...
This can be a problem.
3) Anybody has experience in how to tweak /etc/cachefilesd.conf to limit space used?

Thanks for reading.

Last edited by kokoko3k (2017-01-05 09:18:21)

Trilby · 2017-01-04 16:04:18

120GB is a lot of space. If the majority of the data in your home directory is in a data or media or similar directory, you could have your home directory (with dot files and configs) on the SDD, and just mount a data/media partition from the hard drive under your home.

kokoko3k · 2017-01-04 17:17:57

Well yes, that is an alternative, but i plan to move games to the ssd, so that space will not be that much (what they call AAA games can even take 40GB and cachefilesd should take care of resizing the cache to leave a defined percent of space free).

Arch Linux

#1 2017-01-04 15:25:12

My way of using an ssd to cache hdd reads... use NFS (!)

#2 2017-01-04 16:04:18

Re: My way of using an ssd to cache hdd reads... use NFS (!)

#3 2017-01-04 17:17:57

Re: My way of using an ssd to cache hdd reads... use NFS (!)

Board footer