Arch Linux

brando56894

Member

From: NYC

Registered: 2008-08-03

Posts: 681

Issues With 9PFS

First off, I didn't know whether to put this here or in System Administration.

I'm running FreeNAS 10 (based on FreeBSD 11) on my home server. The OS utilizes bhyve (the BSD version of kvm/qemu) to provide VMs and Docker containers to the user in order to extend the feature set of the OS (there are no "native" plugins anymore, they have been replaced by Docker containers). Bhyve boots up a Boot2Docker VM, which FreeNAS then uses to provide container support, the sharing protocol in use is Plan 9 (9PFS) and some users (like myself) have been experiencing odd issues with Docker that are hard to resolve. The developers of FreeNAS have resolved some issues with 9PFS which makes Docker function better but some issues still remain. I mentioned that I believe the issues lie in 9PFS and not specifically in Docker, in order to prove this I created an Arch VM from one of the canned FreeNAS templates, installed Emby media server, and mounted my 9PFS shares on /mnt/multimedia (for my TV and Movies libraries) and /var/lib/emby so that I can store all the metadata in my ZFS pool instead of inside the VM.

The problem is that Emby gets about 65% through scanning my entire library (about 6300 media files) and will eventually crash with SQLite errors. It takes about 4 hours for it to get to this point. I know it is specifically an issue with 9PFS because I have another VM with Emby and Plex installed, but that VM is using NFS for everything instead of 9PFS, and I have no issues. It scans in my entire library in about 2 hours.

The real problem is that the developers can't reproduce it at all and are relying on myself and other users to provide them with useful info. I've enabled 9PFS logging for them, which I'm assuming is a bhyve logging feature and have provided them with many logs (which they used to fix previous issues, such as Linux not answering mmap calls so things would crash) and my latest log was 26 GB since it was collecting everything from the scan for about 5 hours. Their 9PFS guru, Chris Torek, couldn't find anything wrong though this time. I just spent an hour speaking with Jordan Hubbard, CTO of IX Systems (makers of FreeNAS) and co-founder of the FreeBSD Project, regarding this and he's pretty much at a loss as well since they can't replicate it at all, even using my configurations. They seem to believe that the issue is on the Linux side and not the FreeBSD side, but we can't really test out whether or not Linux is the issue because currently it's the only OS that natively supports 9PFS (out of the current Linux, *BSD and Windows VM offerings).

During the importing of my library, or even when it crashes I see no errors in dmesg/journalctl regarding 9PFS crashing or anything else, the system is still stable, Emby just fails. There is also no useful information in the Emby logs other than it failed with SQLite errors and provided stack traces for them. Is there a way to turn on logging for 9PFS in Linux or maybe get more verbose kernel messages? Is there another 9pfs module that I can use? It seems that the ones Linux uses are 9pnet and 9pnet_virtio

I know next to nothing about 9PFS so any help would be greatly appreciated, it also seems like it's very difficult to find up to date information on 9PFS, specifically for Linux. Here's the bug report in case anyone wants to read up on it: https://bugs.freenas.org/issues/21495

Last edited by brando56894 (2017-03-04 17:50:08)

Lone_Wolf

Administrator

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 15,147

Re: Issues With 9PFS

emby has mono as dependency, so it probably uses .NET .

Does FreeNas have templates for a windows VM ?
If so, try the windows version of emby to see if that has the same problems.

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

brando56894

Member

From: NYC

Registered: 2008-08-03

Posts: 681

Re: Issues With 9PFS

We can't do that, because AFAIK Windows doesn't support 9PFS, which is what we're trying to test

I'm just using Emby because it's reliable and I know I can get it to crash haha People experience problems with Plex over 9P such as shows randomly missing or not being scanned in. Some Docker containers don't work well for odd reasons too and their storage is over 9PFS.

I did find this on kernel.org which mentions a bunch of Linux side mount debugging flags which will probably be useful.