FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

Xyne · 2009-09-27 06:32:07

Info page: http://xyne.archlinux.ca/info/facadefs

This is a result of this thread. There's not much point in explaining it here. Just jump directly to the man page for a description. The rest of this post assumes that you've read the info.

I've tested several things so far but I've probably missed a lot. The python-fuse documentation is sketchy at best. I could use some testers to play around with facadefs to see if any errors show up. Use the "-d" option to get debugging information from fuse itself. That will contain Python errors from facadefs.

I would also like people to try setting up different types of system sandboxes and then try to break out of them. I don't know enough about breaking out of chroot jails or manipulating namespaces and mounts to know how secure this is. One setup that seems appealing is this:

facadefs / /tmp/cache /tmp/mount -o allow_other -c /tmp/config
sudo chroot /tmp/mount
su <non-privileged user>

In this case, /tmp/config can be used to tweak the system (permissions, file access, etc). From what I've read about breaking out of chroot jails, you seem to need root privileges. I'd like to know if it's possible to prevent that with this setup.

Even if that setup isn't completely secure against malicious code, it should work well for testing things out, including installing anything you want on your system to give a live preview. The added bonus is that all modified files can be found in the cache (including new ones). It should be trivial to package anything by adding the right .PKGINFO file to the cache and archiving it. This might be useful for packages like SAGE which require building in the post_install function.

If this turns out to be useful, I'll try to rewrite it in C to make it faster.

Anyway, sorry for babbling. I've just finished coding and am in a rush right now. I'll clean this post up later.

*edit*
Considering what this does I've tried to make the code very clear by using self-explanatory names and lots of comments. As long as you don't use the [live] section of the configuration file, it should never touch anything inside the base filesystem. Obviously I guarantee nothing, but I generally try to be very careful with such things.

Last edited by Xyne (2009-09-27 06:36:09)

Ranguvar · 2009-09-27 07:23:25

The man page link on your web page links to the obfilebrowser man page, might want to fix that

It's really late here, so while this is likely super-total-awesomesauce, as is typical of your work (and thank you for it!) I'm gonna hit the sack and be back to look into it more later.

Last edited by Ranguvar (2009-09-27 17:09:10)

Xyne · 2009-09-27 07:38:28

fixed, thanks

Xyne · 2009-09-27 20:58:27

I've added loads of debugging messages. I've tested installing packages with pacman in a chroot but at the installation stage something blocks. The debugging output is:

QUEUE PATH 59
WAIT ON PATH 59

(path numbers change when rerun)

I haven't figured out what this means yet.

Xyne · 2009-09-30 15:58:20

Nobody?

sctincman · 2009-10-02 01:01:25

This sounds very cool! Not sure what I'd do, but I'd love to play around with it. Will definitely put up my experiences when I get some time to play

Dieter@be · 2009-10-02 10:59:15

how does this differ from unionfs?

Xyne · 2009-10-02 11:32:36

I'm not completely familiar with unionfs or aufs but when I tried them I was unable to mount my root directory with all its nested mountpoints in a separate directory. FUSE has no problem with that.

The configuration file also enables fine-grained control over file permissions, live files and invisible files. I did not find anything equivalent in unionfs or aufs.

FacaseFS does not require root privileges (as it does not need to "mount" anything).

Maybe unionfs/aufs can do all of the above. I wasn't able to find documentation to show me how though and I wanted to learn more about FUSE anyway

bruce · 2009-10-13 01:42:26

xyne, I'm getting "Permission denied" whenever I try and use /dev/null /dev/random or /dev/urandom for reading or writing...
I could well be doing something wrong though :-p
this happens when i create a sandbox with no custom config and chroot into it.
i am able to remove /dev/null from within the sandbox, and then recreate it (with mknod /dev/null c 1 3), however i still get the same message.
This happens with any app i've tried (eg zsh, head, tail...) not sure of the cause etc, but i'm gonna try and keep fiddling and see if i can't see what's happening...
cool app by the way
and as i said, may just be that i'm doing it wrong :-p

Xyne · 2009-10-13 18:54:29

Thanks for reporting the error, bruce.

I looked into it and have the same problem here. Running facadefs in debug mode didn't help. Any operation with those three paths returns the correct (as far as I can tell) attributes but anything further in the operation is denied by the system. I modified the code to make sure that it's stat'ing the file directly but that didn't help. I really have no idea what the cause is. I want to dig into this and figure out what's going on but my free time is non-existent at the moment.

I also tried various combinations of root, e.g. facadefs as root (using "-o allow_other"), chroot into the mount point, etc yet I still got permission errors.

With any luck though this will lead me to the solution to the other problem that I mentioned above.

This is going to irk me into insanity.

p.s. poor documentation = frustration

*edit*
I've also checked some of the built-in fuse options. Using "-o use_ino", the return value of "stat" is identical to that of the "/dev/null" with the exception of st_dev. This shouldn't matter though because python-fuse ignores st_dev.

Last edited by Xyne (2009-10-13 19:11:46)

smartboyathome · 2009-10-13 21:26:18

Xyne, one thing I think could help is recreating the nodes over the ones in the sandbox (ie, mknod). I don't know much about this, but I know those were the three files that LFS created before mount --bind'ing the host's /dev to the chroot in chapter 6.

Xyne · 2009-10-14 16:04:17

Thanks, smartboyathome.

I wasn't able to work around it using mknod. I still get permission errors and I'm begging to suspect that it's a limitation of (python-)fuse. Your mention of binding mounts led me to try binding dev (and later proc) into the facadefs root. After doing that and chrooting in, I was able to get past the previous QUEUE/WAIT ON PATH error and fully install the package. It locks up after that but the files appear as expected in the cache directory.

I also don't know if binding mounts in the chroot pose security issues (presuming that the chroot is set up in such a way that root privileges can be dropped once inside).

smartboyathome · 2009-10-14 16:58:07

Xyne wrote:

I also don't know if binding mounts in the chroot pose security issues (presuming that the chroot is set up in such a way that root privileges can be dropped once inside).

It shouldn't as long as you chroot and then change to a non-root user. It should just act like any normal OS from there on out.

IgnorantGuru · 2010-01-28 18:13:44

Wanted to test this but...

$ facadefs / /tmp/cache /tmp/mount  -o allow_other
$ sudo chroot /tmp/mount
Password:
root:$ su myuser
su: incorrect password

su myuser works okay normally, but once in the chroot just says "incorrect password".

mount reports:

facadefs on /tmp/mount type fuse.facadefs (rw,nosuid,nodev,default_permissions,allow_other,user=myuser)

My ultimate goal is to sandbox apps like Firefox. Any chance of facadefs working for that? Thanks.

Xyne · 2010-01-28 19:27:04

Hi IgnorantGuru

The problem is that you're running facadefs as a regular user so it doesn't have read permissions to gshadow and other system files, which is why "su myuser" fails within the chroot. Even though you are "root" within the chroot, you're still accessing the system via the facadefs interface and the facadefs process is what's actually reading system files, as your normal user.

Basically this is how it works:
run as root, root in chroot: full root privileges
run as root, normal user in chroot: normal user privileges
run as normal user, root in chroot: normal user privileges, even if the chroot thinks you're root
run as normal user, no chroot: normal user privileges

It should work for sandboxing Firefox but there are still some bugs to work out (e.g. it blocks when installing packages in the chroot via pacman). I haven't put much effort into debugging it since I posted it though because there has been very little interest in it and I need more eyes to figure a few things out.

*edit*
From the other thread:

IgnorantGuru wrote:

I assume the fuse mount can't read the password file since facadefs is run as a user, but as a result his su instructions don't work, and I was uncomfortable running it as root.

I can understand why you would be uncomfortable but if you look at the code, you'll see that the fuse option "default_permissions" is hard-coded, which "enable[s] permission checking by kernel". Once you "su myuser" in the chroot, you can't do anything to your system (aside from e.g. deleting your home dir, but there are facadefs options to hide that from the chroot).

Everything I write I test on my main desktop system and I like to think that I'm cautious and thorough when it comes to anything which could be harmful to it. I'll admit that until more knowledgeable users confirm it I can't be sure, but I think the facadefs system, once running as a regular user in the chroot, is completely secure.

Last edited by Xyne (2010-01-28 19:38:00)

IgnorantGuru · 2010-01-28 19:39:19

Xyne wrote:

The problem is that you're running facadefs as a regular user so it doesn't have read permissions to gshadow and other system files, which is why "su myuser" fails within the chroot.

I figured it was something like that. But you suggested:

One setup that seems appealing is this:

facadefs / /tmp/cache /tmp/mount -o allow_other -c /tmp/config
sudo chroot /tmp/mount
su <non-privileged user>

So I don't think those instructions will work. I wasn't sure if I should run facadefs as root. Are you suggesting I do it like this:

sudo facadefs / /tmp/cache /tmp/mount -o allow_other
sudo chroot /tmp/mount
su <non-privileged user>

Or should I run facadefs as user and then do a chroot as user instead of root?

One drawback I see to this is that in order to run say Firefox sandboxed, I would need the root password. Not undoable but it would be nice if there was someway to avoid root in the process of sandboxing.

Also, I think facadefs could use a 'disengage'. IOW a way to end the sandbox session. I just unmounted /tmp/mount as root, but I wasn't sure if that was correct.

This project looks like it has a lot of potential - thanks for your work on it. If you could put together a few examples on how to use it I think that would encourage more testing. I for one am not very familiar with chroot.

IgnorantGuru · 2010-01-28 19:54:35

I was more concerned with my ignorance of chroot and fuse than with your code. I'll play around with it a bit more.

I haven't put much effort into debugging it since I posted it though because there has been very little interest in it and I need more eyes to figure a few things out.

Sometimes people don't know what's good for them. I think sandboxing is the next big improvement to Linux security, long overdue, and there aren't many good tools for it.

Xyne · 2010-01-28 20:23:21

Although security was definitely the main motivation for this, there are other practical benefits. I still hope to use such a system as a universal packaging tool because it would enable the user to install absolutely anything in the chroot, then simply turn the cache into a package*. With the proper settings and some sort of network interception to prevent malicious code from calling home, it would make the build process simple and secure.

*edit: I mean that the installation of beasts such as SAGE, which currently require building in the post_install function, would be easy to package.

Last edited by Xyne (2010-01-28 20:28:04)

IgnorantGuru · 2010-01-28 21:41:20

Okay I did a little testing with this.

I tried two different methods...

$ facadefs / /tmp/cache /tmp/mount  -o allow_other
$ sudo chroot --userspec=myuser:myuser /tmp/mount
$ firefox

and

$ sudo facadefs / /tmp/cache /tmp/mount  -o allow_other
$ sudo chroot /tmp/mount
$ su myuser
$ firefox

In both cases facadefs seemed to be working as far as the folders I could access in the shell. However, firefox reported this error in a pop-up on startup:

Failed to contact configuration server; some possible causes are that you need to enable TCP/IP networking
for ORBit, or you have stale NFS locks due to a system crash. See http://projects.gnome.org/gconf/ for
information. (Details -  1: Failed to get connection to session: /usr/bin/dbus-launch terminated abnormally
without any error message)

Also, Firefox started with a fresh profile, as if it couldn't read /home/*/.mozilla. Later when I unmounted /tmp/mount, /tmp/cache contained a new /home folder with a new .mozilla folder, including a new default profile (different in name from my actual one). So it would appear that Firefox is not reading the files provided by fuse, but it does write files to the cache. I was able to access the web in firefox.

When I su myuser and run firefox without facadefs involved, it works normally. From what I could tell it didn't appear to be an environment variable issue, especially since Firefox wrote to /mnt/cache/home/myuser/.mozilla (so it knew the username and its home folder).

Also, I got the same results if I added a config file with:

[live]
/home/myuser/.mozilla

I also experimented with the configuration file, trying to limit what folders were visible. This worked to an extent. I strongly suggest adding an "[associate]" section which takes precedence over "[dissociate]". Otherwise it is very difficult to, for example, let Firefox see only ~/.mozilla. IOW I would like to specify what folders ARE associated rather than what folders are not. Also, I tried this:

[dissociate]
regex:/home/.*/[^\.]

which worked as expected - only folders and files beginning with "." were visible. However, this...

[dissociate]
regex:/home/.*/[^\.]
regex:/home/.*/\.[^m]

did not work as expected - all of /home/*/ was empty. So it doesn't appear the regex code is working properly, unless I misunderstand something (which wouldn't be a first!)

I would like something like this:

[dissociate]
/

[associate]
/home/myuser/.mozilla
/bin
/etc
/usr
/lib
/var/lib

Also, it seems impossible to effectively dissociate /etc since you then can't use su. An option to dissociate all of /etc except the files needed by su would be helpful - or this could be done using [associate] as I suggested.

So appears Firefox can be sandboxed with the current facadefs, except you will start with a blank configuration.

I still hope to use such a system as a universal packaging tool because it would enable the user to install absolutely anything in the chroot, then simply turn the cache into a package*. With the proper settings and some sort of network interception to prevent malicious code from calling home, it would make the build process simple and secure.

That sounds useful too. I think this can also be a good sandboxing tool for apps. I am not fond of fuse in general though, especially with what I've read on its security problems. A non-fuse solution would be preferable (and I think some of your problems may stem from fuse). It could be pacman isn't working for the same reason Firefox isn't? Seems like you're pretty close to the functionality you want.

Edit: Just after submitting that it occurred to me that sudo might be causing a problem. So I tried it from root without sudo and now firefox seems to be working... still testing. Update: Never mind... I was fooled because I had another firefox session open when I ran firefox. Using root directly seems to work the same as sudo.

Last edited by IgnorantGuru (2010-01-28 21:47:07)

IgnorantGuru · 2010-01-28 22:01:03

Also, I got the same results if I added a config file with:
[live]
/home/myuser/.mozilla

Strike that - it occurred to me that I forgot to specify the config file on the command line.

With this config file:

[dissociate]
/boot
/media
/mnt
/opt
/root
/srv
/sys
regex:/home/.*/[^\.]

[live]
/home/myuser/.mozilla

and this startup:

# facadefs / /tmp/cache /tmp/mount  -o allow_other -c /tmp/config
# chroot /tmp/mount
# su myuser
$ firefox

then firefox gave this pop-up:

An error occurred while loading or saving configuration information for firefox. Some of your
configuration settings may not work properly.

Failed to contact configuration server; some possible causes are that you need to enable TCP/IP
networking for ORBit, or you have stale NFS locks due to a system crash. See 
http://projects.gnome.org/gconf/ for information. 
(Details -  1: Server ping error: IDL:omg.org/CORBA/COMM_FAILURE:1.0)

However, firefox did start with my default profile. In fact I'm typing this post from the sandboxed Firefox. Apparently Firefox needs the [live] on its config folder. Any ideas why that might be? Still might relate to pacman's problem.

But this seems pretty usable now, if a big sluggish.

IgnorantGuru · 2010-01-28 22:05:38

Also, none of the Save As... dialogs in the sandboxed Firefox seem to work. Clicking on Save Page As... or Save Image As... has no effect. But, the Browse button in the "preferences|Downloads|Save files to" does open a chooser and is able to navigate the (limited) filesystem.

Last edited by IgnorantGuru (2010-01-28 22:16:11)

sdolim · 2010-01-28 22:31:38

Possible extra config file option:

[link]
/facade/path = /real/path

To move sections of the filesystem around inside the facade. Use cases:

- To splice a sub-directory of a [dissociated] path back into the facade
- To relocate files or directories to paths that are hardcoded in build scripts
(I just ran across one that expected 'lsmod' to be in /sbin.)
- If both paths are visible in the facade, then it could be made to look like a hard link.

For future consideration: allow links to also be configured in the [chmod] and [live] sections?

Last edited by sdolim (2010-01-28 22:33:31)

Xyne · 2010-01-28 23:10:22

I'm posting from inside a facadefs chroot right now. I get the same errors about session connection using this config:

[live]
glob:/proc/*

I don't know what Firefox is doing behind the scenes but I suspect that it's trying to communicate with something via files or fifos and that this is failing because whatever it's communicating with is outside of the sandbox. Once a file has been modified in the sandbox, it's dissociated from the file on the system. The file remains unmodified outside of the sandbox.

I tried adding /usr to the live paths but this did not solve the problem

For what it's worth, I am able to save files within the sandbox using the "save as" and other dialogues.

Unfortunately, some of the technical problems with this system are beyond me. It requires a deep understanding of how the system works, how FUSE works, and how applications behave within a chroot. Debugging is difficult, partly because of the way Python binds the FUSE library.

@sdolim
That should be easy to do. I'll look into it.

*edit*
I just tested it with

[live]
/

and still got the error, which indicates that the problem is not with file communication but rather the implementation itself. I need to check that I've implemented all necessary functions (correctly). It may also be a limitation of FUSE itself, but I really don't know.

*wishes the FusePython documentation were better*

Last edited by Xyne (2010-01-28 23:17:35)

IgnorantGuru · 2010-01-28 23:26:34

Xyne wrote:

*wishes the FusePython documentation were better*

I know the feeling - I'm looking at aufs to see how I might make a sandbox with chroot, since I don't really need the caching abilities of facadefs. I can't find any good how-tos on aufs. But I'm making some slight progress.

As for firefox under facadefs, the error on startup is less troubling than the save as dialogs not working, but that may be preventable if you're not getting it. Don't spend too much time on that for now, unless you think it may give insight into the pacman problem, as I'm leaning toward using aufs and chroot for what I want to do - providing I can figure out how to use them and it works. That might prove simpler than fuse for this purpose since aufs is supposed to present a 'real' filesystem to apps and the kernel.

IgnorantGuru · 2010-01-28 23:34:55

Now that you got me thinking about it, I tried it again with no dissociate section and just the live section for ~/.mozilla, and that fixed the problem with the save as dialogs. Probably was my dissociating /sys or something? At any rate this makes fsfacade usable for a Firefox sandbox because the startup error doesn't lead to any dysfunction that I can see.

Arch Linux

#1 2009-09-27 06:32:07

FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#2 2009-09-27 07:23:25

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#3 2009-09-27 07:38:28

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#4 2009-09-27 20:58:27

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#5 2009-09-30 15:58:20

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#6 2009-10-02 01:01:25

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#7 2009-10-02 10:59:15

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#8 2009-10-02 11:32:36

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#9 2009-10-13 01:42:26

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#10 2009-10-13 18:54:29

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#11 2009-10-13 21:26:18

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#12 2009-10-14 16:04:17

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#13 2009-10-14 16:58:07

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#14 2010-01-28 18:13:44

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#15 2010-01-28 19:27:04

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#16 2010-01-28 19:39:19

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#17 2010-01-28 19:54:35

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#18 2010-01-28 20:23:21

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#19 2010-01-28 21:41:20

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#20 2010-01-28 22:01:03

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#21 2010-01-28 22:05:38

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#22 2010-01-28 22:31:38

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#23 2010-01-28 23:10:22

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#24 2010-01-28 23:26:34

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

#25 2010-01-28 23:34:55

Re: FacadeFS: A FUSE-based filesystem sandbox. [TESTERS NEEDED]

Board footer