STUFFS - a Semantically-Tagged, Unorganized, Future File-System

aaronlaursen · 2013-08-30 19:57:31

Hey everybody,

I've been a long time Arch user (but not a bbs poster), and the community has always had a few people willing to try something new...

... and so I present STUFFS: a Semantically-Tagged, Unorganized, Future File-System.

I've put it up on the AUR: https://aur.archlinux.org/packages/stuffs-git/
and you can find the code on GitHub: https://github.com/aaronlaursen/STUFFS

This experimental file-system does away with the old hierarchal model in favor of a dynamic, tag-based scheme. However, it also strives to ease the transition through full backwards compatibility.

I've been working on it for a little while. It's now stable and functional and I would love to get some feedback.

Find out more below, or in the README

How it works

Files and tags are stored in a SQLite database, with a schema similar to what might be used for storing posts and tags on a blog, etc. Thanks to some fuse magic, this gets mounted as a virtual file-system and dynamically generates "directories" based on tags and the files tagged as such.

How to install it

If you use yaourt, it's as simple as:

yaourt -S stuffs-git

otherwise, you can find it on the AUR using your favorite method.
you will also need some requirements:

extra/python
extra/sqlite
community/python-sqlalchemy
aur/python-fusepy-git

How to run it

STUFFS.py <mountpoint> [database_file]

How you use it

tags are created with `mkdir`
tags are added to a file with `mv`
searching by tag is accomplished via `ls`

actual output from a user session:

[aaron@eustathios2 STUFFS]$ STUFFS.py mnt/ fs.db &
[1] 4406
[aaron@eustathios2 STUFFS]$ cd mnt
[aaron@eustathios2 mnt]$ ls
[aaron@eustathios2 mnt]$ mkdir tag1 tag2 tag3
[aaron@eustathios2 mnt]$ ls
tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ touch file1 file2 file3
[aaron@eustathios2 mnt]$ ls
file1@1@  file2@2@  file3@3@  tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ echo "foo">file1
[aaron@eustathios2 mnt]$ cat file1
foo
[aaron@eustathios2 mnt]$ cat file1@1@
foo
[aaron@eustathios2 mnt]$ cp file1 file2
[aaron@eustathios2 mnt]$ ls
file1@1@  file2@2@  file3@3@  tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ cat file2
foo
[aaron@eustathios2 mnt]$ mv file1 tag1
[aaron@eustathios2 mnt]$ ls
file1@1@  file2@2@  file3@3@  tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ ls tag1
file1@1@
[aaron@eustathios2 mnt]$ ls tag2
[aaron@eustathios2 mnt]$ mv file1 tag2/tag3
[aaron@eustathios2 mnt]$ ls
file1@1@  file2@2@  file3@3@  tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ ls tag1
file1@1@  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ ls tag2
file1@1@  tag1%1%  tag3%3%
[aaron@eustathios2 mnt]$ ls tag3
file1@1@  tag1%1%  tag2%2%
[aaron@eustathios2 mnt]$ ls tag3/tag1
file1@1@  tag2%2%
[aaron@eustathios2 mnt]$ ls
file1@1@  file2@2@  file3@3@  tag1%1%  tag2%2%  tag3%3%
[aaron@eustathios2 mnt]$ fg
STUFFS.py mnt/ fs.db	(wd: ~/proj/STUFFS)
^C[aaron@eustathios2 mnt]$

Some things to note

Files and tags get unique names, so
```
mkdir mytag; ls
```
may output
```
mytag%9876%
```
where "9876" is an internal ID, and
```
touch myfile;ls
```
may output
```
myfile@8360@
```
However, referencing just the file or tag name (not the ID) should work as expected, assuming there are no duplicates. ie.
```
cat myfile
```

On the plus side, this means that you don't need the full path, so

/tag1%907%/tag2%5267%/tag3%83%/file@47@
/tag1%907%/file@47@
/file@47@
/someTagThatDoesntEvenExist/file@47@

all point to the same "file@47@"

This also means that something pointing to "/file@47@" will always point to "file@47@" regardless of what you tag it as...

nourathar · 2013-08-30 20:48:06

that is a very interesting idea !

J.

aaronlaursen · 2013-08-30 21:16:52

nourathar wrote:

that is a very interesting idea !
J.

Thanks!

If you get a chance to try it out, I'd love to hear how it goes.

HalosGhost · 2013-08-31 01:26:35

Interesting idea. I have a question about function though. You mentioned that you assign tags using `mv`, would the same be true of `cp`? To be more clear, if I have file "pacman.conf" tagged to "/etc/" and I wanted it to also be tagged "/configs/", would I `mv /pacman.conf /configs/`, or would that remove it from its current tags? If so, would `cp /pacman.conf /configs/` perform the operation I'm hoping for?

All the best,

-HG

aaronlaursen · 2013-08-31 02:24:03

HalosGhost wrote:

IYou mentioned that you assign tags using `mv`, would the same be true of `cp`? To be more clear, if I have file "pacman.conf" tagged to "/etc/" and I wanted it to also be tagged "/configs/", would I `mv /pacman.conf /configs/`

Yup, that would do it.

`cp` actually copies the files as one might expect.

so `cp /pacman.conf /configs/` would result in a different "pacman.conf" tagged "configs" (with a different ID and everything)

aaronlaursen · 2013-08-31 19:47:08

Short update.

A bug regarding tag removal was brought to my attention.
This has now been fixed.

To unset a tag:

mv myfile !mytag

note: the '!' may need to be escaped, depending on your shell.

Last edited by aaronlaursen (2013-08-31 21:15:31)

aaronlaursen · 2013-09-17 17:54:30

Update:

STUFFS now only lists tags in the base directory (keeps things from getting too cluttered)
all files can be also be found under the aptly named `ALLFILES` tag
write speed has improved by 4000% !!!! kind of a big deal...
STUFFS now plays nice with GUI file managers!

there are also some scripts in the git repo that make the tagging process a little easier.
more to come...

Arch Linux

#1 2013-08-30 19:57:31