Question about inotify watch limits and real-time backup (SpiderOak)

cfr · 2013-10-15 01:59:18

I use SpiderOak to backup various things and I have run into a problem. Essentially some of my backed up files and directories are not backed up when they change. If I manually initiate a scan, the changes are identified and the files backed up. But the changes should be noticed automatically.

I'm not positive but I think this is due to the following kind of error:

inotify_add_watch <path-to-some-file-or-directory> 28 No space left on device

So I know that the number of watches per user is set as

$ cat /proc/sys/fs/inotify/max_user_watches 
8192

I also know how to increase this temporarily or (through configuring sysctl) persistently.

However, I'm not sure what implications this has. Moreover, I'm puzzled by the following facts:

1) SpiderOak fails to set a watch on files and directories which it is not supposed to be watching. That is, the errors above concern files which are not, and never have been, in my backup list on any device associated with my account. I find it a bit creepy it is trying to watch things I haven't asked it to. (And I don't want it watching *everything*!) In fact, these files are in a directory under /home/<dir> which includes no items which have ever been in my backup list on any device.

2) I assume SpiderOak is running as me (ps seems to confirm this). But in that case, if my explanation above is correct, the application should have exhausted my ability to set watches. However, I can add new watches on things with no issue. That is, I don't get an error saying there's no space or anything - the system just watches as asked. So I am concerned that I'm misunderstanding what's going on here and I'd rather not go changing system resource limits without making sure I have understood what's going on.

I did email SpiderOak but I haven't (yet?) had any response.

Can anybody cast any light on (1) or (2)? Or have any general insight into the advisability or otherwise of increasing max_user_watches?

lightstream · 2013-10-15 10:44:23

It does sound like this could be the inotify watch limit being reached.

One easy way to test is to call the tail command with the follow switch, e.g.

tail -F /tmp/madeupfile

If you have run out of watches, you will see this error:

tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

From what I've read, there is no harm on increasing the max number of watches, as each watch consumes only a very small amount of system resources (of the order of a few tens of bytes). However, as well as a limit on watches, inotify also has limits on events and instances:

cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances

So maybe if watches are not yet all used up, then one of these is?

cfr · 2013-10-15 22:50:40

Thanks. My watches are definitely not all used up. I can tail -f. I can also set a watch manually on a directory.

I'm not sure how to check for the other two - queued events and user instances. I've only come across methods to check for watches being used up so far (i.e. by web searching).

SpiderOak's initial response is that Arch is not really supported but they'll have a look. So I've sent them a copy of my backup list and log files.

I still think the error is weird - that it concerns something spideroak shouldn't have any reason to be concerned with!

Arch Linux

#1 2013-10-15 01:59:18

Question about inotify watch limits and real-time backup (SpiderOak)

#2 2013-10-15 10:44:23

Re: Question about inotify watch limits and real-time backup (SpiderOak)

#3 2013-10-15 22:50:40

Re: Question about inotify watch limits and real-time backup (SpiderOak)

Board footer