How to easily navigate a large filesystem?

kasprosian · 2013-08-02 04:02:35

I programmed something that will help navigate a large filesystem.

First, the program crawls the whole filesystem, then you can simply type "j [query]" and then it will return the top 5 results matching the query.

https://github.com/mallochine/chestnut

Is anybody interested?

kasprosian · 2013-08-02 04:03:10

If there is already a solution to the problem of navigating large file system, then please let me know!

jasonwryan · 2013-08-02 04:04:43

kasprosian wrote:

If there is already a solution to the problem of navigating large file system, then please let me know!

mlocate?

progandy · 2013-08-02 04:55:03

Simple and effective.

Since you asked for existing tools: mlocate, recoll, regain-desktop, tracker, pinot, ...
There are also immature implementations of tagging file systems.

karol · 2013-08-02 11:42:05

progandy already listed a couple good examples, I'll add a link or two.

I'd call such a tool an indexer or a desktop search engine - Arch wiki and the wikipedia agree with me: https://wiki.archlinux.org/index.php/Li … s#Indexers ;P

kasprosian · 2013-08-02 13:19:02

mlocate returns all the results instead of the top 5, so it's very hard to get what you want for a 1 or 2 word nav query on a large filesystem.

If a search engine used your cwd and other heuristics to give you only the most relevant results, would that be a large enough improvement? (just wondering)

kasprosian · 2013-08-02 13:21:15

An additional problem (at least in my case) is that mlocate is often installed globally so it's difficult to get mlocate just for your own local account.

tomk · 2013-08-02 13:31:01

kasprosian wrote:

An additional problem (at least in my case) is that mlocate is often installed globally so it's difficult to get mlocate just for your own local account.

Not really:

locate -A ~ foo

In other words, pass your home dir as one of the patterns, and tell locate to match all patterns. locate(1) for more details.

Good work though, nothing beats scratching your own itch.

karol · 2013-08-02 13:31:59

Please use the 'Edit' functionality (link is in the bottom right of your post) instead of bumping the thread.

I don't have very many files, so I use locate or find - or find + grep if I want to search inside the text files. My memory isn't perfect so returning just top 5 or 10 results is usually not what I want. I use simple queries to find e.g. 10 biggest or oldest files when I'm doing some cleanup, but that's not what you're after.
I use aliases for stuff that I access often. If the contents of your system don't change very often, I think coming up with a hierarchy or categories / tags could help.

kasprosian · 2013-08-02 14:15:54

karol wrote:

Please use the 'Edit' functionality (link is in the bottom right of your post) instead of bumping the thread.
I don't have very many files, so I use locate or find - or find + grep if I want to search inside the text files. My memory isn't perfect so returning just top 5 or 10 results is usually not what I want. I use simple queries to find e.g. 10 biggest or oldest files when I'm doing some cleanup, but that's not what you're after.
I use aliases for stuff that I access often. If the contents of your system don't change very often, I think coming up with a hierarchy or categories / tags could help.

Right, when you're using just a few files at a time that are mostly nearby, navigating to those files is not really a pain.

It's just that from my experience working with a software company's very large codebase, I found it excruciating to simply go between two files that have very far distance from each other in the file system tree. That's why I wrote chestnut.

kasprosian · 2013-08-02 14:19:29

tomk wrote:

kasprosian wrote:
An additional problem (at least in my case) is that mlocate is often installed globally so it's difficult to get mlocate just for your own local account.
Not really:
locate -A ~ foo
In other words, pass your home dir as one of the patterns, and tell locate to match all patterns. locate(1) for more details.
Good work though, nothing beats scratching your own itch.

Haha thanks

On the other though....:

[~]$ locate -A ~ nacl
locate: warning: database `/usr/software/var/locatedb' is more than 8 days old

[~]$ updatedb --localpaths="/home/ec2-user"
/usr/software/bin/updatedb: line 223: /usr/software/var/locatedb.n: Read-only file system

I guess most users on this forum use Linux on their own desktop, so privileges won't be an issue for the most part. Hmph.

**************************

Ideally, a user should not even have to lookup/know which flags to use. Ideally, it should be "Google like" -- just type in your query, and you got what you want.

The problem might be that it's not that big of an upgrade, i.e. not a big deal.

Could you look at https://bbs.archlinux.org/viewtopic.php?id=167567 ? this is a tangential offshoot to what I did here.

Last edited by kasprosian (2013-08-02 14:22:21)

kokoko3k · 2013-08-02 14:27:06

What's the strategy to sort the results?

kasprosian · 2013-08-02 14:40:22

kokoko3k wrote:

What's the strategy to sort the results?

Shorter paths over longer paths. The heuristic seems naive, but has worked well in my programming environment (~30,000 files).

Last edited by kasprosian (2013-08-02 14:47:30)

karol · 2013-08-02 15:12:05

kasprosian wrote:

[~]$ locate -A ~ nacl
locate: warning: database `/usr/software/var/locatedb' is more than 8 days old

I think you should run updatedb e.g. daily via cron or systemd timer.

kasprosian wrote:

Ideally, a user should not even have to lookup/know which flags to use. Ideally, it should be "Google like" -- just type in your query, and you got what you want.

I seldom use Google to search for 'foo', usually my queries rely on slightly more advanced tools, like site:, inurl:, I'm interested in results only from last week or month etc.

tomk · 2013-08-02 15:15:03

kasprosian wrote:

Ideally, a user should not even have to lookup/know which flags to use

Ehmm... get to know your audience. We're Archers, we make a point of knowing which flags to use.

kasprosian · 2013-08-02 15:43:19

karol wrote:

kasprosian wrote:
[~]$ locate -A ~ nacl
locate: warning: database `/usr/software/var/locatedb' is more than 8 days old
I think you should run updatedb e.g. daily via cron or systemd timer.

I would do that, except I'm just an employee at a software company, not a sysadmin :\

kasprosian · 2013-08-02 15:46:12

tomk wrote:

kasprosian wrote:
Ideally, a user should not even have to lookup/know which flags to use
Ehmm... get to know your audience. We're Archers, we make a point of knowing which flags to use.

It's the old idealism vs reality thing again.

graysky · 2013-08-02 17:36:49

Might wanna change the trigger from the single letter j to something else since autojump used j to invoke itself.

Arch Linux

#1 2013-08-02 04:02:35

How to easily navigate a large filesystem?

#2 2013-08-02 04:03:10

Re: How to easily navigate a large filesystem?

#3 2013-08-02 04:04:43

Re: How to easily navigate a large filesystem?

#4 2013-08-02 04:55:03

Re: How to easily navigate a large filesystem?

#5 2013-08-02 11:42:05

Re: How to easily navigate a large filesystem?

#6 2013-08-02 13:19:02

Re: How to easily navigate a large filesystem?

#7 2013-08-02 13:21:15

Re: How to easily navigate a large filesystem?

#8 2013-08-02 13:31:01

Re: How to easily navigate a large filesystem?

#9 2013-08-02 13:31:59

Re: How to easily navigate a large filesystem?

#10 2013-08-02 14:15:54

Re: How to easily navigate a large filesystem?

#11 2013-08-02 14:19:29

Re: How to easily navigate a large filesystem?

#12 2013-08-02 14:27:06

Re: How to easily navigate a large filesystem?

#13 2013-08-02 14:40:22

Re: How to easily navigate a large filesystem?

#14 2013-08-02 15:12:05

Re: How to easily navigate a large filesystem?

#15 2013-08-02 15:15:03

Re: How to easily navigate a large filesystem?

#16 2013-08-02 15:43:19

Re: How to easily navigate a large filesystem?

#17 2013-08-02 15:46:12

Re: How to easily navigate a large filesystem?

#18 2013-08-02 17:36:49

Re: How to easily navigate a large filesystem?

Board footer