Issues with slow file indexing

_JZA_ · 2020-03-14 07:09:08

I am using Plasma5 on a hybrid disk. I have around 150 GB on SSD and around 1 TB on HHD. I was able to map my user home to the HHD so I can use my storage for my files and the fast storage for my applications and other things on my system.

My issue right now is that even thought booting the computer or restarting my desktop is very quick as well as starting sometimes some applications other things are very slow when I am trying to let say, open my Pictures folder and has to display the thousands of files.

Doing some research it seems index is taken care by baloo, and even thought I understand that reading all those files should be a memory intesive process, I wonder if there is a way that it can read from the index and not have to go through the process every single time I want to open a file chooser.

Here is why I wonder if I can either, move the indexing temporary files on the SSD (which btw /tmp/ is on SSD) or configure my system so it can map those specific locations on the SSD mount points.

If there is any other suggestion I am all ears.

mpan · 2020-03-14 19:18:23

The process is not memory-intensive by its nature. In fact baloo is designed and advertised to have low memory footprint as its goal. However, it is I/O-intensive, because at least the header of each and every file has to be read. That requires a lot of random access reads, which are slower than sequential reads even on SSD and can be a tragedy on spinning rust — in particular if applications compete for access.

No baloo here, so I can’t compare performance, but you may try to:

V1del · 2020-03-14 20:48:35

I don't think plain directory listings go over the index, so if that's what's slow you are likely not to be able to get around using the SSD or organizing your images into more digestible chunks via directories.

_JZA_ · 2020-03-14 22:29:07

mpan wrote:

The process is not memory-intensive by its nature. In fact baloo is designed and advertised to have low memory footprint as its goal. However, it is I/O-intensive, because at least the header of each and every file has to be read. That requires a lot of random access reads, which are slower than sequential reads even on SSD and can be a tragedy on spinning rust — in particular if applications compete for access.
No baloo here, so I can’t compare performance, but you may try to:
Index only names in baloo;
Disable baloo indexing completely;
Exclude directories from baloo indexing.

Maybe baloo has not much to do with the sluggishness, if that's the case, any way I can increase the performance. I have disable baloo indexing with no real upside or downside. So wondering if maybe is just an issue on dolphin or plasma file manager chocking on large number of files. I have around 14k files on the directory and sorting them takes up to 15 seconds.

mpan · 2020-03-14 23:57:18

Using the file utility⁽¹⁾ on over 6000 files here required 80s for uncached data and 40s for cached data. Even worse times for PHP’s mime_content_type in a loop, so it’s not a matter of xargs executing file slowly. This is a decade-old PC, so I expect on a newer computer it will be faster.

Therefore my second guess is that detecting file type is to blame. If that’s not the issue, perhaps thumbnail creation is the culprit — that obviously must be slow.

Note that with “thousands of files” in a single directory you shuld not expect fast operation. What’s the reason you must keep such an unmanageable number of files in one location?

____
¹ find . -type f | xargs file --mime-type

_JZA_ · 2020-03-15 02:15:15

mpan wrote:

Note that with “thousands of files” in a single directory you shuld not expect fast operation. What’s the reason you must keep such an unmanageable number of files in one location?

Well I do, and this folder is not new, I have used it many times in the past and I dont remember it being this slow on this machine, maybe the disk is about to die and having IO issues. I might need to replace it, or maybe is the filesystem.

I am on an i7 with 16GB RAM and a Seagate Mobile ST1000LM035-1RK172 1TB

Arch Linux

#1 2020-03-14 07:09:08

Issues with slow file indexing

#2 2020-03-14 19:18:23

Re: Issues with slow file indexing

#3 2020-03-14 20:48:35

Re: Issues with slow file indexing

#4 2020-03-14 22:29:07

Re: Issues with slow file indexing

#5 2020-03-14 23:57:18

Re: Issues with slow file indexing

#6 2020-03-15 02:15:15

Re: Issues with slow file indexing

Board footer