You are not logged in.

#1 2012-09-24 23:39:05

aero
Member
Registered: 2012-09-24
Posts: 5

[SOLVED] ls on specific directory takes very long

Hello guys, I hope I posted this at the right place. I'm having a strange problem and I don't know what is causing it.

The story
I needed to upload a 900MB video so I tried to split it up first using the 'split' command in my home directory. After half an hour of execution my whole system started to lag... I then terminated the process and found out that I had probably run it with wrong arguments (stupid me). As a result I had millions of tiny files in my home directory (ranging form mv_xaa to mv_zzzbscltunz or similar...). I had a really bad time removing them. Lots of applications didn't work anymore because you couldn't list files in my home directory, 'rm mv_*' didn't work because it said that the argument list was too big. I ended up running commands like

find . -name 'mv_*' -exec rm {} \;
while true; do rm `echo m_* | head -c 1000000`; echo deleted 1 block; done # note: not totally safe
find . -name 'mv_*' -delete
Using filezilla GUI to remove the files

for one whole night and then the files were gone. (By the way, which of those command do you think performs faster?)

The problem
Then, relieved that I finally got rid of those files, I saw that I still experience a long lag. The problem is that after I boot up and try to ls or use pcmanfm in my home directory, it takes a very long time. Other directories work fast as usual. If I 'ls' again in my home directory it works fast, probably due to some form of caching.

$cd # SOMEDIR #
$time ls
# OUTPUT #

real	0m0.225s
user	0m0.000s
sys	0m0.003s
$cd ~
$time ls
# OUTPUT (~20 files) #

real	1m46.271s
user	0m0.003s
sys	0m0.547s

My guess is that something on the filesystem is wrong. I use ext4. I though that I maybe have to defragment the home partition. Do you think that can help? How do I do it?
What do you think is causing the big ls lag?

Thanks for reading the long post.

Last edited by aero (2012-09-29 19:27:34)

Offline

#2 2012-09-25 04:28:57

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 20,727

Re: [SOLVED] ls on specific directory takes very long

Any chance you are getting I/O errors while talking to the disk?  Check your logs.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way

Offline

#3 2012-09-27 03:20:11

aero
Member
Registered: 2012-09-24
Posts: 5

Re: [SOLVED] ls on specific directory takes very long

I guess there are no disk I/O errors, because I didn't find anything in /var/log/messages.log. Is there another place I can look for errors?

Offline

#4 2012-09-27 06:46:18

x33a
Forum Fellow
Registered: 2009-08-15
Posts: 4,587

Re: [SOLVED] ls on specific directory takes very long

try /var/log/everything.log

Offline

#5 2012-09-27 08:13:01

skunktrader
Member
From: Brisbane, Australia
Registered: 2010-02-14
Posts: 1,681

Re: [SOLVED] ls on specific directory takes very long

I suspect that the problem is because EXT filesystems do not ever reuse directory inodes once they are allocated.  To check to see the space being used try ls -l .. while in the problem directory.  If the size of the directory seems massive, your only solution is to create a new directory and move all the files to it and then delete the old directory.  Obviously if this is your home directory you will need to take appropriate steps to make sure there are no login shells accessing the directory at the time.

Offline

#6 2012-09-27 16:25:49

aero
Member
Registered: 2012-09-24
Posts: 5

Re: [SOLVED] ls on specific directory takes very long

skunktrader wrote:

I suspect that the problem is because EXT filesystems do not ever reuse directory inodes once they are allocated.  To check to see the space being used try ls -l .. while in the problem directory.  If the size of the directory seems massive, your only solution is to create a new directory and move all the files to it and then delete the old directory.  Obviously if this is your home directory you will need to take appropriate steps to make sure there are no login shells accessing the directory at the time.

I think that might be exactly the problem. ls -l .. in ~ tells me

drwx-----x 92 myname users 63397888 Sep 27 12:15 myname

I will try to recreate the directory later today at home, where I can make backups first. Will then post the results here.

Offline

#7 2012-09-29 19:27:13

aero
Member
Registered: 2012-09-24
Posts: 5

Re: [SOLVED] ls on specific directory takes very long

It worked!

time ls
# OUTPUT #

real	0m0.260s
user	0m0.003s
sys	0m0.000s

What I did:
1. Backup all my files of my home directory in a separate hard drive
2. init 1
3. cd /home
4. mkdir foo
5. chown aero:users foo
6. chmod go-r foo
7. chmod g-x foo
8. mv aero/* foo
9. mv aero/.* foo # don't do that, it's wrong! use something like 'find . -mindepth 1 -maxdepth 1 -exec mv -t.. -- {} +'
10. rmdir aero
11. mv foo aero
12. reboot

Size is much better now:

$ ls -l /home
total 20
drwx-----x 91 aero users  4096 Sep 29 15:29 aero
drwx------  2 root root  16384 Jun 29  2011 lost+found

Thank you skunktrader!

Last edited by aero (2012-09-29 19:31:29)

Offline

Board footer

Powered by FluxBB