simple defragmenter

Pajaro · 2006-12-16 14:33:00

UPDATED: fixed a bug with filenames with spaces
UPDATED: now you can defragment a single file
UPDATED: made it preserve permissions and ownership (using cp -a)
UPDATED: changed disclaimer

hi, i created a simple defragmenter. I tested it with my /opt dir and my /var/lib/pacman dir. Everything still works fine (kde, swiftfox, pacman -Qs, pacman -Ss).

It needs you to install filefrag. (I supose everybody has md5sum, cp, mv, grep, sed, find, mv and cp).

#!/bin/bash

ME=$0;
FILE="$1";

function MD5move {
    newFile="$1";
    oldFile="$2";
    OLD_MD5SUM=$(md5sum "$oldFile"|sed 's/ .*//');
    NEW_MD5SUM=$(md5sum "$newFile"|sed 's/ .*//');
    if [ "$OLD_MD5SUM" != "$NEW_MD5SUM" ]; then
        echo ":: MD5SUM check for file $FILE failed"
    else
        mv "$newFile" "$oldFile";
        return 1;
    fi
}

function defragSingleFile {
    TMP_DIR="/tmp/defrag.pajaro/$$"
    mkdir $TMP_DIR
    TMP_FILE="$TMP_DIR/new_file"
    NEW_NUM_EXTENDS=9999
    TMP_NUM_EXTENDS="$NUM_EXTENDS";
    LOOPS=0;
    while [ "$NEW_NUM_EXTENDS" != "1" ]; do
        cp -a "$FILE" "$TMP_FILE"
        NEW_NUM_EXTENDS=$(filefrag "$TMP_FILE"|grep found|sed 's/.*: //'|sed 's/ .*//');
        if [ $NEW_NUM_EXTENDS -lt $TMP_NUM_EXTENDS ]; then
            MD5move "$TMP_FILE" "$FILE";
            if [ "$?" != "" ]; then
                TMP_NUM_EXTENDS="$NEW_NUM_EXTENDS"
            fi
        fi
        let LOOPS="$LOOPS + 1";
        if [ "$LOOPS" == "5" ]; then
            break;
        fi
    done;
    if [ "$NEW_NUM_EXTENDS" != "1" ]; then
        echo ":: We couldn't reduce the extends of $FILE to 1. It stayed in $TMP_NUM_EXTENDS (it had $NUM_EXTENDS)";
    else
        #echo "==> Now we should md5sum $FILE and $TMP_FILE";
        echo "==> $FILE defragmented! (it had $NUM_EXTENDS)";
    fi
    if [ -f $TMP_FILE ]; then
        rm $TMP_FILE;
    fi
    rmdir $TMP_DIR;
}

if ! [ -d /tmp/defrag.pajaro ]; then
    mkdir /tmp/defrag.pajaro
fi

if [ "$FILE" != "" ] && [ "$FILE" != "--i-understood" ]; then
    EXTENDS=$(filefrag "$FILE"|grep found);
    if [ "$EXTENDS" == "" ]; then
        echo "$ME: Error getting filefrag information for file $FILE";
    else
        NUM_EXTENDS=$(echo $EXTENDS|sed 's/.*: //'|sed 's/ .*//');
        if [ "$NUM_EXTENDS" != "1" ]; then
            #echo "Defragmenting file $FILE because it has $NUM_EXTENDS extends";
            defragSingleFile $FILE, $NUM_EXTENDS;
        fi
    fi
elif [ "$FILE" == "--i-understood" ]; then
    find -mount -type f -exec /usr/bin/defrag {} ;
else
    echo "If you use it, you have to be sure that no other process is accessing"
    echo "the files that are cointained in the directory that you run it."
    echo
    echo "Use it at your own risk."
    echo
    echo "Usage:"
    echo "defrag --i-understood - Defragments the current directory. Does not descent into"
    echo "                        other mounts."
    echo "defrag [file]         - Defragments [file]"
fi

Moo-Crumpus · 2006-12-16 17:03:45

wooot

Pajaro · 2006-12-16 17:47:28

What does "wooot" mean?

Pajaro · 2006-12-16 17:56:55

I removed the "is really dangerous" because it wasn't true. I did lots of checks with no problem. You only have to be sure that no process is modifying the files you are defragmenting.

so you can defrag
/opt
/usr
/bin
/lib
/var/lib/pacman
/var/cache/pacman
/var/abs

with no problem, because only you will change them.

I defragment my /opt dir while i am in kde and using swiftfox. Everthing is fine.

noriko · 2006-12-16 18:16:03

i'd suggest u replace the disclaimer, this tool tested well enough and someone might have bad luk using it..
u don't wanna be blamed do ya?

Pajaro · 2006-12-16 19:18:35

noriko wrote:

i'd suggest u replace the disclaimer, this tool tested well enough and someone might have bad luk using it..
u don't wanna be blamed do ya?

What do you propose the disclaimer to be?

kth5 · 2006-12-16 19:37:17

unfortunately this tools meses up permissions on the files it copies to temp and back. this scripts needs to be run as root, so all files will be recreated with root:root. cp -a will help.

Pajaro · 2006-12-16 20:04:44

kth5 wrote:

unfortunately this tools meses up permissions on the files it copies to temp and back. this scripts needs to be run as root, so all files will be recreated with root:root. cp -a will help.

Thank you! Updated.

noriko · 2006-12-16 21:44:31

it doesn't matter lol..
i think a simple retraction of any implied warranty would suffice....

Pajaro · 2006-12-16 21:59:41

so you consider that the disclaimer is too extense? (keep in mind that english is no my native language)

noriko · 2006-12-16 23:21:44

yeh, what i meant was to remove the explanation of what thre script does and place it in a --help option or something... that wat it makes the disclaimr more visible

Pajaro · 2006-12-17 00:27:50

Disclaimer updated!

I also found that in aur we can find 2 programs, defrag and shake. These are the differences that I saw:

- defrag: it doesnt check the extends of the file, it just copies all files. It either md5sums.
- shake: it seems more interesting and powerful, but it needs the kernel to be specially configured and it needs you to remount the partition with a certain parameter.

The defrag script that i wrote keeps trying to copy the file for 5 times in order to reduce the extends at maximum.

lilsirecho · 2006-12-17 03:17:51

Pajaro;

Was able to defrag files using the script from cmd line.

Please advise as to where the script should be installed to enable a directory to be defragged.

The command .... defrag --i -understood.... is not recognized .....from the desktop script file.

Pardon my stumbling over simple things!!

skottish · 2006-12-17 07:18:37

Minor bump...

[This is a question I'd like an answer for]

Under what scenario does a Linux box need to be defragged?

Pajaro · 2006-12-17 12:01:45

lilsirecho wrote:

Pajaro;
Was able to defrag files using the script from cmd line.
Please advise as to where the script should be installed to enable a directory to be defragged.
The command .... defrag --i -understood.... is not recognized .....from the desktop script file.
Pardon my stumbling over simple things!!

"defrag --i -understood" has an extra space between --i and -understood.
What is the desktop script file?
I have defrag in /usr/bin folder.

Pajaro · 2006-12-17 12:09:50

skottish wrote:

Minor bump...
[This is a question I'd like an answer for]
Under what scenario does a Linux box need to be defragged?

Escenario: you hd is getting fragmented.

The reason why there are no windows-like defragmenters for Linux is that in Linux this kind of operation is too dangerous. So we have simple defragmenters, that just copy files hoping that the fragmentation gets reduced. Another example of this is pacman-optimize.

It is said that with ext4 fragmentation will disappear.

iphitus · 2006-12-17 12:27:38

Pajaro wrote:

The reason why there are no windows-like defragmenters for Linux is that in Linux this kind of operation is too dangerous.

No, not because the operation is dangerous, but because it's not needed.

Fragmentation, is when a *single file's* parts, are spread across a drive, not continuous.

Unlike fat and ntfs, the major linux filesystems do not suffer this to such a degree. Fragmentation is generally very low because of their design. Not sure whether it's ext3, or reiserfs, but the fsck tool for that fs spits out the fragmentation of a partition when it's done -- i've never seen it rise above 3 or so percent.

So we have simple defragmenters, that just copy files hoping that the fragmentation gets reduced. Another example of this is pacman-optimize.
It is said that with ext4 fragmentation will disappear.

These "defragmenters" just rewrite files, in an attempt to have them saved in the same area of the drive.

This is not something any filesystem can easily do itself, as a filesystem has no knowledge of the meaning of the contents of a file and it's relevance of location to any other file.

This is an 'issue' that will exist as long as we use hard drives like those today, where location on the platter affects speed. On flash devices, fragmentation isnt an issue, it makes no difference where data is located to read speed -- and that change will probably happen before filesystems gain the knowledge of which files make up firefox and thus must be stored near each other.

As it is, the pacman optimisers use this to work around the flaw in pacman, that it's database is built from thousands of files.

James

Pajaro · 2006-12-17 12:46:54

Unlike fat and ntfs, the major linux filesystems do not suffer this to such a degree. Fragmentation is generally very low because of their design. Not sure whether it's ext3, or reiserfs, but the fsck tool for that fs spits out the fragmentation of a partition when it's done -- i've never seen it rise above 3 or so percent.

ReiserFS is very good against fragmentation. ext4 is very good too. (ext4 is now in the kernel, but it is experimental).

i've never seen it rise above 3 or so percent.

how do you measure the degree of fragmentation under Linux?

It is really good the fact that fragmentation is gonna disappear when flash drives arrive.

I need to access my hd from linux, so I can't use ReiserFS. I am gonna use defrag until ext4 of a flash hd becomes part of my computer. I suffer of slow hds. I want to do as much as I can.

lilsirecho · 2006-12-17 16:09:37

Last trial gave this result; (/var/local)

[root@n6re local]# defrag --i -understood
filefrag: invalid option -- -
Usage: filefrag [-v] file ...
/usr/bin/defrag: Error getting filefrag information for file --i

Gullible Jones · 2006-12-17 16:38:12

It's ext3 that shows the fragmentation on fsck, FWIW. I've seen that go up to about 20% on partitions with lots of (relatively) large files, but it never seems to increase, and doesn't decrease if you do the copy/wipe/paste routine.

Pajaro wrote:

ReiserFS is very good against fragmentation. ext4 is very good too. (ext4 is now in the kernel, but it is experimental).

ReiserFS actually can get some degree of file fragmentation due to tail-packing, IIRC. Also it is (IIRC) more subject to "fragmentation" of directories than other filesystems. When using it, I've generally found that pacman performance starts out fast but drops rather quickly.

ext4 can reduce fragmentation by using extents, but JFS and XFS already support extents and are not experimental.

It is really good the fact that fragmentation is gonna disappear when flash drives arrive.

If you're referring to hard drives with flash buffers, those won't help fragmentation. If you mean pure flash memory, we've got that, but it's a lot more expensive than HD space, wears out after a certain number of writes, and uses different types of filesystems.

I need to access my hd from linux, so I can't use ReiserFS. I am gonna use defrag until ext4 of a flash hd becomes part of my computer. I suffer of slow hds. I want to do as much as I can.

Not quite sure I understand you...

Pajaro · 2006-12-17 16:56:27

lilsirecho wrote:

Last trial gave this result; (/var/local)
[root@n6re local]# defrag --i -understood
filefrag: invalid option -- -
Usage: filefrag [-v] file ...
/usr/bin/defrag: Error getting filefrag information for file --i

use defrag --i-understood, with no extra spaces

Pajaro · 2006-12-17 17:00:44

ext4 can reduce fragmentation by using extents, but JFS and XFS already support extents and are not experimental.

Not quite sure I understand you...

I searched a lot about XFS, and i found that XFS blanks all you open files when the partition is not clearly unmounted.

What I mean is that, as far as i know, if I want to access my linux partition with read/write permissions i need to use ext3.

lilsirecho · 2006-12-17 17:35:34

Pajaro;

Successfully ran your program on /opt and /var/local with the corrected command line.

Most responses gave 2 as the best it could do.

this completes test of your script.

Thanks...

Pajaro · 2006-12-17 18:28:22

You are welcome.

Pajaro · 2006-12-18 11:33:39

I forgot to say something i found about defragmenting by copying.

When I was doing the initial tests i saw that many times copying a file increased the number of extends instead of reducing them.

This is another thing that makes this script good: it only replaces the original file if the number of extends is gonna be reduced.

Arch Linux

#1 2006-12-16 14:33:00

simple defragmenter

#2 2006-12-16 17:03:45

Re: simple defragmenter

#3 2006-12-16 17:47:28

Re: simple defragmenter

#4 2006-12-16 17:56:55

Re: simple defragmenter

#5 2006-12-16 18:16:03

Re: simple defragmenter

#6 2006-12-16 19:18:35

Re: simple defragmenter

#7 2006-12-16 19:37:17

Re: simple defragmenter

#8 2006-12-16 20:04:44

Re: simple defragmenter

#9 2006-12-16 21:44:31

Re: simple defragmenter

#10 2006-12-16 21:59:41

Re: simple defragmenter

#11 2006-12-16 23:21:44

Re: simple defragmenter

#12 2006-12-17 00:27:50

Re: simple defragmenter

#13 2006-12-17 03:17:51

Re: simple defragmenter

#14 2006-12-17 07:18:37

Re: simple defragmenter

#15 2006-12-17 12:01:45

Re: simple defragmenter

#16 2006-12-17 12:09:50

Re: simple defragmenter

#17 2006-12-17 12:27:38

Re: simple defragmenter

#18 2006-12-17 12:46:54

Re: simple defragmenter

#19 2006-12-17 16:09:37

Re: simple defragmenter

#20 2006-12-17 16:38:12

Re: simple defragmenter

#21 2006-12-17 16:56:27

Re: simple defragmenter

#22 2006-12-17 17:00:44

Re: simple defragmenter

#23 2006-12-17 17:35:34

Re: simple defragmenter

#24 2006-12-17 18:28:22

Re: simple defragmenter

#25 2006-12-18 11:33:39

Re: simple defragmenter

Board footer