You are not logged in.

#1 2009-08-27 08:32:44

lman
Member
From: CZ
Registered: 2007-12-18
Posts: 255

Filesystem for 'bigger' partitions

Hi,

I have a 500 GB partition currently with ext3, but the fsck is painfully slow on it. So I decided to change the filesystem.
There are mainly movies, music on it, and also my documents for school. The only crucial thing is performance and that my documents won't go away tongue. I use this partition on local file sharing network (100Mbit), so both read and write performance is important.
Currently I'm deciding between ext4 or btrfs. I'm definitely going to wait for kernel 2.6.31 to be released. I wonder how stable btrfs is going to be until then.
I'm open to any suggestions. I will also be glad if you could recommend file system options/tweaks - in the discussions on phoronix about arch benchmarks there was sthing about data=writeback for ext4.
Thx in advance

Last edited by lman (2009-08-27 08:33:06)

Offline

#2 2009-08-27 10:19:25

mikesd
Member
From: Australia
Registered: 2008-02-01
Posts: 788
Website

Re: Filesystem for 'bigger' partitions

I don't think btrfs is ready for prime time just yet. ext4, xfs or jfs would be better choices. I think reiserfs still has a following but I haven't used it for ages.

Offline

#3 2009-08-27 10:23:30

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Filesystem for 'bigger' partitions

reiserfs is mainly useful for a whole lot of small files. I guess ext4 with writeback and noatime + nodiratime would be good.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#4 2009-08-27 11:03:14

fukawi2
Ex-Administratorino
From: .vic.au
Registered: 2007-09-28
Posts: 6,217
Website

Re: Filesystem for 'bigger' partitions

mikesd wrote:

I don't think btrfs is ready for prime time just yet. ext4, xfs or jfs would be better choices.

+1
I use XFS for a partition similar to what you've described. JFS is very good with fsck times because it only journals the meta data.

Offline

#5 2009-08-27 12:47:21

bobdob
Member
Registered: 2008-06-13
Posts: 138

Re: Filesystem for 'bigger' partitions

I have XFS on my 500GB hard drive, and the fsck time is pretty quick.

Offline

#6 2009-08-27 13:50:17

steevols
Member
From: US - SC
Registered: 2009-05-27
Posts: 26
Website

Re: Filesystem for 'bigger' partitions

I'm using ext4 with the tweaks mentioned above on a 1TB drive that holds music, movies, and backups of my game CDs and have no problems with it at all.

Offline

#7 2009-08-27 14:36:26

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: Filesystem for 'bigger' partitions

I would keep my school documents on a separate partition with a tried and tested file system wink

On the fsck times then ext4 will do better than ext3 (never tried other file system so I can't say anything). In my humble opinion ext4 is not yet quite ready for prime time (as far as important things are concerned), just check some of the problems being corrected and you will see why, also some improvements being implemented (or recently implemented) need you to reformat you drive to be able to take advantage of them so waiting is a little longer might not be a bad idea.

The tips about ext4 may very well apply to ext3 too (noatime + nodiratime, writeback is already the default if nothing is overriding that) and a 100Mb connection will hardly stress your hard disk / file system unless you are trying to access many files at once, but in that case what matters most is how the hard disk behaves I believe.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#8 2009-08-27 15:01:56

gnud
Member
Registered: 2005-11-27
Posts: 182

Re: Filesystem for 'bigger' partitions

It was my understanding that the ext4 on-disk format was frozen with 2.6.28? What has happened since then that requires a reformat?

Offline

#9 2009-08-27 15:18:25

lman
Member
From: CZ
Registered: 2007-12-18
Posts: 255

Re: Filesystem for 'bigger' partitions

R00KIE wrote:

On the fsck times then ext4 will do better than ext3 (never tried other file system so I can't say anything). In my humble opinion ext4 is not yet quite ready for prime time (as far as important things are concerned), just check some of the problems being corrected and you will see why, also some improvements being implemented (or recently implemented) need you to reformat you drive to be able to take advantage of them so waiting is a little longer might not be a bad idea.

Can you please give me a link, or tell which improvements smile I couldn't find anything useful on google just your other post mentioning reformat tongue

ROOKIE wrote:

The tips about ext4 may very well apply to ext3 too (noatime + nodiratime, writeback is already the default if nothing is overriding that) and a 100Mb connection will hardly stress your hard disk / file system unless you are trying to access many files at once, but in that case what matters most is how the hard disk behaves I believe.

This is kinda out of topic... Well it's the file sharing that stresses my hdd sometimes - 2-3 guys downloading from me each with 4-5 MB/s + I'm downloading sthing 10 MB/s + watching 720/1080p movie + sometimes burning dvd.
Sometimes all this at once big_smile, but usually just the uploading part plus one of the others.
Oh and I forgot to mention when dc++ starts hashing...

Last edited by lman (2009-08-27 15:23:18)

Offline

#10 2009-08-27 17:12:57

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: Filesystem for 'bigger' partitions

As for links here goes one http://osdir.com/ml/linux-ext4/2009-06/msg00240.html check for the ext4 fixes due in kernel 2.6.31.
I have also seen somewhere (can't remember exactly where) that there was going to be a new allocator implemented for ext4 (but I think that was already done for kernel 2.6.30, I'm not sure if this is where I've seen it, I think not but check here http://www.mjmwired.net/kernel/Document … s/ext4.txt and search for orlov).

On the disk usage .... you can't possibly be doing all that  remotely with a 100Mb/s connection, the peak speed would be 12.5MB/s to each side with a FD link so I guess I misunderstood something, but the problem is still the same, I believe that what will limit you the most is how the hard disk performs when there are many different requests, even one of the latest disks with a good ncq algorithm can't save you from bad performance if you need to read/write many MB of data that are scattered all over the disk (which seems to be your case).

Don't take me wrong but it seems you are asking too much from a setup that isn't the most adequate for what you want to do, the fsck times can indeed be solved with a change in the file system but any concurrent access speed issues need more than that I believe, I don't have experience solving those problems, those are more akin to what you would need to solve in a server environment I guess.

There are other people here with a lot more knowledge about that than me and they may be able to give you good tips.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#11 2009-08-27 17:42:49

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: Filesystem for 'bigger' partitions

ext4 would be fine, if you use ext3 now then just convert it.

Offline

#12 2009-08-27 18:57:10

lman
Member
From: CZ
Registered: 2007-12-18
Posts: 255

Re: Filesystem for 'bigger' partitions

R00KIE wrote:

On the disk usage .... you can't possibly be doing all that  remotely with a 100Mb/s connection, the peak speed would be 12.5MB/s to each side with a FD link so I guess I misunderstood something, but the problem is still the same, I believe that what will limit you the most is how the hard disk performs when there are many different requests, even one of the latest disks with a good ncq algorithm can't save you from bad performance if you need to read/write many MB of data that are scattered all over the disk (which seems to be your case).

Thanks for the links.
(Why would it be impossible? Full duplex - 2 guys downloading from me with 5MB/s or let's say 3guys with ~4MB/s (12M), but the peak is usually arund 11.2 due protocol overhead. That much for the upload, and download 10MB/s. Thats ~11 up and ~10 down)

The speed of the hdd is fine, I'm not complaining. I was just explaining why is the performance important to me. Anyway as soon as ssd hdds mature a bit, I'll get one of those smile.

Offline

#13 2009-08-27 20:07:51

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: Filesystem for 'bigger' partitions

It was just my misunderstanding about what you said, I understood 2~3 guys downloading at 4~5MB/s plus watching an hd movie plus burning, all this going through the network, something had to go slowly if all that was going though a 100Mb/s connection wink

On the disk side of things, you know what I mean, ask too many files at the same time from the disk and things will slowdown to a halt unless files are already cached in the ram tongue
Some file systems may be a bit better due to less overhead or slightly different caching strategies but you will eventually reach the limits imposed by the disk or storage subsystem.

The disk can't read/write while moving the heads around to the right place so if one is trying to read/write many big files at the same time then after all caches are full and no requests can be delayed/reordered things go at snail pace tongue SSDs are great in this, very very low access times but unfortunately for now they are too expensive and too small for mass storage.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

Board footer

Powered by FluxBB