Best way to share 40,000+ files?

diesel1 · 2010-07-21 23:42:14

Hi all,

I have approximately 40,000 files (documents, photos, videos and audio) and need to share them on my local network. I have tried nfs and samba but I am having serious delays from connecting to displaying the files.

Do you have any ideas about the best way to serve these file and be able to browse these files?

I am thinking about putting them on an intranet server, with gallery pages for image/photos.

Do you think this is a good idea?

TOH wants 'instant access to the photos'.

Any ideas or comments would be greatly appreciated.

Diesel1.

falconindy · 2010-07-22 00:40:03

How many files in a single sub directory? At some point, regardless of the sharing method, the internal readdir() calls get to be substantial. You can reduce the latency through reorganization.

fukawi2 · 2010-07-22 01:50:12

diesel1 wrote:

I have approximately 40,000 files (documents, photos, videos and audio) and need to share them on my local network.

That shouldn't be a problem... I have over 120,000 files on one server shared.

As falconindy mentioned, they're not all in one directory are they?

Last edited by fukawi2 (2010-07-22 01:50:24)

ngoonee · 2010-07-22 03:53:40

40K files in one directory, sounds like a real nightmare.... Reorganization is almost mandatory in such a case.

Wintervenom · 2010-07-22 03:54:43

#!/bin/bash

find . -maxdepth 1 -type f | while read file; do
  upper_name=`basename "$file" | tr '[:lower:]' '[:upper:]'`
  new_path="`dirname "$file"`/${upper_name:0:1}"
  mkdir -p "$new_path"
  mv "$file" "$new_path"
done

deej · 2010-07-22 04:02:47

Wintervenom wrote:

#!/bin/bash

find . -maxdepth 1 -type f | while read file; do
  upper_name=`basename "$file" | tr '[:lower:]' '[:upper:]'`
  new_path="`dirname "$file"`/${upper_name:0:1}"
  mkdir -p "$new_path"
  mv "$file" "$new_path"
done

Brilliant

Deej

fukawi2 · 2010-07-22 06:24:40

Or....

#!/bin/bash

find . -maxdepth 1 -type f | while read file; do
  ext=`echo ${file#*.} | tr '[:lower:]' '[:upper:]'`
  new_path="`dirname "$file"`/${ext}"
  mkdir -p "$new_path"
  mv "$file" "$new_path"
done

Last edited by fukawi2 (2010-07-22 06:25:10)

deej · 2010-07-22 07:03:34

fukawi2 wrote:

Or....

#!/bin/bash

find . -maxdepth 1 -type f | while read file; do
  ext=`echo ${file#*.} | tr '[:lower:]' '[:upper:]'`
  new_path="`dirname "$file"`/${ext}"
  mkdir -p "$new_path"
  mv "$file" "$new_path"
done

Ah, dare I say - even better !

To save others the studying / testing, this script creates folders named after the file and puts the file within it's own folder, whilst Wintervenoms' creates folders based on
the first character of the file and puts all files beginning with that character within.

I smell a contest looming...

Deej

Last edited by deej (2010-07-22 07:04:42)

karol · 2010-07-22 08:34:45

> I am thinking about putting them on an intranet server, with gallery pages for image/photos.
How many people are expected to access it - avg & tops?

diesel1 · 2010-07-22 09:48:56

Hi all,

The most files in a single directory is approximately 500 and definitely not more than 1000.

The maximum number of users at any one time would be 2or 3 but usually just the wife.

Should I use a CMS or just point apache at the files, or should I try again with nfs or samba?

Diesel1.

karol · 2010-07-22 09:57:22

How ofter would the files change:, the dir structure / filenames, how many GB / per day new data?
I'm _really_ not into networking but Apache sounds a bit heavyweight - aren't there any simple fileservers?

ProgDan · 2010-07-22 09:59:43

What about simple anonymous FTP server with access restricted only to local network?

karol · 2010-07-22 10:00:50

ProgDan wrote:

What about simple anonymous FTP server with access restricted only to local network?

Does it support thumbnails of photos / videos?

ProgDan · 2010-07-22 10:09:47

karol wrote:

ProgDan wrote:
What about simple anonymous FTP server with access restricted only to local network?
Does it support thumbnails of photos / videos?

That depends on the client. By default, FTP does not provide any feature like that. But for example Dolphin is capable of creating and displaying thumbnails for FTP images: it downloads the file, creates thumbnail and displays it (maybe it's not Dolphin's feature but KIO's feature). I'm not 100% sure, but I think that it even stores the thumbs in local cache.

EDIT: videos are not supported I think, it would be quite silly to download hundreds of MB of video just to cut off a simple frame and then throw it all away

Last edited by ProgDan (2010-07-22 10:10:51)

diesel1 · 2010-07-22 10:15:11

Hi all,

The files do not change very often but some editing and copying does take place.

Sometimes I add a few MB and sometimes I might add 20-30GB but that would usually be 10-15 videos.

Diesel1.

karol · 2010-07-22 10:21:45

> TOH wants 'instant access to the photos'.
How big is your photo collection?

Videos are huge, but maybe every user can have his / her own copy locally?

whacath · 2010-07-22 10:38:31

what about sshfs?

diesel1 · 2010-07-22 10:55:07

karol wrote:

> TOH wants 'instant access to the photos'.
How big is your photo collection?
Videos are huge, but maybe every user can have his / her own copy locally?

The collection is photos and videos in folders by category and date, there are 20,000 files taking 200GB.

I would rather not duplicate the collection (other than for backups).

Diesel1.

karol · 2010-07-22 11:03:06

Would it be possible to duplicate just the photos?

diesel1 · 2010-07-22 11:57:21

karol wrote:

Would it be possible to duplicate just the photos?

I really do not want to duplicate any files. I have the original content and if any user wishes to edit/modify photos they copy them locally. I don't want multiple versions of files on multiple systems.

Diesel1.

skanky · 2010-07-22 12:24:18

It might be worth looking at the bottlenecks, hard drive speed, network card, filesystem, etc. as 500 files in a folder shouldn't be too slow, I'd have thought.

I have some image folders on my server with approaching that many and there's no obvious delay using samba (for me or for the missus's xp netbook). I put them all up on flickr too (additional bonus of off-site back-up), which used to be her access to them, but the local access is far quicker - as you'd expect.

There is a pause sometimes in downloading a 5+mb image (or multiple) when moving/viewing/editing etc. but that's probably wireless speeds as much as anything as with moving at least there's 10+mb being shifted.

diesel1 · 2010-07-22 21:10:48

skanky wrote:

It might be worth looking at the bottlenecks, hard drive speed, network card, filesystem, etc. as 500 files in a folder shouldn't be too slow, I'd have thought.
I have some image folders on my server with approaching that many and there's no obvious delay using samba (for me or for the missus's xp netbook). I put them all up on flickr too (additional bonus of off-site back-up), which used to be her access to them, but the local access is far quicker - as you'd expect.
There is a pause sometimes in downloading a 5+mb image (or multiple) when moving/viewing/editing etc. but that's probably wireless speeds as much as anything as with moving at least there's 10+mb being shifted.

Hey, nice to hear from another Yorkshire Arch user!

I used to have no problems with nfs and samba but in the last year things seem to be so slow as to be unusable. I think you are probably right about wireless speeds and also I have been using lots of cpu time with folding clients on the file sharing system. also my photo files are now 5-7MB (Canon 500D camera) where before they were around 1-2MB. Maybe I should set up a dedicated nfs/samba file server.

Diesel1.

karol · 2010-07-22 21:13:39

> I have been using lots of cpu time with folding clients on the file sharing system. also my photo files are now 5-7MB (Canon 500D camera)
I think I can help you with the latter problem: send me the camera and go back to the 1-2 MB files :-)

diesel1 · 2010-07-22 21:16:42

karol wrote:

> I have been using lots of cpu time with folding clients on the file sharing system. also my photo files are now 5-7MB (Canon 500D camera)
I think I can help you with the latter problem: send me the camera and go back to the 1-2 MB files :-)

That sounds like a really good idea, send me your full contact details and I will forward it to you without delay!

Last edited by diesel1 (2010-07-22 21:16:58)

skanky · 2010-07-23 08:25:58

Hey, nice to hear from another Yorkshire Arch user!

I hope we're not the only two.

Maybe I should set up a dedicated nfs/samba file server.

Yes, I should have said that mine is dedicated - it's a (not especially old) laptop with a fried video card. I wired it to the router, so only the client sides are wireless. Any flickr uploading is initiated via ssh from the server, and at some point within the rest of my lifetime I'll probably set-up an abs and/or pacman cache etc. etc. on it.

Arch Linux

#1 2010-07-21 23:42:14

Best way to share 40,000+ files?

#2 2010-07-22 00:40:03

Re: Best way to share 40,000+ files?

#3 2010-07-22 01:50:12

Re: Best way to share 40,000+ files?

#4 2010-07-22 03:53:40

Re: Best way to share 40,000+ files?

#5 2010-07-22 03:54:43

Re: Best way to share 40,000+ files?

#6 2010-07-22 04:02:47

Re: Best way to share 40,000+ files?

#7 2010-07-22 06:24:40

Re: Best way to share 40,000+ files?

#8 2010-07-22 07:03:34

Re: Best way to share 40,000+ files?

#9 2010-07-22 08:34:45

Re: Best way to share 40,000+ files?

#10 2010-07-22 09:48:56

Re: Best way to share 40,000+ files?

#11 2010-07-22 09:57:22

Re: Best way to share 40,000+ files?

#12 2010-07-22 09:59:43

Re: Best way to share 40,000+ files?

#13 2010-07-22 10:00:50

Re: Best way to share 40,000+ files?

#14 2010-07-22 10:09:47

Re: Best way to share 40,000+ files?

#15 2010-07-22 10:15:11

Re: Best way to share 40,000+ files?

#16 2010-07-22 10:21:45

Re: Best way to share 40,000+ files?

#17 2010-07-22 10:38:31

Re: Best way to share 40,000+ files?

#18 2010-07-22 10:55:07

Re: Best way to share 40,000+ files?

#19 2010-07-22 11:03:06

Re: Best way to share 40,000+ files?

#20 2010-07-22 11:57:21

Re: Best way to share 40,000+ files?

#21 2010-07-22 12:24:18

Re: Best way to share 40,000+ files?

#22 2010-07-22 21:10:48

Re: Best way to share 40,000+ files?

#23 2010-07-22 21:13:39

Re: Best way to share 40,000+ files?

#24 2010-07-22 21:16:42

Re: Best way to share 40,000+ files?

#25 2010-07-23 08:25:58

Re: Best way to share 40,000+ files?

Board footer