Long-term storage solution

dav7 · 2008-10-08 03:31:10

This isn't "immediate", per se, and won't be "soon". I'm still at the lots-of-planning stage here with this, so you don't need to get into a race or anything to quickly help me before I make my decision just so you know.

Now, for my question: what would make a good, long-term, backup/archival system that isn't read only?

I want something fast (so not tapes), long-term (so not hard disks) and not fragile (so probably not CDs). What else do I have available?

I could use CDs if something akin to the CD changer one finds in more elaborate hi-fi systems and car stereos existed for data discs - so, therefore, a device that was equivalent to a clean-room environment and could hold like 500 discs. One that accepted standard 5.25" burners would be awesome, because I could switch it to bluray once prices went down and popularity and Linux support went up

I'm basically looking for something that lets me access data like 10 years after storing it, maybe make a couple of changes, etc. 10 years would be the absolute maximum I'd probably need though

Oh, and the medium I want to use needs to scale high. Very high. Into the order of more than a decent amount of TBs... like, say, 40 or 50. Of course that nice, round figure wouldn't pop up out of nowhere the day I have my plans in order, it'd grow - slowly - over time. So not only do I need something scalable, I need something that can scale slowly.

So kinda like a NAS or external HDD type of solution, except one that doesn't use failure-prone media like HDDs. Unless there's a kind of hard disk out there that doesn't fail very "easily" and lasts in an order of years?

-dav7

Last edited by dav7 (2008-10-08 03:34:28)

fukawi2 · 2008-10-08 04:09:40

Will 50,000 CD's be OK? Link 100 of these units together...
http://www.imation.com.au/products/disc … /index.htm

Last edited by fukawi2 (2008-10-08 04:10:09)

dav7 · 2008-10-08 09:46:15

Oh, right. I forgot one important point: the process needs to be automated. The method suggested above works great except it, like, can only hold the CDs. It doesn't have the capability to read or write them.

I mentioned the term "CD changer" for a reason - the CD changer doesn't hold the CDs and, when it's time to change discs, eject the disc and expect you to insert it into the playback device. It contains the mechanics to hold a collection of discs and play the one you want without more than hitting a button.

And, on another note, 50k CDs would be too much! CD-RWs can only be re-burned so many times before they give out.

To be honest, if CD media was all that good, I would probably go and get myself an enclosure, install some fans in it to keep the dust out, and mount a slot-loading CD-RW right in front of that changer's slot, so when the changer ejected, the disk ejected right into the CD-RW, and vice versa. It would work, but it would be slow, and... meh. I guess that method could be a last resort.

I was thinking I could maybe get a good brand of hard disk, and keep an inventory of what I've bought and when, so I have an idea of when to test the disks for stability (say 1 year after purchase, and then every 6 months after that) and replace them as required.

-dav7

Last edited by dav7 (2008-10-08 09:52:00)

Garns · 2008-10-08 09:51:50

I wouldn't write off harddisks that fast.

Afaik you won't get more than reliable two to three years out of burnable cds/dvds after that it comes basically down to luck and with that much data you are talking about the chances of failure are high. With the right storage, a harddisk should be able to last at least this long, when you give it a spin at least every year. Furthermore disks are relatively cheap and easy to handle.

The numbers are taken out of my head, based on an article about long term archiving I read recently, so they might be slightly wrong. But I clearly remember the gist, that even with high quality DVDs you will have to recopy every other year.

Of course it always comes down to the question how much you are willing to invest. I bet there are some really good commercial grade archiving systems out there, which I have got no clue about, if you can afford one of those is another question.

Edit: Just saw your second post, if you want the data to be readily avaiable (no swapping of disks/tapes) you should probably go for harddisks and you might want to read this article

Last edited by Garns (2008-10-08 09:55:01)

molom · 2008-10-08 11:29:50

dav7 wrote:

Oh, and the medium I want to use needs to scale high. Very high. Into the order of more than a decent amount of TBs... like, say, 40 or 50. Of course that nice, round figure wouldn't pop up out of nowhere the day I have my plans in order, it'd grow - slowly - over time. So not only do I need something scalable, I need something that can scale slowly.
So kinda like a NAS or external HDD type of solution, except one that doesn't use failure-prone media like HDDs. Unless there's a kind of hard disk out there that doesn't fail very "easily" and lasts in an order of years?
-dav7

I believe there is nothing that exists in the world, unless you pay huge money for Solid State Hard Drives. The only solution I see is using a bunch of rewritable double layer Blu-ray discs. I don't think there is any proper solution for what your looking for. Your looking for the perfect that doesn't exist.

Garns · 2008-10-08 13:37:43

molom wrote:

dav7 wrote:
Oh, and the medium I want to use needs to scale high. Very high. Into the order of more than a decent amount of TBs... like, say, 40 or 50. Of course that nice, round figure wouldn't pop up out of nowhere the day I have my plans in order, it'd grow - slowly - over time. So not only do I need something scalable, I need something that can scale slowly.
So kinda like a NAS or external HDD type of solution, except one that doesn't use failure-prone media like HDDs. Unless there's a kind of hard disk out there that doesn't fail very "easily" and lasts in an order of years?
-dav7
I believe there is nothing that exists in the world, unless you pay huge money for Solid State Hard Drives. The only solution I see is using a bunch of rewritable double layer Blu-ray discs. I don't think there is any proper solution for what your looking for. Your looking for the perfect that doesn't exist.

I'm not sure how long SSDs will last and I don't think there is any reliable information out there, wether an SSD I plugin now will still work in ten years. Plus you would need one hell of a lot SSDs to store Terabytes of data. BlueRay is probably worse than DVD regarding reliability, until the manufacturers get more experience with them.

Am I the only one to see a pattern with dav7 opening threads looking for things which don't exist?

Last edited by Garns (2008-10-08 13:39:29)

Ramses de Norre · 2008-10-08 14:00:31

A giant RAID array.

moljac024 · 2008-10-08 15:33:44

Ramses de Norre wrote:

A giant RAID array.

+1
No other solution...

Dusty · 2008-10-08 15:39:03

Amazon S3.

skottish · 2008-10-08 15:40:36

A friend of mine works for NetApp and levels of redundancy that have built into everything is amazing. No matter what you do redundancy is key.

dav7 · 2008-10-08 17:06:12

Am I the only one to see a pattern with dav7 opening threads looking for things which don't exist?

it isn't my fault my brain got teleported back from the year 2187!

Okay, so, no SSDs or blu-ray discs, that makes sense.

Amazon S3's rack storage probably wouldn't take too kindly to my duplicates of triplicates anyway (my data is a mess atm, I need to sort it out but to sort it out I need space first ) and I in turn wouldn't take kindly to having to buy my data out of archival. I'll bet that in 20 years everyone will be doing that sort of thing anyway...

NetApp... interesting.

So it looks like a managed RAID array, then, with a monitoring program perhaps.

-dav7

Last edited by dav7 (2008-10-08 17:08:17)

Profjim · 2008-10-08 19:45:21

Or why not rethink your need for speed? Could you use a combination of tape for storage and a big hd for caching the part of the data you need quick access to?

crouse · 2008-10-08 22:36:36

Arch Linux

#1 2008-10-08 03:31:10

Long-term storage solution

#2 2008-10-08 04:09:40

Re: Long-term storage solution

#3 2008-10-08 09:46:15

Re: Long-term storage solution

#4 2008-10-08 09:51:50

Re: Long-term storage solution

#5 2008-10-08 11:29:50

Re: Long-term storage solution

#6 2008-10-08 13:37:43

Re: Long-term storage solution

#7 2008-10-08 14:00:31

Re: Long-term storage solution

#8 2008-10-08 15:33:44

Re: Long-term storage solution

#9 2008-10-08 15:39:03

Re: Long-term storage solution

#10 2008-10-08 15:40:36

Re: Long-term storage solution

#11 2008-10-08 17:06:12

Re: Long-term storage solution

#12 2008-10-08 19:45:21

Re: Long-term storage solution

#13 2008-10-08 22:36:36

Re: Long-term storage solution

#14 2008-10-09 04:48:10

Re: Long-term storage solution

#15 2008-10-09 11:38:45

Re: Long-term storage solution

Board footer