You are not logged in.

#1 2020-09-03 13:00:07

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,014

[SOLVED] Safely refresh hdd content

I've some old hdds witrh data on them and i'd like to refresh theiur content by rewriting their content with the same data.
I've read i can use tools like badblocks, but i'd like to have everything under control, so i'm thinking to copy files from the hdd to itself, maybe into another directory.
unfortunately i cannot do this in one shot, because the free space is not enough.
I could move the files, instead, so that space is freed continuosly, but mv does not "copy-then-remove" if the disk filesystem is the same.

Final question: is there a tool that copy and delete files instead of just "renaming" them?

Thanks.

-edit-
seems the followind woud work:
rsync --dry-run --remove-source-files -avHAX /old_data/ /refreshed_data/

Last edited by kokoko3k (2020-09-16 19:00:20)

Offline

#2 2020-09-03 13:33:26

mpan
Member
Registered: 2012-08-01
Posts: 564
Website

Re: [SOLVED] Safely refresh hdd content

That single attempt to “refresh” the data may bring more degradation than would come from years of storage. So word “safely” should already be removed from the equation.

If you really insist on “refreshing”, you may simply read all the data from the disk: literally `cat` the drive into “/dev/null” once (not the opposite! wink). If the drive can detect magnetic field degradation beyond acceptable level during read, it will automatically rewrite the data. If it does not, touching the drive in any way may worsen things.

What you really want to avoid data loss from bit rot is providing error correction for your data. For that you may use parchive (from the community/par2cmdline package).


Sometimes I seem a bit harsh — don’t get offended too easily! PGP: 7C848198AE93D3BB

Offline

#3 2020-09-03 14:57:42

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,014

Re: [SOLVED] Safely refresh hdd content

Thanks mpan, but i've read the opposite "here and there"
https://en.wikipedia.org/wiki/Data_degr … In_storage

Magnetic media, such as hard disk drives, floppy disks and magnetic tapes, may experience data decay as bits lose their magnetic orientation. Periodic refreshing by rewriting the data can alleviate this problem. In warm/humid conditions these media, especially those poorly protected against ambient air, are prone to the physical decomposition of the storage medium.[2]

...Also, just reading the the data may trigger or not the rewrite, and since i don't plan to do this operation often, i'd like to "touch" everything.

parchive is nice, but it is a cure; i'd like to prevent the problems.

What are your sources, anyway?

Offline

#4 2020-09-03 18:24:52

mpan
Member
Registered: 2012-08-01
Posts: 564
Website

Re: [SOLVED] Safely refresh hdd content

Parchive is not a cure. It’s a different way of encoding data to prevent its complete loss in case of limited damage in the underlying medium. This is not different to any ECC your drive already uses to keep the data intact during its normal operation. It also does exactly what you asked for, at least to the extent of percentage of data that may be recovered in an event of a damage.


I believe I should end this post here, but since I was challenged and I used my time to re-check my claim, let’s continue. big_smile

I am not exactly sure what sources you are asking about. For which claim?

That, subjectively, to me the “refreshing” idea⁽¹⁾ smells like bullshit? It does, because it has all the features of a typical hearsay and lore myth. I do not know if any researcher published a paper about this particular instance of the phenomenon, much less if there is any meta-analysis on the subject. So for that I can provide as much sources as for the fact that the tea I drink right now is tasty. Gut feeling.

That, objectively, the “refreshing” idea is actual bullshit? Requires none, because the original positive claim has no sources.

That HDDs use error correcting codes, that more modern ones detect possible failures before they actually happen or that they rewrite data in such scenarios? Again, requires no sources, because that’s a piece of knowledge described in any authoritative book on a subject. But, if you really want: a first random find from some 2019 IEEE’s conference: section III.B of “Characterizing Disk Health Degradation and Proactively Protecting Against Disk Failures for Reliable Storage Systems” by Song Huang et al.

____
⁽¹⁾ As referenced in this thread, taking into account the context. To make it clear: my counterclaim is that it is not needed.

Last edited by mpan (2020-09-03 18:27:47)


Sometimes I seem a bit harsh — don’t get offended too easily! PGP: 7C848198AE93D3BB

Offline

#5 2020-09-03 18:39:39

loqs
Member
Registered: 2014-03-06
Posts: 11,633

Re: [SOLVED] Safely refresh hdd content

You would normally scrub a drive checking if a block can be read and if the contents of the block matches that calculated from parity e.t.c..
You would only write to the block if the data was bad to see if the block can be healed or needs to be marked as bad.

Last edited by loqs (2020-09-13 12:18:29)

Offline

#6 2020-09-13 09:15:55

ua4000
Member
Registered: 2015-10-14
Posts: 214

Re: [SOLVED] Safely refresh hdd content

if you use BTRFS filesystem, scrub and crc checksums are included - a check can be started anytime.
A more general way is to calculate md5 checksums, e.g. one md5 file per folder. They don't use much storage space but you can veify the integrity of your data.
Anyway, the best way is to have a backup at hand :-)

Offline

#7 2020-09-13 15:18:08

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,014

Re: [SOLVED] Safely refresh hdd content

This thread is all about preserving a backup :-)

Offline

#8 2020-09-16 02:27:48

mpan
Member
Registered: 2012-08-01
Posts: 564
Website

Re: [SOLVED] Safely refresh hdd content

ua4000 wrote:

if you use BTRFS filesystem, scrub and crc checksums are included - a check can be started anytime.
A more general way is to calculate md5 checksums, e.g. one md5 file per folder. They don't use much storage space but you can veify the integrity of your data.

That detects bit rot, not limits its effects. Sure, one will know the data is damaged, but that means it is already lost. The thread is about minimizing chances of data being lost in the first place.

Last edited by mpan (2020-09-16 02:28:30)


Sometimes I seem a bit harsh — don’t get offended too easily! PGP: 7C848198AE93D3BB

Offline

#9 2020-09-16 16:34:45

ua4000
Member
Registered: 2015-10-14
Posts: 214

Re: [SOLVED] Safely refresh hdd content

kokoko3k wrote:

I've some old hdds witrh data on them and i'd like to refresh theiur content by rewriting their content with the same data.

After reconsidering: No I would not recommend this.
If you have important data on it, the only way to keep them safe is a backup on another medium (hdd/optical).

Using the "old hdd" in any way could trigger a mechanical or electrical issue, or a filesystem corruption, ransomware encrypting of your data, user error. Bit rot is possible but a minor problem here I think.

Make yourself a plan for a backup strategy, e.g. copy the data from 2 old hdd to one new bigger one. Repeat this step every 2-3 years.
Google for this topic to get some good ideas, 3 mediums, different storage types, different locations...

Remember: all hdd will die, some later some tomorrow. Fiddling with the old hdd will increases the chances for a fail.

Offline

#10 2020-09-16 18:59:31

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,014

Re: [SOLVED] Safely refresh hdd content

Ok guys, thank you for your time.

Time to remove the dust from my dvd writer then?

It seems optical media are the ones that can keep the data for longer time if you keep them cool and dry inside a drawer.

They are even cheap nowdays, Verbatim's single layer DVD-R costs about 0.3 euros each.
Now it is just a matter of choosing to buy a new hdd to backup the backup or using dvds.

Offline

#11 2020-09-17 14:48:53

ua4000
Member
Registered: 2015-10-14
Posts: 214

Re: [SOLVED] Safely refresh hdd content

without knowing more details about your data I cannot give you a proper advice.
But in general consider for your backup strategy:

- how valueable are your data really ?
- how much data are we taking about: GB or TB ?
- will the data be updated over time: no, little, yes, only file additions ? Revision required ?
- do you need to read the data seldom, often ? Fast random acces required ?
- how long do you have to keep the data preserved ?

Offline

#12 2020-09-17 18:42:43

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,014

Re: [SOLVED] Safely refresh hdd content

I've photos and videos mostly, about 100GB (8 years, growing) and i don't need to read the data, since it is a mere backup we're speaking about.
How long? Life time!
DVD seems the better choice if i plan to make a new copy every... 5 years?

Offline

Board footer

Powered by FluxBB