diff -r question.

89c51 · 2014-08-18 11:31:45

I use diff -r in two directories containing flac files on different drives (different filesystems in case it matters) and in some cases i get the message "Binary files X1.flac and X1.flac differ".

However when i test them with flac -t (on both drives) they seem to be ok.

Is there a reason that this happens? An underlying problem maybe?

ewaller · 2014-08-18 15:59:45

They may be different lengths. Compare the file lengths reported by ls -l
Also, you can pipe them through od to see the contents in octal. (Unless I am using an 18 bit computer, I despise octal, so I would pipe those through od -x -Ax to see hex data and hex addresses)

In any event, you could then see if one or the other are padded with 00, or you can diff the outputs of od for the two files and see exactly which bytes are different.

Last edited by ewaller (2014-08-18 16:00:10)

alphaniner · 2014-08-18 16:12:46

Maybe different tags? But then I don't understand why you assume the difference must be related to corruption.

89c51 · 2014-08-18 20:18:39

alphaniner wrote:

Maybe different tags? But then I don't understand why you assume the difference must be related to corruption.

I have no idea where the difference lies ΤΒΗ. I just want my music to be "safe" until i get money to invest in a NAS (synology etc). They are files that are copied from the same source to both drives or from one drive to another.

The thing is that each time i run diff -r (and without changing anything) different files appeared to differ. No idea what can cause this.

alphaniner · 2014-08-18 20:45:44

diff determines different differences... definitely disturbing.

Sorry, couldn't resist. I've been using diff for this kind of thing for a while, and in my experience it's been reliable. Nevertheless, try a different tool. Use 'cmp' to compare the files manually eg.

$ cd /mnt/foo
$ find . -name "*flac" >/tmp/foo.files
$ cd /mnt
$ while read i; do cmp "/mnt/foo/$i" "/mnt/bar/$i"; done </tmp/foo.files

or generate md5sums from one source and check them at the other eg.

$ cd /mnt/foo
$ find . -name "*flac" -exec md5sum {} \; >/tmp/foo.sums
$ cd /mnt/bar
$ md5sum -c /tmp/foo.sums

ewaller · 2014-08-18 21:16:02

I do also suggest the md5sum solution.

If they are different, you can use my suggestion to find out how. Like this:

ewaller$@$odin ~/devel 1041 %cat hello.c

#include <stdio.h>
#include <stdlib.h>
int main (int argc, char *argv[])
{
  printf ("Hello world!\n");
}
ewaller$@$odin ~/devel 1042 %gcc hello.c
ewaller$@$odin ~/devel 1043 %cp a.out a.broke
ewaller$@$odin ~/devel 1044 %echo 'Blat' >> a.broke
ewaller$@$odin ~/devel 1045 %od -x -Ax a.out >a.hex
ewaller$@$odin ~/devel 1046 %od -x -Ax a.broke >a.broke.hex
ewaller$@$odin ~/devel 1047 %diff a.hex a.broke.hex
417,418c417,418
< 001a60 695f 696e 0074
< 001a66
---
> 001a60 695f 696e 0074 6c42 7461 000a
> 001a6b
ewaller$@$odin ~/devel [1]1048 %

89c51 · 2014-08-18 22:38:23

Generating md5 wont work i think because i don't know which is the correct to start with. And i think flac -t compares an md5 generated during encoding to one of the current file. With metaflac you can compare fingerprints (md5) between flacs.

BTW could it be something RAM related? I get some weird crashes from time to time and people suggested ram issues even though memtest comes clean after several passes (3-4).

alphaniner · 2014-08-19 03:34:37

Well, FF ate my post because my session timed out. So to summarize:

Generate md5sums anyway. It won't tell you which of a pair is 'correct' but it will be useful as another means of comparison. Also if you generate sums of foo and check them against both foo and bar multiple times, you could find that foo always checks out but bar always gives different results. In which case foo is probably sound and something is wrong with bar.

If flac stores an md5hash, it can't be of the entire file, because storing the hash in the file would change the hash of the file. Maybe it hashes the 'data' (eg. encoded audio) in 'metadata' (eg. a tag or something). But if so and the hash checks out then the data must be sound. Which would make the situation even more confusing.

Issues with RAM, CPU, disk, cable, and storage controller all could cause miscompares. But I'd expect problems to be more widespread. OTOH 'diff -r' involves a prolonged period of stress which could be relevant.

R00KIE · 2014-08-19 07:57:03

Just want to add a +1 to the idea of storing a md5sum of the files somewhere. Personally I store it in extended attributes. I have made my own custom script but there are other alternatives like md5deep [1].

You mention you plan to get yourself a NAS, since you also mention data safety I suppose you will be using some raid level. Now Imagine you are using raid 1 and you have some problem with the array, now you have (at least) 2 copies and no way to know which one is the good one(1) unless you have an md5sum (or other *sum) to compare with.

(1) Btrfs raid and checksums might alleviate this problem but I wouldn't count on any NAS supporting btrfs just yet, besides, btrfs is still rough around the edges so not the best bet to store important data imho.

[1] https://aur.archlinux.org/packages/md5deep

89c51 · 2014-08-19 10:23:43

No. No raid or anything at this point. Just two drives that i copy stuff. Not much of a job -in order to automate stuff- since i don't manage a huge number of files.

And Yes, i know i need a better/safer solution but no money at this point. When the time comes i'll probably consult people here.

R00KIE · 2014-08-19 12:20:02

89c51 wrote:

No. No raid or anything at this point. Just two drives that i copy stuff. Not much of a job -in order to automate stuff- since i don't manage a huge number of files.
And Yes, i know i need a better/safer solution but no money at this point. When the time comes i'll probably consult people here.

You can still do raid without NAS. I use 2 identical(1) usb drives in raid1 for stuff I really don't want to lose, that is why I mentioned raid. Before anyone points out that for really important stuff I should also have offsite backups, I do know that, but for now I can't afford to maintain multi gigagbyte backups offsite, plus I don't trust online very much.

(1) Different drives will also do but will be limited by the capacity of the smaller drive.

alphaniner · 2014-08-19 13:17:27

Cost isn't so much of an issue if you're a bit creative. Aside from what I already had lying around, most of the components in my server (NAS/SAN) came from eBay:

Six drive Intel hot swap enclosure (newish): $15
Four drive Intel hot swap enclosure (new): $25
SuperMicro mobo w/6 SATA ports (used): $20
LSI SAS HBA w/4 ports (used): $15
CPU (used): $12

Granted, I'm still in the process of designing a cooling solution for the enclosures and figuring out how to mount them in my old Antec Twelve Hundred case...

ROOKIE wrote:

Personally I store it in extended attributes.

I wasn't aware this was possible. Can you point me to some resources on this subject?
Nevermind, figured it out.

Last edited by alphaniner (2014-08-19 13:45:05)

R00KIE · 2014-08-19 13:50:00

alphaniner wrote:

ROOKIE wrote:
Personally I store it in extended attributes.
I wasn't aware this was possible. Can you point me to some resources on this subject?

First you need a filesystem that supports extended attributes, ext4 does.

I don't recall if I searched about it in google, the resources you already have are 'man setfattr' and 'man getfattr'.

What I do is:
1. determine md5sum and save result in variable
2. setfattr -n user.md5sum -v $md5sum_result -- $file

If you want to read back the stored attribute you can do the following:
getfattr --only-values -n user.md5sum -- $file
or
getfattr -d -- $file

The -- is only to safeguard against filenames that might have dashes or anything else that may be misinterpreted. Usually I use the second form, the output is more verbose but easier to remember and you can do 'getfattr -d *'. The first form is useful if you script it and want to check the md5sum later.

Of course this supports other hashes too, I selected md5sum because it should be fast and good enough.

Edit:
It seems I took too long to write a reply I'll leave this here in case anyone else wonders about it.

Last edited by R00KIE (2014-08-19 13:55:43)

alphaniner · 2014-08-19 14:40:39

@ROOKIE

I made some changes to the File permissions and attributes wiki page to include some basic info on extended attributes.

Last edited by alphaniner (2014-08-19 14:42:07)

R00KIE · 2014-08-19 15:59:59

@alphaniner
I have steered off using attr because of what it says in the man page (although it does work fine with ext4):

It is thus aimed specifically at users of the XFS filesystem - for filesystem independent extended attribute manipulation, consult the getfattr(1) and setfattr(1) documentation.

alphaniner · 2014-08-19 16:31:02

@ROOKIE

See discussion tab of the wiki page. I think we've OT'd this thread enough.

Arch Linux

#1 2014-08-18 11:31:45

diff -r question.

#2 2014-08-18 15:59:45

Re: diff -r question.

#3 2014-08-18 16:12:46

Re: diff -r question.

#4 2014-08-18 20:18:39

Re: diff -r question.

#5 2014-08-18 20:45:44

Re: diff -r question.

#6 2014-08-18 21:16:02

Re: diff -r question.

#7 2014-08-18 22:38:23

Re: diff -r question.

#8 2014-08-19 03:34:37

Re: diff -r question.

#9 2014-08-19 07:57:03

Re: diff -r question.

#10 2014-08-19 10:23:43

Re: diff -r question.

#11 2014-08-19 12:20:02

Re: diff -r question.

#12 2014-08-19 13:17:27

Re: diff -r question.

#13 2014-08-19 13:50:00

Re: diff -r question.

#14 2014-08-19 14:40:39

Re: diff -r question.

#15 2014-08-19 15:59:59

Re: diff -r question.

#16 2014-08-19 16:31:02

Re: diff -r question.

Board footer