[Fixed] Command `file` does not recognise WebM as video any more

stefan · 2018-05-26 15:33:53

Hi.

Some of my video files (mostly from Youtube via `youtube-dl`) are not recognised as videos anymore. This problem occurs for `.mkv` and `.webm`, but not for `.mp4` files.

$ file --version
file-5.33
magic file from /usr/share/file/misc/magic

$ file --mime film.webm 
film.webm: application/octet-stream; charset=binary

$ file film.webm 
film.webm: , created: Thu Jul 11 00:22:26 1974, modified: Sun Jul 17 00:26:42 2033

This is a regression from `file`s previous version which runs fine. I've verified this by downgrading:

$ file --version
file-5.32
magic file from /usr/share/file/misc/magic

$ file film.webm 
film.webm: WebM

$ file --mime film.webm 
film.webm: video/webm; charset=binary

Unfortunately the mailing list and bug tracking system [1] mentioned at the `file` homepage [2] are down.

Any idea whom to contact or how to solve this issue?

____________________
[1] bugs.gw.com
[2] https://www.darwinsys.com/file/

Last edited by stefan (2018-05-26 19:23:26)

loqs · 2018-05-26 15:50:53

You might want to change the title from find to file.
https://bugs.archlinux.org/task/58626#comment169455 notes similar issues contacting upstream and has one working email address.

stefan · 2018-05-26 16:12:52

loqs wrote:

You might want to change the title from find to file.

Oooohcrap! Fixed, thank you!

loqs wrote:

https://bugs.archlinux.org/task/58626#comment169455

Thanks for the pointer, although I'm not sure how seccomp relates to my problem in detail...

loqs · 2018-05-26 16:16:54

I do not think seccomp is related but the post I linked to had a working email address for upstream as the website / bug tracker was down then as well.
You could also try bisecting between 5.32 and 5.33 and see if you can find a commit that caused the issue.

Trilby · 2018-05-26 16:20:24

FWIW, I can't replicate the problem:

$ file -v
file-5.33
magic file from /usr/share/file/misc/magic

$ file out.webm
out.webm: WebM

$ file --mime out.webm
out.webm: video/webm; charset=binary

loqs · 2018-05-26 16:45:52

You are both using same release of file 5.33?

stefan · 2018-05-26 16:54:59

Trilby wrote:

FWIW, I can't replicate the problem:

Maybe it fails only on some WebM files? I do not know aboout variations of this container structure...

I've added pacman output:

$ file film.webm 
film.webm: WebM
$ file --version
file-5.32
magic file from /usr/share/file/misc/magic
$ pacman -Q file
file 5.32-1

versus

$ file film.webm 
film.webm: , created: Thu Jul 11 00:22:26 1974, modified: Sun Jul 17 00:26:42 2033
$ file --version
file-5.33
magic file from /usr/share/file/misc/magic
$ pacman -Q file
file 5.33-1

Last edited by stefan (2018-05-26 16:56:32)

Trilby · 2018-05-26 17:45:14

I didn't have any webm files around, but I grabbed one from here to test. Stefan, does `file` correctly identify webm files from that site for you?

stefan · 2018-05-26 18:31:40

Now this really is becoming entertaining: It works on your file, but not on mine.

$ curl -OL 'http://dl3.webmfiles.org/big-buck-bunny_trailer.webm'
$ sha1sum big-buck-bunny_trailer-big-buck-bunny_trailer.webm
cafcc2df4c3998ba5ab94b5262ef3369502488f7  big-buck-bunny_trailer-big-buck-bunny_trailer.webm

$ file --version
file-5.33
magic file from /usr/share/file/misc/magic

$ file big-buck-bunny_trailer-big-buck-bunny_trailer.webm
big-buck-bunny_trailer-big-buck-bunny_trailer.webm: WebM

$ file --mime big-buck-bunny_trailer-big-buck-bunny_trailer.webm
big-buck-bunny_trailer-big-buck-bunny_trailer.webm: video/webm; charset=binary

But this fails with *some* videos I download with `youtube-dl`. I do not know how `youtube-dl` decides on the output format, but it sometimes produces `.mp4`, `.mkv`, and `.webm`.

For the following example, `youtube-dl` extracts two files, one for audio and one for video (both WebM), and merges them to a new file (WebM). I've just now learned about the `-k` switch which keeps the first two files after the merge:

$ youtube-dl -k --restrict-filenames -o 'video.%(ext)s' 'https://www.youtube.com/watch?v=Cld0p3a43fU'
$ sha1sum video*
7ff6e05bd454a089b97b3efe4864c0ae1107362c  video.f248.webm
e1a4e59498ab7fd67fd3a35913383818e66200f4  video.f251.webm
4773dacbf99d87a1a5abc2d8e3965b1d6e71c9cf  video.webm

Now `file` correctly guesses for the first two files, but *not* for the merged one:

$ file video.*
video.f248.webm: WebM
video.f251.webm: WebM
video.webm:      , created: Thu Jul 11 00:22:26 1974, modified: Sun Jul 17 00:26:42 2033

$ file --mime video.*
video.f248.webm: video/webm; charset=binary
video.f251.webm: video/webm; charset=binary
video.webm:      application/octet-stream; charset=binary

One could argue that the merge is broken, but `mplayer` happily plays all three of them, reporting

[lavf] stream 0: video (vp9), -vid 0
[lavf] stream 1: audio (opus), -aid 0, -alang eng

for the merged file, and just one of the streams for the extracted files.

So I guess there's a variant of the WebM container format that `file` does not grok any more. It does work wit the old version of `file` though.

loqs · 2018-05-26 18:43:21

I would suggest using file-git you bisect between 5.33 and 5.32 and see what the cause is.
Edit:
Will see if I can do it on this system.
Edit2:
Seems to be fixed by https://github.com/file/file/commit/4fb … e6de7d574f
Edit3:
Fixed by applying https://github.com/file/file/commit/4fb … 574f.patch

Last edited by loqs (2018-05-26 19:08:48)

Trilby · 2018-05-26 19:01:07

It looks like the ones that are recognized are vp8 while those that aren't are vp9.

stefan · 2018-05-26 19:20:17

Ok, this seems to be a known fixed bug in the magic file. See git log of https://github.com/file/file, commit 4fb6a59d reads “fix bogus magic”. Checking out the commit before, 87c533bf, and just making and using the magic file from that, displays the error. Also, the `file` packed by ArchLinux works with the magic file from commit `4fb6a59d`.

I'll flag ArchLinux' `file` outdated.

Trilby · 2018-05-26 19:21:48

stefan wrote:

I'll flag ArchLinux' `file` outdated.

Please don't. It is not outdated.

EDIT: it seems I was too late. Please do not missuse flags this way, it just decreases the signal to noise ratio so real out-of-date flags may not be noticed.

Last edited by Trilby (2018-05-26 19:24:05)

stefan · 2018-05-26 19:24:47

Trilby wrote:

stefan wrote:
I'll flag ArchLinux' `file` outdated.
Please don't. It is not outdated.

Oh no, too late. Why not? I've left a message to the packager with a link to this discussion. You think thatwas a bad idea? What now?

Trilby · 2018-05-26 19:26:21

Why not, because it's not out of date. A bug/feature-request might be appropriate if you were hoping the maintainer might backport the relevant change, but the out-of-date flag is meant for out of date packages (hence the name) where our version is behind upstreams last versioned/tagged release.

Arch Linux

#1 2018-05-26 15:33:53

[Fixed] Command `file` does not recognise WebM as video any more

#2 2018-05-26 15:50:53

Re: [Fixed] Command `file` does not recognise WebM as video any more

#3 2018-05-26 16:12:52

Re: [Fixed] Command `file` does not recognise WebM as video any more

#4 2018-05-26 16:16:54

Re: [Fixed] Command `file` does not recognise WebM as video any more

#5 2018-05-26 16:20:24

Re: [Fixed] Command `file` does not recognise WebM as video any more

#6 2018-05-26 16:45:52

Re: [Fixed] Command `file` does not recognise WebM as video any more

#7 2018-05-26 16:54:59

Re: [Fixed] Command `file` does not recognise WebM as video any more

#8 2018-05-26 17:45:14

Re: [Fixed] Command `file` does not recognise WebM as video any more

#9 2018-05-26 18:31:40

Re: [Fixed] Command `file` does not recognise WebM as video any more

#10 2018-05-26 18:43:21

Re: [Fixed] Command `file` does not recognise WebM as video any more

#11 2018-05-26 19:01:07

Re: [Fixed] Command `file` does not recognise WebM as video any more

#12 2018-05-26 19:20:17

Re: [Fixed] Command `file` does not recognise WebM as video any more

#13 2018-05-26 19:21:48

Re: [Fixed] Command `file` does not recognise WebM as video any more

#14 2018-05-26 19:24:47

Re: [Fixed] Command `file` does not recognise WebM as video any more

#15 2018-05-26 19:26:21

Re: [Fixed] Command `file` does not recognise WebM as video any more

Board footer