A question about long term storage of video

nordog · 2010-04-14 13:15:37

I figured that even though this question doesn't relate to Arch directly and might be better suited for other forums, I would really appreciate the input of Archers.

So here is the backstory: I got married last year and a friend recorded the ceremony on his handheld camera. The file containing the video is several tens of gigabytes and at the moment it is stored on the backupserver at work (running Arch!). However, I can't keep it there for ever and at the moment the file is quite useless anyway since due to its size it is unplayable on most devices. It therefore needs to be encoded and compressed into a reasonable size and format, and this is the crux of the question.

I would of course like the video to be of high quality, but I also need to be able to play it for the forseeable future. I've done some reading on the various container formats and codecs and it looks like H264 is becomnig some sort of a standard and seems to produce nice quality videos. I think it would therefore be a safe bet that the video will be watchable in the years to come. However, I really don't like the whole patent issue and don't want to wake up one day and find out that I need to pay for a decoder to watch my own wedding video. I would therefore prefer something more free, like Theora.

It's difficult to get a completely objective comparison of the various video formats on the internet. It seems that by most accounts, H264 produces better quality videos, but by others Theora is said to have H264 beat. Furthermore, these comparisons also do not take into account the recent advances in Theora. For these reasons I'm leaning towards using Theora, but I have a nagging feeling that in say 20 years time I will neither be able to read it or reencode it as something else. If I were to constantly reencode my video once every 10 years for the next decades, I'm guessing it will start to look like crap down the line.

So, I'm wondering whether anyone has some useful insight into the matter?

Best regards

skottish · 2010-04-14 13:24:36

You're at no risk at all of having H264 unplayable because of any patents. 20 years from now that will remain true.

H264 is far superior to Theora. Theora was not designed for this level of compression. This is a fact that even Theora's devs agree with.

I highly recommend that you look into x264 for encoding the video. Since the input source is of such high quality, you should get great results out of it. I'd also recommend going over to one of the following forums:

http://forum.doom9.org
http://doom10.org

olvar · 2010-04-14 14:23:46

Not long ago, Google bought On2, the owners of a very nice video codec, VP8 (theora being VP3). It is possible that they release it in a (royalty-)free licence (read e.g. http://newteevee.com/2010/04/12/google- … l5-video/). It is very likely they will also start using it as the default codec in the html5 version of youtube, thus probably making it the new de facto standard.

I would recommend you to try to keep you original video for a bit longer (even if you decide to make another version using other codec) and wait and see what happens next. From the technical point of view, it seems that VP8 is comparable or even better than H.264

nordog · 2010-04-14 16:56:16

Thank you for your replies. I will hang onto the original copy by some means (perhaps an external hard drive) until things become more clear with VP8. In the meantime I think I'll go with the x264 encoder.

Video encoding isn't very easy to get into though. The number of options regarding the container format, codecs, compression and so forth are astounding, not to mention all the different CLI tools that are available: ffmpeg, transcode, mplayer. I tried to 'cheat' and use Avidemux to convert the video but it crashed, perhaps due to the file size. Do you guys recommend any particular tool for x264 encoding? I think I can probably figure out the necessary options by my self, but I need some sort of starting point.

As a side point: Any views on the actual mechanism of storage? I've had countless hard drives fail on me over the years and I usually keep two backups: one nonlocal backup that gets updated every month or so, and another local backup that is refreshed a few times a week. However, I thought I'd make two DVD copies of the video, but as we know they deteriorate over time (disc rot). Furthermore, the 'cloud' may evaporate, magnetic tape also deteriorates and so forth. It all sort of reminds me of Asimov's book (Prelude to Foundation?) where he touches upon the physical limits of data storage.

ngoonee · 2010-04-15 00:13:39

Its a wedding video, something you'd want to keep forever. Why not keep the original (10s of GB is quite cheap nowadays, buy 2 hard-disks and mirror them)?

Berticus · 2010-04-15 04:22:29

I'm assuming nordog will always keep the original, but is asking to compress and encode copies so he can watch it on other devices. There's nothing wrong with that.

Anyway, there isn't really a sure way of storing it and ensuring it lasts forever. The only thing you can really do is make several back-ups (preferrably of the original), and hope they don't all fail at the same time.

ngoonee · 2010-04-15 04:55:43

Berticus wrote:

I'm assuming nordog will always keep the original, but is asking to compress and encode copies so he can watch it on other devices. There's nothing wrong with that.
Anyway, there isn't really a sure way of storing it and ensuring it lasts forever. The only thing you can really do is make several back-ups (preferrably of the original), and hope they don't all fail at the same time.

If that's the case, he probably wouldn't be concerned about re-encoding? My impression was that he wanted to convert it to some format and keep it in that format.

Darksoul71 · 2010-04-15 08:38:20

@nordog:
Not shure about your source resolution but if we speak about PAL D1 (720x576) I would go for MPEG2 in DVD compatibility. Video bitrate set to 8 MBit and audio bitrate set to 384 for MP2 audio should be more then sufficent. Keep the video interlaced ! Some HQ denoiser might not hurt.

For encoding I would rather use HCEnc under Windows/WINE than any libavcodec based encoder (mencoder/VLC/avidemux). For optimal multicore usage search for HCEnc^n (which I coded *shameless plug* ). IMO HCEnc is superior in quality and does not share those bitrate spike issues which libav is well known for although I think that those issues are mostly adressed by using the XVid rate control algorithm.

I have the same concerns with the videos of my little one (PAL DV) and will switch over to an SDCard-based HD Cam soon. I think I'll use h264/AAC encoded videos inside an MP4 container for archiving purposes in future. In terms of storage I would go at least for double backups on external medias (e.g. rsyncing to two big external USB HDDs) and/or using RAID1 (or better RAID5) in your system. When it boils down to mostly read access and only partial write access even a pure software-based RAID5 works "ok".

Edit: For h264 encoding I've often used ffmpeg. I can only suggest to read the following guide:
http://rob.opendot.cl/index.php/useful- … ing-guide/

Last edited by Darksoul71 (2010-04-15 09:16:59)

Themaister · 2010-04-15 14:32:30

x264 <- Simple as that

Darksoul71 · 2010-04-15 15:14:13

Themaister wrote:

x264 <- Simple as that

For the codec it may be that simple but to someone with limited video knowledge a tool which encodes also audio (e.g. ffmpeg) or may be even provides some decent filters (e.g. avidemux) might be more suitable.

Esp. since x264 (the plain video encoding tool) does not encode any audio.

Last edited by Darksoul71 (2010-04-15 15:19:06)

beroal · 2010-04-15 20:22:47

You may have issues in the future with format for which only proprietary codecs exist. Because of deprecation. For example, there is Voxware audio format which I can not play on 64-bit Arch Linux because the codec is 32-bit Windows DLL. And that media file is written in 2006, only 4 years old.

Darksoul71 · 2010-04-15 20:28:38

In regard to compatibility in future I think that XVid/DivX with MP3 audio in AVI container, MPEG2 video with MP2 or AC3 audio in MPEG2 PS and h264 with various audio formats (AC3 / AAC) in various container formats (MP4 / MKV) can be considered "safe". I would rather not go through the hassle encoding something with a free codec simply because its free.

Simply compare ogg to MP3: Technically ogg is really superior in terms of features, quality and so on. But what about playback ? Pretty much every cheap portable player or cell phone can play back MP3 but (unfortunately) you can count the few players with ogg support.

Just my 2 cents,
D$

skottish · 2010-04-16 00:25:53

We do have to be careful in this thread in that the best of the best, software like x264 that is designed to do only one thing, are far beyond the scope of any swiss-army-knife tool. FFmpeg, while being one of my all time favorite pieces of software, does not specialize in encoders; It specializes in streaming and decoding. While it's true that some good quality encodes can come out of it, it's no where near the quality of the do-one-thing-only professional encoders like x264. Hell, even neroAacEnc, by far the best AAC encoder for Linux, is behind some if it's commercial competitors.

I bring this up because if high quality encodes is the ultimate goal of this thread, then learning the tools is helpful. And really, splitting the stream to encode separately is simple. Encoding with x264 with professional level settings is now simple. If Nero is the choice for AAC audio, then using it is simple. mkvtoolnix can put them back together with no fear of A/V sync issues.

DrZaius · 2010-04-27 00:51:57

The process skottish just described is the same as my method, but there are several ways to do it. You can use a decoder such as FFmpeg or MPlayer to pipe your input to x264. Alternately, x264 can be built with lavf input support allowing you to skip the need of another decoder. The repo x264 is not compiled with lavf support (nor is it compiled to allow .mp4 output, but that hardly matters).

My method to get lavf input support in x264 is to compile a minimal FFmpeg (without --enable-libx264) and then compile x264-git. I then uninstall FFmpeg and re-compile a more "normal" FFmpeg that includes --enable-libx264. I do this because I sometimes just use FFmpeg to encode with libx264 instead of x264 itself. For reference, here is my FFmpeg PKGBUILD.MINIMAL (just a hastily stripped down ffmpeg-svn):

pkgname=ffmpeg-svn
pkgver=22963
pkgrel=1
pkgdesc="Complete and free Internet live audio and video broadcasting solution for Linux/Unix"
arch=('i686' 'x86_64')
url="http://ffmpeg.mplayerhq.hu/"
license=('GPL')
depends=('zlib' 'imlib2')
makedepends=('subversion')
provides=("ffmpeg=`date +%Y%m%d`")
conflicts=('ffmpeg')
source=()
md5sums=()

_svntrunk=svn://svn.ffmpeg.org/ffmpeg/trunk
_svnmod=ffmpeg

build() {
  cd "$srcdir"

  if [ -d $_svnmod/.svn ]; then
    (cd $_svnmod && svn up -r $pkgver)
  else
    svn co $_svntrunk --config-dir ./ -r $pkgver $_svnmod
  fi

  msg "SVN checkout done or server timeout"
  msg "Starting make..."

  rm -rf "$_svnmod-build"
  mkdir "$_svnmod-build"
  cd "$_svnmod-build"
  "$srcdir/$_svnmod/configure" \
  --prefix=/usr \
  --disable-everything \
  --enable-gpl \
  --enable-decoders \
  --enable-demuxers \
  --enable-protocols \
  --enable-pthreads \
  --arch=`uname -m` \
  || return 1

  make || return 1
  make DESTDIR="$pkgdir" install || return 1
}

Now for the x264 encoding example:

x264 --crf 18 --preset veryslow -o x264output.mkv input.foo

Encoding the audio with neroAacEnc:

ffmpeg -i input.foo -f wav - | neroAacEnc -ignorelength -if - -of audio.aac

Combining with mkvtoolnix:

mkvmerge -o final.mkv x264output.mkv audio.aac

You could also use FFmpeg or MP4Box to combine as well.

Last edited by DrZaius (2010-04-27 01:18:58)

skottish · 2010-04-27 01:13:25

Nice tutorial DrZaius.

To illustrate what one gets by using x264's presets, this is what --veryslow passed (with --threads 0 on my machine):

 cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=8 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / wpredp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00

It doesn't get more simple than that.

DrZaius · 2010-04-27 16:31:06

skottish wrote:

Nice tutorial DrZaius.

Thanks, skottish.

skottish wrote:

To illustrate what one gets by using x264's presets, this is what --veryslow passed (with --threads 0 on my machine):
...

I may be wrong, but I don't think you need to declare --threads anymore:

$ x264 --crf 22 --preset fast -o z-none.mp4 IronMan.mkv
$ strings z-none.mp4 | grep x264
x264 - core 94 r1564 a927654 - H.264/MPEG-4 AVC codec - Copyleft 2003-2010 - http://www.videolan.org/x264.html - options: cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=6 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 /
threads=12 /
sliced_threads=0 nr=0 decimate=1 interlaced=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 wpredb=1 wpredp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=30 rc=crf mbtree=1 crf=22.0 qcomp=0.60 qpmin=10 qpmax=51 qpstep=4 ip_ratio=1.40 aq=1:1.0

Which leads me to another tip for our new x264 users (as skottish already displayed): you can use strings as in my above example to see the settings used on a video encoded by x264.

skottish · 2010-04-28 00:16:16

Wow! I didn't know about strings in this context. Between that and ffprobe from FFmpeg, I don't need mediainfo anymore. Thanks doc!

Arch Linux

#1 2010-04-14 13:15:37

A question about long term storage of video

#2 2010-04-14 13:24:36

Re: A question about long term storage of video

#3 2010-04-14 14:23:46

Re: A question about long term storage of video

#4 2010-04-14 16:56:16

Re: A question about long term storage of video

#5 2010-04-15 00:13:39

Re: A question about long term storage of video

#6 2010-04-15 04:22:29

Re: A question about long term storage of video

#7 2010-04-15 04:55:43

Re: A question about long term storage of video

#8 2010-04-15 08:38:20

Re: A question about long term storage of video

#9 2010-04-15 14:32:30

Re: A question about long term storage of video

#10 2010-04-15 15:14:13

Re: A question about long term storage of video

#11 2010-04-15 20:22:47

Re: A question about long term storage of video

#12 2010-04-15 20:28:38

Re: A question about long term storage of video

#13 2010-04-16 00:25:53

Re: A question about long term storage of video

#14 2010-04-27 00:51:57

Re: A question about long term storage of video

#15 2010-04-27 01:13:25

Re: A question about long term storage of video

#16 2010-04-27 16:31:06

Re: A question about long term storage of video

#17 2010-04-28 00:16:16

Re: A question about long term storage of video

Board footer