You are not logged in.

#1 2019-04-08 12:17:09

darrenldl
Member
Registered: 2013-06-04
Posts: 31

Blockyarchive - archive with FEC and sector level recoverability

EDIT : A asciicast demo is available here.

Blockyarchive/blkar (pronounced "bloc-kar") is a comprehensive utility for creating, rescuing, and general handling of SeqBox archives, with optional forward error correction

SeqBox is a single-file archive format designed by Marco Pontello that facilitates sector level data recovery for when file system metadata is corrupted/missing, while the archive itself still exists as a normal file on file system

Please visit the official SeqBox repo for the original implementation and technical details on this

Why SeqBox?
A SeqBox container/archive stores your file in SBX blocks internally. Each SBX block has its own header, allowing identification of the data position and the container the block belongs to. On supported file systems (file systems where the logical sector size is multiple of physical sector size, i.e. most file systems), the block is aligned to the sector boundary and sits inside a sector (if the selected block size is smaller than the sector size).

This means even when file system metadata is corrupted, or even with the partition table wiped, SeqBox containers can still be retrieved by simply reading at block/sector level. Retrieval of general data or archives in the case of faulty file systems, however, becomes very difficult as the file may have been spread out to multiple possibly non-consecutive sectors, and may even be out of order. In these cases, without file system metadata, there is no certain way to tell which chunk of data belongs to which file, and where it sits in the file.

The official SeqBox repository contains an excellent demo/illustration of a typical data recovery scenario, which you can see here.

Why blkar?
The original SeqBox implementation and specification provides sector level recoverability as described above. However, it does not provide protection against bit-rot or data corruption in general. This makes it less suitable for long term or robust data storage, since if certain SBX blocks are missing or corrupted, you still end up with only partial copy of your original data.

blkar adds forward error correction into the mix, allowing you to repair your SBX container. This makes it much more likely to recover your original data fully. Additionally, blkar supports arranging the SBX blocks in a burst error resistant way, allowing the SBX container to potentially survive burst (sector) errors (this depends on your specific parameters obviously).

blkar is also a lot more versatile and user friendly to use than the original SeqBox implementation. You can see the following wiki pages for how data rescue might look like with blkar

Features overall

  • Data recovery that does not depend on file system metadata (sector level recovery)

    • This allows data recovery even when data is fragmented and out of order

  • Supports error correction (via Reed-Solomon erasure code)

  • Supports burst (sector) error resistance

  • JSON mode

    • Outputs information in JSON format instead of human readable text, allowing easy integration with scripts

Installation
blkar is available on AUR, and also on crates.io

Links

Last edited by darrenldl (2019-04-13 03:00:58)

Offline

Board footer

Powered by FluxBB