Arch Linux

ua4000

Member

Registered: 2015-10-14

Posts: 557

[SOLVED] bsdtar : ok with file, corrupt with pipe output ?

Hi, can someone have a look and spot my mistake ?
bsdtar works with a file output (01), but has a corrupt output in pipe mode (02).

$ file=textfile.txt
$ echo 'Hello World.' > ${file}

$ bsdtar --create --verbose --zstd --options zstd:threads=4  --file "${file}-01.tar.zst" "${file}"
a textfile.txt

$ bsdtar --create --verbose --zstd --options zstd:threads=4  --file -                    "${file}" > "${file}-02.tar.zst"
a textfile.txt

$ bsdtar tf "${file}-01.tar.zst"
textfile.txt

$ bsdtar tf "${file}-02.tar.zst"
bsdtar: Error opening archive: Zstd decompression failed: Unknown frame descriptor

$ ls -la ${file}*
-rw------- 1 s s    13 Dec 31 12:30 textfile.txt
-rw------- 1 s s   111 Dec 31 12:30 textfile.txt-01.tar.zst
-rw------- 1 s s 10240 Dec 31 12:30 textfile.txt-02.tar.zst

Comparing the two tar.zst files: both have the same binary data at the beginning, 02 has then a lot 00-bytes extra.

This issue matters here, because I plan to pipe the data then into gpg encryption... therefore I want to skip the in-between file, something like

bsdtar ... | gpg --symmetric 

Thanks a lot.

Edit: [SOLVED] for me with a workaround for bsdtar: --file /dev/stdout

Last edited by ua4000 (2025-01-04 10:31:33)

schard

Forum Moderator

From: Hannover

Registered: 2016-05-06

Posts: 2,647

Website

Re: [SOLVED] bsdtar : ok with file, corrupt with pipe output ?

I can reproduce this here. The file from the pipe has excess trailing zero bytes.
You can fix the file via

truncate -s 111 textfile.txt-02.tar.zst

As to why this happens, I have no idea.

---

Inofficial first vice president of the Rust Evangelism Strike Force

frostschutz

Member

Registered: 2013-11-15

Posts: 1,646

Re: [SOLVED] bsdtar : ok with file, corrupt with pipe output ?

I'm not familiar with bsdtar. This could simply be a bug. The file works fine if you truncate the surplus \0 bytes.

--block-size seems to affect it somehow so it could be a tape alignment thing? No idea really.

ua4000

Member

Registered: 2015-10-14

Posts: 557

Re: [SOLVED] bsdtar : ok with file, corrupt with pipe output ?

Thanks for looking into this.

I think I will stay with a regular tar.zst file as intermediate step,  then invoke gpg, then wipe the tar.zst

Also I encountered the next pipe issue: bsdtar ... | gpg --symmetric won't work:
symmetric gpg with pinentry-tty (password entry on console/ no X) seems to use both stdin...

$ date |  gpg --symmetric --output "test.gpg"
gpg: cancelled by user
gpg: error creating passphrase: Operation cancelled
gpg: symmetric encryption of '[stdin]' failed: Operation cancelled

Invoking gpg with a regular file prompts for the password twice in terminal and works fine.

ua4000

Member

Registered: 2015-10-14

Posts: 557

Re: [SOLVED] bsdtar : ok with file, corrupt with pipe output ?

Info from a developer "kientzle" about the extra 00-bytes:
"The padding default is legacy from the days when people wrote backup archives to tape drives which absolutely demanded that the archive size be padded in a certain way."

Workaround for bsdtar:
While "--file - " adds padding, "--file /dev/stdout" does not.

He's thinking about patching bsdtar, because tape drive are usually not used anymore directly with tar, but the patch may break  things for other users /usecases.
Another idea is: make zstd more robust, to ignore padding bytes...

Last edited by ua4000 (2025-01-04 10:32:37)