[BUG] File compression unexpectedly stops and does not create anything

byjove01 · 2023-12-23 01:16:22

Hi.
I'm archiving heavy important data (on a scale from few to hundred Gb) and I need to nicely compress it to gain disk space.
I chose LZMA as compression format because AFAIK it's the best one adaptated to this goal... but because life can often be malicious, I could only create a few compressed archives before Dolphin (or Ark) suddenly stops the process and send a notification that asserts it was a success but returns a broken link that doesn't lead to anything... because there's no file. I just lost my time.
What the hell?
It even happens to the progress bar to go on a infinite loop with no defined max value, until a) the issue happens again b) it goes back to a defined progress value for a short time.

Kernel is 6.6.7-arch1-1. Using KDE.

Last edited by byjove01 (2024-01-11 14:23:33)

mpan · 2023-12-23 07:05:05

Hello. Invoke the compression program directly from a terminal and see, what is the outcome.

I’m not using Ark, but apparently it is a frontend to libarchive and/or p7zip. LZMA is handled by xz and 7z, so you may try either.

As for choosing the tool:

xz is compressing a data stream, not archiving files. You must first archive files using another tool, e.g. tar. This makes archive scanning and decompression a linear process (slow), however tar stores file attributes better than 7zip.
7z is an archiver and compressor, so it handles both. This means scanning and decompression can use radom access (fast). However, it may not store some file attributes

byjove01 · 2023-12-23 13:39:49

According to the Github page, Ark is using the 7z binary for all 7-Zip-related archive formats. I opened a terminal and tried to create a test archive via CLI, but I got this output:

[byjove@pchost-1 GameFolder]$ 7z a "data.tar.lzma" "folder1" "folder2" "folder3" "file.txt"
7-Zip (a) 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
64-bit locale=fr_FR.UTF-8 Threads:32 OPEN_MAX:1024, ASM



System ERROR:
E_NOTIMPL : Not implemented

mpan · 2023-12-23 13:56:28

7z a data.7z FILEs…

To set compression level to N (0…9):

7z a -mxN data.7z FILEs…

Last edited by mpan (2023-12-23 13:56:38)

byjove01 · 2023-12-23 16:36:57

[byjove@pchost-1 GameFolder]$ 7z a -mx9 "test.7z" "data"

7-Zip (a) 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
64-bit locale=fr_FR.UTF-8 Threads:32 OPEN_MAX:1024, ASM

Scanning the drive:
52 folders, 388 files, 10002607434 bytes (9540 MiB)

Creating archive: test.7z

Add new data to archive: 52 folders, 388 files, 10002607434 bytes (9540 MiB)


Files read from disk: 388
Archive size: 7656770550 bytes (7303 MiB)
Everything is Ok

However, the data was compressed into a traditional 7z archive, and not a LZMA-formatted one.

Head_on_a_Stick · 2023-12-23 20:10:59

byjove01 wrote:

the data was compressed into a traditional 7z archive, and not a LZMA-formatted one

man 7z wrote:

       -m0=lzma
              lzma method

byjove01 · 2023-12-23 21:37:32

It worked, but it does not solve my Dolphin-related problem at all. It would be annoying to ignore the Ark-based GUI and use only CLI to compress files.
By the way, with the file command, I noticed that the output archive was considered as...

file1: 7-zip archive data, version 0.4

while previous LZMA archives, that I made via the Dolphin/Ark GUI, were classified like this:

file2: LZMA compressed data, streamed

How can I check if this recent archive was definitely compressed with LZMA? Maybe I should have used XZ instead of 7zip?

Last edited by byjove01 (2023-12-23 21:38:49)

seth · 2023-12-24 00:08:26

According to the Github page, Ark is using the 7z binary for all 7-Zip-related archive formats.
LZMA archives, that I made via the Dolphin/Ark GUI, were classified like "LZMA compressed data, streamed"

so why ar we focussing on 7z?

Also

Dolphin (or Ark) suddenly stops the process and send a notification that asserts it was a success but returns a broken link that doesn't lead to anything... because there's no file

What makes you believe the compression format is relevant tbw? Did you try a different algorithm?

Then I'd point out

I'm archiving heavy important data (on a scale from few to hundred Gb) my porn collection and I need to nicely compress it to gain disk space.

Where's the destionation for the compressed files? The same partition where you need to "gain disk space"?
Ie. could the problem simply be that you're running out of space here?

byjove01 · 2023-12-24 06:19:27

seth wrote:

so why ar we focussing on 7z?

Because I don't know what program produces the last data type... that's all. 7z being a very popular archiver, I thought it could just be the one Dolphin/Ark was using to compress files.

What makes you believe the compression format is relevant tbw? Did you try a different algorithm?

I obviously can't be confident about the issue being specifically related to LZMA but I didn't have this problem with other algorithms. However, LZMA is the one I want to use.

I'm archiving heavy important data (on a scale from few to hundred Gb) my porn collection

https://tenor.com/bvOlN.gif

Where's the destionation for the compressed files? The same partition where you need to "gain disk space"?
Ie. could the problem simply be that you're running out of space here?

I just do not run of out of space. I would've noticed it if it happened, but I'm far to reach the 100% used disk space.

Last edited by byjove01 (2023-12-24 06:19:49)

seth · 2023-12-24 08:16:47

Your current theory is that ark/dolphin uses some subprocess to compress data and that specific subprocess fails the compression.
To figure whether and which subprocess is used, check the process tree ("ps fax" might be helpful), I'd assume the compression runs long enough for that?
To figure whether it is an actual problem with this specific compression, test the same (entire) data against some other algorithm.

I didn't have this problem with other algorithms

is underspecific or meaningless in this case.
If you used some other algorithm 3 years ago on a 1MB text file, there're obviously a lot of variables when compared to the situation at hand (volume, available disk space and updates across the entire software stack)

I just do not run of out of space. I would've noticed it if it happened, but I'm far to reach the 100% used disk space.

By the power of makebelieve?
Did you monitor the disk space while compressing?
Did you estimate the target data size and compare that w/ the available space of the destination partition and/or possible quotas?
Are there any IO issues in dmesg (ie. is the available space maybe not what you think it is b/c the HW is falling apart)?
For an unprivileged user and extfs the magical limit is btw. 95%

https://tenor.com/bvOlN.gif

https://bbs.archlinux.org/help.php#bbcode
There's however actually a point to this: "heavy important data" is meaningless pretentious noise - and there's a huge difference whether you're compressing some low-entropy statistical datasets (where you can expect compression rates north, possibly *way* north of 10:1) or some high-end video codec (which is virtually incompressible, the resulting file may get slightly bigger) when it comes to estimations of the target size.

byjove01 · 2023-12-24 11:04:27

seth wrote:

Your current theory is that ark/dolphin uses some subprocess to compress data and that specific subprocess fails the compression.
To figure whether and which subprocess is used, check the process tree ("ps fax" might be helpful), I'd assume the compression runs long enough for that?

As I said in op, the notification mostly goes in a loop (so you can believe it's somehow working) before to return a false positive. It seems like I was wrong about the Dolphin subprocess. "ps fax" (http://0x0.st/Hg6f.txt) didn't output anything like that.

> To figure whether it is an actual problem with this specific compression, test the same (entire) data against some other algorithm.
> "I didn't have this problem with other algorithms" is underspecific or meaningless in this case.
If you used some other algorithm 3 years ago on a 1MB text file, there're obviously a lot of variables when compared to the situation at hand (volume, available disk space and updates across the entire software stack)

Yesterday isn't 3 years ago. But you're right : I ran several archivation processes with different compression algorithms and most (tar, tar.gz, tar.lz4, tar.zst) are correctly working while other aren't (tar.lzma, tar.lz and tar.xz). As I'm writing this post, the first ones I mentioned finished to compress a while ago, whileas the LZMA one sent the false positive. tar.xz and tar.lz didn't even do that and are still reported as working.

And they'll probably never finish.

By the power of makebelieve?
Did you monitor the disk space while compressing?
Did you estimate the target data size and compare that w/ the available space of the destination partition and/or possible quotas?
Are there any IO issues in dmesg (ie. is the available space maybe not what you think it is b/c the HW is falling apart)?
For an unprivileged user and extfs the magical limit is btw. 95%

No, because the disk space widget at the bottom of my file explorer window didn't report anything like this and it's anyways improbable because I'm getting stuck on a 9Gb data package. My total disk space is 275Gb. I don't expect to run out of free space unless some serious crap happens. But if you want more precise data, here is my dmesg output (https://0x0.st/Hg6g.txt).
TL;DR => Didn't found special dmesg errors.

There's however actually a point to this: "heavy important data" is meaningless pretentious noise - and there's a huge difference whether you're compressing some low-entropy statistical datasets (where you can expect compression rates north, possibly *way* north of 10:1) or some high-end video codec (which is virtually incompressible, the resulting file may get slightly bigger) when it comes to estimations of the target size.

It's me naturally telling this data is heavy and important. No pretentious tone intended at all.

Last edited by byjove01 (2023-12-24 11:05:36)

seth · 2023-12-24 12:55:10

the notification mostly goes in a loop (so you can believe it's somehow working)

Here's a conclusive list of things you can infer from from some spinning throbber or a progressbar:
1. a throbber is spinning/progressbar is moving
Dolphin might just kick off the notification, then falls on its nose and the notification is simply sitting there, waiting for a timeout.

=> Does the CPU load rise (significantly, you'd expect at least one core to be fully charged) while we're not believing that something's happening?
=> If you run dolphin/ark from an interactive shell, does it yell errors at you when trying to compress sth.?

Because the pattern you discribe fits one of two common scenarios:
1. writing errors leading to an abort (and cleanup) - typically because you're running out of space since the requirement cannot be perfectly predicted
This can be ruled out if a different algorithm on the same data and destination works.
2. nothing actually ever happens (hence check the CPU load)

most (tar, tar.gz, tar.lz4, tar.zst) are correctly working

Do those kick off subprocesses?
If https://github.com/KDE/ark is relevant and correct, most algorithms just use https://archlinux.org/packages/core/x86_64/libarchive/ - incl. gz, lz4 and zstd

----

It's me naturally telling this data is heavy and important.

What is an entirely meaningless noise of random words.
Data is not "heavy", bytes have no mass and "important" is subjective at best.
Unless you're going to detail on the nature of the data and how that might be relevant the only thing that's remotely relevant is the filesize. The rest is noise.

I'll try to explain that: every line sombody writes consists of information, silent assumptions and noise.
If you want to solve a problem - any problem - to need to test the assumptions to turn them into information and convert the noise through relevant details or just dump it.

Forming meaningful questions about a problem quite often makes the solution or path there rather obvious.

mpan · 2023-12-25 01:51:34

byjove01 wrote:

However, the data was compressed into a traditional 7z archive, and not a LZMA-formatted one.

byjove01 wrote:

How can I check if this recent archive was definitely compressed with LZMA? Maybe I should have used XZ instead of 7zip?

The algorithm used by both 7z and xz is LZMA, unless you explicitly ask them to use a different compression method. So there is nothing to check.

byjove01 wrote:

It worked, but it does not solve my Dolphin-related problem at all. It would be annoying to ignore the Ark-based GUI and use only CLI to compress files.

I asked to use xz or 7z to receive the error (if any). But no error is a bit puzzling, as it suggests a bug is either in Ark or libarchive. I believe you may test libarchive by invoking its bsdtar:

bsdtar -czf ARCHIVE_FILE INPUT_FILEs…

byjove01 wrote:

file2: LZMA compressed data, streamed

This is a non-archive(!) format representing a stream of octets compressed uzing LZMA. This is a legacy format used by xz predecessor, LZMA Utils. If for some reason you insist on using that instead of current formats, you may ask xz to use this format by passing `--format=lzma`.

byjove01 · 2023-12-25 08:38:31

seth wrote:

=> Does the CPU load rise (significantly, you'd expect at least one core to be fully charged) while we're not believing that something's happening?
=> If you run dolphin/ark from an interactive shell, does it yell errors at you when trying to compress sth.?

According to KSysGuard, the Dolphin window doesn't use less or more than 8% of my CPU, with 15 threads. Nothing significant. The only output I could get in the shell is this:

[byjove@pchost-1 Desktop]$ dolphin
true
actions!!!

Do those kick off subprocesses?
If https://github.com/KDE/ark is relevant and correct, most algorithms just use https://archlinux.org/packages/core/x86_64/libarchive/ - incl. gz, lz4 and zstd

No. I didn't see anything related with compression or even Dolphin (if not the bash shell I opened into the window).

mpan wrote:

I asked to use xz or 7z to receive the error (if any). But no error is a bit puzzling, as it suggests a bug is either in Ark or libarchive. I believe you may test libarchive by invoking its bsdtar:
bsdtar -czf ARCHIVE_FILE INPUT_FILEs…

It created a nicely compressed gzip file... weirdly, it didn't create any tar file ; by curiosity I tried to create a classical tar.gz file and I noticed the issue was happening. Could this be an issue with tar instead of LZMA?

This is a non-archive(!) format representing a stream of octets compressed uzing LZMA. This is a legacy format used by xz predecessor, LZMA Utils. If for some reason you insist on using that instead of current formats, you may ask xz to use this format by passing `--format=lzma`.

I was asking this because it is the format used by Dolpĥin when I created the first LZMA-compressed archives. As far as I can use the LZMA algorithm, the format doesn't matter.

seth · 2023-12-25 09:04:40

The only output I could get in the shell is this:

You might have to "killall dolphin" first because it'll likely just IPC-open a new window on an existing process.
And try ark as well.

From the CPU situation this doesn't sound like it's even trying to compress anything.
Oh, and use "top" or when using ksysguard, check the per-process CPU load, not the overall one, there're too many variables with the latter.

It created a nicely compressed gzip file... weirdly, it didn't create any tar file

What command did you exactly issue?
What's

file compressed_gzip_file.gz
gunzip compressed_gzip_file.gz
file compressed_gzip_file

by curiosity I tried to create a classical tar.gz file and I noticed the issue was happening.

Please don't paraphrase, https://bbs.archlinux.org/viewtopic.php?id=57855
Tried *how* exactly (dolphin, tar or bsdtar) and what exactly was the outcome (errors?)

byjove01 · 2023-12-25 11:48:05

seth wrote:

You might have to "killall dolphin" first because it'll likely just IPC-open a new window on an existing process.
And try ark as well.
From the CPU situation this doesn't sound like it's even trying to compress anything.
Oh, and use "top" or when using ksysguard, check the per-process CPU load, not the overall one, there're too many variables with the latter.

I ran two compression process to make a comparison. Dolphin is doing LZIP compression via GUI, bsdtar was ran from Dolphin's terminal without any special arguments.

While I'm writing this line, bsdtar almost finished the job whileas Dolphin's archive file wasn't even created. About the terminal log, the file explorer was a few more talkative than it was previously.

Cannot initialize model with data QJsonObject(). missing: QJsonValue(string, "urls")
true
actions!!!
kf.service.services: KMimeTypeTrader: MIME type "" not found
true
actions!!!
kf.service.services: KMimeTypeTrader: MIME type "" not found
true
actions!!!
kf.service.services: KMimeTypeTrader: MIME type "" not found

What command did you exactly issue?

The one mpan just wrote. Just replaced "ARCHIVE_FILE" with the archive name and "INPUT_FILEs" with three folder names as strings. These folders are the same I used during previous archive attempts.

What's

file compressed_gzip_file.gz
gunzip compressed_gzip_file.gz
file compressed_gzip_file

> bsdtararchive: gzip compressed data, last modified: Mon Dec 25 11:34:01 2023, from Unix, original size modulo 2^32 1573787648 gzip compresseddata, reserved method, ASCII, has CRC, extra field, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 1573787648
> gunzip bsdtararchive.tar.gz # turned input file into a .tar file
> bsdtararchive.tar: POSIX tar archive

Tried *how* exactly (dolphin, tar or bsdtar) and what exactly was the outcome (errors?)

I created these archives with Dolphin. And the error in question is the one this thread is about: notification is happening, progress bar going almost indefinitely, but no archive is created at the end.

Last edited by byjove01 (2023-12-25 11:50:08)

seth · 2023-12-25 14:19:02

Ok, dolphin is certainly charging one core while you're trying to compress something.

more talkative

Unfortunately that looks like randm debug output that was forgotten to be deacitivated.

Just replaced "ARCHIVE_FILE" with the archive name

The actual archive name would have been interesting but the subsequent test already showed that you got a tar.gz as expected.

So dolphin also cannot create a tar.gz

Can you lzma compress a single file w/ dolphin

dd if=/dev/urandom of=/tmp/foo count=100k

will create a 50MB test file of random data as /tmp/foo, try to compress that.

Edit: did you try ark?

Last edited by seth (2023-12-25 14:19:14)

byjove01 · 2023-12-25 16:51:01

seth wrote:

Just replaced "ARCHIVE_FILE" with the archive name
The actual archive name would have been interesting but the subsequent test already showed that you got a tar.gz as expected.

I am calling my test files by the subtle name of "test", nothing with special characters or spaces in it, in the case you need that info later.

Can you lzma compress a single file w/ dolphin
dd if=/dev/urandom of=/tmp/foo count=100k
will create a 50MB test file of random data as /tmp/foo, try to compress that.

I was able to compress it with Dolphin to .tar.lzma and .7z.

Edit: did you try ark?

I did with your 50MB foo file. It worked.

So, at least, we can speculate there's a problem with my data, or with how Dolphin/Ark is interacting with it...

Last edited by byjove01 (2023-12-25 20:25:40)

seth · 2023-12-25 20:28:45

The filename thing was just about your assertion it wasn't a tar.gz and whether that was maybe down to you assuming the filename would somehow change to reflect that.
But that's been settled, you get a tar.gz as expected.

Since compressing the single foo file worked, can you generate an archive when "cp /tmp/foo /tmp/bar" and compressing both files?

mpan · 2023-12-26 11:06:23

byjove01 wrote:

mpan wrote:
I asked to use xz or 7z to receive the error (if any). But no error is a bit puzzling, as it suggests a bug is either in Ark or libarchive. I believe you may test libarchive by invoking its bsdtar:
bsdtar -czf ARCHIVE_FILE INPUT_FILEs…
It created a nicely compressed gzip file... weirdly, it didn't create any tar file ; by curiosity I tried to create a classical tar.gz file and I noticed the issue was happening. Could this be an issue with tar instead of LZMA?

I mixed up invocations. I meant:

bsdtar -c --lzma -f ARCHIVE_FILE INPUT_FILES…

Sorry for confusion.

byjove01 wrote:

I was asking this because it is the format used by Dolpĥin when I created the first LZMA-compressed archives. As far as I can use the LZMA algorithm, the format doesn't matter.

Then both 7z and xz use LZMA by default.

byjove01 · 2023-12-26 11:45:10

seth wrote:

Since compressing the single foo file worked, can you generate an archive when "cp /tmp/foo /tmp/bar" and compressing both files?

I can. But the archive size is a bit weird. How, even with overcompression, can an archive containing two 50Mb files only weights 50,7Mb?

mpan wrote:

I mixed up invocations. I meant:
bsdtar -c --lzma -f ARCHIVE_FILE INPUT_FILES…
Sorry for confusion. hmm

No problem, I noticed that by myself
The program ran properly and created the archive. No issues at the horizon.

Last edited by byjove01 (2023-12-26 11:46:58)

seth · 2023-12-26 11:52:40

"cp foo bar" - think about it

However, it seems the problem is related to your specific usecase.
Are the files you're trying to archive
- many
- huge
- in weird paths (non-ascii, spaces, recursive symlinks)
You're not trying to archive stuff from sysfs/devtmpfs, are you?

Last edited by seth (2023-12-26 11:52:54)

byjove01 · 2023-12-26 12:26:19

seth wrote:

"cp foo bar" - think about it

Oooh, I got it.

seth wrote:

However, it seems the problem is related to your specific usecase.
Are the files you're trying to archive
- many
- huge
- in weird paths (non-ascii, spaces, recursive symlinks)
You're not trying to archive stuff from sysfs/devtmpfs, are you?

Yes, there are many huge files. I'm compressing my games in order to stock them somewhere so I could reinstall them later, this is why I need a high-level compression algorithm like LZMA.
I thought for a moment the problem could be related to colons in some folder names, but removing them didn't alter the issue.

seth · 2023-12-26 19:33:13

Please quantify "many" and "huge" and does it work if you try to compress only some part of the game files?

mpan · 2023-12-27 02:33:01

If neither 7z nor bsdtar give an error, and Ark simply misbehaves with no messages, then I am out of ideas how to debug the situation without unreasonable effort. mpan switches to stand-by mode.

Arch Linux

#1 2023-12-23 01:16:22

[BUG] File compression unexpectedly stops and does not create anything

#2 2023-12-23 07:05:05

Re: [BUG] File compression unexpectedly stops and does not create anything

#3 2023-12-23 13:39:49

Re: [BUG] File compression unexpectedly stops and does not create anything

#4 2023-12-23 13:56:28

Re: [BUG] File compression unexpectedly stops and does not create anything

#5 2023-12-23 16:36:57

Re: [BUG] File compression unexpectedly stops and does not create anything

#6 2023-12-23 20:10:59

Re: [BUG] File compression unexpectedly stops and does not create anything

#7 2023-12-23 21:37:32

Re: [BUG] File compression unexpectedly stops and does not create anything

#8 2023-12-24 00:08:26

Re: [BUG] File compression unexpectedly stops and does not create anything

#9 2023-12-24 06:19:27

Re: [BUG] File compression unexpectedly stops and does not create anything

#10 2023-12-24 08:16:47

Re: [BUG] File compression unexpectedly stops and does not create anything

#11 2023-12-24 11:04:27

Re: [BUG] File compression unexpectedly stops and does not create anything

#12 2023-12-24 12:55:10

Re: [BUG] File compression unexpectedly stops and does not create anything

#13 2023-12-25 01:51:34

Re: [BUG] File compression unexpectedly stops and does not create anything

#14 2023-12-25 08:38:31

Re: [BUG] File compression unexpectedly stops and does not create anything

#15 2023-12-25 09:04:40

Re: [BUG] File compression unexpectedly stops and does not create anything

#16 2023-12-25 11:48:05

Re: [BUG] File compression unexpectedly stops and does not create anything

#17 2023-12-25 14:19:02

Re: [BUG] File compression unexpectedly stops and does not create anything

#18 2023-12-25 16:51:01

Re: [BUG] File compression unexpectedly stops and does not create anything

#19 2023-12-25 20:28:45

Re: [BUG] File compression unexpectedly stops and does not create anything

#20 2023-12-26 11:06:23

Re: [BUG] File compression unexpectedly stops and does not create anything

#21 2023-12-26 11:45:10

Re: [BUG] File compression unexpectedly stops and does not create anything

#22 2023-12-26 11:52:40

Re: [BUG] File compression unexpectedly stops and does not create anything

#23 2023-12-26 12:26:19

Re: [BUG] File compression unexpectedly stops and does not create anything

#24 2023-12-26 19:33:13

Re: [BUG] File compression unexpectedly stops and does not create anything

#25 2023-12-27 02:33:01

Re: [BUG] File compression unexpectedly stops and does not create anything

Board footer