[SOLVED] incron not working with 7z and tar

LinuxUser15939 · 2022-02-20 11:38:56

incrontab -e

Format: path mask command (With a tab in between each)

FULL_PATH_TO_DOWNLOADS_DIRECTORY IN_CREATE 7z x ~/Downloads/*.zip -oFULL_PATH_TO_DOWNLOADS_DIRECTORY && find ~/Downloads -name *.zip -delete
FULL_PATH_TO_DOWNLOADS_DIRECTORY IN_CREATE tar -xf ~/Downloads/*.tar.gz -C ~/Downloads && find ~/Downloads -name *.tar.gz -delete
FULL_PATH_TO_DOWNLOADS_DIRECTORY IN_CREATE tar -xf ~/Downloads/*.tar.xz -C ~/Downloads && find ~/Downloads -name *.tar.xz -delete

I'm trying to automatically extract .zip, .tar.gz and .tar.xz files as soon as they reach my Downloads folder. And delete the archive/zip file afterwards. But the commands don't seem to work when used with incron, with or without the parts after &&. I've already done systemctl start incrond. And I've put my username in /etc/incron.allow. The commands work perfectly fine when run directly from the command line, so I dont' think they're the problem.

I'm already successfully using incron for a certain task where files of a certain file extension are deleted as soon as they are created in a certain directory. Incron is working perfectly for this task already. So, I know at least that incron isn't completely broken.

Can someone help me fix this?

Edit: according to journalctl -f -u incrond.service, the commands aren't even being run. IN_CREATE events are logged, but incron is only executing one of my rules, the wrong rule:

find ~/Downloads -type f -iname "*.crx" -delete

Last edited by LinuxUser15939 (2022-02-21 15:19:01)

lmn · 2022-02-20 12:11:44

My suspicion is that the ~ is not expanded at all or to the incorrect HOME directory. Have you tried replacing them with absolute paths?
In the same vein the glob (*) could not expanded correctly. Are those commands passed to a shell?
I assume the commands are passed verbatim to a shell running as the root user (or the user of the icron process, which is probably not your user).

Also this setup seems to delete all tar.gz files every time. This could result in a race condition where one is deleted before unpacking it is complete.

LinuxUser15939 · 2022-02-20 12:17:36

I tried changing the paths to absolute paths. It still doesn't work, unfortunately. As for the race conditions, I think it should be fine because I tested it in the command line and nothing bad happened. As for the "passed to a shell" issue, I'm not really sure. I don't think the glob is the problem, because I'm using it in another incron job without any issues. That incron job is

FULL_PATH_TO_DOWNLOADS_DIRECTORY IN_CREATE find ~/Downloads -type f -iname "*.crx"

Last edited by LinuxUser15939 (2022-02-20 12:23:08)

V1del · 2022-02-20 12:27:01

What's your "working rule that shows this totally works" ? Of course this works on the command line because that's what interprets ~ and * and &&. But it's not entirely clear that incrontab will invoke a shell, so what's likely to happen is that it runs 7z with an invalid path. Did you check your journal log for errors?

The "minimal" fix to ensure this actually runs through a shell is wrap them and pass them to a shell i.e.

sh -c "7z x ~/Downloads/*.zip -oFULL_PATH_TO_DOWNLOADS_DIRECTORY && find ~/Downloads -name *.zip -delete"

that still doesn't fix the race condition concerns.

lmn · 2022-02-20 12:29:09

Have you checked the system log or any related log files for the output of those commands i.e. are they run at all?

Regarding a race condition: Some browsers/utilities create dummy files inside the Download directory while downloading into another (hidden) file. This could trigger an inotify event. Then this could fail the extraction.
Or any other reason a tar.gz file is put in the Downloads directory before extraction is complete.

Also the * in the find command needs to be in quotes as it should probably not be expanded by a shell but be passed to find.

find ~/Downloads -name "*.zip" -delete

I've already done systemctl start incron

As a sanity check, does icron need a restart after the configuration has changed?

Edit: glob for find not as intended
Edit: restart service

Last edited by lmn (2022-02-20 12:48:11)

LinuxUser15939 · 2022-02-20 13:06:12

I've tried restarting it with systemctl restart incrond and incrontab --reload. I've tested the commands in the command line by having a .zip, tar.gz or tar.xz file present in my downloads directory. The files were successfully extracted and the archives were deleted afterwards without errors. I've tried quoting the command like you mentioned but it didn't work. As for the system logs, I tried journalctl -f -u incrond.service. It seems the commands aren't even being run. Only my "working" .crx rule was being run after I copy and pasted a .zip file to the Downloads directory. The IN_CREATE events are being picked up, but incron only executes my .crx rule, not my .zip rule, for some reason.

Last edited by LinuxUser15939 (2022-02-20 13:11:55)

LinuxUser15939 · 2022-02-20 13:24:18

Thanks for the replies guys. I think I figured it out. The manpage incrontab(5) says

Please remember that the same path may occur only once per table (otherwise only the first occurrence takes effect and an error message is emitted to the system log).

I appended all of the commands with && to my "working" rule. But for some reason, the zip rules work but the tar rules don't work. Weird. Maybe it's the race conditions you mentioned. I swear those commands worked before. I'm trying them on the command line now and I get the error "Not found in archive".

Last edited by LinuxUser15939 (2022-02-20 13:34:33)

lmn · 2022-02-20 13:37:13

Have you fixed the quoting in the find commands. If you have more than one one file with the same extension the find command will otherwisefail.
With that the whole chain of commands from that point on will fail as they are connected by && (AKA and).

Please be more specific: Which parts did fail? The extraction or the deletion parts? Did you create more than one file with the same extension in the Downloads directory?
The creation event as far as I can tell is fired before writing the file is complete, so this could be a point of failure.

Edit:
The Error stems from tar as you can't extract multiple archives that way

tar xf *tar.gz   expands to    tar xf file1.tar.gz file2.tar.gz

which is valid syntax for tar but does not what you expect it to do. It tries to extract file2.tar.gz from the archive file1.tar.gz.
You need some form of looping construct to go over the archives in the Dowloads directory, something like the -exec flag in find or xargs or shell looping.
This will still be exposed to the race condition problems described above though.

Last edited by lmn (2022-02-20 13:55:42)

LinuxUser15939 · 2022-02-20 14:44:13

Hmm. Well it seems that the zip command only works when one zip is being unzipped at a time. When run from the command line, with multiple zip files, the archives and the contents are both deleted. The tar commands work when run on the command line, but only on a single .tar.gz file. When multiple .tar.gz files are copied, I get the "Not found in archive" error.

When run with incron, I've only tested downloading 1 file at a time, but the incron job seems to work on zip files, but not .tar.gz files. Sorry if this is confusing. I don't really know much programming so I can't really do much here. I might just resort to using the right-click menu to extract the archives. It's better than nothing I guess.

Thanks for your help by the way.

Last edited by LinuxUser15939 (2022-02-20 14:50:14)

lmn · 2022-02-20 15:12:38

First off I suggest you start writing a proper script to handle all the complexity. While it is possible to cram everything inside the incrontab it will be difficult to maintain and debug.

Like already said, the tar command can't extract multiple files at once. You need a proper looping construct. For example this loop will unpack all *.tar.gz and *.tar.xz files and delete them afterwards

for archive in ~/Downloads/*.tar.{gz,xz}
do
    tar -xf "$archive"-C ~/Downloads && rm "$archive"
done

This should fix the error with the tar command. See this as an example to build from and don't just copy it. There are some good resources on the internet to learn shell programming (I think some users recommend this bash guide if I'm not mistaken. I cannot speak to it's quality. This is specifically for bash so keep that in mind)
Something similar can be done for the zip archives as well, if 7z can't handle more than one archive.

But this still does not solve the problem you get when a matching archive is created while another one is being extracted. In this case all archives will be extracted again or deleted halfway through and error out.
This is a concurrency problem and you would need to implement a locking mechanism to solve that. There probably is a solution to this out there, but I'm not familiar with one.

Is there a specific reason you want to automatically extract all archives?

Firefox for example allows to run programs to open downloaded files with. Maybe that is a more promising avenue to take?

Last edited by lmn (2022-02-20 15:17:20)

LinuxUser15939 · 2022-02-20 16:40:42

I just wanted to get rid of the mild inconvenience of having to extract the files manually, which requires quite a few mouse clicks. Oh well. Thanks for all the help.

V1del · 2022-02-21 01:13:50

Since incrontab also catches the file name leading to the event you could fairly easily do a script "just" for the filename it gets an event on which would solve all the concurrency concerns as you'd be manipulating just that file.

LinuxUser15939 · 2022-02-21 07:41:06

OH!!! I'm so glad you gave me the link to that bash guide. I've finally solved the problem. It's because I used && instead of ; to separate the consecutive commands. Command A && Command B means that Command B only runs if Command A succeeds. However, when I was testing, I didn't have any .zip files in my Downloads directory, so the 7z commands would fail, which would cause the subsequent tar commands to fail. I replaced && with ; and now everything works, at least, when extracting 1 archive at at time. I think that should cover most use cases.

Last edited by LinuxUser15939 (2022-02-21 08:25:15)

LinuxUser15939 · 2022-02-21 14:05:26

By the way, does anyone know why this incron job is giving me permanent 100% CPU usage after running? Also memory usage gradually goes up and up until it's used up like 10-20% of my memory. Task manager is showing a bunch of "cat", "tar", "find", and "7z". I have to stop incrond.service to get my CPU usage and memory to go back down. It seems like the processes aren't being killed or something.

7z x "/FULL_PATH_TO_DOWNLOADS_DIRECTORY/*.zip" -oFULL_PATH_TO_DOWNLOADS_DIRECTORY ; cat ~/Downloads/*.tar.gz | tar -zxvf - -i -C ~/Downloads ; cat ~/Downloads/*.tar.xz | tar -Jxvf - -i -C ~/Downloads ; find ~/Downloads -name "*.zip" -delete ; find ~/Downloads -name *.tar.gz -delete ; find ~/Downloads -name *.tar.xz -delete

Also, it seems like if I copy and paste multiple .zip files into ~/Downloads, my CPU usage doesn't spike. If I do it with multiple .tar.gz files, my CPU usage doesn't spike either. But when I do it with .tar.xz (2 .tar.xz files by themselves), my CPU usage goes to 100%, but only for a couple of seconds. Then it goes back down. And as soon as I copy and paste either .tar.gz or .zip along with .tar.xz files at the same time, the problem occurs and CPU usage stays at 100% indefinitely. I think that there must be something wrong with tar -J since .tar.xz seems to be the culprit here.

journalctl -f -u incrond is giving me a bunch of EVENT (IN_CREATE) with the contents of the archives (some .svg files, etc.)

The command by itself works to extract multiple archives at once, while deleting the original archives, and works both in the command line and in incron. But after the incron job is executed, my CPU usage stays at 100%. This doesn't happen when I run this command on the command line.

Last edited by LinuxUser15939 (2022-02-21 14:35:48)

LinuxUser15939 · 2022-02-21 15:18:46

Okay, I've finally figured it out. incrontab needs you to write IN_CREATE,recursive=false in order to not fire commands for every extracted file created. That seems to have been the reason behind the 100% CPU usage. I've managed to write multiple for loops in a bash script to run these commands. I now have a working incron job that automatically extracts multiple archives and deletes the archives afterwards. CPU usage remains normal. Thanks for the help.

Edit: For anyone with the same problem, the above incron job also works. There's not much difference between using a bash script with incron or just using the incron job above. They both work. Just make sure you use recursive=false.

Last edited by LinuxUser15939 (2022-02-21 15:43:25)

Arch Linux

#1 2022-02-20 11:38:56

[SOLVED] incron not working with 7z and tar

#2 2022-02-20 12:11:44

Re: [SOLVED] incron not working with 7z and tar

#3 2022-02-20 12:17:36

Re: [SOLVED] incron not working with 7z and tar

#4 2022-02-20 12:27:01

Re: [SOLVED] incron not working with 7z and tar

#5 2022-02-20 12:29:09

Re: [SOLVED] incron not working with 7z and tar

#6 2022-02-20 13:06:12

Re: [SOLVED] incron not working with 7z and tar

#7 2022-02-20 13:24:18

Re: [SOLVED] incron not working with 7z and tar

#8 2022-02-20 13:37:13

Re: [SOLVED] incron not working with 7z and tar

#9 2022-02-20 14:44:13

Re: [SOLVED] incron not working with 7z and tar

#10 2022-02-20 15:12:38

Re: [SOLVED] incron not working with 7z and tar

#11 2022-02-20 16:40:42

Re: [SOLVED] incron not working with 7z and tar

#12 2022-02-21 01:13:50

Re: [SOLVED] incron not working with 7z and tar

#13 2022-02-21 07:41:06

Re: [SOLVED] incron not working with 7z and tar

#14 2022-02-21 14:05:26

Re: [SOLVED] incron not working with 7z and tar

#15 2022-02-21 15:18:46

Re: [SOLVED] incron not working with 7z and tar

Board footer