makepkg doesn't seem to extract a xz file [SOLVED]

graysky · 2021-04-01 20:46:31

Can someone please try to build linux-ck from the AUR: https://aur.archlinux.org/cgit/aur.git/ … -ck.tar.gz

It seems that makepkg fails to extract a xz file from the source array which leads to a failure in prepare. if I untar the file manually, everything proceeds as expected.

==> Starting prepare()...
Setting version...
Applying patch 0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-C.patch...
patching file include/linux/user_namespace.h
patching file init/Kconfig
patching file kernel/fork.c
patching file kernel/sysctl.c
patching file kernel/user_namespace.c
Applying patch 0002-iommu-amd-Don-t-initialise-remapping-irqdomain-if-IO.patch...
patching file drivers/iommu/amd/init.c
Hunk #1 succeeded at 2998 (offset 1 line).
Setting config...
sed: can't read ../patch-5.11-ck1: No such file or directory
==> ERROR: A failure occurred in prepare().
    Aborting...

Last edited by graysky (2021-04-02 13:34:36)

Awebb · 2021-04-01 20:51:51

EDIT: I've added the key, now it's running.

EDIT2: I can reproduce your error.

Last edited by Awebb (2021-04-01 20:59:57)

eschwartz · 2021-04-02 03:36:37

Works for me, but... as expected, you skipped the actually important lines that come before "==> Starting prepare()..."

Why do people try to outthink the "post complete logs" thing? There aren't so many lines there that reading a code box becomes untenable.

progandy · 2021-04-02 08:51:45

Interestingly, file does something strange if you ask for the mime type:

$ file -S -bzL -- patch-5.11-ck1.xz 
unified diff output, ASCII text (XZ compressed data, checksum CRC64)
$ file -S -bizL -- patch-5.11-ck1.xz                                       
text/x-diff; charset=us-ascii compressed-encoding=application/octet-stream; charset=binary

Edit: A downgrade to file 5.39 seems to "fix" it.

Last edited by progandy (2021-04-02 08:56:15)

graysky · 2021-04-02 09:32:11

eschwartz wrote:

Works for me, but... as expected, you skipped the actually important lines that come before "==> Starting prepare()..."
Why do people try to outthink the "post complete logs" thing? There aren't so many lines there that reading a code box becomes untenable.

You said it worked for you? Here is the entire output. The tarred patch is omitted from the extraction routine.

==> Making package: linux-ck 5.11.11-2 (Fri 02 Apr 2021 05:26:26 AM EDT)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found linux-5.11.11.tar.xz
  -> Found linux-5.11.11.tar.sign
  -> Found config
  -> Found more-uarches-20210401.tar.gz
  -> Found patch-5.11-ck1.xz
  -> Found 0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-C.patch
  -> Found 0002-iommu-amd-Don-t-initialise-remapping-irqdomain-if-IO.patch
==> Validating source files with b2sums...
    linux-5.11.11.tar.xz ... Passed
    linux-5.11.11.tar.sign ... Skipped
    config ... Passed
    more-uarches-20210401.tar.gz ... Passed
    patch-5.11-ck1.xz ... Passed
    0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-C.patch ... Passed
    0002-iommu-amd-Don-t-initialise-remapping-irqdomain-if-IO.patch ... Passed
==> Verifying source file signatures with gpg...
    linux-5.11.11.tar ... Passed
==> Extracting sources...
  -> Extracting linux-5.11.11.tar.xz with bsdtar
  -> Extracting more-uarches-20210401.tar.gz with bsdtar
==> Starting prepare()...
Setting version...
Applying patch 0001-ZEN-Add-sysctl-and-CONFIG-to-disallow-unprivileged-C.patch...
patching file include/linux/user_namespace.h
patching file init/Kconfig
patching file kernel/fork.c
Hunk #2 succeeded at 1876 (offset 8 lines).
Hunk #3 succeeded at 2949 (offset 8 lines).
patching file kernel/sysctl.c
patching file kernel/user_namespace.c
Applying patch 0002-iommu-amd-Don-t-initialise-remapping-irqdomain-if-IO.patch...
patching file drivers/iommu/amd/init.c
Hunk #1 succeeded at 2998 (offset 1 line).
Setting config...
sed: can't read ../patch-5.11-ck1: No such file or directory
==> ERROR: A failure occurred in prepare().
    Aborting...

progandy wrote:

Interestingly, file does something strange if you ask for the mime type:
$ file -S -bzL -- patch-5.11-ck1.xz 
unified diff output, ASCII text (XZ compressed data, checksum CRC64)
$ file -S -bizL -- patch-5.11-ck1.xz                                       
text/x-diff; charset=us-ascii compressed-encoding=application/octet-stream; charset=binary
Edit: A downgrade to file 5.39 seems to "fix" it.

If I downgrade to file-5.39-1 as you pointed out, makepkg runs as expected here too.

EDIT: I wrote a hacky fix for this in linux-ck-5.11.11-3. Not sure if it's a bug with file, if I downgrade to 5.39-1 and get bsdtar to test the file, it still fails, yet building with makepkg and the downgraded file works as expected without my hacky workaround in the PKGBUILD.

% bsdtar -tf patch-5.11-ck1.xz -q '*'   
bsdtar: Error opening archive: Unrecognized archive format

Last edited by graysky (2021-04-02 10:31:09)

Allan · 2021-04-02 10:47:13

Can someone report the bug against the file package?

figue · 2021-04-02 11:46:34

I see the same in my linux-xanmod package. XZ patch is not uncompressed anymore. I'll add a workaround for now until "file" is fixed. Thanks!

graysky · 2021-04-02 12:05:01

Allan wrote:

Can someone report the bug against the file package?

I registered on their bug tracker system but am a little unclear what to report. Incorrect MIME type with 5.40? I don't understand why I cannot get bsdtar to properly work with 5.39.

figue · 2021-04-02 12:15:53

graysky wrote:

Allan wrote:
Can someone report the bug against the file package?
I registered on their bug tracker system but am a little unclear what to report. Incorrect MIME type with 5.40? I don't understand why I cannot get bsdtar to properly work with 5.39.

No, I think bsdtar can't decompress a pure xz file... makepkg is using "xz -dcf -- $file", not bsdtar.

If MIME changed upstream, perhaps we need to patch makepkg xz detection.

bash -x makepkg with file-5.39:

+ local filepath=/DATA/packages/linux-xanmod/patch-5.11.10-xanmod1.xz
+ rm -f /DATA/packages/linux-xanmod/src/patch-5.11.10-xanmod1.xz
+ ln -s /DATA/packages/linux-xanmod/patch-5.11.10-xanmod1.xz /DATA/packages/linux-xanmod/src/
+ in_array patch-5.11.10-xanmod1.xz
+ local needle=patch-5.11.10-xanmod1.xz
+ shift
+ local item
+ return 1
++ file -S -bizL -- patch-5.11.10-xanmod1.xz
+ local 'file_type=text/x-diff; charset=utf-8 compressed-encoding=application/x-xz; charset=binary'
+ local ext=xz
+ local cmd=
+ case "$file_type" in
+ case "$ext" in
+ cmd=xz
+ local ret=0
++ gettext 'Extracting %s with %s'
+ msg2 'Extrayendo %s con %s' patch-5.11.10-xanmod1.xz xz
+ ((  QUIET  ))
+ local 'mesg=Extrayendo %s con %s'
+ shift
+ printf '  -> Extrayendo %s con %s\n' patch-5.11.10-xanmod1.xz xz
  -> Extrayendo patch-5.11.10-xanmod1.xz con xz
+ [[ xz = \b\s\d\t\a\r ]]
+ rm -f -- patch-5.11.10-xanmod1
+ xz -dcf -- patch-5.11.10-xanmod1.xz
+ ((  ret  ))
+ ((  EUID == 0  ))

Last edited by figue (2021-04-02 12:19:24)

progandy · 2021-04-02 12:28:33

graysky wrote:

Allan wrote:
Can someone report the bug against the file package?
I registered on their bug tracker system but am a little unclear what to report. Incorrect MIME type with 5.40?

Yes, that seems to be the way to go. file reports incorrect mime type strings for xz compressed files (The human readable output is fine). Also add an example like

$ echo "test" | xz - >test.xz
$file -b test.xz
XZ compressed data, checksum CRC64
$ file -bi test.xz
application/octet-stream; charset=binary

I don't understand why I cannot get bsdtar to properly work with 5.39.

bsdtar requires an archive, pure compression without an archive format is not supported.

I bisected file, and found the problematic commit. Maybe we should revert it for the arch package, I'll file an arch bug for that. Edit: https://bugs.archlinux.org/task/70261

3ebd747de661b244f1acec29495d676dd52e71d5 - Add checksum for XZ

This seems to fix it.

From 94ab14414606b794b7ccf60595493cd8715e195b Mon Sep 17 00:00:00 2001
From: ProgAndy <code@progandy.de>
Date: Fri, 2 Apr 2021 14:33:18 +0200
Subject: [PATCH] Fix mime type XZ after "Add checksum for XZ"

---
 magic/Magdir/compress | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/magic/Magdir/compress b/magic/Magdir/compress
index 64d11a73..aef4bb49 100644
--- a/magic/Magdir/compress
+++ b/magic/Magdir/compress
@@ -265,14 +265,13 @@
 
 # http://tukaani.org/xz/xz-file-format.txt
 0	ustring		\xFD7zXZ\x00		XZ compressed data, checksum
+!:strength * 2
+!:mime	application/x-xz
 >7	byte&0xf	0x0			NONE
 >7	byte&0xf	0x1			CRC32
 >7	byte&0xf	0x4			CRC64
 >7	byte&0xf	0xa			SHA-256
 
-!:strength * 2
-!:mime	application/x-xz
-
 # https://github.com/ckolivas/lrzip/blob/master/doc/magic.header.txt
 0	string		LRZI			LRZIP compressed data
 >4	byte		x			- version %d
-- 
2.31.1

Last edited by progandy (2021-04-02 12:46:59)

Awebb · 2021-04-02 13:23:46

graysky wrote:

Allan wrote:
Can someone report the bug against the file package?
I registered on their bug tracker system but am a little unclear what to report. Incorrect MIME type with 5.40? I don't understand why I cannot get bsdtar to properly work with 5.39.

I think he meant on our flyspray against the package.

EDIT:

progandy wrote:

I bisected file, and found the problematic commit. Maybe we should revert it for the arch package, I'll file an arch bug for that. Edit: https://bugs.archlinux.org/task/70261

Missed that! Cool.

Last edited by Awebb (2021-04-02 13:24:49)

graysky · 2021-04-02 13:34:25

Thanks for the discussion, all. I see FS#70261 has been created. I will mark this thread.

eschwartz · 2021-04-02 14:58:18

graysky wrote:

You said it worked for you? Here is the entire output. The tarred patch is omitted from the extraction routine.

I tried running it as a non-chroot build on a box that had the old file(1) version.

The relevant bit is the tarred patch being omitted from the extraction route -- that is a definitive problem in makepkg pointing to an error occurring during the runtime of the /usr/share/makepkg/source/file.sh extract_file() function. file needs to be fixed.

"Something happened during prepare()" is less precise, could mean a number of things... and given the extraction routine output is there anyway it makes sense to provide it.

graysky wrote:

EDIT: I wrote a hacky fix for this in linux-ck-5.11.11-3.

Why do you unlink the symlink in $srcdir? Why do you use $startdir which is marked as deprecated?

graysky wrote:

Not sure if it's a bug with file, if I downgrade to 5.39-1 and get bsdtar to test the file, it still fails, yet building with makepkg and the downgraded file works as expected without my hacky workaround in the PKGBUILD.
% bsdtar -tf patch-5.11-ck1.xz -q '*'   
bsdtar: Error opening archive: Unrecognized archive format

The clue is in the word "archive".

graysky · 2021-04-03 11:09:41

eschwartz wrote:

Why do you unlink the symlink in $srcdir? Why do you use $startdir which is marked as deprecated?

Not sure why I unlinked...
Didn't realize $startdir was deprecated. Is there a reference to that (maybe in the source itself)?

progandy · 2021-04-03 11:31:24

graysky wrote:

Not sure why I unlinked...
Didn't realize $startdir was deprecated. Is there a reference to that (maybe in the source itself)?

It is in the manpage: https://man.archlinux.org/man/core/pacm … _FUNCTIONS

Arch Linux

#1 2021-04-01 20:46:31

makepkg doesn't seem to extract a xz file [SOLVED]

#2 2021-04-01 20:51:51

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#3 2021-04-02 03:36:37

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#4 2021-04-02 08:51:45

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#5 2021-04-02 09:32:11

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#6 2021-04-02 10:47:13

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#7 2021-04-02 11:46:34

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#8 2021-04-02 12:05:01

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#9 2021-04-02 12:15:53

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#10 2021-04-02 12:28:33

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#11 2021-04-02 13:23:46

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#12 2021-04-02 13:34:25

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#13 2021-04-02 14:58:18

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#14 2021-04-03 11:09:41

Re: makepkg doesn't seem to extract a xz file [SOLVED]

#15 2021-04-03 11:31:24

Re: makepkg doesn't seem to extract a xz file [SOLVED]

Board footer