Replacing XZ with ZSTD?

RainbowZephyr · 2017-08-27 06:20:26

Is there anyway to force makepkg to use ZSTD instead of XZ? When I added the .zst format to PKGEXT it said it was unrecognized format and refused to compress using ZSTD.

The reason I want to use ZSTD is because it uses much lower cpu to compress the files.

eschwartz · 2017-08-27 07:11:47

Unrecognized formats won't work on account of being unrecognized. Sure, you could modify makepkg to recognize it, but you'd also have to modify pacman, or rather libarchive, to recognize it too.

https://github.com/libarchive/libarchive/pull/905 does add upstream support for ZSTD in libarchive, merged 14 days ago and not yet in a release. So assuming you are willing to wait on this, you should get it.

graysky · 2017-08-27 09:21:19

Eschwartz wrote:

https://github.com/libarchive/libarchive/pull/905 does add upstream support for ZSTD in libarchive, merged 14 days ago and not yet in a release. So assuming you are willing to wait on this, you should get it.

Build libarchive-git?

Allan · 2017-08-27 09:28:31

There are multiple steps...

1) have libarchive support
2) have makepkg support.

Using libarchive-git will fix #1 (but be careful, anything linked to libarchive may need rebuilt, including pacman... be VERY careful).

For #2, you will need to edit makepkg. Look for this bit of the code:

	case "$PKGEXT" in
		*tar.gz)  ${COMPRESSGZ[@]:-gzip -c -f -n} ;;

Add a case statement for you compression of choice. You can then set it as the default in makepkg.conf

eschwartz · 2017-08-27 14:47:38

Allan, I looked at this to write a simple patch adding support to makepkg, and I noticed something confusing. Why does create_package() use a pipeline in combination with the COMPRESS* array but later on create_srcpackage() uses `bsdtar -cL ${TAR_OPT} -f "$pkg_file"` ?

Jristz · 2017-08-28 06:08:26

Allan wrote:

There are multiple steps...
1) have libarchive support
2) have makepkg support.
Using libarchive-git will fix #1 (but be careful, anything linked to libarchive may need rebuilt, including pacman... be VERY careful).
For #2, you will need to edit makepkg. Look for this bit of the code:
	case "$PKGEXT" in
		*tar.gz)  ${COMPRESSGZ[@]:-gzip -c -f -n} ;;
Add a case statement for you compression of choice. You can then set it as the default in makepkg.conf

It sould be something like

*tar.zst)   ${COMPRESSZST[@]:-zstd -d -qq -f} ;;

I don't know if there are more referencess to be checked.

eschwartz · 2017-08-28 14:35:57

Actually, no since those aren't the actual zstd flags.

https://github.com/eli-schwartz/pacman/tree/zst-support adds makepkg and repo-add support.
(repo-add uses bsdtar's --zstd option which relies on libarchive-git, makepkg support uses a compression filter for bsdtar's *.pkg.tar output.)

I was a bit late writing it, as I wanted to refactor source archive and package archive handling into one unified compression filter.

Last edited by eschwartz (2017-08-28 14:43:42)

Jristz · 2017-08-29 06:25:29

Eschwartz wrote:

Actually, no since those aren't the actual zstd flags.

I used what was used on libarchive... so maybe is not totally good idea to use zstd until libarchive 3,4,1 just to be safe.

Trilby · 2017-08-29 11:19:26

Can I ask, why zstd? I assume the built packages are not to be shared/distributed as tar.zst packages would not be useable by other archers for a bit yet.

If this is just for your own use (building and installing AUR packages) and the goal is simply to reduce the cpu-use in the compression step, there are far easier ways. Just disable compression all together.

eschwartz · 2017-08-29 11:57:03

Well, I posted patches on pacman-dev, so this should hopefully be available in the next libarchive/pacman releases.

RainbowZephyr · 2017-08-30 20:15:41

Allan wrote:

There are multiple steps...
1) have libarchive support
2) have makepkg support.
Using libarchive-git will fix #1 (but be careful, anything linked to libarchive may need rebuilt, including pacman... be VERY careful).
For #2, you will need to edit makepkg. Look for this bit of the code:
	case "$PKGEXT" in
		*tar.gz)  ${COMPRESSGZ[@]:-gzip -c -f -n} ;;
Add a case statement for you compression of choice. You can then set it as the default in makepkg.conf

I guess I will play it safe here and wait for the support

RainbowZephyr · 2017-08-30 20:18:02

Trilby wrote:

Can I ask, why zstd? I assume the built packages are not to be shared/distributed as tar.zst packages would not be useable by other archers for a bit yet.
If this is just for your own use (building and installing AUR packages) and the goal is simply to reduce the cpu-use in the compression step, there are far easier ways. Just disable compression all together.

Because I run Arch on some low end machines which are not capable of compiling large programs from AUR, thus I compile them on a higher machine and distribute them to the lower machines, but even then do they struggle extracting heavily compressed XZ archives. I tested ZSTD on the low end machines and it takes practically no resources compared to XZ and it is as well much faster than XZ

txtsd · 2017-09-24 19:49:26

Any updates on this?

eschwartz · 2017-09-24 20:06:48

https://lists.archlinux.org/pipermail/p … 22114.html and followup patches for my patches to add zst support

https://lists.archlinux.org/pipermail/p … 22126.html someone else's patch to add lz4 support, where allan said to "hold off on resubmitting the patch until after I deal with the patches unifying the compression".

...

pacman/makepkg development happens when allan gets time to review patches, there are no guarantees but I see no reason to think this won't make it in.

The major block is libarchive support, which is merged but requires a libarchive 3.3.3 release with no ETA: https://github.com/libarchive/libarchive/issues/940

It's easier to test pacman-git than libarchive-git.

Arch Linux

#1 2017-08-27 06:20:26

Replacing XZ with ZSTD?

#2 2017-08-27 07:11:47

Re: Replacing XZ with ZSTD?

#3 2017-08-27 09:21:19

Re: Replacing XZ with ZSTD?

#4 2017-08-27 09:28:31

Re: Replacing XZ with ZSTD?

#5 2017-08-27 14:47:38

Re: Replacing XZ with ZSTD?

#6 2017-08-28 06:08:26

Re: Replacing XZ with ZSTD?

#7 2017-08-28 14:35:57

Re: Replacing XZ with ZSTD?

#8 2017-08-29 06:25:29

Re: Replacing XZ with ZSTD?

#9 2017-08-29 11:19:26

Re: Replacing XZ with ZSTD?

#10 2017-08-29 11:57:03

Re: Replacing XZ with ZSTD?

#11 2017-08-30 20:15:41

Re: Replacing XZ with ZSTD?

#12 2017-08-30 20:18:02

Re: Replacing XZ with ZSTD?

#13 2017-09-24 19:49:26

Re: Replacing XZ with ZSTD?

#14 2017-09-24 20:06:48

Re: Replacing XZ with ZSTD?

Board footer