Arch Linux Delta Repositories

nkukard · 2019-01-08 18:58:04

Background:

I just love Arch Linux and delta support intrigued me.

TLDR; Currently generating deltas for last 5 versions 3 months of packages directly to the latest version (traditionally A->B->C->D->E->F, now A->F, B->F, C->F, D->F, E->F), up to a 0.99 ratio. (+IPv6!)

Usage:

1) Edit /etc/pacman.d/mirrorlist and add the following as the top repository... (http and https supported, replace 'uk' with 'za' if you're closer to South Africa)

# Delta repository
Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

2) Edit /etc/pacman.conf and uncomment "UseDelta=0.7" in the [options] section
Deltas are generated up to 0.99 ratio (for when every MiB matters), so if you're on a mobile connection, you may want to increase the default 0.7 if you see fit.

3) Install xdelta3...

pacman -S xdelta3

4) Do whatever you normally do

5) PROFIT

Additional Information

I decided to setup two Arch Linux mirrors. I then spun some changes to repo-add to allow the creation of deltas from version A->B->C->D in the fashion of A->D, B->D, C->D and added a cleanup for when there is more than 5 the version is older than 3 months. This means that updates from the last 5 versions 3 months to the latest version will have a delta available. There is an upper limit of 15 deltas max per package, after this number of deltas, they will be force removed even if within the 3 month period.

I'm currently generating deltas for core, community and extra. I'm open to others if there is anyone interested.

The delta path is /archlinux-deltarepo/ instead of /archlinux/.

The ratio is set to 0.99, the reasoning behind this is when you're on mobile and the package is huge, you may see value in saving the extra few MiB.

Mirroring occurs every few mins, after which the deltas are generated. The delta repo is behind the official /archlinux/ by only a few mins depending on how long delta generation takes.

The delta repo can also be mirrored using rsync.

This is a longer term project of mine and not likely to *disappear*.

Feedback/Suggestions

Always welcome!

Updates
2019-01-13 - Deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas

Last edited by nkukard (2019-01-28 10:18:00)

nkukard · 2019-01-13 17:34:03

Hi guys,

Thanks for the suggestions and feedback sent to me in pm.

I've made a slight modification, deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas per package.

This means:
- If there is more than 15 versions in the past 3 months, only the latest 15 versions delta files will be kept around
- If a package only has 1 update in 3 months, only that 1 update will have a delta file kept around

So if you're updating your machine at least once every 3 months, the mirror should have a delta for it.

-N

atemu · 2019-01-28 10:15:51

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

nkukard · 2019-01-28 10:18:21

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

Thanks for pointing that out, I've updated the above post.

katie · 2019-02-11 00:26:24

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

nkukard · 2019-02-11 06:14:51

katie wrote:

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

As soon as I get some time to clean up the latest changes I made I will be doing so.

In the meantime you could always rsync off one of my UK mirror.

katie · 2019-02-12 18:44:05

nkukard wrote:

katie wrote:
This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?
As soon as I get some time to clean up the latest changes I made I will be doing so.
In the meantime you could always rsync off one of my UK mirror.

Awesome! I'm mirroring the UK mirror at

Server = http://mirror.katie.host/archlinux/$repo/os/$arch

if anyone wants to use it <:^)

I also went ahead and started mirroring multilib, multilib-testing, and community-testing from a tier 1 mirror, since nkukard isn't generating deltas for those.

Last edited by katie (2019-02-12 18:56:36)

bartus · 2019-03-11 14:28:22

What's a plan for this `delta` repos, now after `libalpm` and Pacman drops support for deltas commit
Can they be facilitated with the external script, or a fork of pacman/libalpm would be needed?

Last edited by bartus (2019-03-11 14:32:28)

nkukard · 2019-03-11 14:34:07

bartus wrote:

What's a plan for this `delta` repos, now after `libalpm` and Pacman drops support for deltas commit

Oh shucks, that sucks.

Not much I can do from my side. I'm willing to keep the deltas available for maybe 3 months, or until I stop seeing downloads. After that, I'll swing the virtual host over to my mirror without deltas.

progandy · 2019-03-11 15:56:20

Not much I can do from my side.

If you are serious about deltas, then you could create a tool for XferCommand which can do delta-resolution and in the end throws the complete new package at pacman. I question whether the required effort would be worth it, though.

Last edited by progandy (2019-03-11 15:57:42)

nkukard · 2019-03-11 16:25:21

progandy wrote:

Not much I can do from my side.
If you are serious about deltas, then you could create a tool for XferCommand which can do delta-resolution and in the end throws the complete new package at pacman. I question whether the required effort would be worth it, though.

Yea, I spent about 2 weeks with the repo scripts to get the delta support I'm using at the moment. I don't have much more time to spend on additional tooling . I only bargained on the server admin portion being recurring.

If anyone has any ideas how we can use XferCommand, or has anything I can run on my side to generate packages to save download time, I'm all for it.

My first idea is to use rsync and generate batch updates.

mmdamin · 2019-03-30 19:36:20

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?

error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds

nkukard · 2019-03-30 20:01:39

mmdamin wrote:

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?

error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds

UK is United Kingdom.

ZA is South Africa.

There is currently no issues on my side with either.

mmdamin · 2019-03-30 20:23:07

nkukard wrote:

mmdamin wrote:
atemu wrote:
The mirrorlist entries need to be
Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch
and
Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch
respectively.
is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?
error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
UK is United Kingdom.
ZA is South Africa.
There is currently no issues on my side with either.

I'm in Iran. Is there any better repo for me? I've found a french one that I'm going to try but is there any other?

EDIT: I've changed my connection to another provider, in that one the uk mirror is ~100K/s (sometime reach 1K/s even 100B/s) and the delta.archlinux.fr doesn't connect at all.

Last edited by mmdamin (2019-03-31 07:20:15)

Arch Linux

#1 2019-01-08 18:58:04

Arch Linux Delta Repositories

#2 2019-01-13 17:34:03

Re: Arch Linux Delta Repositories

#3 2019-01-28 10:15:51

Re: Arch Linux Delta Repositories

#4 2019-01-28 10:18:21

Re: Arch Linux Delta Repositories

#5 2019-02-11 00:26:24

Re: Arch Linux Delta Repositories

#6 2019-02-11 06:14:51

Re: Arch Linux Delta Repositories

#7 2019-02-12 18:44:05

Re: Arch Linux Delta Repositories

#8 2019-03-11 14:28:22

Re: Arch Linux Delta Repositories

#9 2019-03-11 14:34:07

Re: Arch Linux Delta Repositories

#10 2019-03-11 15:56:20

Re: Arch Linux Delta Repositories

#11 2019-03-11 16:25:21

Re: Arch Linux Delta Repositories

#12 2019-03-30 19:36:20

Re: Arch Linux Delta Repositories

#13 2019-03-30 20:01:39

Re: Arch Linux Delta Repositories

#14 2019-03-30 20:23:07

Re: Arch Linux Delta Repositories

Board footer