You are not logged in.

#1 2019-01-08 18:58:04

nkukard
Member
Registered: 2018-12-26
Posts: 24

Arch Linux Delta Repositories

Background:

I just love Arch Linux and delta support intrigued me.

TLDR; Currently generating deltas for last 5 versions 3 months of packages directly to the latest version (traditionally A->B->C->D->E->F, now A->F, B->F, C->F, D->F, E->F), up to a 0.99 ratio. (+IPv6!)

Usage:

1) Edit /etc/pacman.d/mirrorlist and add the following as the top repository... (http and https supported, replace 'uk' with 'za' if you're closer to South Africa)

# Delta repository
Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

2) Edit /etc/pacman.conf and uncomment "UseDelta=0.7" in the [options] section
Deltas are generated up to 0.99 ratio (for when every MiB matters), so if you're on a mobile connection, you may want to increase the default 0.7 if you see fit.

3) Install xdelta3...

pacman -S xdelta3

4) Do whatever you normally do

5) PROFIT


Additional Information

I decided to setup two Arch Linux mirrors. I then spun some changes to repo-add to allow the creation of deltas from version A->B->C->D in the fashion of A->D, B->D, C->D and added a cleanup for when there is more than 5 the version is older than 3 months. This means that updates from the last 5 versions 3 months to the latest version will have a delta available. There is an upper limit of 15 deltas max per package, after this number of deltas, they will be force removed even if within the 3 month period.

I'm currently generating deltas for core, community and extra. I'm open to others if there is anyone interested.

The delta path is /archlinux-deltarepo/ instead of /archlinux/.

The ratio is set to 0.99, the reasoning behind this is when you're on mobile and the package is huge, you may see value in saving the extra few MiB.

Mirroring occurs every few mins, after which the deltas are generated. The delta repo is behind the official /archlinux/ by only a few mins depending on how long delta generation takes.

The delta repo can also be mirrored using rsync.

This is a longer term project of mine and not likely to *disappear*.


Feedback/Suggestions

Always welcome!


Updates
2019-01-13 - Deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas

Last edited by nkukard (2019-01-28 10:18:00)


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#2 2019-01-13 17:34:03

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

Hi guys,

Thanks for the suggestions and feedback sent to me in pm.

I've made a slight modification, deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas per package.

This means:
- If there is more than 15 versions in the past 3 months, only the latest 15 versions delta files will be kept around
- If a package only has 1 update in 3 months, only that 1 update will have a delta file kept around

So if you're updating your machine at least once every 3 months, the mirror should have a delta for it.

-N


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#3 2019-01-28 10:15:51

atemu
Member
Registered: 2019-01-28
Posts: 1

Re: Arch Linux Delta Repositories

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

Offline

#4 2019-01-28 10:18:21

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

Thanks for pointing that out, I've updated the above post.


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#5 2019-02-11 00:26:24

katie
Member
Registered: 2017-06-04
Posts: 2

Re: Arch Linux Delta Repositories

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

Offline

#6 2019-02-11 06:14:51

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

katie wrote:

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

As soon as I get some time to clean up the latest changes I made I will be doing so.

In the meantime you could always rsync off one of my UK mirror.


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#7 2019-02-12 18:44:05

katie
Member
Registered: 2017-06-04
Posts: 2

Re: Arch Linux Delta Repositories

nkukard wrote:
katie wrote:

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

As soon as I get some time to clean up the latest changes I made I will be doing so.

In the meantime you could always rsync off one of my UK mirror.

Awesome! I'm mirroring the UK mirror at

Server = http://mirror.katie.host/archlinux/$repo/os/$arch

if anyone wants to use it <:^)

I also went ahead and started mirroring multilib, multilib-testing, and community-testing from a tier 1 mirror, since nkukard isn't generating deltas for those.

Last edited by katie (2019-02-12 18:56:36)

Offline

#8 2019-03-11 14:28:22

bartus
Member
Registered: 2013-05-13
Posts: 50

Re: Arch Linux Delta Repositories

What's a plan for this `delta` repos, now after `libalpm` and Pacman drops support for deltas hmm commit
Can they be facilitated with the external script, or a fork of pacman/libalpm would be needed?

Last edited by bartus (2019-03-11 14:32:28)

Offline

#9 2019-03-11 14:34:07

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

bartus wrote:

What's a plan for this `delta` repos, now after `libalpm` and Pacman drops support for deltas hmm commit

Oh shucks, that sucks.

Not much I can do from my side. I'm willing to keep the deltas available for maybe 3 months, or until I stop seeing downloads. After that, I'll swing the virtual host over to my mirror without deltas.


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#10 2019-03-11 15:56:20

progandy
Member
Registered: 2012-05-17
Posts: 5,256

Re: Arch Linux Delta Repositories

Not much I can do from my side.

If you are serious about deltas, then you could create a tool for XferCommand which can do delta-resolution and in the end throws the complete new package at pacman. I question whether the required effort would be worth it, though.

Last edited by progandy (2019-03-11 15:57:42)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#11 2019-03-11 16:25:21

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

progandy wrote:

Not much I can do from my side.

If you are serious about deltas, then you could create a tool for XferCommand which can do delta-resolution and in the end throws the complete new package at pacman. I question whether the required effort would be worth it, though.

Yea, I spent about 2 weeks with the repo scripts to get the delta support I'm using at the moment. I don't have much more time to spend on additional tooling sad. I only bargained on the server admin portion being recurring.

If anyone has any ideas how we can use XferCommand, or has anything I can run on my side to generate packages to save download time, I'm all for it.

My first idea is to use rsync and generate batch updates.


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#12 2019-03-30 19:36:20

mmdamin
Member
From: Iran
Registered: 2019-02-19
Posts: 205
Website

Re: Arch Linux Delta Repositories

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?

error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds

Offline

#13 2019-03-30 20:01:39

nkukard
Member
Registered: 2018-12-26
Posts: 24

Re: Arch Linux Delta Repositories

mmdamin wrote:
atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?

error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds

UK is United Kingdom.

ZA is South Africa.

There is currently no issues on my side with either.


irc.libera.chat ~ nkukard
Discord ~ discord.gg/linuxchat ~ OpenSourceCoder

Offline

#14 2019-03-30 20:23:07

mmdamin
Member
From: Iran
Registered: 2019-02-19
Posts: 205
Website

Re: Arch Linux Delta Repositories

nkukard wrote:
mmdamin wrote:
atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

is there any other repo?
do the 'uk' and 'za' in repo's address shows a special region for example uk means United Kingdom?

error: failed retrieving file 'core.db' from archlinux.uk.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'core.db' from archlinux.za.mirror.allworldit.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds

UK is United Kingdom.

ZA is South Africa.

There is currently no issues on my side with either.

I'm in Iran. Is there any better repo for me? I've found a french one that I'm going to try but is there any other?

EDIT: I've changed my connection to another provider, in that one the uk mirror is ~100K/s (sometime reach 1K/s even 100B/s) and the delta.archlinux.fr doesn't connect at all.

Last edited by mmdamin (2019-03-31 07:20:15)

Offline

Board footer

Powered by FluxBB