You are not logged in.

#1 2019-01-08 18:58:04

nkukard
Member
Registered: 2018-12-26
Posts: 12

Arch Linux Delta Repositories

Background:

I just love Arch Linux and delta support intrigued me.

TLDR; Currently generating deltas for last 5 versions 3 months of packages directly to the latest version (traditionally A->B->C->D->E->F, now A->F, B->F, C->F, D->F, E->F), up to a 0.99 ratio. (+IPv6!)

Usage:

1) Edit /etc/pacman.d/mirrorlist and add the following as the top repository... (http and https supported, replace 'uk' with 'za' if you're closer to South Africa)

# Delta repository
Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

2) Edit /etc/pacman.conf and uncomment "UseDelta=0.7" in the [options] section
Deltas are generated up to 0.99 ratio (for when every MiB matters), so if you're on a mobile connection, you may want to increase the default 0.7 if you see fit.

3) Install xdelta3...

pacman -S xdelta3

4) Do whatever you normally do

5) PROFIT


Additional Information

I decided to setup two Arch Linux mirrors. I then spun some changes to repo-add to allow the creation of deltas from version A->B->C->D in the fashion of A->D, B->D, C->D and added a cleanup for when there is more than 5 the version is older than 3 months. This means that updates from the last 5 versions 3 months to the latest version will have a delta available. There is an upper limit of 15 deltas max per package, after this number of deltas, they will be force removed even if within the 3 month period.

I'm currently generating deltas for core, community and extra. I'm open to others if there is anyone interested.

The delta path is /archlinux-deltarepo/ instead of /archlinux/.

The ratio is set to 0.99, the reasoning behind this is when you're on mobile and the package is huge, you may see value in saving the extra few MiB.

Mirroring occurs every few mins, after which the deltas are generated. The delta repo is behind the official /archlinux/ by only a few mins depending on how long delta generation takes.

The delta repo can also be mirrored using rsync.

This is a longer term project of mine and not likely to *disappear*.


Feedback/Suggestions

Always welcome!


Updates
2019-01-13 - Deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas

Last edited by nkukard (2019-01-28 10:18:00)


IRC ~ WildPikachu

Offline

#2 2019-01-13 17:34:03

nkukard
Member
Registered: 2018-12-26
Posts: 12

Re: Arch Linux Delta Repositories

Hi guys,

Thanks for the suggestions and feedback sent to me in pm.

I've made a slight modification, deltas are now generated for the last 3 months of package versions, with an upper limit of 15 deltas per package.

This means:
- If there is more than 15 versions in the past 3 months, only the latest 15 versions delta files will be kept around
- If a package only has 1 update in 3 months, only that 1 update will have a delta file kept around

So if you're updating your machine at least once every 3 months, the mirror should have a delta for it.

-N


IRC ~ WildPikachu

Offline

#3 2019-01-28 10:15:51

atemu
Member
Registered: 2019-01-28
Posts: 1

Re: Arch Linux Delta Repositories

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

Offline

#4 2019-01-28 10:18:21

nkukard
Member
Registered: 2018-12-26
Posts: 12

Re: Arch Linux Delta Repositories

atemu wrote:

The mirrorlist entries need to be

Server = http://archlinux.uk.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

and

Server = http://archlinux.za.mirror.allworldit.com/archlinux-deltarepo/$repo/os/$arch

respectively.

Thanks for pointing that out, I've updated the above post.


IRC ~ WildPikachu

Offline

#5 2019-02-11 00:26:24

katie
Member
Registered: 2017-06-04
Posts: 2

Re: Arch Linux Delta Repositories

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

Offline

#6 2019-02-11 06:14:51

nkukard
Member
Registered: 2018-12-26
Posts: 12

Re: Arch Linux Delta Repositories

katie wrote:

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

As soon as I get some time to clean up the latest changes I made I will be doing so.

In the meantime you could always rsync off one of my UK mirror.


IRC ~ WildPikachu

Offline

#7 2019-02-12 18:44:05

katie
Member
Registered: 2017-06-04
Posts: 2

Re: Arch Linux Delta Repositories

nkukard wrote:
katie wrote:

This is wonderful! I'd love to set up a US delta repo, and it'd be great if I didn't have to reinvent the wheel. Any plans to publish your scripts?

As soon as I get some time to clean up the latest changes I made I will be doing so.

In the meantime you could always rsync off one of my UK mirror.

Awesome! I'm mirroring the UK mirror at

Server = http://mirror.katie.host/archlinux/$repo/os/$arch

if anyone wants to use it <:^)

I also went ahead and started mirroring multilib, multilib-testing, and community-testing from a tier 1 mirror, since nkukard isn't generating deltas for those.

Last edited by katie (2019-02-12 18:56:36)

Offline

Board footer

Powered by FluxBB