You are not logged in.

#1 2022-05-20 09:46:41

pritam_poseidon
Member
Registered: 2021-07-13
Posts: 21

[SOLVED] Error serving local private mirror files

I have created a private local mirror following this Archlinux guidelines. Used the repo-syncing template mentioned in the same guidelines. Added a few parameters :

#!/bin/bash
#
########
#
# Copyright © 2014-2019 Florian Pritz <bluewind@xinu.at>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
#
########
#
# This is a simple mirroring script. To save bandwidth it first checks a
# timestamp via HTTP and only runs rsync when the timestamp differs from the
# local copy. As of 2016, a single rsync run without changes transfers roughly
# 6MiB of data which adds up to roughly 250GiB of traffic per month when rsync
# is run every minute. Performing a simple check via HTTP first can thus save a
# lot of traffic.

# Directory where the repo is stored locally. Example: /srv/repo
target="/home/mirror"

# Lockfile path
lock="/var/lock/syncrepo.lck"

# If you want to limit the bandwidth used by rsync set this.
# Use 0 to disable the limit.
# The default unit is KiB (see man rsync /--bwlimit for more)
bwlimit=0

# The source URL of the mirror you want to sync from.
# If you are a tier 1 mirror use rsync.archlinux.org, for example like this:
# rsync://rsync.archlinux.org/ftp_tier1
# Otherwise chose a tier 1 mirror from this list and use its rsync URL:
# https://www.archlinux.org/mirrors/
source_url='rsync://mirror.rackspace.com/archlinux'

# An HTTP(S) URL pointing to the 'lastupdate' file on your chosen mirror.
# If you are a tier 1 mirror use: https://rsync.archlinux.org/lastupdate
# Otherwise use the HTTP(S) URL from your chosen mirror.
lastupdate_url='rsync://mirror.rackspace.com/archlinux/lastupdate'

#### END CONFIG

[ ! -d "${target}" ] && mkdir -p "${target}"

exec 9>"${lock}"
flock -n 9 || exit

# Cleanup any temporary files from old run that might remain.
# Note: You can skip this if you have rsync newer than 3.2.3
# not affected by https://github.com/WayneD/rsync/issues/192
find "${target}" -name '.~tmp~' -exec rm -r {} +

rsync_cmd() {
	local -a cmd=(rsync -rlptH --safe-links --delete-delay --delay-updates
		"--timeout=600" "--contimeout=60" --no-motd)

	if stty &>/dev/null; then
		cmd+=(-h -v --progress)
	else
		cmd+=(--quiet)
	fi

	if ((bwlimit>0)); then
		cmd+=("--bwlimit=$bwlimit")
	fi

	"${cmd[@]}" "$@"
}


# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
	# keep lastsync file in sync for statistics generated by the Arch Linux website
	rsync_cmd "$source_url/lastsync" "$target/lastsync"
	exit 0
fi

rsync_cmd \
	--exclude='*.links.tar.gz*' \
	--exclude='/other' \
	--exclude='/sources' \
        --exclude='/archive' \
        --exclude='/community-debug' \
        --exclude='/community-staging-debug' \
        --exclude='/community-staging' \
        --exclude='/community-testing-debug' \
        --exclude='/community-testing' \
        --exclude='/core-debug' \
        --exclude='/extra-debug' \
        --exclude='/gnome-unstable-debug' \
        --exclude='/gnome-unstable' \
        --exclude='/images' \
        --exclude='/kde-unstable-debug' \
        --exclude='/kde-unstable' \
        --exclude='/multilib-debug' \
        --exclude='/multilib-staging-debug' \
        --exclude='/multilib-staging' \
        --exclude='/multilib-testing-debug' \
        --exclude='/multilib-testing' \
        --exclude='/multilib' \
        --exclude='/pool' \
        --exclude='/staging-debug' \
        --exclude='/staging' \
        --exclude='/testing-debug' \
        --exclude='/testing' \
	"${source_url}" \
	"${target}"

#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"

Basically I am only downloading the :
- core
- community
- extra
- iso
directories only.

I have downloaded all those files, was around 5.5GB

sudo ./syncrepo-template.sh

Used python's http server to serve that directory :

cd /home/mirror
python -m http.server 8000

At this point, I could access all those files (and download them) through going to a web browser (on some other computer in my local network) and using that servers IP address : http://192.168.130.137:8000

Now on another Archlinux system, I have setup the mirrorlist as :

# /etc/pacman.d/mirrorlist
Server = http://192.168.130.137:8000/$repo/os/$arch

The Problem
When I start upgrading the packages (the same system where the local mirror was added) :

sudo pacman -Syu
:: Synchronizing package databases...
core is up to date
extra                                1723.7 KiB   168 MiB/s 00:00 [#####################################] 100%
community is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (3) linux-5.17.9.arch1-1  llvm-libs-13.0.1-4  mesa-22.0.4-1

Total Download Size:   198.39 MiB
Total Installed Size:  337.85 MiB
Net Upgrade Size:       -0.03 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
linux-5.17.9.arch1-1-x86_64.pkg.tar.zst failed to download
llvm-libs-13.0.1-4-x86_64.pkg.tar.zst failed to download
mesa-22.0.4-1-x86_64.pkg.tar.zst failed to download
Total (3/3)                           198.4 MiB  48.4 GiB/s 00:00 [#####################################] 100%
error: failed retrieving file 'linux-5.17.9.arch1-1-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requested URL returned error: 404
error: failed retrieving file 'llvm-libs-13.0.1-4-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requestedURL returned error: 404
error: failed retrieving file 'mesa-22.0.4-1-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requested URL returned error: 404
warning: too many errors from 192.168.130.137:8000, skipping for the remainder of this transaction
warning: failed to retrieve some files
error: failed to commit transaction (failed to retrieve some files)
Errors occurred, no packages were upgraded.

And at the meantime, the response on the server serving the mirror :
Server Error

Where is the problem ? if any have done this kind of setup can guide me pls.

Last edited by pritam_poseidon (2022-05-25 16:23:20)

Offline

#2 2022-05-23 14:12:48

pritam_poseidon
Member
Registered: 2021-07-13
Posts: 21

Re: [SOLVED] Error serving local private mirror files

Silly me smile
it is required to include the pool directory for download...
which i had excluded unknowingly :

New syncrepo-template.sh

#!/bin/bash
#
########
#
# Copyright © 2014-2019 Florian Pritz <bluewind@xinu.at>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
#
########
#
# This is a simple mirroring script. To save bandwidth it first checks a
# timestamp via HTTP and only runs rsync when the timestamp differs from the
# local copy. As of 2016, a single rsync run without changes transfers roughly
# 6MiB of data which adds up to roughly 250GiB of traffic per month when rsync
# is run every minute. Performing a simple check via HTTP first can thus save a
# lot of traffic.

# Directory where the repo is stored locally. Example: /srv/repo
target="/home/mirror"

# Lockfile path
lock="/var/lock/syncrepo.lck"

# If you want to limit the bandwidth used by rsync set this.
# Use 0 to disable the limit.
# The default unit is KiB (see man rsync /--bwlimit for more)
bwlimit=0

# The source URL of the mirror you want to sync from.
# If you are a tier 1 mirror use rsync.archlinux.org, for example like this:
# rsync://rsync.archlinux.org/ftp_tier1
# Otherwise chose a tier 1 mirror from this list and use its rsync URL:
# https://www.archlinux.org/mirrors/
source_url='rsync://mirror.rackspace.com/archlinux'

# An HTTP(S) URL pointing to the 'lastupdate' file on your chosen mirror.
# If you are a tier 1 mirror use: https://rsync.archlinux.org/lastupdate
# Otherwise use the HTTP(S) URL from your chosen mirror.
lastupdate_url='rsync://mirror.rackspace.com/archlinux/lastupdate'

#### END CONFIG

[ ! -d "${target}" ] && mkdir -p "${target}"

exec 9>"${lock}"
flock -n 9 || exit

# Cleanup any temporary files from old run that might remain.
# Note: You can skip this if you have rsync newer than 3.2.3
# not affected by https://github.com/WayneD/rsync/issues/192

# I have also closed this check, 
# (in simple english) as due to huge size of the repo with slow download speed, 
# if for some reason the syncing gets interrupted and killed,
# then closing the following command, this script will not have to download those huge packages again.
# as described in the previous comments.
# find "${target}" -name '.~tmp~' -exec rm -r {} +


rsync_cmd() {
	local -a cmd=(rsync -rlptH --safe-links --delete-delay --delay-updates
		"--timeout=600" "--contimeout=60" --no-motd)

	if stty &>/dev/null; then
		cmd+=(-h -v --progress)
	else
		cmd+=(--quiet)
	fi

	if ((bwlimit>0)); then
		cmd+=("--bwlimit=$bwlimit")
	fi

	"${cmd[@]}" "$@"
}


# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
	# keep lastsync file in sync for statistics generated by the Arch Linux website
	rsync_cmd "$source_url/lastsync" "$target/lastsync"
	exit 0
fi

rsync_cmd \
	--exclude='*.links.tar.gz*' \
	--exclude='/other' \
	--exclude='/sources' \
        --exclude='/archive' \
        --exclude='/community-debug' \
        --exclude='/community-staging-debug' \
        --exclude='/community-staging' \
        --exclude='/community-testing-debug' \
        --exclude='/community-testing' \
        --exclude='/core-debug' \
        --exclude='/extra-debug' \
        --exclude='/gnome-unstable-debug' \
        --exclude='/gnome-unstable' \
        --exclude='/images' \
        --exclude='/kde-unstable-debug' \
        --exclude='/kde-unstable' \
        --exclude='/multilib-debug' \
        --exclude='/multilib-staging-debug' \
        --exclude='/multilib-staging' \
        --exclude='/multilib-testing-debug' \
        --exclude='/multilib-testing' \
        --exclude='/multilib' \
#     --exclude='/pool' \
        --exclude='/staging-debug' \
        --exclude='/staging' \
        --exclude='/testing-debug' \
        --exclude='/testing' \
	"${source_url}" \
	"${target}"

#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"

Offline

Board footer

Powered by FluxBB