You are not logged in.
I have created a private local mirror following this Archlinux guidelines. Used the repo-syncing template mentioned in the same guidelines. Added a few parameters :
#!/bin/bash
#
########
#
# Copyright © 2014-2019 Florian Pritz <bluewind@xinu.at>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
#
########
#
# This is a simple mirroring script. To save bandwidth it first checks a
# timestamp via HTTP and only runs rsync when the timestamp differs from the
# local copy. As of 2016, a single rsync run without changes transfers roughly
# 6MiB of data which adds up to roughly 250GiB of traffic per month when rsync
# is run every minute. Performing a simple check via HTTP first can thus save a
# lot of traffic.
# Directory where the repo is stored locally. Example: /srv/repo
target="/home/mirror"
# Lockfile path
lock="/var/lock/syncrepo.lck"
# If you want to limit the bandwidth used by rsync set this.
# Use 0 to disable the limit.
# The default unit is KiB (see man rsync /--bwlimit for more)
bwlimit=0
# The source URL of the mirror you want to sync from.
# If you are a tier 1 mirror use rsync.archlinux.org, for example like this:
# rsync://rsync.archlinux.org/ftp_tier1
# Otherwise chose a tier 1 mirror from this list and use its rsync URL:
# https://www.archlinux.org/mirrors/
source_url='rsync://mirror.rackspace.com/archlinux'
# An HTTP(S) URL pointing to the 'lastupdate' file on your chosen mirror.
# If you are a tier 1 mirror use: https://rsync.archlinux.org/lastupdate
# Otherwise use the HTTP(S) URL from your chosen mirror.
lastupdate_url='rsync://mirror.rackspace.com/archlinux/lastupdate'
#### END CONFIG
[ ! -d "${target}" ] && mkdir -p "${target}"
exec 9>"${lock}"
flock -n 9 || exit
# Cleanup any temporary files from old run that might remain.
# Note: You can skip this if you have rsync newer than 3.2.3
# not affected by https://github.com/WayneD/rsync/issues/192
find "${target}" -name '.~tmp~' -exec rm -r {} +
rsync_cmd() {
local -a cmd=(rsync -rlptH --safe-links --delete-delay --delay-updates
"--timeout=600" "--contimeout=60" --no-motd)
if stty &>/dev/null; then
cmd+=(-h -v --progress)
else
cmd+=(--quiet)
fi
if ((bwlimit>0)); then
cmd+=("--bwlimit=$bwlimit")
fi
"${cmd[@]}" "$@"
}
# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
# keep lastsync file in sync for statistics generated by the Arch Linux website
rsync_cmd "$source_url/lastsync" "$target/lastsync"
exit 0
fi
rsync_cmd \
--exclude='*.links.tar.gz*' \
--exclude='/other' \
--exclude='/sources' \
--exclude='/archive' \
--exclude='/community-debug' \
--exclude='/community-staging-debug' \
--exclude='/community-staging' \
--exclude='/community-testing-debug' \
--exclude='/community-testing' \
--exclude='/core-debug' \
--exclude='/extra-debug' \
--exclude='/gnome-unstable-debug' \
--exclude='/gnome-unstable' \
--exclude='/images' \
--exclude='/kde-unstable-debug' \
--exclude='/kde-unstable' \
--exclude='/multilib-debug' \
--exclude='/multilib-staging-debug' \
--exclude='/multilib-staging' \
--exclude='/multilib-testing-debug' \
--exclude='/multilib-testing' \
--exclude='/multilib' \
--exclude='/pool' \
--exclude='/staging-debug' \
--exclude='/staging' \
--exclude='/testing-debug' \
--exclude='/testing' \
"${source_url}" \
"${target}"
#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"
Basically I am only downloading the :
- core
- community
- extra
- iso
directories only.
I have downloaded all those files, was around 5.5GB
sudo ./syncrepo-template.sh
Used python's http server to serve that directory :
cd /home/mirror
python -m http.server 8000
At this point, I could access all those files (and download them) through going to a web browser (on some other computer in my local network) and using that servers IP address : http://192.168.130.137:8000
Now on another Archlinux system, I have setup the mirrorlist as :
# /etc/pacman.d/mirrorlist
Server = http://192.168.130.137:8000/$repo/os/$arch
The Problem
When I start upgrading the packages (the same system where the local mirror was added) :
sudo pacman -Syu
:: Synchronizing package databases...
core is up to date
extra 1723.7 KiB 168 MiB/s 00:00 [#####################################] 100%
community is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
Packages (3) linux-5.17.9.arch1-1 llvm-libs-13.0.1-4 mesa-22.0.4-1
Total Download Size: 198.39 MiB
Total Installed Size: 337.85 MiB
Net Upgrade Size: -0.03 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
linux-5.17.9.arch1-1-x86_64.pkg.tar.zst failed to download
llvm-libs-13.0.1-4-x86_64.pkg.tar.zst failed to download
mesa-22.0.4-1-x86_64.pkg.tar.zst failed to download
Total (3/3) 198.4 MiB 48.4 GiB/s 00:00 [#####################################] 100%
error: failed retrieving file 'linux-5.17.9.arch1-1-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requested URL returned error: 404
error: failed retrieving file 'llvm-libs-13.0.1-4-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requestedURL returned error: 404
error: failed retrieving file 'mesa-22.0.4-1-x86_64.pkg.tar.zst' from 192.168.130.137:8000 : The requested URL returned error: 404
warning: too many errors from 192.168.130.137:8000, skipping for the remainder of this transaction
warning: failed to retrieve some files
error: failed to commit transaction (failed to retrieve some files)
Errors occurred, no packages were upgraded.
And at the meantime, the response on the server serving the mirror :
Server Error
Where is the problem ? if any have done this kind of setup can guide me pls.
Last edited by pritam_poseidon (2022-05-25 16:23:20)
Offline
Silly me
it is required to include the pool directory for download...
which i had excluded unknowingly :
New syncrepo-template.sh
#!/bin/bash
#
########
#
# Copyright © 2014-2019 Florian Pritz <bluewind@xinu.at>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
#
########
#
# This is a simple mirroring script. To save bandwidth it first checks a
# timestamp via HTTP and only runs rsync when the timestamp differs from the
# local copy. As of 2016, a single rsync run without changes transfers roughly
# 6MiB of data which adds up to roughly 250GiB of traffic per month when rsync
# is run every minute. Performing a simple check via HTTP first can thus save a
# lot of traffic.
# Directory where the repo is stored locally. Example: /srv/repo
target="/home/mirror"
# Lockfile path
lock="/var/lock/syncrepo.lck"
# If you want to limit the bandwidth used by rsync set this.
# Use 0 to disable the limit.
# The default unit is KiB (see man rsync /--bwlimit for more)
bwlimit=0
# The source URL of the mirror you want to sync from.
# If you are a tier 1 mirror use rsync.archlinux.org, for example like this:
# rsync://rsync.archlinux.org/ftp_tier1
# Otherwise chose a tier 1 mirror from this list and use its rsync URL:
# https://www.archlinux.org/mirrors/
source_url='rsync://mirror.rackspace.com/archlinux'
# An HTTP(S) URL pointing to the 'lastupdate' file on your chosen mirror.
# If you are a tier 1 mirror use: https://rsync.archlinux.org/lastupdate
# Otherwise use the HTTP(S) URL from your chosen mirror.
lastupdate_url='rsync://mirror.rackspace.com/archlinux/lastupdate'
#### END CONFIG
[ ! -d "${target}" ] && mkdir -p "${target}"
exec 9>"${lock}"
flock -n 9 || exit
# Cleanup any temporary files from old run that might remain.
# Note: You can skip this if you have rsync newer than 3.2.3
# not affected by https://github.com/WayneD/rsync/issues/192
# I have also closed this check,
# (in simple english) as due to huge size of the repo with slow download speed,
# if for some reason the syncing gets interrupted and killed,
# then closing the following command, this script will not have to download those huge packages again.
# as described in the previous comments.
# find "${target}" -name '.~tmp~' -exec rm -r {} +
rsync_cmd() {
local -a cmd=(rsync -rlptH --safe-links --delete-delay --delay-updates
"--timeout=600" "--contimeout=60" --no-motd)
if stty &>/dev/null; then
cmd+=(-h -v --progress)
else
cmd+=(--quiet)
fi
if ((bwlimit>0)); then
cmd+=("--bwlimit=$bwlimit")
fi
"${cmd[@]}" "$@"
}
# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
# keep lastsync file in sync for statistics generated by the Arch Linux website
rsync_cmd "$source_url/lastsync" "$target/lastsync"
exit 0
fi
rsync_cmd \
--exclude='*.links.tar.gz*' \
--exclude='/other' \
--exclude='/sources' \
--exclude='/archive' \
--exclude='/community-debug' \
--exclude='/community-staging-debug' \
--exclude='/community-staging' \
--exclude='/community-testing-debug' \
--exclude='/community-testing' \
--exclude='/core-debug' \
--exclude='/extra-debug' \
--exclude='/gnome-unstable-debug' \
--exclude='/gnome-unstable' \
--exclude='/images' \
--exclude='/kde-unstable-debug' \
--exclude='/kde-unstable' \
--exclude='/multilib-debug' \
--exclude='/multilib-staging-debug' \
--exclude='/multilib-staging' \
--exclude='/multilib-testing-debug' \
--exclude='/multilib-testing' \
--exclude='/multilib' \
# --exclude='/pool' \
--exclude='/staging-debug' \
--exclude='/staging' \
--exclude='/testing-debug' \
--exclude='/testing' \
"${source_url}" \
"${target}"
#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"
Offline