Making archlinux (local) mirror

3nd3r · 2007-07-01 05:54:01

Before I post the issue PLEASE dont send me to the wiki page, I have been there and it needs to be updated big time!

okay, I followed the steps on the wiki and I cannot get the mirror script to function, I just get access denied. I believe that I get access denied due to the fact that I am not an official mirror. There is nowhere on the wiki that shows any other servers that can be used properly for the script. I tried the ibiolo mirror on the page but it does not work. If anyone has an unofficial mirror, can you please paste your whole sync.sh? or can someone give me an alternative mirror to rsync.archlinux.org? Thanks

Last edited by 3nd3r (2007-07-01 05:54:50)

kishd · 2007-07-02 12:53:39

try mirror.pacific.net.au

rsync -avzy --delete-after --progress rsync://mirror.pacific.net.au::linux/archlinux/current/os/i686 /home/kishd/mirror/current/
rsync -avzy --delete-after --progress rsync://mirror.pacific.net.au::linux/archlinux/extra/os/i686 /home/kishd/mirror/extra/
# --delete to delete old files remove it if you want to keep them
rsync -avzy --delete-after --progress rsync://mirror.pacific.net.au::linux/archlinux/community/os/i686 /home/kishd/mirror/community/
rsync -avzy --delete-after --progress rsync://mirror.pacific.net.au::linux/archlinux/testing/os/i686 /home/kishd/mirror/testing/
rsync -avzy --delete-after --progress rsync://mirror.pacific.net.au::linux/archlinux/unstable/os/i686 /home/kishd/mirror/unstable/

Last edited by kishd (2007-07-02 12:54:49)

dtw · 2007-07-02 13:32:00

Do you REALLY need to mirror the whole thing?

I did some work on creating a shared pkg cache, to share at least the most common applications, and this will shortly be supported within pacman.

The basic idea is that you share the pkg cache via NFS on your "main" machine i.e. the one the has more pkgs than the others, and this cache is checked by all of your other machines _before_ it downloads anything from the server.

If the mirror is an attempt to save bandwidth for your organisation I'd be very surprised if it doesn't use more because your going to be getting hundreds of pkgs you don't need. And if you are running these systems in some sort of corporate/restricted environment (only case I can imagine where you have enough machines that a mirror would be required) then you don't actually need access to EVERY pkg.

Even if your "main" machine just has all of the pkgs you need for the other machines installed and is used for nothing except cron updates of those pkgs, a "cache-server", you're still going to save a shed tonne of bandwidth over mirroring.

The idea of local mirrors is pretty outmode, hence the out of date wiki page.

Stalwart · 2007-07-03 18:02:10

When i'll get a bigger hdd for my server - sure i'll mirror entire i686+x86_64. Because there are no mirrors in Latvia and i install and maintain some servers and workstations - i never know what package i'll need, having fast access to entire repo will be great

elgatofelix · 2007-07-03 19:44:45

i got the same with that script u say.. the thing is dir hierarchy errs made a changes in it and works

#!/bin/bash
# 
# This script is used to sync a local mirror of the Arch Linux repositories.
#
# Copyright (c)2007 Woody Gilk <woody@archlinux.org>
# Licensed under the GNU GPL (version 2)

# Original modificado  !

# Filesystem locations for the sync operations
SYNC_HOME="$HOME/archmirror"
SYNC_LOGS="$SYNC_HOME/logs"
SYNC_PKGS="$SYNC_HOME/packages"
  
EXCLUDE_FROM=rsync_arch.excludes
  
# This allows you to set which repositories you would like to sync
# Valid options are: current, extra, community, unstable, testing, release
SYNC_REPO=(current extra community )

# servidor oficial SYNC_SERVER="rsync.archlinux.org::"
#francia    rsync://distrib-coffee.ipsl.jussieu.fr/pub/linux/archlinux/ rsync
#francia    rsync://mir1.archlinuxfr.org/archlinux rsync
SYNC_SERVER="distro.ibiblio.org::distros/archlinux/"

# directorio almacenar descargas temporales ( rync )
PARTIAL_DIR="$SYNC_HOME/rsync_partial"
# eliminar ficheros que no existan en el servidor ..
DELETE=true
  
# The following line is the format of the log file name, this example will
# output something like this: pkgsync_20070201-8.log
LOG_FILE="pkgsync_$(date +%Y%m%d-%H).log"
  
# Do not edit the following lines, they protect the sync from running more than
# one instance at a time.
if [ ! -d $SYNC_HOME ]; then
      echo "$SYNC_HOME does not exist, please create it, then run this script again."
        exit 1
    fi

    SYNC_LOCK="$SYNC_HOME/sync_running.lock"
    if [ -f $SYNC_LOCK ] ; then
        echo "aparentemente existe una instancia de este programa en ejecucion... desea continuar? (s):"
        read continua
        if test "$continua" != "s"; then
            exit 1
        fi
    fi
     
    touch "$SYNC_LOCK"
    # End of non-editable lines  

    # Create the log file and insert a timestamp  
    
#[c]touch "$SYNC_LOGS/$LOG_FILE"
#[c]    echo "==========================================" >> "$SYNC_LOGS/$LOG_FILE"
#[c]    echo ">> Starting sync on $(date)" >> "$SYNC_LOGS/$LOG_FILE"
#[c]    echo ">> ---" >> "$SYNC_LOGS/$LOG_FILE"

        
    # Rsync each of the repos set in $SYNC_REPO
    for repo in ${SYNC_REPO[@]}; do
          repo=$(echo $repo |tr [:upper:] [:lower:])
            echo "Syncing $repo to $SYNC_PKGS/$repo"
          
          repourl=$repo
                
              # If you only want to mirror 32bit packages, you can add
                # " --exclude=os/x86_64" after "--delete"
                    # If you only want to mirror 64bit packages, use "--exclude=os/i686"
                  # If you want both 32bit and 64bit, leave the following line as it is
                    #rsync -rptv --delete --exclude=os/x86_64  rsync.archlinux.org::$repourl "$SYNC_PKGS/$repo" >> "$SYNC_LOGS/$LOG_FILE"

                      parametros=" -av --progress -hh -k --stats --partial --partial-dir="$PARTIAL_DIR
                    # parametros para eliminar ficheros que no existan en el servidor ..
                      if $DELETE; then
                          parametros=$parametros" --delete --delete-excluded"
                      fi
                      
                      if test -e $EXCLUDE_FROM ; then
                              parametros="$parametros --exclude-from=$EXCLUDE_FROM"
                      fi
                      echo $parametros
                      rsync $parametros --ipv4 --exclude=os/x86_64  $SYNC_SERVER$repourl "$SYNC_PKGS/" 
                        
                        # If you know of a rsync mirror that is geographically closer to you than archlinux.org
                      # simply replace ".archlinux.org" with the domain of that mirror in the above line
    done
                    
    # Remove the lock file and exit
    rm -f "$SYNC_LOCK"
    exit 0

and if there is a file rsync_arch.excludes in the dir where app is running then it must be like this one (man rsync)
where every line that matches a file in repo will be excluded in the download, otherside if prefixed with a + it will be
inckuded (man rsync)

# solo descargando paquetes, 
# si se quiere descargar tambien las imagenes de instalalcion, comentar
current/iso/

wesnoth*
nexuiz*
flightgear*
tremulous*
sauerbraten*
scorched3d*
#torcs-*

+ koffice-l10n-es*
koffice-l10n-*

+ kde-i18n-es*
kde-i18n-*

festival-*

eclipse-*

rfc-*

skype-*

+ openoffice-base*
+ openoffice-es*
+ openoffice-spell-en*
+ openoffice-spell-es*
openoffice-*

3nd3r · 2007-07-04 04:01:40

Thank you for all of the help, I will attempt to use this script ^^ tomorrow after I get home from work. Thanks again

MindTooth · 2008-12-05 09:25:21

Can this be used for creating a public mirror?

Birger

Arch Linux

#1 2007-07-01 05:54:01

Making archlinux (local) mirror

#2 2007-07-02 12:53:39

Re: Making archlinux (local) mirror

#3 2007-07-02 13:32:00

Re: Making archlinux (local) mirror

#4 2007-07-03 18:02:10

Re: Making archlinux (local) mirror

#5 2007-07-03 19:44:45

Re: Making archlinux (local) mirror

#6 2007-07-04 04:01:40

Re: Making archlinux (local) mirror

#7 2008-12-05 09:25:21

Re: Making archlinux (local) mirror

Board footer