Internal Arch repository?

Winkie · 2004-10-22 14:12:58

Ok, a summary first, I'm the systems administrator for a small company with a callcentre, few servers and a bunch of desktop machines. I plan to replace the antiquated SuSe and Mandrake installs with some nice fresh Arch installs.

Although arch is designed to be updated to bleeding edge at all points, i want to use it for
1. Pacman
2. Stability
3. I like the way everything is done

However, I won't blindly roll out updates to the whole callcentre, my plan is therefore to sync my local repository to the global one every month, sync one machine to it and 2 weeks later, if the machine is uncrashed, sync the rest. This keeps machines within a month and a half of the latest (obviously i'll update security related packages instantly) but it requires an internal repository, I have NO idea of the best way to set one up, so I come to you for advice.

Is there an easy way to produce an internal arch repository that 40-50 machines can use as a base without having to rsync the entire repository from the server / otherwise needlessly consume mirrors' bandwidth?

phrakture · 2004-10-22 14:21:54

well, you can use the "download only, don't install" option of pacman which escapes me at the moment.... if you run gensync on your pacman cache, you can expose it through ftp or nfs or smb to the local network and have them all sync to that server/share....
however, seeing as i haven't done this, you'd have to check if the "download only" option will, in effect, register it so it will download a newer version if available.... I am assuming you only want specific packages too.... if you want to sync the whole current tree... if all machines are isntalled identically, here is what I would do:

designate one machine as the staging machine
expose the pacman cache of this machine as mentioned
sync all other machines to this one
unexpose the cache
sync staging to live servers.... make sure the other machines don't do this (edit pacman.conf)
assuming the staging machine is stable, expose the cache again and sync all machines to it

in effect, you are allowing all other machines to sync only to the staging machine, and you shut off the repo, this way accidental syncs to the other machines do nothing (probably fail connecting)[/code]

apeiro · 2004-10-22 23:13:17

Winkie wrote:

Is there an easy way to produce an internal arch repository that 40-50 machines can use as a base without having to rsync the entire repository from the server / otherwise needlessly consume mirrors' bandwidth?

I keep a local mirror of Current with the following:

# rsync -a --delete rsync.archlinux.org::current /home/ftp

It's not that big (~500mb).

i3839 · 2004-10-23 14:19:52

Perhaps you're interested in my patch:

http://bugs.archlinux.org/index.php?do=details&id=1477

It gives the option to only upgrade the packages which are a certain age, so that you can avoid installing the real bleeding-edge packages, only installing the packages which have proven themselves somewhat.

A patches gensync script is needed because the standard one doesn't preserve the file dates (yet?), so it will work only on self made package databases for now. :-(

apeiro · 2004-10-23 17:30:26

I've modified dragon's gensync to preserve the mtimes, so you guys can use the UpgradeDelay patch if you want.

i3839 · 2004-10-23 18:15:17

Thanks!

kleptophobiac · 2004-11-30 23:51:28

Well, I used the rsync method, and it seems my IP has been banned from rsync.archlinux.org

I figured I'd save bandwidth and rsync, then use my local file server. I install arch very regularly on different boxen, especially as I experiment with all sorts of stuff. Having a local mirror is tremendously important to me, and probably cuts the bandwidth I'd use on ftp.archlinux.org quite a bit. Netselect repeatedly comes up with that mirror second only to my local server. Having the rsync mirror at my place probably reduces my usage on the central servers by 2000%

Is there some way to continue to rsync, or otherwise sync to the master without getting banned?

iphitus · 2004-12-01 06:39:00

Just my two cents, but in a business situation, the workers wouldnt need the most up to date version. a basic suse install and the required software then just leave it running. update if they are needed. The workers probably wouldnt care whether they have the latest KDE or whatever you use, nor do they need the latest kernel.

What I guess I am saying is if it aint broke, dont fix it.

iphitus

kleptophobiac · 2004-12-01 10:13:15

I'm running my house, where I have about 11 arch machines, plus a few experimental computers that I reformat several times each week. Keeping the internal server is massively convenient and reduces bandwidth.

Business? ha! Arch is my newest hobby.

Winkie · 2004-12-02 15:24:00

I've been rsyncing for a while now and don't seem to be IP banned, I rsync every night (GMT) at midnight for current and 2am for extra.

Xentac · 2004-12-02 21:13:06

Someone's rsync hasn't been playing nicely and wouldn't let anyone else rsync. Since your IP was blocked, it was probably you

Talk to Judd to figure this one out. My guess is that your rsync isn't set up properly.

apeiro · 2004-12-02 21:29:35

kleptophobiac wrote:

Well, I used the rsync method, and it seems my IP has been banned from rsync.archlinux.org

That's correct. Your cronjob was rsyncing every single minute, so basically you were eating up all the rsync slots so the larger, public mirrors could not update. Email me when you've fixed your cronjob and I'll un-ban you.

TO ALL RSYNC'ERS:

Please don't mirror more than once or twice a day, and please don't mirror if you only have one or two machines running Arch. Extra by itself is ~2gb, so enough of you pulling the entire repository can eat up our bandwidth Real Fast.

If you have a lot of machines, then it's in all our best interests for you to run an internal mirror, as you'll probably save us bandwidth in the long run. But if you only install ~300mb worth of packages on two machines, then rsync'ing 2gb is hardly worth it.

Thanks.

kleptophobiac · 2004-12-03 10:18:33

I could've sworn I wrote it for once daily.... I have no idea why it would run more often than that. :?:

here's what's in the crontab:

* * */1 * * /scripts/arch-sync > /dev/null

and here's the script that it runs:

/usr/bin/rsync -av --delete rsync.archlinux.org::current /home/ftp/archlinux/current
/usr/bin/rsync -av --delete rsync.archlinux.org::extra /home/ftp/archlinux/extra

Don't unblock me yet, because that's what's been running - which seems to be causing problems.

JGC · 2004-12-03 11:46:35

You're still running it every minute. Change the first two *'s to numbers and it will run at that minute and that hour at every day.

Winkie · 2004-12-03 17:03:56

Here are the appropriate parts of my crontab

30 0 * * * rsync -rlptq --delete rsync://rsync.archlinux.org/current /home/arch/current
30 2 * * * rsync -rlptq --delete rsync://rsync.archlinux.org/extra /home/arch/extra

kleptophobiac · 2004-12-03 21:43:34

How's that? 3:01 AM EST...

01 03 * * * /scripts/arch-sync > /dev/null

kleptophobiac · 2004-12-04 14:46:21

ehh, I said screw it and just put it in /etc/cron.daily

kleptophobiac · 2004-12-07 23:05:26

OK, mine is definitely working properly now. Care to unban me?

hugelmopf · 2004-12-17 23:37:38

Would it not be a much easier solution for most people, to (smb,nfs-)mount the local repository on /var/cache/pacman/pkg on each machine, or is there any disadvantage to that? This way you would not have any overhead like rsyncing a lot of packages, that are not going to be needed.

kleptophobiac · 2004-12-18 13:48:42

I have a local ftp server in my current and extra files, and the local ftp server rsyncs. It is the most efficient solution as it does not require any permanent mounts and doesn't waste excess bandwidth.

lanrat · 2004-12-18 15:23:00

It has just came to my mind:

I've made shfs pkgbuild yesterday which (I think) could also be used to mount /var/cache/pacman/pkg remotely (and securely) on client machines. It's very easy to configure it and you don't have to keep mounts all the time (maybe only when you're doing an upgrade?). That's just an idea.

BTW: It's my 666th post on the forums so be careful with this advice :twisted:

jw · 2005-01-08 16:56:33

hi all,

i am new to these forums, so i'd like to start by saying thanks for this great distro! i've been using arch for about two months now and i find it by far the best distro i've tried. arch is always up to date, maintenance is mostly hassle-free and pacman is a breeze to use. i _really_ like the ABS build system. oh, and it also runs quite decently on my 500Mhz 192Mb laptop. wow!

*ahem*, i suppose i will get on topic now:

as i do not have internet at home, i like to keep a local repository. this way i don't have to drag my laptop to a connection each time i want to install a package. the easiest way to do this is by sync'ing from rsync.archlinux.org every two months or so.

i fully understand that you guys are not to eager to have home users rsync from your server. so my question is: are there mirrors that can be used for rsync-ing?

i already tried syncing from a ftp.gwdg.de/pub/linux/metalab/distributions/archlinux. this works ok, except for the fact that it leaves me with a mirror containing several versions of each package. i guess i could sync from a mirror and then delete superfluous packages by doing

rsync -av ftp.gwdg.de::pub/linux/metalab/distributions/archlinux/current ./current
rsync -av --delete rsync.archlinux.org::current ./current

but this method wastes gigabytes of bandwith. anyone have a better idea?

cactus · 2005-01-08 19:59:06

rsync is not great for large binary files. In fact, I think I recall somewhere reading that the author(s) suggested not using it for such a purpose! lol

It might be more advantageous to use wget in this situation.
wget --mirror --cut-dirs=4 http://ftp.gwdg.de/pub/linux/metalab/di … /archlinux
maybe.
you can even add a
--limit-rate=20K
or something, to be kind on the site.

note: I might not have used the proper number for cut-dirs, experimentation with a specific site will yield better results.

jw · 2005-01-11 11:20:59

thanks for the reply. however, i think this does not solve my original problem. perhaps i was not clear enough in my last post. my problem is the following: the mirrors contain several versions of most packages (for example: the are 11 different versions of abiword!). i do not want to download all of these, just the most recent. how can i do this?

cactus · 2005-01-11 20:03:44

hmm...
you could do a packman download..I think it is pacman -Sw

so, you could pacman -Sw
and it should get (download only) only the current packages.

Arch Linux

#1 2004-10-22 14:12:58

Internal Arch repository?

#2 2004-10-22 14:21:54

Re: Internal Arch repository?

#3 2004-10-22 23:13:17

Re: Internal Arch repository?

#4 2004-10-23 14:19:52

Re: Internal Arch repository?

#5 2004-10-23 17:30:26

Re: Internal Arch repository?

#6 2004-10-23 18:15:17

Re: Internal Arch repository?

#7 2004-11-30 23:51:28

Re: Internal Arch repository?

#8 2004-12-01 06:39:00

Re: Internal Arch repository?

#9 2004-12-01 10:13:15

Re: Internal Arch repository?

#10 2004-12-02 15:24:00

Re: Internal Arch repository?

#11 2004-12-02 21:13:06

Re: Internal Arch repository?

#12 2004-12-02 21:29:35

Re: Internal Arch repository?

#13 2004-12-03 10:18:33

Re: Internal Arch repository?

#14 2004-12-03 11:46:35

Re: Internal Arch repository?

#15 2004-12-03 17:03:56

Re: Internal Arch repository?

#16 2004-12-03 21:43:34

Re: Internal Arch repository?

#17 2004-12-04 14:46:21

Re: Internal Arch repository?

#18 2004-12-07 23:05:26

Re: Internal Arch repository?

#19 2004-12-17 23:37:38

Re: Internal Arch repository?

#20 2004-12-18 13:48:42

Re: Internal Arch repository?

#21 2004-12-18 15:23:00

Re: Internal Arch repository?

#22 2005-01-08 16:56:33

Re: Internal Arch repository?

#23 2005-01-08 19:59:06

Re: Internal Arch repository?

#24 2005-01-11 11:20:59

Re: Internal Arch repository?

#25 2005-01-11 20:03:44

Re: Internal Arch repository?

Board footer