lastsync file format

rockin turtle · 2010-08-04 04:22:26

I don't know if this is a bug, or if there is a better place to post this.

I have a bash script that I use to run pacman. This script uses wget to obtain the "lastsync" file from mirrors listed under selected countries in /etc/pacman.d/mirrorlist. It then selects one of those mirrors as the "best" and runs pacman -Sy.

This morning I noticed that the returned lastsync files are inconsistent.

On some mirrors, the file appears to be generated by:

$date +%s >lastsync

On other mirrors, it appears to be generated by:

$date --rfc-3339=seconds >lastsync

I realize that the people who maintain the pacman mirrors probably don't intend for people like me to use the lastsync file as they may periodically want to change their processes (which would then break a script like mine). That's fine; I have already fixed my script and I don't care if they change things again. I will adapt.

But I do believe that they may desire that this file be consistent on all mirrors, and are perhaps unaware that this inconsistency is out there. So if anyone reading this has any involvement with the pacman mirrors, you may want to re-evaluate the mechanism by which these files are generated.

Of course, they may be exactly as intended. In that case, never mind.

fukawi2 · 2010-08-04 05:51:26

Every mirror is maintained by their own hostmaster, and I'm guessing only a *very* small number (if any) of them read the forums.

brain0 · 2010-08-04 11:22:39

The mirror admins don't generate that file, the master server does. Note that this file moved from the individual repository directories to the root of the server. It is generated by this cronjob:

# lastsync timestamp, every 10 minutes
*/10 * * * * date +%s > /srv/ftp/lastsync

Pierre · 2010-08-04 11:30:41

The problem is that at least one tier 1 mirror overwrites our lastsync with his own. (I have contacted the admin and also send a mail to the mirror list) See http://mailman.archlinux.org/pipermail/ … 00051.html

Btw: You should have a look at reflector. There is no need to probe every mirror on your own...these data is already collected by the MirrorStatus scripts.

Edit: If you want to keep using your own script you may have a look at the reflector api though: https://www.archlinux.de/?page=MirrorStatusReflector

fukawi2 · 2010-08-04 11:55:41

brain0 wrote:

The mirror admins don't generate that file, the master server does. Note that this file moved from the individual repository directories to the root of the server. It is generated by this cronjob:

As Pierre mentioned, obviously some mirrors are generating their own, otherwise there wouldn't be a difference in the format between mirrors

rockin turtle · 2010-08-04 19:40:07

Thanks for the responses. I will continue to use my script; it is small and I generally only run it once a day. By querying each mirror myself, I gain some insight not only to the response time status of each mirror, but also to the response time health of each router/gateway etc. between me and each mirror. That is, it gives me an end-to-end figure of merit for each mirror that is applicable specifically to me. Reflector doesn't give me this information.

I know that this information changes each second on the internet and is thus not that useful; but I would use more overall network bandwidth by just looking at https://www.archlinux.de/?page=MirrorStatus or https://www.archlinux.de/?page=MirrorStatusReflector than by running my script. In my script, I only query North American mirrors, and each mirror responds with only a small (~50 bytes) static file.

Anyway, this inconsistency seemed wrong to me and I hope the information was useful to someone.

fphillips · 2010-08-04 20:00:22

@Pierre

Is the delay column that useful when so many are 1 year or more? It can be misunderstood as servers being outdated.
https://www.archlinux.de/?page=MirrorSt … ;sort=desc

Pierre · 2010-08-05 09:50:40

No its not useful but should fix itself soon. I need to figure out which mirrors overwrite our file and contact them.

SanskritFritz · 2010-08-13 10:48:34

Dear rockin turtle, can you make that bash script of yours available for us?

rockin turtle · 2010-08-14 05:28:19

@SanskritFritz, you can find my script here.

It is currently set up to query only North American mirrors, but that is easy to change. Just change line 39 to list the countries you want. For example, you could use:

countries=("Hungary" "Austria" "Switzerland" "Germany" "Czech Republic")

You can have any number of countries, but the country names have to match the country names listed in the mirrorlist file. Bash syntax requires that there be no spaces in the 'countries=(' at the beginning of that line, and that there be no commas between the actual country names - just spaces. If there is a mirror that you do NOT want to use but it is in a country that you have selected, you can cause this script to ignore that mirror by starting the line in the mirrorlist file containing the mirror with a ##.

Each time you run this script, it will generate (or overwrite) a file named /etc/pacman.d/mirrors. This file will contain the last sync information for each mirror queried. For example, the following mirror would have been updated 0 days, 15 hours, and 23 minutes prior to running this script:

#  0d 15h 23m  http://lug.mtu.edu/

The last line of the file will be your choosen mirror, such as:

Server = http://mirrors.kernel.org/archlinux/$repo/os/$arch

This script will choose a (potentially different) "best" mirror each time it is run.

You will need to change your /etc/pacman.conf file to use this file for all this to work. My pacman.conf file looks like this:

[core]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrors

[extra]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrors

[community]
# Add your preferred servers here, they will be used first
Include = /etc/pacman.d/mirrors

Name this script something like 'pac-sync', do a "chmod +x pac-sync", and put it somewhere in your path (I put mine in ~/bin). I generally run it once a day. It does a 'pacman -Syy' followed by a 'pacman -Qu' to show you what needs to be updated. You will then have to run a 'pacman -Su' to actually do the update.

This script should be owned (and run as) root since it does a 'pacman -Sy' and it writes to /etc/pacman.d.

Let me know if this works for you or if you have any problems.

Arch Linux

#1 2010-08-04 04:22:26

lastsync file format

#2 2010-08-04 05:51:26

Re: lastsync file format

#3 2010-08-04 11:22:39

Re: lastsync file format

#4 2010-08-04 11:30:41

Re: lastsync file format

#5 2010-08-04 11:55:41

Re: lastsync file format

#6 2010-08-04 19:40:07

Re: lastsync file format

#7 2010-08-04 20:00:22

Re: lastsync file format

#8 2010-08-05 09:50:40

Re: lastsync file format

#9 2010-08-13 10:48:34

Re: lastsync file format

#10 2010-08-14 05:28:19

Re: lastsync file format

Board footer