Mirror Package Stats - Information about packages as seen by a mirror

tyzoid · 2017-08-18 20:30:36

So I run a mirror over at https://arlm.tyzoid.com/, and I decided it would be cool to see if I can't get some interesting information back from the logs generated.

Currently, I have a list of the most downloaded packages (compiled nightly):

And a graph of network traffic:

https://stats.arlm.tyzoid.com/

I'm wondering if there's any other information that might be interesting to make available.

Some caveats:

IP addresses will be available, nor any IP Prefix.
Geographic data will be made available at no more than a weekly granularity (exception for summary data which spans multiple weeks)
Geographic data will not be made available with more granularity than state/province.

Please leave a comment for any requests you may have.

dmerej · 2017-08-21 17:54:21

  58553 cryptsetup
   4850 linux

Why is there so many cryptsetup downloads ?

graysky · 2017-08-21 18:15:20

Likely due to systemd's somewhat recent requirement of it, no?

tyzoid · 2017-08-21 19:29:42

My original suspicion is that it's because of some monitoring software package. I'm seeing the user agent of a lot of these requests as Python-urllib/3.6, which is different than the user agent for pacman:

$ grep cryptsetup access.log |tail -n 5 | cut -f4- -d' '
[21/Aug/2017:15:23:06 -0400] "GET /core/os/x86_64/cryptsetup-1.7.5-1-x86_64.pkg.tar.xz HTTP/1.1" 200 246880 "-" "Python-urllib/3.6"
[21/Aug/2017:15:24:38 -0400] "GET /core/os/x86_64/cryptsetup-1.7.5-1-x86_64.pkg.tar.xz HTTP/1.1" 200 246880 "-" "Python-urllib/3.6"
[21/Aug/2017:15:26:41 -0400] "GET /core/os/x86_64/cryptsetup-1.7.5-1-x86_64.pkg.tar.xz HTTP/1.1" 200 246880 "-" "Python-urllib/3.6"
[21/Aug/2017:15:27:06 -0400] "GET /core/os/x86_64/cryptsetup-1.7.5-1-x86_64.pkg.tar.xz HTTP/1.1" 200 246880 "-" "Python-urllib/3.6"
[21/Aug/2017:15:27:35 -0400] "GET /core/os/x86_64/cryptsetup-1.7.5-1-x86_64.pkg.tar.xz HTTP/1.1" 200 246880 "-" "Python-urllib/3.6"

Not shown is the IP of each request, but they are all unique.

vs

$ grep archlinux-keyring access.log |tail -n 5| cut -f4- -d' '
[21/Aug/2017:02:15:43 -0400] "GET /testing/os/x86_64/archlinux-keyring-20170611-1-any.pkg.tar.xz HTTP/1.1" 404 555 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
[21/Aug/2017:03:32:48 -0400] "GET /core/os/x86_64/archlinux-keyring-20170611-1-any.pkg.tar.xz HTTP/1.1" 200 678980 "-" "pacman/5.0.2 (Linux x86_64) libalpm/10.0.2"
[21/Aug/2017:06:44:55 -0400] "GET /core/os/x86_64/archlinux-keyring-20170611-1-any.pkg.tar.xz.sig HTTP/1.1" 200 4412 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
[21/Aug/2017:10:51:59 -0400] "GET /core/os/x86_64/archlinux-keyring-20170611-1-any.pkg.tar.xz HTTP/1.1" 200 677669 "-" "pacman/5.0.2 (Linux x86_64) libalpm/10.0.2"
[21/Aug/2017:15:02:03 -0400] "GET /core/os/x86_64/archlinux-keyring-20170611-1-any.pkg.tar.xz HTTP/1.1" 200 677725 "-" "python-requests/2.18.1"

That said, this is still something I'm very much interested in finding out what the root cause is.

Last edited by tyzoid (2017-08-21 19:31:50)

Arch Linux

#1 2017-08-18 20:30:36

Mirror Package Stats - Information about packages as seen by a mirror

#2 2017-08-21 17:54:21

Re: Mirror Package Stats - Information about packages as seen by a mirror

#3 2017-08-21 18:15:20

Re: Mirror Package Stats - Information about packages as seen by a mirror

#4 2017-08-21 19:29:42

Re: Mirror Package Stats - Information about packages as seen by a mirror

Board footer