You are not logged in.

#1 2020-11-23 19:29:07

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Proposal to improve the mirror list to get the faster server.

I see a big opportunity to improve the local assignation for the servers in `pacman`. I posted here because I want to receive feedback of my solution and get guidance of how I can propose the implementation in Arch Linux, `rankmirrors` or `reflector`.

Context:
I am using `reflector` to upgrade the mirror list also I tried `rankmirrors`. Basically, I put the filter to returns the first 8 servers with the highest score and set the country to the United States at the end I sort by the highest download rate.

Problem:
I live in Guadalajara City, Mexico and 95% of the times it set the “faster” server but it's not true. I synchronize the package databases (`pacman -Syy`) and it's slow.

Proposal:
Take the real synchronization time of your country, ISP, etcetera.
Steps:

  1. Download the mirror list in the temporal directory.

  2. Take the first server and add as unique server in the `pacman` mirror list.

  3. Test the synchronization time.

  4. Repeat the step 2: take the next server until get these.

  5. Compare and assign the faster server to the `pacman` mirror list.

Conclusion:
The difference is huge, I tested for a couple of weeks and it makes the difference when try to update the Linux kernel or Nvidia packages. In fact, in the evidence shows the best result in reflector is the worst result in the real life.

  • Server: mirror.dc02.hackingand.coffee vs mirror.mia11.us.leaseweb.net

  • Reflector: 1519.58 KiB/s vs 1082.99 KiB/s

  • `pacman -Syy`: 0m3.494s vs 0m1.206s

  • Synchronization speed for community: 2.64 MiB/s vs 41.6 MiB/s

My solution:
I created the Reflector Script and it's working.

#!/bin/bash

FILE="/etc/pacman.d/mirrorlist"
FILE_TEMP="/tmp/.mirrorlist-tmp"

# --------------------------------------
# Reflector: Request the servers and
#   stores to the temporal folder.
# --------------------------------------
reflector --verbose --threads 8 --country "United States" --protocol https --completion-percent 100 --age 72 --score 8 --sort rate --save "${FILE_TEMP}"
echo ""
echo ""

# --------------------------------------
# Mirrorlist: Store all the servers and
#   download rate into the array.
# --------------------------------------
INDEX=0
SERVERS=()
TIMER=()

while IFS= read -r LINE
do
  if [[ "${LINE}" == Server* ]]
  then
    echo "${LINE}"
    echo "${LINE}" > "${FILE}"
    SERVERS[${INDEX}]=$(echo "${LINE}")
    TIME=$({ time pacman -Syy --noconfirm ; } 2>&1 | grep real | sed "s/real\s*//")
    TIMER[${INDEX}]=${TIME}
    echo "INDEX: ${INDEX}"
    echo "SERVERS[${INDEX}]: ${SERVERS[${INDEX}]}"
    echo "TIMER[${INDEX}]: ${TIMER[${INDEX}]}"
    echo ""
    INDEX=$((INDEX+1))
  fi
done < "${FILE_TEMP}"

rm -f "${FILE_TEMP}"

echo ""
echo "No. of SERVERS: ${#SERVERS[@]}"
echo "TIMER:          ${TIMER[@]}"
echo ""
echo ""

# --------------------------------------
# Array: Get the smallest time.
# --------------------------------------

if [[ "${INDEX}" -gt 0 ]]
then
  # --------------------------------------
  # Real Time: Extract the real time
  #   separate by minutes.
  # --------------------------------------
  INDEX=0
  MINUTES=()

  echo "# Get minutes:"
  echo ""
  for TIME in "${TIMER[@]}"
  do
    echo "$TIME"
    MINUTES[${INDEX}]=$(echo "$TIME" | sed -E "s/([0-9]*)m.*/\1/")
    INDEX=$(( INDEX + 1 ))
  done
  echo ""
  echo "MINUTES:   ${MINUTES[@]}"

  # --------------------------------------
  # Real Time: Get the smallest minute and
  #   create a new array for servers.
  # --------------------------------------
  SMALLEST=9999

  for MINUTE in "${MINUTES[@]}"
  do
    IS_SMALLER=$(echo "${MINUTE} < ${SMALLEST}" | bc)
    if [ ${IS_SMALLER} -eq 1 ]
      then
        SMALLEST="${MINUTE}"
      fi
  done
  echo "SMALLEST:  ${SMALLEST}"

  INDEX=0
  NEW_SERVERS=()
  NEW_TIMER=()

  for MINUTE in "${MINUTES[@]}"
  do
    if [[ "${MINUTE}" == "${SMALLEST}" ]]
      then
        SMALLEST="${MINUTE}"
        NEW_SERVERS[${INDEX}]="${SERVERS[${INDEX}]}"
        NEW_TIMER[${INDEX}]="${TIMER[${INDEX}]}"
      fi
      INDEX=$(( INDEX + 1 ))
  done
  SERVERS=("${NEW_SERVERS[@]}")
  TIMER=("${NEW_TIMER[@]}")
  echo "TIMER:     ${TIMER[@]}"
  echo""
  echo""

  # --------------------------------------
  # Real Time: Extract the real time
  #   separate by seconds.
  # --------------------------------------
  INDEX=0
  SECONDS=()

  echo "# Get seconds:"
  echo ""
  for TIME in "${TIMER[@]}"
  do
    echo "$TIME"
    SECONDS[${INDEX}]=$(echo "$TIME" | sed -E "s/.*m([0-9]*)\..*/\1/")
    INDEX=$(( INDEX + 1 ))
  done
  echo ""
  echo "SECONDS:   ${SECONDS[@]}"

  # --------------------------------------
  # Real Time: Get the smallest second and
  #   create a new array for servers.
  # --------------------------------------
  SMALLEST=9999

  for SECOND in "${SECONDS[@]}"
  do
    IS_SMALLER=$(echo "${SECOND} < ${SMALLEST}" | bc)
    if [ ${IS_SMALLER} -eq 1 ]
      then
        SMALLEST="${SECOND}"
      fi
  done
  echo "SMALLEST:  ${SMALLEST}"

  INDEX=0
  NEW_SERVERS=()
  NEW_TIMER=()

  for SECOND in "${SECONDS[@]}"
  do
    if [[ "${SECOND}" == "${SMALLEST}" ]]
      then
        SMALLEST="${SECOND}"
        NEW_SERVERS[${INDEX}]="${SERVERS[${INDEX}]}"
        NEW_TIMER[${INDEX}]="${TIMER[${INDEX}]}"
      fi
      INDEX=$(( INDEX + 1 ))
  done
  SERVERS=("${NEW_SERVERS[@]}")
  TIMER=("${NEW_TIMER[@]}")
  echo "TIMER:     ${TIMER[@]}"
  echo""
  echo""

  # --------------------------------------
  # Real Time: Extract the real time
  #   separate by milliseconds.
  # --------------------------------------
  INDEX=0
  MILLISECONDS=()

  echo "# Get milliseconds:"
  echo ""
  for TIME in "${TIMER[@]}"
  do
    echo "$TIME"
    MILLISECONDS[${INDEX}]=$(echo "$TIME" | sed -E "s/.*\.([0-9]*)s/\1/")
    INDEX=$(( INDEX + 1 ))
  done
  echo ""
  echo "MILLISECONDS: ${MILLISECONDS[@]}"

  # --------------------------------------
  # Real Time: Get the smallest milliseconds
  #   and create a new array for servers.
  # --------------------------------------
  SMALLEST=9999

  for MILLISECOND in "${MILLISECONDS[@]}"
  do
    IS_SMALLER=$(echo "${MILLISECOND} < ${SMALLEST}" | bc)
    if [ ${IS_SMALLER} -eq 1 ]
      then
        SMALLEST="${MILLISECOND}"
      fi
  done
  echo "SMALLEST:  ${SMALLEST}"

  INDEX=0
  NEW_SERVERS=()
  NEW_TIMER=()

  for MILLISECOND in "${MILLISECONDS[@]}"
  do
    if [[ "${MILLISECOND}" == "${SMALLEST}" ]]
      then
        SMALLEST="${MILLISECOND}"
        NEW_SERVERS[${INDEX}]="${SERVERS[${INDEX}]}"
        NEW_TIMER[${INDEX}]="${TIMER[${INDEX}]}"
      fi
      INDEX=$(( INDEX + 1 ))
  done
  SERVERS=("${NEW_SERVERS[@]}")
  TIMER=("${NEW_TIMER[@]}")
  echo "TIMER:     ${TIMER[@]}"
  echo""
  echo""

  # --------------------------------------
  # Save the best servers in the mirror file.
  # --------------------------------------
  echo "" > "${FILE}"
  for SERVER in "${SERVERS[@]}"
  do
    echo "${SERVER}" >> "${FILE}"
  done
  echo "SERVERS:   ${SERVERS[@]}"
  echo""

  # --------------------------------------
  # Upgrade the packages.
  # --------------------------------------
  echo ""
  time pacman -Syy --noconfirm

  echo ""
  pacman -Su --noconfirm

  echo ""
  pacman -Rns $(pacman -Qtdq) --noconfirm

  echo ""
  sudo -u wolf yay -Sau --noconfirm
fi

Evidence:

Execute `reflector`.

reflector --verbose --threads 8 --country "United States" --protocol https --completion-percent 100 --age 72 --score 8 --sort rate --save /tmp/.mirrorlist-tmp`

Result of `reflector`.

Server                                            Rate             Time
https://mirror.dc02.hackingand.coffee/arch/      1519.58 KiB/s     0.09 s
https://mirror.mia11.us.leaseweb.net/archlinux/  1082.99 KiB/s     0.12 s
https://mirror.wdc1.us.leaseweb.net/archlinux/   1048.52 KiB/s     0.12 s
https://arch.mirror.square-r00t.net/              900.99 KiB/s     0.14 s
https://arch.hu.fo/archlinux/                     862.52 KiB/s     0.15 s
https://arlm.tyzoid.com/                          807.76 KiB/s     0.16 s
https://arch.mirror.constant.com/                 664.49 KiB/s     0.19 s
https://mirrors.lug.mtu.edu/archlinux/            576.54 KiB/s     0.22 s

Testing the synchronization in real time with command `time pacman -Syy`.

Real 0m1.206s: https://mirror.mia11.us.leaseweb.net/archlinux/
Real 0m1.242s: https://mirror.wdc1.us.leaseweb.net/archlinux/
Real 0m1.313s: https://arch.mirror.constant.com/
Real 0m2.331s: https://mirrors.lug.mtu.edu/archlinux/
Real 0m1.505s: https://arch.mirror.square-r00t.net/
Real 0m1.878s: https://arch.hu.fo/archlinux/
Real 0m1.324s: https://arlm.tyzoid.com/
Real 0m3.494s: https://mirror.dc02.hackingand.coffee/arch/

Testing 5 times.

Server: https://mirror.dc02.hackingand.coffee/arch/

wolf@MSIGT73EVR7RF | 12:45:58 (Mon Nov 23)
~/workspace/projects/archLinux-installer-and-setup/src/laptop_MSI_GT73EVR_7R_Titan_Pro/04-setup/setup-resources
(master=)

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1487 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2.01 MiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB  2.72 MiB/s 00:02 [####################################################################] 100%
 multilib                                                                             152.8 KiB  4.97 MiB/s 00:00 [####################################################################] 100%
real    0m3.244s
user    0m0.088s
sys     0m0.091s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1176 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2.01 MiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB  2.61 MiB/s 00:02 [####################################################################] 100%
 multilib                                                                             152.8 KiB  4.97 MiB/s 00:00 [####################################################################] 100%
real    0m3.340s
user    0m0.100s
sys     0m0.091s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1176 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2047 KiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB  2.64 MiB/s 00:02 [####################################################################] 100%
 multilib                                                                             152.8 KiB  4.39 MiB/s 00:00 [####################################################################] 100%
real    0m3.287s
user    0m0.047s
sys     0m0.087s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1176 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2.01 MiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB  2.61 MiB/s 00:02 [####################################################################] 100%
 multilib                                                                             152.8 KiB  5.53 MiB/s 00:00 [####################################################################] 100%
real    0m3.325s
user    0m0.085s
sys     0m0.071s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1135 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2.08 MiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB  2.76 MiB/s 00:02 [####################################################################] 100%
 multilib                                                                             152.8 KiB  4.39 MiB/s 00:00 [####################################################################] 100%
real    0m3.165s
user    0m0.052s
sys     0m0.095s

Server: https://mirror.mia11.us.leaseweb.net/archlinux/

wolf@MSIGT73EVR7RF | 12:39:35 (Mon Nov 23)
~/workspace/projects/archLinux-installer-and-setup/src/laptop_MSI_GT73EVR_7R_Titan_Pro/04-setup/setup-resources
(master=)

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB  1116 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  8.69 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  41.6 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  49.7 MiB/s 00:00 [####################################################################] 100%
real    0m1.210s
user    0m0.064s
sys     0m0.042s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   731 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  8.89 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  44.8 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.193s
user    0m0.060s
sys     0m0.036s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   748 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  9.09 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  43.7 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.168s
user    0m0.034s
sys     0m0.047s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   735 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  8.89 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  43.7 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.198s
user    0m0.034s
sys     0m0.047s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   735 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  8.89 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  42.6 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.192s
user    0m0.055s
sys     0m0.042s

Last edited by airv_zxf (2020-11-24 17:39:04)

Offline

#2 2020-11-24 13:18:35

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: Proposal to improve the mirror list to get the faster server.

You do realise the pacman sync dbs are rather small and total about 7 MiB ?
Even the files databases are just above 30 Mib .

The size of updates is much larger (especially when things like kernel or libreoffice are updated) and I find the sustained speed uptodate mirror servers can deliver much more important.

The download time for a large file like the archlinux iso is a much better metric.

If you want to automate things and improve updating speed overall, try power pill


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#3 2020-11-24 21:40:16

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Thanks for your comment Lone_Wolf.
A wolf is answering to another wolf.

The Power pill looks very useful, I always wonder why `pacman` is not doing the “Multiple File Downloader”. I have a theory which is to keep the server's bandwidth available keeping a constant medium bandwidth in long times versus a high bandwidth in short times.

My theory is true
Coming back to the “faster” download rate versus the real rate speed. I was thinking and I created some tests but I have bad news for you, my theory is true and it's better to get the real speed if you want faster downloads.

Conclusions:
I am confidence that the download rate is taking from some special servers to connect the mirror or some sort of similar theories. For this reason, the speed rate is not relevant as a user we need to do a quick test synchronizing the database and checking the speed.

Results for the test #1:
Synchronizing package databases 3 times and install the 0AD game (735.66 MiB).

  • Winner: “constant.com”.

  • Servers: “square-r00t.net” versus “constant.com”.

  • Reflector speed: “1026.07 KiB/s” versus “697.40 KiB/s”, difference of “328.67 KiB/s” for “square-r00t.net”.

  • 1st synchronizing package databases: “0m7.689s” versus “0m1.763s”, difference of ~6 seconds for “constant.com”.

  • 2nd synchronizing package databases: “0m9.715s” versus “0m1.739s”, difference of ~8 seconds for “constant.com”.

  • 3rd synchronizing package databases: “0m6.023s” versus “0m1.664s”, difference of ~4 seconds for “constant.com”.

  • 1st installation of the game: “2m42.233s” versus “0m45.559s”, difference of ~2 minutes for “constant.com”.

Results for the test #2:
Install the 0AD game (735.66 MiB) 5 times.

  • Winner: “constant.com”.

  • Servers: “square-r00t.net” versus “constant.com”.

  • Reflector speed: “1026.07 KiB/s” versus “697.40 KiB/s”, difference of “328.67 KiB/s” for “square-r00t.net”.

  • 1st install: “14m6.078s” versus “0m46.887s”, difference of ~13 minutes for “constant.com”.

  • 2nd install: “3m15.181s” versus “0m47.852s”, difference of ~2 minutes for “constant.com”.

  • 3rd install: “13m1.826s” versus “0m48.179s”, difference of ~12 minutes for “constant.com”.

  • 4th install: “15m9.004s” versus “0m45.795s”, difference of ~14 minutes for “constant.com”.

  • 5th install: “19m36.028s” versus “0m44.950s”, difference of ~18 minutes for “constant.com”.

  • Extra : The “square-r00t.net” failed 6 times means the installation wasn't able to complete. I retried until it complete the 5 tests.

Tests:
I removed the Linux game `0AD` then I cleaned the cache. The size of the data is 2,006.67 MiB (`0ad-data`). To discard some of our assumptions the server which download this game faster, should be the winner.

Reflector results:
https://arch.mirror.square-r00t.net/             1026.07 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     864.60 KiB/s     0.15 s
https://arch.mirror.constant.com/                 697.40 KiB/s     0.19 s
https://reflector.luehm.com/arch/                 396.90 KiB/s     0.33 s
https://archmirror1.octyl.net/                    368.09 KiB/s     0.35 s
https://mirrors.lug.mtu.edu/archlinux/            289.30 KiB/s     0.45 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    310.52 KiB/s     0.42 s
https://mirror.dc02.hackingand.coffee/arch/       218.50 KiB/s     0.59 s

Test #1.1: arch.mirror.square-r00t.net

# https://arch.mirror.square-r00t.net/

echo 'Server = https://arch.mirror.square-r00t.net/$repo/os/$arch' | sudo tee /etc/pacman.d/mirrorlist
sudo pacman -Rns 0ad --noconfirm   # Remove the package and the dependencies.
sudo pacman -Sc --noconfirm        # Clean the pacman cache.
time sudo pacman -Syy
time sudo pacman -Syy
time sudo pacman -Syy
time sudo pacman -S 0ad --noconfirm

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   657 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  2.65 MiB/s 00:01 [####################################################################] 100%
 community                                                                              5.2 MiB   920 KiB/s 00:06 [####################################################################] 100%
 multilib                                                                             152.8 KiB   776 KiB/s 00:00 [####################################################################] 100%
real    0m7.689s
user    0m0.066s
sys     0m0.105s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   667 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  1088 KiB/s 00:02 [####################################################################] 100%
 community                                                                              5.2 MiB   775 KiB/s 00:07 [####################################################################] 100%
 multilib                                                                             152.8 KiB   817 KiB/s 00:00 [####################################################################] 100%
real    0m9.715s
user    0m0.091s
sys     0m0.101s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   657 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1637.9 KiB  3.40 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  1239 KiB/s 00:04 [####################################################################] 100%
 multilib                                                                             152.8 KiB   792 KiB/s 00:00 [####################################################################] 100%
real    0m6.023s
user    0m0.101s
sys     0m0.087s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB  0.00   B/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  4.84 MiB/s 02:30 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  4.08 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  2.33 MiB/s 00:04 [####################################################################] 100%
real    2m42.233s
user    0m7.253s
sys     0m7.388s

Test #1.2: arch.mirror.constant.com

# https://arch.mirror.constant.com/

echo 'Server = https://arch.mirror.constant.com/$repo/os/$arch' | sudo tee /etc/pacman.d/mirrorlist
sudo pacman -Rns 0ad --noconfirm   # Remove the package and the dependencies.
sudo pacman -Sc --noconfirm        # Clean the pacman cache.
time sudo pacman -Syy
time sudo pacman -Syy
time sudo pacman -Syy
time sudo pacman -S 0ad --noconfirm

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   660 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1636.8 KiB  3.94 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  17.5 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.763s
user    0m0.054s
sys     0m0.053s

:: Synchronizing package databases...
 core                                                                                 129.4 KiB   657 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1636.8 KiB  4.57 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  18.3 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.739s
user    0m0.047s
sys     0m0.064s

$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                                                                                 129.4 KiB   731 KiB/s 00:00 [####################################################################] 100%
 extra                                                                               1636.8 KiB  5.46 MiB/s 00:00 [####################################################################] 100%
 community                                                                              5.2 MiB  20.4 MiB/s 00:00 [####################################################################] 100%
 multilib                                                                             152.8 KiB  0.00   B/s 00:00 [####################################################################] 100%
real    0m1.664s
user    0m0.079s
sys     0m0.065s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   510 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  20.4 MiB/s 00:36 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  3.61 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  7.72 MiB/s 00:01 [####################################################################] 100%
real    0m45.559s
user    0m5.594s
sys     0m4.533s

Test #2.1: arch.mirror.square-r00t.net

# https://arch.mirror.square-r00t.net/

echo 'Server = https://arch.mirror.square-r00t.net/$repo/os/$arch' | sudo tee /etc/pacman.d/mirrorlist
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   494 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB   906 KiB/s 13:41 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB   632 KiB/s 00:01 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB   586 KiB/s 00:16 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    14m6.078s
user    0m10.364s
sys     0m11.203s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   393 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  3.91 MiB/s 03:06 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  8.85 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  8.81 MiB/s 00:01 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    3m15.181s
user    0m7.903s
sys     0m8.603s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   130 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB   978 KiB/s 12:40 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  1442 KiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB   766 KiB/s 00:12 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    13m1.826s
user    0m9.908s
sys     0m10.946s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB  0.00   B/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB   839 KiB/s 14:46 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  1632 KiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB   672 KiB/s 00:14 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    15m9.004s
user    0m9.796s
sys     0m11.067s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   393 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB   643 KiB/s 19:17 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB   877 KiB/s 00:01 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB   934 KiB/s 00:10 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    19m36.028s
user    0m10.310s
sys     0m11.597s

Test #2.2: arch.mirror.constant.com

# https://arch.mirror.constant.com/

echo 'Server = https://arch.mirror.constant.com/$repo/os/$arch' | sudo tee /etc/pacman.d/mirrorlist
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm


$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   447 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  19.3 MiB/s 00:38 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  15.6 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  12.7 MiB/s 00:01 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m46.887s
user    0m5.640s
sys     0m4.469s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   103 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  18.9 MiB/s 00:38 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  11.3 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  15.6 MiB/s 00:01 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m47.852s
user    0m5.765s
sys     0m4.713s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   441 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  18.6 MiB/s 00:39 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  6.10 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  18.0 MiB/s 00:01 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m48.179s
user    0m5.655s
sys     0m4.587s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   447 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  19.6 MiB/s 00:37 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  40.8 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  22.9 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m45.795s
user    0m5.843s
sys     0m4.335s

$ time sudo pacman -S 0ad --noconfirm
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-11
Total Download Size:    735.66 MiB
Total Installed Size:  2039.28 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   424 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  20.2 MiB/s 00:36 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  8.43 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-11-x86_64                                                                    9.0 MiB  16.9 MiB/s 00:01 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m44.950s
user    0m5.674s
sys     0m4.546s

Offline

#4 2020-11-24 22:44:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Proposal to improve the mirror list to get the faster server.

You can look at precisely what reflector is doing starting on line 244 of Reflector.py.  It determines the speed/rate by downloading the core repo database.  This is susceptible to the Lone_Wolf's critique, but as your tests show, the reflector determined rate doesn't even seem to be related to the core db sync speed for pacman.

The real problem is not just that the "small" core.db is the only thing tested, but rather that Relfector defaults to running as many threads as there are CPU cores.  This is problematic for two reasons - the main problem is that this will often result in the bandwidth bottleneck being at the local client end rather than the server: so the metric obtained doesn't actually say much about the rate of the mirrors being tested.  This is drastically compounded by likely often giving a huge boost to the last mirror(s) tested due to them running with fewer "competitors".  For example, if you have 4 cores, and test 13 mirrors, the first ~12 will all run in parallel, and the 13th runs on it's own (yes, this is making a big assumption that they all take very similar time ... but the logic holds regardless).  So whichever mirror just coincidentally is tested last, will almost certainly get the best score.

The second problem is less of a bias-source, but still odd: what is being parallelized is IO bound not CPU bound, so setting the number of threads to the number of CPU cores is rather arbitrary.  There is no reason a 12 core machine should be running more IO bound processes at once than a 4 core machine if they both only have one network interface.  I suspect this would create an even greater disconnect between reflector vs pacman rates in machines with greater core counts (the low extreme being on a 1 core machine where reflector's results should perfectly align with pacman's core db sync).

You can pretty easily test the effects of the parallelization by simply changing DEFAULT_N_THREADS on line 65 to 1 rather than os.cpu_count() edit: oops, just use the --threads argument, that's easier.  I suspect the results you get from reflector will then be a much better representation of what you get from pacman - at least from the syncronization of the core database.

Last edited by Trilby (2020-11-24 23:01:23)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#5 2020-11-25 02:54:38

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Thanks for your deep answer Inspector Parrot.

Let me understand your point, basically the problem is that `reflector` download a file `core.base` and take this as a measure rather than synchronize the database online. The other problem is that locally `reflector` does the multi-threading to download these files in the different servers which cause a slow download rate. I have one network cards, means the result is not pure.

I did some tests trying to check your facts. The only change that I did was modified the `--threads` parameter from 8 to 1.

reflector --verbose --threads 1 --country "United States" --protocol https --completion-percent 100 --age 48 --score 8 --sort rate --save /tmp/mirrorlsit

Conclusion:

  1. I can't test the correct behavior if it is downloading a file rather than synchronize to take the download rate.

  2. It's correct your assumption about `reflector` download 8 sites simultaneously and then the rest of the 3 servers take a different bandwidth. I tested with `--thread 1` and I don't get any crazy download rate, only the `square-r00t.net` still giving big numbers.

  3. It's not working as the real user experience.  The `reflector` stats show that the first position is `square-r00t.net`with 7.4 then `hu.fo` with 7.2 and `constant.com` with 6.3. The optimized script keep as the first position all the time the `constant.com` with 8.0 then `hu.fo` with 6.8 and `square-r00t.net` with 6.1 both these with a huge difference comparing with the first place.

  4. Why is so different when `reflector` try to download files versus synchronize the database? Both process seems that it's downloading something at the end.

  5. Keeping the focus in the main goal. Do you have some comments? (I want to receive feedback of my solution and get guidance of how I can propose the implementation in Arch Linux, `rankmirrors` or `reflector`.)

Read the values in this context the minimum is 1.0 the maximum is 8.0. High average is better.

|--- RELFLECTOR -------------------------------|- Tries -|- Sum -|- Average -|
https://arch.mirror.square-r00t.net/                10       74       7.4
https://arch.hu.fo/archlinux/                       10       72       7.2
https://arch.mirror.constant.com/                   10       63       6.3
https://mirror.wdc1.us.leaseweb.net/archlinux/      10       38       3.8
https://reflector.luehm.com/arch/                   10       36       3.6
https://mirrors.lug.mtu.edu/archlinux/              10       33       3.3
https://mirror.dc02.hackingand.coffee/arch/         10       28       2.8
https://archmirror1.octyl.net/                      10       16       1.6

|--- OPTIMIZED SCRIPT--------------------------|- Tries -|- Sum -|- Average -|
https://arch.mirror.constant.com/                   10       80       8.0
https://arch.hu.fo/archlinux/                       10       68       6.8
https://arch.mirror.square-r00t.net/                10       61       6.1
https://reflector.luehm.com/arch/                   10       50       5.0
https://mirror.wdc1.us.leaseweb.net/archlinux/      10       35       3.5
https://archmirror1.octyl.net/                      10       30       3.0
https://mirror.dc02.hackingand.coffee/arch/         10       26       2.6
https://mirrors.lug.mtu.edu/archlinux/              10       10       1.0

Evidence:
The next table show the `reflector` download rate sorted by highest bandwidth then the real time with my scrip sorted by lowest time. I tried 10 times to get a better stats.

Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/              982.23 KiB/s     0.13 s
https://arch.mirror.constant.com/                 901.39 KiB/s     0.14 s
https://arch.hu.fo/archlinux/                     874.91 KiB/s     0.15 s
https://mirror.dc02.hackingand.coffee/arch/       308.38 KiB/s     0.42 s
https://mirrors.lug.mtu.edu/archlinux/            286.92 KiB/s     0.45 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    203.52 KiB/s     0.64 s
https://archmirror1.octyl.net/                    184.22 KiB/s     0.70 s
https://reflector.luehm.com/arch/                 181.94 KiB/s     0.71 s

Optimized Script
0m1.752s   https://arch.mirror.constant.com/
0m2.146s   https://arch.hu.fo/archlinux/
0m12.397s  https://arch.mirror.square-r00t.net/
0m16.308s  https://reflector.luehm.com/arch/
0m28.068s  https://archmirror1.octyl.net/
0m31.265s  https://mirror.dc02.hackingand.coffee/arch/
0m31.894s  https://mirror.wdc1.us.leaseweb.net/archlinux/
1m6.826s   https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.hu.fo/archlinux/                     860.88 KiB/s     0.15 s
https://arch.mirror.square-r00t.net/              690.62 KiB/s     0.19 s
https://arch.mirror.constant.com/                 664.57 KiB/s     0.19 s
https://mirrors.lug.mtu.edu/archlinux/            287.13 KiB/s     0.45 s
https://reflector.luehm.com/arch/                 250.93 KiB/s     0.52 s
https://mirror.dc02.hackingand.coffee/arch/       230.29 KiB/s     0.56 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    303.29 KiB/s     0.43 s
https://archmirror1.octyl.net/                    184.55 KiB/s     0.70 s

Optimized Script
0m1.660s   https://arch.mirror.constant.com/
0m2.194s   https://arch.hu.fo/archlinux/
0m3.533s   https://arch.mirror.square-r00t.net/
0m13.847s  https://reflector.luehm.com/arch/
0m22.926s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m31.153s  https://archmirror1.octyl.net/
0m32.929s  https://mirror.dc02.hackingand.coffee/arch/
1m9.921s   https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.hu.fo/archlinux/                     863.43 KiB/s     0.15 s
https://arch.mirror.constant.com/                 592.53 KiB/s     0.22 s
https://arch.mirror.square-r00t.net/              486.12 KiB/s     0.27 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    422.97 KiB/s     0.31 s
https://reflector.luehm.com/arch/                 316.42 KiB/s     0.41 s
https://mirrors.lug.mtu.edu/archlinux/            286.39 KiB/s     0.45 s
https://archmirror1.octyl.net/                    230.33 KiB/s     0.56 s
https://mirror.dc02.hackingand.coffee/arch/       228.43 KiB/s     0.57 s

Optimized Script
0m1.796s   https://arch.mirror.constant.com/
0m2.169s   https://arch.hu.fo/archlinux/
0m3.685s   https://arch.mirror.square-r00t.net/
0m17.550s  https://reflector.luehm.com/arch/
0m22.164s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m27.990s  https://archmirror1.octyl.net/
0m32.009s  https://mirror.dc02.hackingand.coffee/arch/
1m10.892s  https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/             1028.76 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     861.24 KiB/s     0.15 s
https://arch.mirror.constant.com/                 698.04 KiB/s     0.19 s
https://reflector.luehm.com/arch/                 313.78 KiB/s     0.41 s
https://mirrors.lug.mtu.edu/archlinux/            288.12 KiB/s     0.45 s
https://mirror.dc02.hackingand.coffee/arch/       229.37 KiB/s     0.56 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    211.98 KiB/s     0.61 s
https://archmirror1.octyl.net/                    180.25 KiB/s     0.72 s

Optimized Script
0m1.614s   https://arch.mirror.constant.com/
0m2.009s   https://arch.hu.fo/archlinux/
0m14.763s  https://arch.mirror.square-r00t.net/
0m20.519s  https://reflector.luehm.com/arch/
0m30.680s  https://mirror.dc02.hackingand.coffee/arch/
0m31.843s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m38.558s  https://archmirror1.octyl.net/
1m12.394s  https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/              971.73 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     859.29 KiB/s     0.15 s
https://arch.mirror.constant.com/                 592.66 KiB/s     0.22 s
https://mirror.dc02.hackingand.coffee/arch/       309.12 KiB/s     0.42 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    302.20 KiB/s     0.43 s
https://mirrors.lug.mtu.edu/archlinux/            287.93 KiB/s     0.45 s
https://archmirror1.octyl.net/                    209.95 KiB/s     0.62 s
https://reflector.luehm.com/arch/                 172.20 KiB/s     0.75 s

Optimized Script
0m1.658s   https://arch.mirror.constant.com/
0m2.158s   https://arch.hu.fo/archlinux/
0m3.110s   https://arch.mirror.square-r00t.net/
0m19.352s  https://reflector.luehm.com/arch/
0m27.091s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m28.739s  https://archmirror1.octyl.net/
0m33.500s  https://mirror.dc02.hackingand.coffee/arch/
1m9.787s   https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/             1027.52 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     858.97 KiB/s     0.15 s
https://arch.mirror.constant.com/                 587.99 KiB/s     0.22 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    360.89 KiB/s     0.36 s
https://mirrors.lug.mtu.edu/archlinux/            287.18 KiB/s     0.45 s
https://reflector.luehm.com/arch/                 219.52 KiB/s     0.59 s
https://mirror.dc02.hackingand.coffee/arch/       219.45 KiB/s     0.59 s
https://archmirror1.octyl.net/                    224.35 KiB/s     0.58 s

Optimized Script
0m1.787s   https://arch.mirror.constant.com/
0m2.189s   https://arch.hu.fo/archlinux/
0m3.773s   https://arch.mirror.square-r00t.net/
0m14.254s  https://reflector.luehm.com/arch/
0m15.760s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m31.372s  https://archmirror1.octyl.net/
0m32.541s  https://mirror.dc02.hackingand.coffee/arch/
1m8.185s   https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/             1024.78 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     859.59 KiB/s     0.15 s
https://arch.mirror.constant.com/                 658.96 KiB/s     0.20 s
https://reflector.luehm.com/arch/                 397.19 KiB/s     0.33 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    301.44 KiB/s     0.43 s
https://mirror.dc02.hackingand.coffee/arch/       291.51 KiB/s     0.44 s
https://mirrors.lug.mtu.edu/archlinux/            287.10 KiB/s     0.45 s
https://archmirror1.octyl.net/                    196.97 KiB/s     0.66 s

Optimized Script
0m1.737s   https://arch.mirror.constant.com/
0m2.166s   https://arch.hu.fo/archlinux/
0m13.215s  https://reflector.luehm.com/arch/
0m15.282s  https://arch.mirror.square-r00t.net/
0m27.392s  https://archmirror1.octyl.net/
0m29.714s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m33.655s  https://mirror.dc02.hackingand.coffee/arch/
1m12.342s  https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.hu.fo/archlinux/                     857.92 KiB/s     0.15 s
https://arch.mirror.constant.com/                 655.59 KiB/s     0.20 s
https://reflector.luehm.com/arch/                 393.71 KiB/s     0.33 s
https://arch.mirror.square-r00t.net/              392.57 KiB/s     0.33 s
https://mirrors.lug.mtu.edu/archlinux/            287.68 KiB/s     0.45 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    263.71 KiB/s     0.49 s
https://mirror.dc02.hackingand.coffee/arch/       231.62 KiB/s     0.56 s
https://archmirror1.octyl.net/                    204.83 KiB/s     0.63 s

Optimized Script
0m1.636s   https://arch.mirror.constant.com/
0m2.172s   https://arch.hu.fo/archlinux/
0m7.530s   https://arch.mirror.square-r00t.net/
0m16.852s  https://reflector.luehm.com/arch/
0m22.218s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m22.692s  https://archmirror1.octyl.net/
0m30.953s  https://mirror.dc02.hackingand.coffee/arch/
1m11.022s  https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/              891.51 KiB/s     0.15 s
https://arch.hu.fo/archlinux/                     860.83 KiB/s     0.15 s
https://arch.mirror.constant.com/                 748.37 KiB/s     0.17 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    717.07 KiB/s     0.18 s
https://reflector.luehm.com/arch/                 395.94 KiB/s     0.33 s
https://mirrors.lug.mtu.edu/archlinux/            287.32 KiB/s     0.45 s
https://mirror.dc02.hackingand.coffee/arch/       272.16 KiB/s     0.48 s
https://archmirror1.octyl.net/                    184.32 KiB/s     0.70 s

Optimized Script
0m1.499s   https://arch.mirror.constant.com/
0m1.547s   https://arch.mirror.square-r00t.net/
0m2.009s   https://arch.hu.fo/archlinux/
0m15.579s  https://mirror.dc02.hackingand.coffee/arch/
0m17.068s  https://reflector.luehm.com/arch/
0m17.784s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m20.136s  https://archmirror1.octyl.net/
0m44.095s  https://mirrors.lug.mtu.edu/archlinux/


Reflector                                                 Rate       Time
https://arch.mirror.square-r00t.net/              972.19 KiB/s     0.13 s
https://arch.hu.fo/archlinux/                     864.98 KiB/s     0.15 s
https://arch.mirror.constant.com/                 735.49 KiB/s     0.18 s
https://mirror.wdc1.us.leaseweb.net/archlinux/    526.06 KiB/s     0.25 s
https://archmirror1.octyl.net/                    470.73 KiB/s     0.27 s
https://reflector.luehm.com/arch/                 394.77 KiB/s     0.33 s
https://mirror.dc02.hackingand.coffee/arch/       334.87 KiB/s     0.39 s
https://mirrors.lug.mtu.edu/archlinux/            287.41 KiB/s     0.45 s

Optimized Script
0m1.604s   https://arch.mirror.constant.com/
0m1.699s   https://arch.mirror.square-r00t.net/
0m1.909s   https://arch.hu.fo/archlinux/
0m11.611s  https://reflector.luehm.com/arch/
0m15.793s  https://mirror.wdc1.us.leaseweb.net/archlinux/
0m15.793s  https://archmirror1.octyl.net/
0m18.743s  https://mirror.dc02.hackingand.coffee/arch/
1m3.652s   https://mirrors.lug.mtu.edu/archlinux/

Offline

#6 2020-11-25 03:29:10

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Proposal to improve the mirror list to get the faster server.

From your last output you seem to be getting reasonably good agreement now between reflector and your script.  Note in your attempted summary statistics you are taking the average of an ordinal variable which is misleading, those "averages" don't really mean much of anything and they can't be appropriately compared.  This is particularly odd as you have ratio values - the actual measurement of seconds to download or KiB/s downloaded which is what you should compare.

But in the string of data tables, it looks like the output from your script and reflector are well correlated.  I'd not expect them to be exact matches as bandwidth will contiinually fluctuate and be impacted by countless random factors.

Also note I don't have much feedback on your script except that it is WAY WAY WAY too long for what it does.  Like ridiculously way too long.  Why are you parsing the output of time in that way breaking down minutes, seconds, milliseconds?  Just use `time -f %e` to get the numerical seconds rather than minutes and seconds.

EDIT: FYI, based on your ten repetitions in the last post, reflector and your script show a correlation between the average results for each server with an R^2 of 0.48, p = 0.057.  This p value drops to 2.6*10^-6 by leaving out the slower half of the data points indicating that among the reasonably scoring servers there is a clear correlation between the results from reflector and your script.  Bad mirrors, in contrast, are - almost by definition - inconsistent: they get different scores from reflector and your script, but they get wildly different scores from the same tool on subsequent runs.  Or, long story very short: your data shows very good agreement between reflector run with 1 thread and your script's results.

Last edited by Trilby (2020-11-25 04:17:56)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#7 2020-11-25 04:38:31

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Thanks again Inspector Parrot / Trilby. I'm got all your points, and I agree with these.

Yes, I noticed that my script and `reflector` have the first three servers and make sense that they have different order based on some variables. Thanks for the feedback, in my Arch Linux the `-f` option not exists only the `-p` option but I'll keep in mind that refactor for next versions.

Regards,
Israel Roldan

Offline

#8 2020-11-25 04:47:30

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Proposal to improve the mirror list to get the faster server.

The 'time' builtin in bash does not have the format option, but [extra]/time does.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2020-12-02 23:49:12

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Proposal to improve the mirror list to get the faster server.

Sorry, I missed this discussion. Thanks for pinging me, Trilby.

Some comments, in no particular order.

  • I completely agree that parallel speed testing makes absolutely no sense if the user connection is the bottleneck. The options directly state that the results may be skewed.

  • Downloading a small file is indeed a bad test, but given that the download uses server cpu cycles and bandwidth, the smallest file was deliberately chosen to avoid even more overhead for what is essentially an unimportant metric (see next point).

  • The metric is unimportant because powerpill can be used (with internal reflector support) to download synchronously from enough servers to saturate most users' bandwidth. I personally never bother ranking my mirrorlist by rate because of this. I pick my main server based on reliable updates and then user powerpill+reflector to saturate my bandwidth with 50-100 servers (which incidentally also spread the load across those servers and off the main one).

  • Parallel downloads are in the works for pacman. I've skimmed some discussions on the pacman-dev mailing list but I haven't followed it closely.

I'll change the default number of threads to 1 but leave the option for those with better connections.
If there's any real demand for it, I could add functionality to select a different remote file for the test, perhaps with an option to download the first x MB of a larger file for more reliable results. The Arch iso would be a good target, but I don't think all mirrors host it, although that may have changed since I last checked. My concern though is that people will start downloading large chunks of data from multiple servers every day via automated ranking tests just so that they can shave off a minute or two from their weekly updates.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#10 2020-12-03 08:59:41

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Thanks for your answer Xyne. I'll start to use the Powerpill package, looks awesome. I see that you are working in the optimization of Pacman, thanks in advance many users will be happy. All the information that you provided was interested.

Offline

#11 2020-12-03 17:54:01

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Proposal to improve the mirror list to get the faster server.

I'm pushing an update today that removes threading and which uses the community instead of the core database for rating the mirrors. I noticed that core.db is currently only 133K, which is useless as a speed test. community.db is just over 5 MB, which is reasonable both as a test and with regard to server overhead.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#12 2020-12-08 16:10:15

vaal
Member
Registered: 2020-12-08
Posts: 1

Re: Proposal to improve the mirror list to get the faster server.

Xyne wrote:

I'm pushing an update today that removes threading and which uses the community instead of the core database for rating the mirrors. I noticed that core.db is currently only 133K, which is useless as a speed test. community.db is just over 5 MB, which is reasonable both as a test and with regard to server overhead.

Hello, Xyne.

What about users with good landline internet? Can you reverse thread support as the option? I think if you set "don't use threads by default" it can help you reach your goal.

Pls, don't think  I'm rude, just isn't a native English speaker.

Best wishes, Alex


Yet another IT man

Offline

#13 2020-12-11 15:45:46

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

vaal wrote:

Hello, Xyne.

What about users with good landline internet? Can you reverse thread support as the option? I think if you set "don't use threads by default" it can help you reach your goal.

Pls, don't think  I'm rude, just isn't a native English speaker.

Best wishes, Alex

Hello, Alex.

I'm not the expert in this subject as Xyne but regarding all my research, the `threads` option was a not accurate measure talking about getting the faster Mirror Server. Maybe the next scenario should be got the faster server; In this case image we have 4 Ethernet cards in the same computer and `reflector` has the option to put the thread related to a specific `interface` in this case we can have something like that `--threads 4 --thread eth0  --thread eth1  --thread eth2  --thread eth3`. Another case is if we don't care about the `threads` options, but we can use, it is when the faster server not matters for example test all the servers around the world with all the protocols and IP versions, it will retrieve around 550 servers. In this case, avoiding the `threads` options I highly recommend using the filters, in my case the Germany connections looks the faster but it is not stable, the best servers that I got was the USA and make sense because Mexico is near to the USA. Here is the example of my `reflector` with filters, excluding the faster server: `reflector --verbose --country "United States" --protocol https --completion-percent 100 --age 48 --score 8 --sort rate --save /etc/pacman.d/mirrorlist` this command checks 30 servers and get the 8 best scored.

Summary: If you don't care about the faster server then filter by `country` and `protocol`, optional use `completion-percent`, `age` and `score`.

Xyne, I have an idea to improve the `reflector`:

  • Only when the user adds the option `--fastest` it will test by 1 thread all the servers with the `community` database. As it's working now.

  • If the option `--fastest` is not there, then use automatically the `--threads 8` option but not test the databases only search for the server information which should be very fast, maybe 5 milliseconds.

  • When the user adds the option `--fastest` and another options as `score` then it comes back to the 1 thread and test the databases.

  • Summary, only when the `--fastest` options is there, it will use 1 thread and test the `community` database otherwise it will only retrieve the mirror information with multi-threads.

Regards,
Israel Roldan

Offline

#14 2020-12-11 15:58:01

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Proposal to improve the mirror list to get the faster server.

Israel, I'm not sure I understand your logic.  First, the multithreaded test is either valid or useless.  Which it is may well depend on the local connection: e.g. an ethernet connection and a high-bandwidth ISP should allow for accurate results with the multithreaded test.  The concern is that if either the local machine to router or ISP bandwidth could be less than $(nproc) times the mirror speeds, then this test gives faulty (not just biased, but potentially completely meaningless) results.

So if your hardware and internet service supports using the multithread test, then go for it (and this could be an argument for keeping a --threads option but defaulting to single-threaded which is what I understood to be the most recent change.)

But here's the important part: if your hardware / ISP would not be sufficient for the multithread test, do not do the multithread test!  It's a complete waste and may give bad information.  So running single threaded with "--fastest" as you propose is fine, but in the absence of that flag, do not run the speed test at all.  Running the speed test under conditions expected to be problematic is pretty silly.

In other words, flags should give the user the option to "do it right" or "not to do it", but there is no point in an option to "do it all wrong but do it anyways".  If you want the results to be sorted by an unreliable and virtually meaningless metric, just pipe the results through `shuf' and leave the mirror servers alone.

EDIT: note the initial caveat: for people with good local networks and good ISPs all of this may be irrelevant and the multithreaded test could be great.  But if we assume single threaded is needed to get accurate results on a given set up, that set up should never do the multithreaded test, whether the --fastest flag is past or not.  The real problem is the assumption that $(nproc) would correlate in any way with the local/ISP bandwidth - if your local/ISP bandwidth is expected to be a bit over twice the best mirror's bandwidth, then you could/should use up to 2 threads for the speed test, if your local bandwidth is expected to be 4 times the best mirror's, then you should use up to 4 threads.  This is the case regardless of whether your machine has 4 cpu cores, or 128, or anything in between.

Last edited by Trilby (2020-12-11 16:07:24)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#15 2020-12-11 19:34:19

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Trilby, Inspector Parrot.
Yes, I was talking about two different approaches, and maybe it's the confusing part.

All of this is coming from the Alex post, referring to the last `reflector` upgrade which deleted the `--threads` option. I think so the `--threads` option is important and it will save time.

Let's forget for a moment the action of get the fastest server.

Actual behavior:

  1. Executing `reflector` and inspect inside it: `reflector --completion-percent 100 --age 48 --sort rate --save /etc/pacman.d/mirrorlist`.

  2. I'm not seen the source code but I understand that it take a list of mirrors like Pacman Mirror list Generator.

  3. Then it goes server by server retrieving the specific information like Mirror Status which has the competition, delay and score.

  4. After this, `reflector` does the filters and sorts.

If it's correct, let me continue. If we apply the threads and keep avoiding the fastest servers, this is the improvement that I propose.

  1. Executing `reflector` and inspect inside it: `reflector --completion-percent 100 --age 48 --sort rate --save /etc/pacman.d/mirrorlist`.

  2. Take a list of mirrors like Pacman Mirror list Generator.

  3. `reflector` could be use the multi-threads to retrieve the specific information for every server (Mirror Status), if we have the option `--threads 24` it will take more information severs in less time. For example, the 552 servers which  takes 1 second to get the specific information. Using 24 threads will retrieve the information in 23 seconds for all servers versus not use threads which will take 552 seconds.

  4. After this, `reflector` does the filters and sorts.

What happens if the user set the `--fastest`.

  1. Executing `reflector` and inspect inside it: `reflector --completion-percent 100 --age 48 --sort rate --fastest --save /etc/pacman.d/mirrorlist`.

  2. Take a list of mirrors like Pacman Mirror list Generator.

  3. `reflector` will be use the multi-threads `--threads 24` to retrieve the specific information for every server (Mirror Status). It will take 23 seconds.

  4. After this, `reflector` does the filters but NOT the sorts.

  5. Get the fastest servers testing the community database after this action then `reflector` will sort by the fastest server. Right now `reflector` is doing it.

  6. When the `--fastest` option is set, the `sort` options will be discarded.

In other words, reflector will be split in two main sections, one is the `filters` and the other is the `sort's`in fact, `reflector` is only available to sort by one category (we can split this in the documentation). The first step always will be the filters because it's cheap to take the specific information also it's possible to use multi-threads. The second step is the sort's which will be easy for parameters as score, age and little more advance for fastest which has the `powerpill` implementation and it's testing the `community` database to get the better server.

NOTE: All of this explanation will be discard if `reflector` take the JSON file with the Mirror Status, in this case we don't need multi-threads to get the specific information for every server.

Did I explain myself?

Last edited by airv_zxf (2020-12-11 19:46:48)

Offline

#16 2020-12-12 02:14:41

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Proposal to improve the mirror list to get the faster server.

airv_zxf wrote:

Then it goes server by server retrieving the specific information like Mirror Status which has the competition, delay and score...

If it's correct, let me continue. If we apply the threads and keep avoiding the fastest servers, this is the improvement that I propose.

No, that is not correct.  All that information is retreived in a single download from the mirror status list.  The only time threading is relevant is in measuing actual download speeds from mirrors.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#17 2020-12-12 03:23:14

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Trilby wrote:

No, that is not correct.  All that information is retreived in a single download from the mirror status list.  The only time threading is relevant is in measuing actual download speeds from mirrors.

Thank you. Talk with all of you is my pleasure as always.

Offline

#18 2020-12-12 03:51:26

airv_zxf
Member
From: Guadalajara City, Mexico
Registered: 2017-08-19
Posts: 54
Website

Re: Proposal to improve the mirror list to get the faster server.

Hi again.

Obviously, it was pending the classic test, `powerpill` vs `my script`.

The test repeats 5 times, it does a synchronization of the databases and install the 0AD game (735.64 MiB), the cache was cleaned in every iteration means that it is the same scenario of the latter.

Results:
Lower is better.

App: | Powerpill | My Script
-----+-----------+----------
#1   | 0m32.609s | 0m25.667s
#2   | 0m33.357s | 0m25.450s
#3   | 0m34.751s | 0m26.321s
#4   | 0m34.646s | 0m25.762s
#5   | 0m34.421s | 0m24.916s

Summary:

  • My script is good but not at all because sometime take the “fastest” server and other time it's wrong because was faster by 200 milliseconds in the test but when I'm using it is slow.

  • Congratulations with your project `powerpill`, it is stable and gets an excellent speed.

  • Talking about `powerpill` I was expecting that it split every package in parts, and download simultaneous. In 1998 existed a program, similar to “Download Accelerator Plus” and DownThemAll they were awesome. In fact, download a webpage was faster because it got the file and split in 8 parts then it downloads every part, similar to torrent (multipart download).


My script: complete tests.

echo 'Server = https://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch' | sudo tee /etc/pacman.d/mirrorlist
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm
sudo pacman -Rns 0ad --noconfirm
sudo pacman -Sc --noconfirm
time sudo pacman -S 0ad --noconfirm

Server = https://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Download Size:    735.64 MiB
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   424 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  43.1 MiB/s 00:17 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  17.7 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-12-x86_64                                                                    9.0 MiB  36.9 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m25.667s
user    0m5.423s
sys     0m4.267s

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Download Size:    735.64 MiB
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   544 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  43.0 MiB/s 00:17 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  0.00   B/s 00:00 [####################################################################] 100%
 0ad-a23.1-12-x86_64                                                                    9.0 MiB  46.5 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m25.450s
user    0m5.243s
sys     0m4.171s
checking dependencies...

Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Download Size:    735.64 MiB
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   544 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  40.9 MiB/s 00:18 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB   177 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-12-x86_64                                                                    9.0 MiB  44.0 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m26.321s
user    0m5.266s
sys     0m4.075s
checking dependencies...

Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Download Size:    735.64 MiB
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   544 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  42.8 MiB/s 00:17 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB  0.00   B/s 00:00 [####################################################################] 100%
 0ad-a23.1-12-x86_64                                                                    9.0 MiB  49.0 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m25.762s
user    0m5.308s
sys     0m4.224s
checking dependencies...

Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Download Size:    735.64 MiB
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 enet-1.3.16-1-x86_64                                                                  32.6 KiB   544 KiB/s 00:00 [####################################################################] 100%
 0ad-data-a23.1-2-any                                                                 726.1 MiB  44.9 MiB/s 00:16 [####################################################################] 100%
 gloox-1.0.24-1-x86_64                                                                543.6 KiB   133 MiB/s 00:00 [####################################################################] 100%
 0ad-a23.1-12-x86_64                                                                    9.0 MiB  34.9 MiB/s 00:00 [####################################################################] 100%
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m24.916s
user    0m5.517s
sys     0m4.587s

`powerpill`: complete tests.
`reflector --verbose --country "United States" --protocol https --completion-percent 100 --age 48 --score 8 --sort rate --save /etc/pacman.d/mirrorlist`

sudo reflector --verbose --country "United States" --protocol https --completion-percent 100 --age 48 --score 8 --sort rate --save /etc/pacman.d/mirrorlist

[2020-12-11 10:05:31] INFO: rating 8 mirror(s) by download speed
[2020-12-11 10:05:31] INFO: Server                                                    Rate       Time
[2020-12-11 10:05:32] INFO: https://arch.mirror.constant.com/               10473.55 KiB/s     0.51 s
[2020-12-11 10:05:37] INFO: https://mirror.dc02.hackingand.coffee/arch/      1082.65 KiB/s     4.94 s
[2020-12-11 10:05:38] INFO: https://arch.mirror.square-r00t.net/            11851.88 KiB/s     0.45 s
[2020-12-11 10:05:39] INFO: https://arch.hu.fo/archlinux/                   10799.89 KiB/s     0.49 s
[2020-12-11 10:05:43] INFO: https://mirrors.lug.mtu.edu/archlinux/           1463.94 KiB/s     3.65 s
[2020-12-11 10:05:44] INFO: https://archmirror1.octyl.net/                  10545.68 KiB/s     0.51 s
[2020-12-11 10:05:45] INFO: https://mirror.wdc1.us.leaseweb.net/archlinux/  12066.40 KiB/s     0.44 s
[2020-12-11 10:05:46] INFO: https://mirrors.rit.edu/archlinux/               9339.56 KiB/s     0.57 s


sudo powerpill -Rns 0ad --noconfirm
sudo powerpill -Sc --noconfirm
time sudo powerpill -S 0ad --noconfirm
sudo powerpill -Rns 0ad --noconfirm
sudo powerpill -Sc --noconfirm
time sudo powerpill -S 0ad --noconfirm
sudo powerpill -Rns 0ad --noconfirm
sudo powerpill -Sc --noconfirm
time sudo powerpill -S 0ad --noconfirm
sudo powerpill -Rns 0ad --noconfirm
sudo powerpill -Sc --noconfirm
time sudo powerpill -S 0ad --noconfirm
sudo powerpill -Rns 0ad --noconfirm
sudo powerpill -Sc --noconfirm
time sudo powerpill -S 0ad --noconfirm

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
12/11 10:06:22 [NOTICE] Downloading 4 item(s)
12/11 10:06:22 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:06:22 [NOTICE] Download complete: /var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:06:22 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:06:22 [NOTICE] Download complete: /var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
[DL:28MiB][#c4e396 1.4MiB/8.9MiB(16%)][#47d457 21MiB/726MiB(2%)]
12/11 10:06:23 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:06:23 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
[#47d457 726MiB/726MiB(100%) CN:0] [Checksum:#47d457 651MiB/726MiB(89%)]
12/11 10:06:46 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:06:46 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
bd7bfb|OK  |    78KiB/s|/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
3b9813|OK  |   909KiB/s|/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
c4e396|OK  |   6.4MiB/s|/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
47d457|OK  |    39MiB/s|/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Status Legend:
(OK):download completed.
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m32.609s
user    0m9.546s
sys     0m5.483s

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
12/11 10:06:55 [NOTICE] Downloading 4 item(s)
12/11 10:06:56 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:06:56 [NOTICE] Download complete: /var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
[DL:31MiB][#71d372 6.7MiB/8.9MiB(75%)][#46b522 18MiB/726MiB(2%)][#12b47a 416KiB/543KiB(76%)]                                                                                                
12/11 10:06:57 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:06:57 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:06:57 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:06:57 [NOTICE] Download complete: /var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
[#46b522 726MiB/726MiB(100%) CN:0] [Checksum:#46b522 671MiB/726MiB(92%)]
12/11 10:07:21 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:07:21 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
fff0d7|OK  |    76KiB/s|/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
71d372|OK  |   8.9MiB/s|/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12b47a|OK  |   419KiB/s|/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
46b522|OK  |    37MiB/s|/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Status Legend:
(OK):download completed.
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m33.357s
user    0m9.135s
sys     0m5.270s

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
12/11 10:07:30 [NOTICE] Downloading 4 item(s)
12/11 10:07:30 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:07:30 [NOTICE] Download complete: /var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
[DL:41MiB][#d19756 8.3MiB/8.9MiB(92%)][#3acb1f 24MiB/726MiB(3%)][#ca7531 416KiB/543KiB(76%)]                                                                                                
12/11 10:07:31 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:07:31 [NOTICE] Download complete: /var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:07:32 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:07:32 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
[#3acb1f 726MiB/726MiB(100%) CN:0] [Checksum:#3acb1f 672MiB/726MiB(92%)]
12/11 10:07:56 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:07:56 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
596ea3|OK  |    84KiB/s|/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
ca7531|OK  |   444KiB/s|/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
d19756|OK  |   5.5MiB/s|/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
3acb1f|OK  |    35MiB/s|/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Status Legend:
(OK):download completed.
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m34.751s
user    0m9.196s
sys     0m5.084s

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
12/11 10:08:06 [NOTICE] Downloading 4 item(s)
12/11 10:08:06 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:08:06 [NOTICE] Download complete: /var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:08:07 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:08:07 [NOTICE] Download complete: /var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
[DL:38MiB][#22bf48 8.5MiB/8.9MiB(94%)][#f3b168 22MiB/726MiB(3%)]
12/11 10:08:07 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:08:07 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
[#f3b168 726MiB/726MiB(100%) CN:0] [Checksum:#f3b168 676MiB/726MiB(93%)]
12/11 10:08:32 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:08:32 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
52eefd|OK  |    90KiB/s|/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
0eef27|OK  |   743KiB/s|/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
22bf48|OK  |   8.9MiB/s|/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
f3b168|OK  |    35MiB/s|/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
Status Legend:
(OK):download completed.
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m34.646s
user    0m9.503s
sys     0m5.886s

checking dependencies...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Removed Size:  2039.27 MiB
:: Do you want to remove these packages? [Y/n] 
:: Processing package changes...
(1/4) removing 0ad                                                                                                [####################################################################] 100%
(2/4) removing gloox                                                                                              [####################################################################] 100%
(3/4) removing enet                                                                                               [####################################################################] 100%
(4/4) removing 0ad-data                                                                                           [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
Packages to keep:
  All locally installed packages
Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] 
removing old packages from cache...
Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] 
removing unused sync repositories...
12/11 10:08:42 [NOTICE] Downloading 4 item(s)
12/11 10:08:42 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:08:42 [NOTICE] Download complete: /var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
12/11 10:08:42 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
12/11 10:08:42 [NOTICE] Download complete: /var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
[DL:1.8MiB][#bc55a6 8.9MiB/8.9MiB(100%)][#45ddb6 726MiB/726MiB(100%)] [Checksum:#45ddb6 683MiB/726MiB(94%)](+1)                                                                             
12/11 10:09:08 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:09:08 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
12/11 10:09:08 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
12/11 10:09:08 [NOTICE] Download complete: /var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d932f9|OK  |    84KiB/s|/var/cache/pacman/pkg/enet-1.3.16-1-x86_64.pkg.tar.zst
585b54|OK  |   791KiB/s|/var/cache/pacman/pkg/gloox-1.0.24-1-x86_64.pkg.tar.zst
45ddb6|OK  |    35MiB/s|/var/cache/pacman/pkg/0ad-data-a23.1-2-any.pkg.tar.zst
bc55a6|OK  |   446KiB/s|/var/cache/pacman/pkg/0ad-a23.1-12-x86_64.pkg.tar.zst
Status Legend:
(OK):download completed.
resolving dependencies...
looking for conflicting packages...
Packages (4) 0ad-data-a23.1-2  enet-1.3.16-1  gloox-1.0.24-1  0ad-a23.1-12
Total Installed Size:  2039.27 MiB
:: Proceed with installation? [Y/n] 
(4/4) checking keys in keyring                                                                                    [####################################################################] 100%
(4/4) checking package integrity                                                                                  [####################################################################] 100%
(4/4) loading package files                                                                                       [####################################################################] 100%
(4/4) checking for file conflicts                                                                                 [####################################################################] 100%
(4/4) checking available disk space                                                                               [####################################################################] 100%
:: Processing package changes...
(1/4) installing enet                                                                                             [####################################################################] 100%
(2/4) installing 0ad-data                                                                                         [####################################################################] 100%
(3/4) installing gloox                                                                                            [####################################################################] 100%
(4/4) installing 0ad                                                                                              [####################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the desktop file MIME type cache...
real    0m34.421s
user    0m9.076s
sys     0m5.060s

Last edited by airv_zxf (2020-12-12 03:53:34)

Offline

#19 2020-12-12 09:31:33

GSMiller
Member
Registered: 2020-11-23
Posts: 75

Re: Proposal to improve the mirror list to get the faster server.

Thank you airv_zxf and Trilby. This is a great thread. It is wonderful to see such sharp minds working together. I hope you make Archlinux better and better. I want faster downloads too!


A dog is a man's best friend.

Offline

#20 2020-12-13 17:09:11

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Proposal to improve the mirror list to get the faster server.

Just fyi, the speed increase of powerpill will partly depend on the "reflector" options in /etc/powerpill/powerpill.json and the number and size of packages to download. More mirrors, more packages and larger packages should see the greatest increase, up to bandwidth saturation. Internally powerpill uses aria2c so there is overhead involved in setting it up and launching it, plus there is redundant IO overhead due to both aria2c and pacman checking file integrity.

I can re-add the threads option to reflector if users want it but I am not convinced that it would be very useful. As Trilby correctly states, all metadata is retrieved from the mirrorlist page in a single download so threading only affects the database download test. That should be run automatically via a timer without user interaction so it shouldn't matter how fast the actual test is. However, I'm in favor of given users the freedom even if some of them will shoot themselves in the foot by misusing the option.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#21 2020-12-13 18:04:48

fluxboxer
Member
Registered: 2012-12-02
Posts: 118

Re: Proposal to improve the mirror list to get the faster server.

Xyne, not sure if it's related, but I came here as my reflector seems to be broken today listing apparently the slowest mirrors and my download has repeatedly stalled at libreoffice. I've manually updated the list as a workaround.

Last edited by fluxboxer (2020-12-13 18:05:11)

Offline

#22 2020-12-13 18:09:21

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Proposal to improve the mirror list to get the faster server.

@fluxboxer
What command are you running and what's the output?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#23 2020-12-13 18:15:55

fluxboxer
Member
Registered: 2012-12-02
Posts: 118

Re: Proposal to improve the mirror list to get the faster server.

It's in my pacman hook:

/bin/sh -c "reflector --latest 5 --protocol https --sort rate --save /etc/pacman.d/mirrorlist;  rm -f /etc/pacman.d/mirrorlist.pacnew

When I try (without the hook)

sudo reflector --latest 5 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

I get

[2020-12-13 21:14:10] WARNING: failed to rate http(s) download (https://mirror.pseudoform.org/community/os/x86_64/community.db): Download timed out after 5 second(s)

Offline

#24 2020-12-13 18:17:15

fluxboxer
Member
Registered: 2012-12-02
Posts: 118

Offline

#25 2020-12-13 23:09:19

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Proposal to improve the mirror list to get the faster server.

The warning just means that one of the mirrors was so slow that reflector gave up trying to download. When that happens, the mirror is sorted to the end of the list. You can increase the download timeout with the --download-timeout option but the output will not change if only one mirror timed out.

As for downloads stalling, try using different search criteria. If all 5 of the latest mirrors are slow and time out on your connection, then the order will not make any difference. Maybe try filtering by country to use mirrors closer to you. You could also try powerpill.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB