You are not logged in.

Pages: 1

#1 2018-07-15 08:31:32

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Simplifying upgrades

Hi,

After every "pacman -Syu" I inspect its output, see which files were installed as *.pacnew, "mv $f $f.prv", "mv $f.pacnew $f", and apply changes I made in the config. Today, I've written a script:

#!/usr/bin/env bash
set -eu

# * prints last upgrade's output, omitting packages that produced no extra output
# * upgrades config files, new versions of which were installed as *.pacnew
# * backs up config files as <name>.prv-\d{2}, when changed

# assumptions:
# * the upgrade command is "pacman -Syu"
# * all issues had been resolved before running the last upgrade
# * important information appears between /(transaction started|installed|upgraded)/ and /(installed|upgraded|transaction completed)/ lines
# * or after /running/ lines
# * see re_line, re_next_line regexps
# * no two packages with names <name> and <name>-[0-9]* exists

# quotes string for extended regular expression
# args:
#   string
#   delimiter (optional)
quote_ere() {
    local s=$1 delimiter=${2-}
    local re='\.|\*|\[|\^|\$|\\|\+|\?|\{|\||\('   # .*[^$\+?{|(
    if (( ${#delimiter} )); then
        re=$re'|\'$delimiter
    fi
    printf "%s\n" "$s" | sed -E 's/('"$re"')/\\\1/g'
}

# asks user to make a choice
# args:
#   prompt
#   choices: (optional) single-letter characters separated by /,
#     the default one is uppercased
# defaults to Y/n choices, in this case returns 0/1
# otherwise prints user choice (single lowercase letter)
ask() {
    local prompt=$1 choices=${2-}
    local char
    if [[ "$choices" ]]; then
        local _IFS=$IFS
        IFS=/
        local choices_arr=($choices)
        IFS=$_IFS

        while true; do
            read -rsN 1 -p "$prompt [$choices] " char

            # print user choice, possibly choosing the default one if user pressed Enter
            if [ "$char" = $'\n' ]; then
                # user pressed Enter, look for the default choice
                for choice in ${choices_arr[@]+"${choices_arr[@]}"}; do
                    if [ "$choice" != "${choice,,}" ]; then
                        printf "%s\n" "${choice,,}" >&2
                        char=${choice,,}
                        break;
                    fi
                done
            else
                printf "%s\n" "${char,,}" >&2
            fi

            # print the result
            for choice in ${choices_arr[@]+"${choices_arr[@]}"}; do
                if [ "${char,,}" = "${choice,,}" ]; then
                    printf "%s" "${char,,}"
                    return
                fi
            done
        done
    else
        read -rsN 1 -p "$prompt [Y/n] " char
        if [ "${char,,}" = y ] || [ "${char-}" = $'\n' ]; then
            echo yes >&2
            return 0
        else
            echo no >&2
            return 1
        fi
    fi
}

hr=$(printf "=%.0s" $(seq 1 "$(tput cols)"))


# Print last upgrade's output, omitting packages that produced no extra output

# get last upgrade's output from the log
last_upgrade_first_line=`egrep "Running 'pacman -Syu'" /var/log/pacman.log | tail -n 1`
re_line='^\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|running)'
re_next_line='\[[^]]+\] \[ALPM\] (upgraded|installed|running|transaction completed)'
q_last_upgrade_first_line=`quote_ere "$last_upgrade_first_line" /`
last_upgrade=`sed -En "/$q_last_upgrade_first_line/,/Running 'pacman/p" /var/log/pacman.log`

script=$(cat <<'SCRIPT'
    use strict;
    use warnings;
    my $re = qr{\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|transaction completed|running)};
    my $r1;
    my $r = [];
    my $line_found;
    my $extra_output;
    while (<>) {
        if (/$re/) {
            if ($line_found && $extra_output) {
                push @$r, $r1 . $_;
            }
            $r1 = $_;
            $line_found = 1;
            $extra_output = 0;
        } elsif ($line_found) {
            $r1 .= $_;
            $extra_output = 1;
        }
    }
    foreach my $fragment (@$r) {
        print '=' x `tput cols`, "\n";
        print $fragment;
        print "\n";
    }
SCRIPT
)
perl -e "$script" <<< "$last_upgrade"

# sed version (left for historical purposes)
if false; then
printf "%s\n" "$last_upgrade" \
    | sed -En "
        :a /$re_line/ {           # if found install/upgrade line
            N                     # append next line into the pattern space

            # no extra output after install/upgrade line
            /\n$re_next_line/ {   # if next line is also install/upgrade line
                s/.*\n//          # leave only next install/upgrade line
                b a               # jump to :a
            }

            # while not reached next install/upgrade line
            :b N                      # append next line into the pattern space
            /\n$re_next_line/ ! b b   # else jump to :b

            # found next install/upgrade line
            /\n$re_next_line/ {       # if found next install/upgrade line
                # output lines accumulated so far and jump to the next iteration
                h                     # copy pattern space to hold space
                i \
$hr
                p                     # print pattern space
                i \

                x                     # exchange pattern and hold spaces
                s/.*\n//              # leave only next install/upgrade line
                b a                   # jump to :a
            }
        }
    "
echo
fi


# Upgrade config files, new versions of which were installed as *.pacnew

re_file='^\[[^]]+\] \[ALPM\] warning: (\S+).*'

# find files having been installed as *.pacnew during the last upgrade
s_files=`printf "%s\n" "$last_upgrade" \
    | egrep '\binstalled as\b' \
    | sed -E "s/$re_file/\1/"`
IFS_=$IFS
IFS=$'\n'
files=($s_files)
IFS=$IFS_

for f in ${files[@]+"${files[@]}"}; do
    if ! [ -e "$f.pacnew" ]; then
        continue
    fi
    package=`pacman -Qo -- "$f" | awk '{print $(NF - 1)}'`
    package_file=`ls /var/cache/pacman/pkg/$package-[0-9]* | tail -n 2 | head -n 1`
    patch=$(diff -u <(tar xOf "$package_file" -- "${f:1}") "$f" || true)

    printf "%s\n" "$hr"
    printf "%s\n" "$f"
    printf "%s\n" "$patch"
    answer=$(ask 'fix?' "Y/n/q")
    case "$answer" in
    y) # determine the number of the last config file backup
        last_n=`find -name "$f.prv-*" \
            | sed -E 's/.*-//' \
            | sort -nk1 \
            | tail -n 1`

        # move config -> backup, pacnew -> config
        mv "$f" "$f.prv-$(printf "%02u" "$(( last_n + 1 ))")"
        mv "$f.pacnew" "$f"

        # patch config with the changes made by user
        patch "$f" <(echo "$patch")
        ;;
    n) continue
        ;;
    q) exit 1
        ;;
    esac
done

Output:

===============================================================================================
[2018-07-15 07:51] [ALPM] upgraded archlinux-keyring (20180404-1 -> 20180627-1)
[2018-07-15 07:51] [ALPM-SCRIPTLET] ==> Appending keys from archlinux.gpg...
[2018-07-15 07:51] [ALPM-SCRIPTLET] ==> Locally signing trusted keys in keyring...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Locally signing key DDB867B92AA789C165EEFA799B729B06A680C281...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Locally signing key 684148BB25B49E986A4944C55184252D824B18E8...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Locally signing key 91FFE0700E80619CEB73235CA88E23E377514E00...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Locally signing key AB19265E5D7D20687D303246BA1DFB64FFF979E7...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Locally signing key 0E8B644079F599DFC1DDC3973348882F6AC6A4C2...
[2018-07-15 07:51] [ALPM-SCRIPTLET] ==> Importing owner trust values...
[2018-07-15 07:51] [ALPM-SCRIPTLET] ==> Disabling revoked keys in keyring...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 7FA647CD89891DEDC060287BB9113D1ED21E1A55...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key D4DE5ABDE2A7287644EAC7E36D1A9E70E19DAA50...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 40440DC037C05620984379A6761FAD69BA06C6A9...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key B1F2C889CB2CCB2ADA36D963097D629E437520BD...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key BC1FBE4D2826A0B51E47ED62E2539214C6C11350...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 63F395DE2D6398BBE458F281F2DBB4931985A992...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 8F76BEEA0289F9E1D3E229C05F946DED983D4366...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 4FCF887689C41B09506BE8D5F3E1D5C5D30DB0AD...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 81D7F8241DB38BC759C80FCE3A726C6170E80477...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 5E7585ADFF106BFFBBA319DC654B877A0864983E...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 50F33E2E5B0C3D900424ABE89BDCF497A4BBCC7F...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key E7210A59715F6940CF9A4E36A001876699AD6E84...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key F5A361A3A13554B85E57DDDAAF7EF7873CFD4BB6...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 40776A5221EF5AD468A4906D42A1DB15EC133BAD...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 8CF934E339CAD8ABF342E822E711306E3C4F88BC...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 5696C003B0854206450C8E5BE613C09CB4440678...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 9515D8A8EAB88E49BB65EDBCE6B456CAF15447D5...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 4A8B17E20B88ACA61860009B5CED81B7C2E5C0D2...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 0B20CA1931F5DA3A70D0F8D2EA6836E1AB441196...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 34C5D94FE7E7913E86DC427E7FB1A3800C84C0A5...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 39F880E50E49A4D11341E8F939E4F17F295AFBF4...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 66BD74A036D522F51DD70A3C7F2A16726521E06D...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key DBE7D3DD8C81D58D0A13D0E76BC26A17B9B7018A...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 07DFD3A0BC213FA12EDC217559B3122E2FA915EC...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 44D4A033AC140143927397D47EFD567D4C7EA887...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 27FFC4769E19F096D41D9265A04F9397CDFD6BB0...
[2018-07-15 07:51] [ALPM-SCRIPTLET]   -> Disabling key 8840BD07FC24CB7CE394A07CCF7037A4F27FB7DA...
[2018-07-15 07:51] [ALPM-SCRIPTLET] ==> Updating trust database...
[2018-07-15 07:51] [ALPM-SCRIPTLET] gpg: next trustdb check due at 2018-10-19
[2018-07-15 07:51] [ALPM] upgraded libxdmcp (1.1.2-1 -> 1.1.2-2)

===============================================================================================
[2018-07-15 07:51] [ALPM] upgraded libseccomp (2.3.2-2 -> 2.3.3-1)
[2018-07-15 07:51] [ALPM] warning: /etc/systemd/logind.conf installed as /etc/systemd/logind.conf.pacnew
[2018-07-15 07:51] [ALPM] upgraded systemd (238.76-1 -> 239.0-2)

===============================================================================================
[2018-07-15 07:52] [ALPM] upgraded git (2.17.1-1 -> 2.18.0-1)
[2018-07-15 07:52] [ALPM] warning: /usr/bin/pinentry installed as /usr/bin/pinentry.pacnew
[2018-07-15 07:52] [ALPM] upgraded pinentry (1.1.0-3 -> 1.1.0-4)

===============================================================================================
[2018-07-15 07:53] [ALPM] upgraded mono (5.12.0.226-1 -> 5.12.0.260-1)
[2018-07-15 07:53] [ALPM-SCRIPTLET] Mono Certificate Store Sync - version 5.12.0.0
[2018-07-15 07:53] [ALPM-SCRIPTLET] Populate Mono certificate store from a concatenated list of certificates.
[2018-07-15 07:53] [ALPM-SCRIPTLET] Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
[2018-07-15 07:53] [ALPM-SCRIPTLET]
[2018-07-15 07:53] [ALPM-SCRIPTLET] Importing into legacy system store:
[2018-07-15 07:53] [ALPM-SCRIPTLET] I already trust 136, your new list has 135
[2018-07-15 07:53] [ALPM-SCRIPTLET] 1 previously trusted certificates were removed.
[2018-07-15 07:53] [ALPM-SCRIPTLET] Certificate removed: C=TR, L=Ankara, O=TÜRKTRUST Bilgi İletişim ve Bilişim Güvenliği Hizmetleri A.Ş., CN=TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı H5
[2018-07-15 07:53] [ALPM-SCRIPTLET] Import process completed.
[2018-07-15 07:53] [ALPM-SCRIPTLET]
[2018-07-15 07:53] [ALPM-SCRIPTLET] Importing into BTLS system store:
[2018-07-15 07:53] [ALPM-SCRIPTLET] I already trust 136, your new list has 135
[2018-07-15 07:53] [ALPM-SCRIPTLET] 1 previously trusted certificates were removed.
[2018-07-15 07:53] [ALPM-SCRIPTLET] Certificate removed: C=TR, L=Ankara, O=TÜRKTRUST Bilgi İletişim ve Bilişim Güvenliği Hizmetleri A.Ş., CN=TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı H5
[2018-07-15 07:53] [ALPM-SCRIPTLET] Import process completed.
[2018-07-15 07:53] [ALPM] upgraded msmtp (1.6.6-1 -> 1.6.8-1)

===============================================================================================
[2018-07-15 07:53] [ALPM] upgraded openssh (7.7p1-1 -> 7.7p1-2)
[2018-07-15 07:53] [ALPM] warning: /etc/pacman.d/mirrorlist installed as /etc/pacman.d/mirrorlist.pacnew
[2018-07-15 07:53] [ALPM] upgraded pacman-mirrorlist (20180524-1 -> 20180710-1)

===============================================================================================
[2018-07-15 07:54] [ALPM] running '90-linux.hook'...
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Starting build: 4.17.5-1-ARCH
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [autodetect]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Image generation successful
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Starting build: 4.17.5-1-ARCH
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [udev]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [modconf]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [block]
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [filesystems]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [keyboard]
[2018-07-15 07:54] [ALPM-SCRIPTLET]   -> Running build hook: [fsck]
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
[2018-07-15 07:54] [ALPM-SCRIPTLET] ==> Image generation successful
[2018-07-15 07:54] [ALPM] running 'detect-old-perl-modules.hook'...


===============================================================================================
/usr/bin/pinentry

fix? [Y/n/q] n
===============================================================================================
/etc/pacman.d/mirrorlist
--- /dev/fd/63  2018-07-16 11:46:37.110934935 +0300
+++ /etc/pacman.d/mirrorlist    2018-06-02 13:03:44.028189583 +0300
@@ -1,545 +1,20 @@
-##
-## Arch Linux repository mirrorlist
-## Generated on 2018-05-24
-##
-
-## Worldwide
-#Server = http://mirrors.evowise.com/archlinux/$repo/os/$arch
-#Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch
-
-## Australia
-#Server = https://mirror.aarnet.edu.au/pub/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirror.digitalpacific.com.au/$repo/os/$arch
-#Server = http://ftp.iinet.net.au/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.internode.on.net/pub/archlinux/$repo/os/$arch
-#Server = http://archlinux.melbourneitmirror.net/$repo/os/$arch
-#Server = http://ftp.swin.edu.au/archlinux/$repo/os/$arch
-
-## Austria
-#Server = http://mirror.digitalnova.at/archlinux/$repo/os/$arch
-#Server = http://mirror.easyname.at/archlinux/$repo/os/$arch
-#Server = http://mirror1.htu.tugraz.at/archlinux/$repo/os/$arch
-#Server = http://mirror.reisenbauer.ee/archlinux/$repo/os/$arch
-#Server = https://mirror.reisenbauer.ee/archlinux/$repo/os/$arch
-
-## Bangladesh
-#Server = http://mirror.xeonbd.com/archlinux/$repo/os/$arch
-
-## Belarus
-#Server = http://ftp.byfly.by/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.datacenter.by/pub/archlinux/$repo/os/$arch
-
-## Belgium
-#Server = http://archlinux.cu.be/$repo/os/$arch
-#Server = http://archlinux.mirror.kangaroot.net/$repo/os/$arch
-
-## Bosnia and Herzegovina
-#Server = http://burek.archlinux.ba/$repo/os/$arch
-#Server = http://archlinux.mirror.ba/$repo/os/$arch
-
-## Brazil
-#Server = http://br.mirror.archlinux-br.org/$repo/os/$arch
-#Server = http://archlinux.c3sl.ufpr.br/$repo/os/$arch
-#Server = http://linorg.usp.br/archlinux/$repo/os/$arch
-#Server = http://pet.inf.ufsc.br/mirrors/archlinux/$repo/os/$arch
-#Server = http://archlinux.pop-es.rnp.br/$repo/os/$arch
-#Server = http://mirror.ufam.edu.br/archlinux/$repo/os/$arch
-
-## Bulgaria
-#Server = http://mirror.host.ag/archlinux/$repo/os/$arch
-#Server = http://mirrors.netix.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.uni-plovdiv.net/archlinux/$repo/os/$arch
-#Server = https://mirrors.uni-plovdiv.net/archlinux/$repo/os/$arch
-
-## Canada
-#Server = http://mirror.cedille.club/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirror.colo-serv.net/$repo/os/$arch
-#Server = http://mirror.csclub.uwaterloo.ca/archlinux/$repo/os/$arch
-#Server = https://mirror.csclub.uwaterloo.ca/archlinux/$repo/os/$arch
-#Server = http://mirror.its.dal.ca/archlinux/$repo/os/$arch
-#Server = http://muug.ca/mirror/archlinux/$repo/os/$arch
-#Server = https://muug.ca/mirror/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirror.rafal.ca/$repo/os/$arch
-#Server = http://mirror.sergal.org/archlinux/$repo/os/$arch
-#Server = https://mirror.sergal.org/archlinux/$repo/os/$arch
-
-## Chile
-#Server = http://mirror.archlinux.cl/$repo/os/$arch
-
-## China
-#Server = http://mirrors.163.com/archlinux/$repo/os/$arch
-#Server = http://mirror.lzu.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.neusoft.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.neusoft.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.shu.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.shu.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.shu6.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.sjtug.sjtu.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.ustc.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.ustc.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.xjtu.edu.cn/archlinux/$repo/os/$arch
-#Server = https://mirrors.xjtu.edu.cn/archlinux/$repo/os/$arch
-#Server = http://mirrors.zju.edu.cn/archlinux/$repo/os/$arch
-
-## Colombia
-#Server = http://mirror.upb.edu.co/archlinux/$repo/os/$arch
-#Server = http://mirror.venturasystems.tech/archlinux/$repo/os/$arch
-
-## Croatia
-#Server = http://archlinux.iskon.hr/$repo/os/$arch
-
-## Czechia
-#Server = http://mirror.dkm.cz/archlinux/$repo/os/$arch
-#Server = https://mirror.dkm.cz/archlinux/$repo/os/$arch
-#Server = http://ftp.fi.muni.cz/pub/linux/arch/$repo/os/$arch
-#Server = http://ftp.linux.cz/pub/linux/arch/$repo/os/$arch
-#Server = http://gluttony.sin.cvut.cz/arch/$repo/os/$arch
-#Server = https://gluttony.sin.cvut.cz/arch/$repo/os/$arch
-#Server = http://mirrors.nic.cz/archlinux/$repo/os/$arch
-#Server = http://ftp.sh.cvut.cz/arch/$repo/os/$arch
-#Server = https://ftp.sh.cvut.cz/arch/$repo/os/$arch
-#Server = http://mirror.vpsfree.cz/archlinux/$repo/os/$arch
-
-## Denmark
-#Server = http://mirrors.dotsrc.org/archlinux/$repo/os/$arch
-#Server = https://mirrors.dotsrc.org/archlinux/$repo/os/$arch
-#Server = http://ftp.klid.dk/ftp/archlinux/$repo/os/$arch
-#Server = http://mirror.one.com/archlinux/$repo/os/$arch
-#Server = https://mirror.one.com/archlinux/$repo/os/$arch
-
-## Ecuador
-#Server = http://mirror.cedia.org.ec/archlinux/$repo/os/$arch
-#Server = http://mirror.espoch.edu.ec/archlinux/$repo/os/$arch
-#Server = http://mirror.uta.edu.ec/archlinux/$repo/os/$arch
-
-## Finland
-#Server = http://arch.mirror.far.fi/$repo/os/$arch
-#Server = https://mirror.srv.fail/archlinux/$repo/os/$arch
-
-## France
-#Server = http://archlinux.de-labrusse.fr/$repo/os/$arch
-#Server = http://mirror.archlinux.ikoula.com/archlinux/$repo/os/$arch
-#Server = http://archlinux.vi-di.fr/$repo/os/$arch
-#Server = https://archlinux.vi-di.fr/$repo/os/$arch
-#Server = http://mirror.armbrust.me/archlinux/$repo/os/$arch
-#Server = https://mirror.armbrust.me/archlinux/$repo/os/$arch
-#Server = http://mirrors.arnoldthebat.co.uk/archlinux/$repo/os/$arch
-#Server = https://mirrors.arnoldthebat.co.uk/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirrors.benatherton.com/$repo/os/$arch
-#Server = http://fooo.biz/archlinux/$repo/os/$arch
-#Server = https://fooo.biz/archlinux/$repo/os/$arch
-#Server = http://mirror.ibcp.fr/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.lastmikoi.net/archlinux/$repo/os/$arch
-#Server = http://archlinux.mailtunnel.eu/$repo/os/$arch
-#Server = https://archlinux.mailtunnel.eu/$repo/os/$arch
-#Server = http://mir.archlinux.fr/$repo/os/$arch
-#Server = http://mirrors.celianvdb.fr/archlinux/$repo/os/$arch
-#Server = https://mirrors.celianvdb.fr/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.phx.ms/arch/$repo/os/$arch
-#Server = https://mirrors.phx.ms/arch/$repo/os/$arch
-#Server = http://archlinux.mirror.pkern.at/$repo/os/$arch
-#Server = https://archlinux.mirror.pkern.at/$repo/os/$arch
-#Server = http://archlinux.polymorf.fr/$repo/os/$arch
-#Server = http://mirrors.standaloneinstaller.com/archlinux/$repo/os/$arch
-#Server = http://arch.tamcore.eu/$repo/os/$arch
-#Server = https://mirror.thekinrar.fr/archlinux/$repo/os/$arch
-#Server = http://ftp.u-strasbg.fr/linux/distributions/archlinux/$repo/os/$arch
-#Server = https://mirror.wormhole.eu/archlinux/$repo/os/$arch
-#Server = http://arch.yourlabs.org/$repo/os/$arch
-#Server = https://arch.yourlabs.org/$repo/os/$arch
-
-## Germany
-#Server = http://mirror.23media.de/archlinux/$repo/os/$arch
-#Server = http://artfiles.org/archlinux.org/$repo/os/$arch
-#Server = https://mirror.bethselamin.de/$repo/os/$arch
-#Server = http://arch.eckner.net/archlinux/$repo/os/$arch
-#Server = https://arch.eckner.net/archlinux/$repo/os/$arch
-#Server = http://mirror.f4st.host/archlinux/$repo/os/$arch
-#Server = https://mirror.f4st.host/archlinux/$repo/os/$arch
-#Server = http://ftp.fau.de/archlinux/$repo/os/$arch
-#Server = https://ftp.fau.de/archlinux/$repo/os/$arch
-#Server = http://mirror.gnomus.de/$repo/os/$arch
-#Server = http://www.gutscheindrache.com/mirror/archlinux/$repo/os/$arch
-#Server = http://ftp.gwdg.de/pub/linux/archlinux/$repo/os/$arch
-#Server = http://mirror.hactar.xyz/$repo/os/$arch
-#Server = https://mirror.hactar.xyz/$repo/os/$arch
-#Server = http://archlinux.honkgong.info/$repo/os/$arch
-#Server = http://ftp.hosteurope.de/mirror/ftp.archlinux.org/$repo/os/$arch
-#Server = http://ftp-stud.hs-esslingen.de/pub/Mirrors/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirror.iphh.net/$repo/os/$arch
-#Server = http://repo.itmettke.de/archlinux/$repo/os/$arch
-#Server = https://repo.itmettke.de/archlinux/$repo/os/$arch
-#Server = https://mirror.jankoppe.de/archlinux/$repo/os/$arch
-#Server = http://arch.jensgutermuth.de/$repo/os/$arch
-#Server = https://arch.jensgutermuth.de/$repo/os/$arch
-#Server = http://k42.ch/mirror/archlinux/$repo/os/$arch
-#Server = https://k42.ch/mirror/archlinux/$repo/os/$arch
-#Server = https://archlinux.layer8.fail/$repo/os/$arch
-#Server = http://mirror.fra10.de.leaseweb.net/archlinux/$repo/os/$arch
-#Server = https://mirror.fra10.de.leaseweb.net/archlinux/$repo/os/$arch
-#Server = http://mirror.metalgamer.eu/archlinux/$repo/os/$arch
-#Server = https://mirror.metalgamer.eu/archlinux/$repo/os/$arch
-#Server = http://mirror.michael-eckert.net/archlinux/$repo/os/$arch
-#Server = https://mirror.michael-eckert.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.n-ix.net/archlinux/$repo/os/$arch
-#Server = https://mirrors.n-ix.net/archlinux/$repo/os/$arch
-#Server = http://mirror.netcologne.de/archlinux/$repo/os/$arch
-#Server = https://mirror.netcologne.de/archlinux/$repo/os/$arch
-#Server = http://mirrors.niyawe.de/archlinux/$repo/os/$arch
-#Server = https://mirrors.niyawe.de/archlinux/$repo/os/$arch
-#Server = http://archlinux.nullpointer.io/$repo/os/$arch
-#Server = https://archlinux.nullpointer.io/$repo/os/$arch
-#Server = http://mirror.orbit-os.com/archlinux/$repo/os/$arch
-#Server = https://mirror.orbit-os.com/archlinux/$repo/os/$arch
-#Server = http://packages.oth-regensburg.de/archlinux/$repo/os/$arch
-#Server = https://packages.oth-regensburg.de/archlinux/$repo/os/$arch
-#Server = http://mirror.pseudoform.org/$repo/os/$arch
-#Server = https://mirror.pseudoform.org/$repo/os/$arch
-#Server = https://www.ratenzahlung.de/mirror/archlinux/$repo/os/$arch
-#Server = http://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
-#Server = https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
-#Server = http://linux.rz.rub.de/archlinux/$repo/os/$arch
-#Server = http://mirror.selfnet.de/archlinux/$repo/os/$arch
-#Server = http://ftp.spline.inf.fu-berlin.de/mirrors/archlinux/$repo/os/$arch
-#Server = https://ftp.spline.inf.fu-berlin.de/mirrors/archlinux/$repo/os/$arch
-#Server = http://archlinux.thaller.ws/$repo/os/$arch
-#Server = https://archlinux.thaller.ws/$repo/os/$arch
-#Server = http://archlinux.thelinuxnetworx.rocks/$repo/os/$arch
-#Server = https://archlinux.thelinuxnetworx.rocks/$repo/os/$arch
-#Server = http://mirror.thomaskilian.net/archlinux/$repo/os/$arch
-#Server = https://mirror.thomaskilian.net/archlinux/$repo/os/$arch
-#Server = http://ftp.tu-chemnitz.de/pub/linux/archlinux/$repo/os/$arch
-#Server = http://mirror.ubrco.de/archlinux/$repo/os/$arch
-#Server = https://mirror.ubrco.de/archlinux/$repo/os/$arch
-#Server = http://ftp.uni-bayreuth.de/linux/archlinux/$repo/os/$arch
-#Server = http://ftp.uni-hannover.de/archlinux/$repo/os/$arch
-#Server = http://ftp.uni-kl.de/pub/linux/archlinux/$repo/os/$arch
-#Server = http://mirror.united-gameserver.de/archlinux/$repo/os/$arch
-#Server = http://ftp.wrz.de/pub/archlinux/$repo/os/$arch
-#Server = https://ftp.wrz.de/pub/archlinux/$repo/os/$arch
-#Server = http://archlinux.eu.mirror.zoidplex.net/$repo/os/$arch
-#Server = https://archlinux.eu.mirror.zoidplex.net/$repo/os/$arch
-
-## Greece
-#Server = http://ftp.cc.uoc.gr/mirrors/linux/archlinux/$repo/os/$arch
-#Server = http://foss.aueb.gr/mirrors/linux/archlinux/$repo/os/$arch
-#Server = https://foss.aueb.gr/mirrors/linux/archlinux/$repo/os/$arch
-#Server = http://mirrors.myaegean.gr/linux/archlinux/$repo/os/$arch
-#Server = http://ftp.ntua.gr/pub/linux/archlinux/$repo/os/$arch
-#Server = http://ftp.otenet.gr/linux/archlinux/$repo/os/$arch
-
-## Hong Kong
-#Server = http://mirrors.kurnode.com/archlinux/$repo/os/$arch
-#Server = https://mirrors.kurnode.com/archlinux/$repo/os/$arch
-#Server = https://arch-mirror.wtako.net/$repo/os/$arch
-#Server = http://mirror.xtom.com.hk/archlinux/$repo/os/$arch
-#Server = https://mirror.xtom.com.hk/archlinux/$repo/os/$arch
-
-## Hungary
-#Server = http://ftp.energia.mta.hu/pub/mirrors/ftp.archlinux.org/$repo/os/$arch
-#Server = http://archmirror.hbit.sztaki.hu/archlinux/$repo/os/$arch
-#Server = http://arch.mthq.org/mirror/$repo/os/$arch
-#Server = https://arch.mthq.org/mirror/$repo/os/$arch
-
-## Iceland
-#Server = http://mirror.system.is/arch/$repo/os/$arch
-#Server = https://mirror.system.is/arch/$repo/os/$arch
-
-## India
-#Server = http://mirror.cse.iitk.ac.in/archlinux/$repo/os/$arch
-#Server = http://ftp.iitm.ac.in/archlinux/$repo/os/$arch
-#Server = https://ind.mirror.pkgbuild.com/$repo/os/$arch
-
-## Indonesia
-#Server = http://mirror.poliwangi.ac.id/archlinux/$repo/os/$arch
-#Server = http://suro.ubaya.ac.id/archlinux/$repo/os/$arch
-
-## Iran
-#Server = http://repo.iut.ac.ir/repo/archlinux/$repo/os/$arch
-#Server = http://linuxmirrors.ir/pub/archlinux/$repo/os/$arch
-#Server = https://linuxmirrors.ir/pub/archlinux/$repo/os/$arch
-#Server = http://repo.sadjad.ac.ir/arch/$repo/os/$arch
-#Server = https://repo.sadjad.ac.ir/arch/$repo/os/$arch
-
-## Ireland
-#Server = http://ftp.heanet.ie/mirrors/ftp.archlinux.org/$repo/os/$arch
-#Server = https://ftp.heanet.ie/mirrors/ftp.archlinux.org/$repo/os/$arch
-
-## Israel
-#Server = http://mirror.isoc.org.il/pub/archlinux/$repo/os/$arch
-#Server = https://archlinux.mivzakim.net/$repo/os/$arch
-
-## Italy
-#Server = https://archlinux.beccacervello.it/archlinux/$repo/os/$arch
-#Server = http://mi.mirror.garr.it/mirrors/archlinux/$repo/os/$arch
-#Server = http://mirrors.prometeus.net/archlinux/$repo/os/$arch
-#Server = http://archlinux.students.cs.unibo.it/$repo/os/$arch
-
-## Japan
-#Server = http://ftp.tsukuba.wide.ad.jp/Linux/archlinux/$repo/os/$arch
-#Server = http://ftp.jaist.ac.jp/pub/Linux/ArchLinux/$repo/os/$arch
-#Server = https://ftp.jaist.ac.jp/pub/Linux/ArchLinux/$repo/os/$arch
-#Server = https://jpn.mirror.pkgbuild.com/$repo/os/$arch
-#Server = http://archlinux.asia-east.mirror.zoidplex.net/$repo/os/$arch
-#Server = https://archlinux.asia-east.mirror.zoidplex.net/$repo/os/$arch
-
-## Kazakhstan
-#Server = http://mirror.neolabs.kz/archlinux/$repo/os/$arch
-
-## Latvia
-#Server = http://archlinux.koyanet.lv/archlinux/$repo/os/$arch
-
-## Lithuania
-#Server = http://mirrors.atviras.lt/archlinux/$repo/os/$arch
-#Server = https://mirrors.atviras.lt/archlinux/$repo/os/$arch
-
-## Luxembourg
-#Server = http://archlinux.mirror.root.lu/$repo/os/$arch
-
-## Macedonia
-#Server = http://arch.softver.org.mk/archlinux/$repo/os/$arch
-#Server = http://mirror.t-home.mk/archlinux/$repo/os/$arch
-#Server = https://mirror.t-home.mk/archlinux/$repo/os/$arch
-
-## Mexico
-#Server = https://mex.mirror.pkgbuild.com/$repo/os/$arch
-
-## Netherlands
-#Server = http://mirror.i3d.net/pub/archlinux/$repo/os/$arch
-#Server = https://mirror.i3d.net/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.koddos.net/archlinux/$repo/os/$arch
-#Server = https://mirror.koddos.net/archlinux/$repo/os/$arch
-#Server = http://mirror.ams1.nl.leaseweb.net/archlinux/$repo/os/$arch
-#Server = https://mirror.ams1.nl.leaseweb.net/archlinux/$repo/os/$arch
-#Server = http://mirror.neostrada.nl/archlinux/$repo/os/$arch
-#Server = https://mirror.neostrada.nl/archlinux/$repo/os/$arch
-#Server = http://mirror.netrouting.net/archlinux/$repo/os/$arch
-#Server = http://ftp.nluug.nl/os/Linux/distr/archlinux/$repo/os/$arch
-#Server = http://ftp.snt.utwente.nl/pub/os/linux/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirror.wearetriple.com/$repo/os/$arch
-#Server = https://archlinux.mirror.wearetriple.com/$repo/os/$arch
-
-## New Caledonia
-#Server = http://mirror.lagoon.nc/pub/archlinux/$repo/os/$arch
-#Server = http://archlinux.nautile.nc/archlinux/$repo/os/$arch
-#Server = https://archlinux.nautile.nc/archlinux/$repo/os/$arch
-
-## New Zealand
-#Server = http://mirror.smith.geek.nz/archlinux/$repo/os/$arch
-#Server = https://mirror.smith.geek.nz/archlinux/$repo/os/$arch
-
-## Norway
-#Server = http://mirror.archlinux.no/$repo/os/$arch
-#Server = http://archlinux.uib.no/$repo/os/$arch
-#Server = http://mirror.neuf.no/archlinux/$repo/os/$arch
-#Server = https://mirror.neuf.no/archlinux/$repo/os/$arch
-
-## Philippines
-#Server = http://mirror.rise.ph/archlinux/$repo/os/$arch
-
-## Poland
-#Server = http://arch.midov.pl/arch/$repo/os/$arch
-#Server = http://mirror.onet.pl/pub/mirrors/archlinux/$repo/os/$arch
-#Server = http://piotrkosoft.net/pub/mirrors/ftp.archlinux.org/$repo/os/$arch
-#Server = http://ftp.vectranet.pl/archlinux/$repo/os/$arch
-
-## Portugal
-#Server = http://glua.ua.pt/pub/archlinux/$repo/os/$arch
-#Server = https://glua.ua.pt/pub/archlinux/$repo/os/$arch
-#Server = http://ftp.rnl.tecnico.ulisboa.pt/pub/archlinux/$repo/os/$arch
-#Server = https://ftp.rnl.tecnico.ulisboa.pt/pub/archlinux/$repo/os/$arch
-
-## Qatar
-#Server = http://mirror.qnren.qa/archlinux/$repo/os/$arch
-
-## Romania
-#Server = http://mirror.archlinux.ro/archlinux/$repo/os/$arch
-#Server = http://archlinux.mirrors.linux.ro/$repo/os/$arch
-#Server = http://mirrors.m247.ro/archlinux/$repo/os/$arch
-#Server = http://mirrors.nav.ro/archlinux/$repo/os/$arch
-#Server = http://mirrors.nxthost.com/archlinux/$repo/os/$arch
-#Server = https://mirrors.nxthost.com/archlinux/$repo/os/$arch
-#Server = http://mirrors.pidginhost.com/arch/$repo/os/$arch
-#Server = https://mirrors.pidginhost.com/arch/$repo/os/$arch
-
-## Russia
-#Server = http://mirror.aur.rocks/$repo/os/$arch
-#Server = https://mirror.aur.rocks/$repo/os/$arch
-#Server = http://mirror.rol.ru/archlinux/$repo/os/$arch
-#Server = https://mirror.rol.ru/archlinux/$repo/os/$arch
-#Server = http://mirror.truenetwork.ru/archlinux/$repo/os/$arch
-#Server = http://mirror.yandex.ru/archlinux/$repo/os/$arch
-#Server = https://mirror.yandex.ru/archlinux/$repo/os/$arch
-#Server = http://archlinux.zepto.cloud/$repo/os/$arch
-
-## Serbia
-#Server = http://arch.petarmaric.com/$repo/os/$arch
-#Server = http://mirror.pmf.kg.ac.rs/archlinux/$repo/os/$arch
-
-## Seychelles
-#Server = http://mirror.mileswilson.net/arch/$repo/os/$arch
-#Server = https://mirror.mileswilson.net/arch/$repo/os/$arch
-
-## Singapore
-#Server = http://mirror.0x.sg/archlinux/$repo/os/$arch
-#Server = https://mirror.0x.sg/archlinux/$repo/os/$arch
-#Server = http://download.nus.edu.sg/mirror/arch/$repo/os/$arch
-#Server = https://download.nus.edu.sg/mirror/arch/$repo/os/$arch
-#Server = https://sgp.mirror.pkgbuild.com/$repo/os/$arch
-#Server = http://mirror.nus.edu.sg/archlinux/$repo/os/$arch
-
-## Slovakia
-#Server = http://mirror.lnx.sk/pub/linux/archlinux/$repo/os/$arch
-#Server = https://mirror.lnx.sk/pub/linux/archlinux/$repo/os/$arch
-#Server = http://tux.rainside.sk/archlinux/$repo/os/$arch
-
-## Slovenia
-#Server = http://archimonde.ts.si/archlinux/$repo/os/$arch
-#Server = https://archimonde.ts.si/archlinux/$repo/os/$arch
-
-## South Africa
-#Server = http://za.mirror.archlinux-br.org/$repo/os/$arch
-#Server = http://ftp.wa.co.za/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.is.co.za/mirror/archlinux.org/$repo/os/$arch
-#Server = http://mirror.wbs.co.za/archlinux/$repo/os/$arch
-
-## South Korea
-#Server = http://ftp.kaist.ac.kr/ArchLinux/$repo/os/$arch
-#Server = http://ftp.lanet.kr/pub/archlinux/$repo/os/$arch
-#Server = https://ftp.lanet.kr/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.premi.st/archlinux/$repo/os/$arch
-
-## Spain
-#Server = http://osl.ugr.es/archlinux/$repo/os/$arch
-#Server = http://ftp.rediris.es/mirror/archlinux/$repo/os/$arch
-
-## Sweden
-#Server = http://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch
-#Server = https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch
-#Server = http://archlinux.dynamict.se/$repo/os/$arch
-#Server = https://archlinux.dynamict.se/$repo/os/$arch
-#Server = http://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch
-#Server = https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch
-#Server = http://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch
-#Server = https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch
-#Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
-
-## Switzerland
-#Server = http://pkg.adfinis-sygroup.ch/archlinux/$repo/os/$arch
-#Server = https://pkg.adfinis-sygroup.ch/archlinux/$repo/os/$arch
-#Server = https://mirror.datacenterlight.ch/mirror/packages/archlinux/$repo/os/$arch
-#Server = http://mirror.puzzle.ch/archlinux/$repo/os/$arch
-#Server = https://mirror.puzzle.ch/archlinux/$repo/os/$arch
-
-## Taiwan
-#Server = http://archlinux.cs.nctu.edu.tw/$repo/os/$arch
-#Server = http://shadow.ind.ntou.edu.tw/archlinux/$repo/os/$arch
-#Server = http://ftp.tku.edu.tw/Linux/ArchLinux/$repo/os/$arch
-#Server = http://ftp.yzu.edu.tw/Linux/archlinux/$repo/os/$arch
-
-## Thailand
-#Server = http://mirror.adminbannok.com/archlinux/$repo/os/$arch
-#Server = https://mirror.adminbannok.com/archlinux/$repo/os/$arch
-#Server = http://mirror.kku.ac.th/archlinux/$repo/os/$arch
-#Server = https://mirror.kku.ac.th/archlinux/$repo/os/$arch
-#Server = http://mirror2.totbb.net/archlinux/$repo/os/$arch
-
-## Turkey
-#Server = http://ftp.linux.org.tr/archlinux/$repo/os/$arch
-#Server = http://mirror.veriteknik.net.tr/archlinux/$repo/os/$arch
-
-## Ukraine
-#Server = http://archlinux.ip-connect.vn.ua/$repo/os/$arch
-#Server = https://archlinux.ip-connect.vn.ua/$repo/os/$arch
-#Server = http://mirrors.nix.org.ua/linux/archlinux/$repo/os/$arch
-#Server = https://mirrors.nix.org.ua/linux/archlinux/$repo/os/$arch
-
-## United Kingdom
-#Server = http://mirror.bytemark.co.uk/archlinux/$repo/os/$arch
-#Server = https://mirror.bytemark.co.uk/archlinux/$repo/os/$arch
-#Server = http://mirrors.manchester.m247.com/arch-linux/$repo/os/$arch
-#Server = http://www.mirrorservice.org/sites/ftp.archlinux.org/$repo/os/$arch
-#Server = https://www.mirrorservice.org/sites/ftp.archlinux.org/$repo/os/$arch
-#Server = http://arch.serverspace.co.uk/arch/$repo/os/$arch
-#Server = http://archlinux.mirrors.uk2.net/$repo/os/$arch
-#Server = http://mirrors.ukfast.co.uk/sites/archlinux.org/$repo/os/$arch
-#Server = https://mirrors.ukfast.co.uk/sites/archlinux.org/$repo/os/$arch
-
-## United States
-#Server = http://mirrors.acm.wpi.edu/archlinux/$repo/os/$arch
-#Server = http://mirrors.advancedhosters.com/archlinux/$repo/os/$arch
-#Server = http://mirrors.aggregate.org/archlinux/$repo/os/$arch
-#Server = http://ca.us.mirror.archlinux-br.org/$repo/os/$arch
-#Server = http://il.us.mirror.archlinux-br.org/$repo/os/$arch
-#Server = http://archlinux.surlyjake.com/archlinux/$repo/os/$arch
-#Server = https://archlinux.surlyjake.com/archlinux/$repo/os/$arch
-#Server = http://arlm.tyzoid.com/$repo/os/$arch
-#Server = https://arlm.tyzoid.com/$repo/os/$arch
-#Server = http://mirror.as65535.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.cat.pdx.edu/archlinux/$repo/os/$arch
-#Server = http://mirror.cc.columbia.edu/pub/linux/archlinux/$repo/os/$arch
-#Server = http://centos.mbni.med.umich.edu/archlinux/$repo/os/$arch
-#Server = http://arch.mirror.constant.com/$repo/os/$arch
-#Server = https://arch.mirror.constant.com/$repo/os/$arch
-#Server = http://mirror.cs.pitt.edu/archlinux/$repo/os/$arch
-#Server = http://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/$arch
-#Server = http://distro.ibiblio.org/archlinux/$repo/os/$arch
-#Server = http://mirror.epiphyte.network/archlinux/$repo/os/$arch
-#Server = https://mirror.epiphyte.network/archlinux/$repo/os/$arch
-#Server = http://mirror.es.its.nyu.edu/archlinux/$repo/os/$arch
-#Server = http://mirrors.gigenet.com/archlinux/$repo/os/$arch
-#Server = http://mirror.grig.io/archlinux/$repo/os/$arch
-#Server = https://mirror.grig.io/archlinux/$repo/os/$arch
-#Server = http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.hackingand.coffee/arch/$repo/os/$arch
-#Server = https://mirror.hackingand.coffee/arch/$repo/os/$arch
-#Server = http://mirrors.kernel.org/archlinux/$repo/os/$arch
-#Server = https://mirrors.kernel.org/archlinux/$repo/os/$arch
-#Server = http://mirror.dal10.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = http://mirror.sfo12.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = http://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = https://mirror.dal10.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = https://mirror.sfo12.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = https://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.liquidweb.com/archlinux/$repo/os/$arch
-#Server = http://mirror.lty.me/archlinux/$repo/os/$arch
-#Server = https://mirror.lty.me/archlinux/$repo/os/$arch
-#Server = http://mirrors.lug.mtu.edu/archlinux/$repo/os/$arch
-#Server = https://mirrors.lug.mtu.edu/archlinux/$repo/os/$arch
-#Server = http://mirror.math.princeton.edu/pub/archlinux/$repo/os/$arch
-#Server = http://mirror.metrocast.net/archlinux/$repo/os/$arch
-#Server = http://mirror.kaminski.io/archlinux/$repo/os/$arch
-#Server = https://mirror.kaminski.io/archlinux/$repo/os/$arch
-#Server = http://mirrors.ocf.berkeley.edu/archlinux/$repo/os/$arch
-#Server = https://mirrors.ocf.berkeley.edu/archlinux/$repo/os/$arch
-#Server = http://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
-#Server = http://arch.mirrors.pair.com/$repo/os/$arch
-#Server = http://mirrors.rit.edu/archlinux/$repo/os/$arch
-#Server = https://mirrors.rit.edu/archlinux/$repo/os/$arch
-#Server = http://mirrors.rutgers.edu/archlinux/$repo/os/$arch
-#Server = https://mirrors.rutgers.edu/archlinux/$repo/os/$arch
-#Server = http://mirrors.sonic.net/archlinux/$repo/os/$arch
-#Server = https://mirrors.sonic.net/archlinux/$repo/os/$arch
-#Server = http://mirrors.sorengard.com/archlinux/$repo/os/$arch
-#Server = https://mirrors.sorengard.com/archlinux/$repo/os/$arch
-#Server = http://arch.mirror.square-r00t.net/$repo/os/$arch
-#Server = https://arch.mirror.square-r00t.net/$repo/os/$arch
-#Server = http://mirror.stephen304.com/archlinux/$repo/os/$arch
-#Server = https://mirror.stephen304.com/archlinux/$repo/os/$arch
-#Server = http://mirror.umd.edu/archlinux/$repo/os/$arch
-#Server = http://mirror.vtti.vt.edu/archlinux/$repo/os/$arch
-#Server = http://mirrors.xmission.com/archlinux/$repo/os/$arch
-#Server = http://mirror.yellowfiber.net/archlinux/$repo/os/$arch
-#Server = http://archlinux.us-central.mirror.zoidplex.net/$repo/os/$arch
-#Server = http://archlinux.us-east.mirror.zoidplex.net/$repo/os/$arch
-#Server = http://archlinux.us-west.mirror.zoidplex.net/$repo/os/$arch
-#Server = https://archlinux.us-central.mirror.zoidplex.net/$repo/os/$arch
-#Server = https://archlinux.us-east.mirror.zoidplex.net/$repo/os/$arch
-#Server = https://archlinux.us-west.mirror.zoidplex.net/$repo/os/$arch
-
-## Vietnam
-#Server = http://f.archlinuxvn.org/archlinux/$repo/os/$arch
-
+################################################################################
+################# Arch Linux mirrorlist generated by Reflector #################
+################################################################################
+
+# With:       reflector -p http -p https -f 10
+# When:       2018-06-02 09:53:01 UTC
+# From:       https://www.archlinux.org/mirrors/status/json/
+# Retrieved:  2018-06-02 09:48:32 UTC
+# Last Check: 2018-06-02 09:47:09 UTC
+
+Server = http://mirrors.nix.org.ua/linux/archlinux/$repo/os/$arch
+Server = https://mirrors.nix.org.ua/linux/archlinux/$repo/os/$arch
+Server = http://arch.midov.pl/arch/$repo/os/$arch
+Server = http://mirror.datacenter.by/pub/archlinux/$repo/os/$arch
+Server = http://mirrors.atviras.lt/archlinux/$repo/os/$arch
+Server = http://archlinux.koyanet.lv/archlinux/$repo/os/$arch
+Server = http://mirror.easyname.at/archlinux/$repo/os/$arch
+Server = http://mirrors.evowise.com/archlinux/$repo/os/$arch
+Server = http://archmirror.hbit.sztaki.hu/archlinux/$repo/os/$arch
+Server = http://mirror.reisenbauer.ee/archlinux/$repo/os/$arch
fix? [Y/n/q] n

It displays important messages from the last upgrade, then for each such file, it asks if I want to fix it, and fixes if confirmed. One of the points of the script is that it shows what changes you've made, not the difference between current and new version of a config file.

How do you handle this matter? If no better solution exists, and you have ideas how to make the script better, I'll try to find time to realize them.

Last edited by x-yuri (2018-07-19 17:07:19)

Offline

#2 2018-07-15 08:34:59

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Simplifying upgrades

Moving to Community Contributions

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#3 2018-07-15 11:38:58

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,863
Website

Re: Simplifying upgrades

Have you tried pacdiff?

Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#4 2018-07-15 11:45:11

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,597
Website

Re: Simplifying upgrades

Have you tried vimdiff?

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

Offline

#5 2018-07-15 11:54:29

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Simplifying upgrades

graysky wrote:

Have you tried vimdiff?

Which is exactly what pacdiff uses as it's diff back end smile

No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#6 2018-07-15 12:03:12

2ManyDogs
Forum Fellow
Registered: 2012-01-15
Posts: 4,645

Re: Simplifying upgrades

Suggestion: add more comments to your code.

http://www.catb.org/esr/writings/unix-k … odigy.html

Offline

#7 2018-07-15 22:21:20

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,412
Website

Re: Simplifying upgrades

In other words, this code looks like pure line noise. You're sure you didn't want to write perl?

Protip: start using pacutils to save ad-hoc code, e.g. paclog. Also start learning about best practices in Bash, for example you should never parse ls.

Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#8 2018-07-16 09:44:32

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Re: Simplifying upgrades

WorMzy wrote:

Have you tried pacdiff?

graysky wrote:

Have you tried vimdiff?

The thing with those two is that they show the difference between the current version and the new version from repository. In my script I compare the previous version from repository and the current version (the changes I've made), and apply that patch to the new version from repository.

2ManyDogs wrote:

Suggestion: add more comments to your code.

http://www.catb.org/esr/writings/unix-k … odigy.html

I've been doing it for a long time, so I'm well aware about future self. But I must have indeed been too succinct. And I do seek cooperation with other programmers. Updated the first post.

Alad wrote:

In other words, this code looks like pure line noise. You're sure you didn't want to write perl?

Protip: start using pacutils to save ad-hoc code, e.g. paclog. Also start learning about best practices in Bash, for example you should never parse ls.

You figured me out, Perl is the language that has impressed me the most. Also, I've always considered Basic to be a bad language (the one I don't want to deal with). But recently I had to launch a Visual Basic project, and I found myself feeling nostalgic smile

Regarding "paclog", thanks, I'll look into it. As for parsing "ls" output, the main point in the article is that files may contain new lines. Be it as it may, the more edge cases you consider, the less readable the code gets.

Consider the following case:

$ touch profile profile.prv-1 profile.prv-02

How do I find the number of the last config's backup without "ls"? The way I do it now is:

f=profile
re='((.*)\.prv-(.*))'
prv_files=`ls "$f".prv-* 2> /dev/null || true`
last_n=`printf "%s\n" "$prv_files" \
    | sed -E "s/$re/\3 \1/" \
    | sort -nk1 \
    | sed -E 's/\S+ //' \
    | tail -n 1 \
    | sed -E "s/$re/\3/"`

UPD Now that I think about it, "find" for the win.

Last edited by x-yuri (2018-07-16 12:51:01)

Offline

#9 2018-07-16 12:58:17

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,530
Website

Re: Simplifying upgrades

Holy pipelines!

You should never need to pipe through all those text processing tools.  You should definitely not need to pipe from sed to sed to sed.  One sed script could probably do just fine, though it cannot sort.  So perhaps one awk script would do the job.

I was going to try to rewrite all that as one sed or awk command, but even after reading it a few times, I haven't the foggiest idea what it is even supposed to do.  What is the goal of that pipeline?  You seem to just split and rearrange each line so you can sort on the first element in each line ... why not just sort on the 3rd element in the first place?

Wait, if you are doing that for a single base filename, that is ridiculously needlessly complex.  You just want the most recent one, right:

ls -c profile.prv* | head -1

That pipeline does indeed look perl-like.  I've heard talk therapy for perl-recovery has mixed results, but there are some new medications that are proving effective tongue

On a slight tangent, I think the suggestion that your original code needed comments was understandable, but I'd argue it was the wrong solution.  Comments are indeed valuable, but if someone else reading your code needs extensive comments to even have a clue what the code is doing, the code is bad in the first place.  Good code is (generally) self-explanatory.  A brief comment here or there when something out of the ordinary is being done is justified.  But rather than explaining complex code with complex comments, just simplify the code (simplify, refactor, use good variable and function names, etc).  Does my one line example above need any comments?  I'd guess you might not be familiar with the '-c' flag to ls, but if ithat is the case, you don't need a code comment: you can just read the man page.

Last edited by Trilby (2018-07-16 13:23:08)

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#10 2018-07-16 17:29:19

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Re: Simplifying upgrades

Switched to `find`, and simplified the backup part a bit. Thanks for suggestions.

Trilby wrote:

I was going to try to rewrite all that as one sed or awk command, but even after reading it a few times, I haven't the foggiest idea what it is even supposed to do.

Let's say we have a config file (/etc/<basename>). I'd like to have numbered backups. (/etc/<basename>.prv-1, /etc/<basename>.prv-2, and so on). To do that I:

* get a list of all backups (ls "$f".prv-* 2> /dev/null || true)
* add its number to the beginning (sed -E "s/$re/\3 \1/")
* sort by it (sort -nk1)
* remove it (sed -E 's/\S+ //')
* get the last file (tail -n 1)
* remove everything but the number (sed -E "s/$re/\3/")

re='((.*)\.prv-(.*))'
prv_files=`find -name "$f.prv-*"`
printf "%s\n" "$prv_files" \
	| sed -E "s/$re/\3 \1/" \
	| sort -nk1 \
	| sed -E 's/\S+ //' \
	| tail -n 1 \
	| sed -E "s/$re/\3/"

With your help I've found a simpler way, thanks.

Trilby wrote:

why not just sort on the 3rd element in the first place?

If I had known basename wouldn't include a hyphen (-), I would've:

$ find -name "$f.prv-*" | sort -t- -nk2
Trilby wrote:

That pipeline does indeed look perl-like.  I've heard talk therapy for perl-recovery has mixed results, but there are some new medications that are proving effective tongue

Don't take me wrong, I mostly do Ruby/Python development. Not that it matters, I think...

Trilby wrote:

Good code is (generally) self-explanatory.  A brief comment here or there when something out of the ordinary is being done is justified.

That's why my code is generally scarce on comments. A lot of comments are generally in the way of reading the code. But I admit my code is generally way too scarce on them.

Trilby wrote:

On a slight tangent, I think the suggestion that your original code needed comments was understandable, but I'd argue it was the wrong solution.  Comments are indeed valuable, but if someone else reading your code needs extensive comments to even have a clue what the code is doing, the code is bad in the first place.  Good code is (generally) self-explanatory.  A brief comment here or there when something out of the ordinary is being done is justified.  But rather than explaining complex code with complex comments, just simplify the code (simplify, refactor, use good variable and function names, etc).  Does my one line example above need any comments?  I'd guess you might not be familiar with the '-c' flag to ls, but if ithat is the case, you don't need a code comment: you can just read the man page.

I'm totally open to suggestions, the code changes as we speak.

Regarding `ls -c profile.prv* | head -1`, I'm well aware of the `-c` switch (actually, `ls` sorts by default), but I've got to sort numerically.

#!/usr/bin/env bash
set -eu

rm -rf 1   # !!!
mkdir 1
cd 1

prepare_files() {
    local f=$1
    touch "$f" "$f".prv-1 "$f".prv-10
}

Trilby() {
    local f=$1
    ls -c "$f".prv* | head -1 | sed -E 's/.*-//'
}

me1() {
    local f=$1
    re='((.*)\.prv-(.*))'
    prv_files=`find -name "$f.prv-*"`
    printf "%s\n" "$prv_files" \
        | sed -E "s/$re/\3 \1/" \
        | sort -nk1 \
        | sed -E 's/\S+ //' \
        | tail -n 1 \
        | sed -E "s/$re/\3/"
}

me2() {
    local f=$1
    prv_files=`find -name "$f.prv-*"`
    printf "%s\n" "$prv_files" \
        | sed -E 's/.*-//' \
        | sort -nk1 \
        | tail -n 1
}

prepare_files profile

echo --- Trilby
Trilby profile

echo --- me1
me1 profile

echo --- me2
me2 profile

Output:

--- Trilby
1
--- me1
10
--- me2
10

Last edited by x-yuri (2018-07-16 19:44:43)

Offline

#11 2018-07-16 18:33:51

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,530
Website

Re: Simplifying upgrades

Trilby() {
   find -name "$1.prv*" | awk -F- '{n[$NF]=1;} END {for (v in n); print v;}'
}

Last edited by Trilby (2018-07-16 18:34:55)

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#12 2018-07-16 19:43:09

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Re: Simplifying upgrades

Let me bring my current solution to your style:

me() {
    find -name "$1.prv-*" | sed -E 's/.*-//' | sort -nk1 | tail -n 1
}

I'd rather say my solution is more readable (straightforward). What would you as a person do to find the last backup? Sort, and take the last value. It's just that to sort I have to leave just the numbers.

Anyways, that's not the most complex part. The most complex one is the one nobody has touched so far:

last_upgrade_first_line=`egrep "Running 'pacman -Syu'" /var/log/pacman.log | tail -n 1`
re_line='^\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|running)'
re_next_line='\[[^]]+\] \[ALPM\] (upgraded|installed|running|transaction completed)'
q_last_upgrade_first_line=`quote_ere "$last_upgrade_first_line" /`
# last upgrade's output
last_upgrade=`sed -En "/$q_last_upgrade_first_line/,/Running 'pacman/p" /var/log/pacman.log`
printf "%s\n" "$last_upgrade" \
    | sed -En "
        :a /$re_line/ {           # if found install/upgrade line
            N                     # append next line into the pattern space

            # no extra output after install/upgrade line
            /\n$re_next_line/ {   # if next line is a next install/upgrade line
                s/.*\n//          # leave only next install/upgrade line
                b a               # jump to :a
            }

            # while not a next install/upgrade line
            :b N                      # append next line into the pattern space
            /\n$re_next_line/ ! b b   # else jump to :b

            # found next install/upgrade line
            /\n$re_next_line/ {       # if found next install/upgrade line
                # output lines accumulated so far and jump to the next iteration
                h                     # copy pattern space to hold space
                i \
$hr
                p                     # print pattern space
                i \

                x                     # exchange pattern and hold spaces
                s/.*\n//              # leave only next install/upgrade line
                b a                   # jump to :a
            }
        }
    "

Well, this could be rewritten in Ruby or Python, but that would be much more wordy, wouldn't it? And yes, that's sed's both a strong and weak point. You can express yourself in a more succinct way, but it takes some knowledge to understand it. What do you say? And yes, it's probably better have it this much commented.

Last edited by x-yuri (2018-07-16 19:43:41)

Offline

#13 2018-07-16 20:10:19

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,530
Website

Re: Simplifying upgrades

My "style" is not to be written on one line.  It is to be computationally efficient.

x-yuri wrote:

What would you as a person do to find the last backup?

I run into such thinking regularly from those learning to program/script, and the idea behind this question is one that just needs to die once and for all.  It is a completely wrong way of thinking if the goal is to write effecient reliable code.  Computers are useful because they can do things we can't - forcing them to do things as foolishly as we would is really a waste.

x-yuri wrote:

The most complex one is the one nobody has touched so far

I can't speak for others, but that is because as-is, it is unapproachable.  It does something very simple, yet the code is very complex.  There's just nowhere to start.

If you just want to work with the lines from the last upgrade, the following sed script will remove the need for all those variables, subshells, and RE munging:

:start
/Running 'pacman -Syu'/ {
        h
        :loop
        n
        H
        /Running 'pacman/bstart
        $bstart
        # do stuff here with lines from last update
        bloop
}
$ { x; p;  }

Last edited by Trilby (2018-07-16 20:20:08)

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#14 2018-07-19 14:07:58

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Re: Simplifying upgrades

Okay, let's take my version:

#!/usr/bin/env bash
set -eu

quote_ere() {
    local s=$1 delimiter=${2-}
    local re='\.|\*|\[|\^|\$|\\|\+|\?|\{|\||\('   # .*[^$\+?{|(
    if (( ${#delimiter} )); then
        re=$re'|\'$delimiter
    fi
    printf "%s\n" "$s" | sed -E 's/('"$re"')/\\\1/g'
}

hr=$(printf "=%.0s" $(seq 1 "$(tput cols)"))

last_upgrade_first_line=`egrep "Running 'pacman -Syu'" /var/log/pacman.log | tail -n 1`
re_line='\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|transaction completed|running)'
q_last_upgrade_first_line=`quote_ere "$last_upgrade_first_line" /`
last_upgrade=`sed -En "/$q_last_upgrade_first_line/,/Running 'pacman/p" /var/log/pacman.log`

printf "%s\n" "$last_upgrade" \
    | sed -En "
        :a /^$re_line/ {          # if found install/upgrade line
            N                     # append next line into the pattern space

            # no extra output after install/upgrade line
            /\n$re_line/ {        # if next line is also install/upgrade line
                s/.*\n//          # leave only next install/upgrade line
                b a               # jump to :a
            }

            # while not reached next install/upgrade line
            :b N                 # append next line into the pattern space
            /\n$re_line/ ! b b   # else jump to :b

            # found next install/upgrade line
            /\n$re_line/ {            # if found next install/upgrade line
                # output lines accumulated so far and jump to the next iteration
                h                     # copy pattern space to hold space
                i \
$hr
                p                     # print pattern space
                i \

                x                     # exchange pattern and hold spaces
                s/.*\n//              # leave only next install/upgrade line
                b a                   # jump to :a
            }
        }
    "
echo

It takes 0.87 sec (average across 10 runs) to complete. Next let's take yours that simply adds "- " to each line of the last upgrade:

#!/usr/bin/env bash
set -eu

sed -En "
    :start
    /Running 'pacman -Syu'/ {           # if found start of an upgrade
        h                               # copy first line to the hold space
        :loop
        n; H                            # copy next line to the hold space
        /Running 'pacman/ b start       # if found next 'running' line, branch to :start
        \$ b start                      # if reached EOF, branch to start
        x; s/(.*\n)(.*)/\1- \2/; x      # process last line in the hold space
        b loop                          # branch to :loop
    }
    $ {
        x                                   # move last upgrade's output
                                            # plus possibly extra line
                                            # from hold space to pattern space
        s/^(.*)/- \1/                       # process first line
        /\n[^\n]+Running 'pacman[^\n]+$/ {  # if there's a line
                                            # after last upgrade's output
            s/(.*)(\n.*)/\1/                # remove that line
        }
        p                                   # print result
    }
" /var/log/pacman.log

Now that takes 12.81 sec. As you suggested, I put processing at the "do stuff here" line. I hope you'll point me what I did wrong. Well, that probably has to do we me being forced to process not a single line, but a buffer that grows bigger and bigger. And let me make it clear here. All upgrades get processed. But only the last one is printed. Talk about computational efficiency.

Then, let's make it a little bit more approachable:

#!/usr/bin/env bash
set -eu

sed -En "
    :start
    /Running 'pacman -Syu'/ {           # if found start of an upgrade
        x; s/.*//; x                    # erase hold space

        :loop
        s/(.*\n|^)(.*)/\1- \2/          # process a line
        H                               # copy the line to the hold space
        $ b end                         # avoid exiting prematurely
        n                               # get next line
        /Running 'pacman/ ! b loop      # if not 'running' line, branch to :loop

        /Running 'pacman/ b start       # if 'running' line, branch to :start
    }
    :end
    $ {
        x           # move hold space to pattern space
        s/^\n//     # erase newline resulting from the first H to hold space
        p           # print pattern space
    }
" /var/log/pacman.log

Now we're basically back to normal (1.26 sec). Also it came to me, that using Ruby/Python wouldn't make it much wordy. So I decided to rewrite it in Perl smile Hey, I don't often get to make use of it.

#!/usr/bin/env bash
set -eu

quote_ere() {
    local s=$1 delimiter=${2-}
    local re='\.|\*|\[|\^|\$|\\|\+|\?|\{|\||\('   # .*[^$\+?{|(
    if (( ${#delimiter} )); then
        re=$re'|\'$delimiter
    fi
    printf "%s\n" "$s" | sed -E 's/('"$re"')/\\\1/g'
}

last_upgrade_first_line=`egrep "Running 'pacman -Syu'" /var/log/pacman.log | tail -n 1`
re_line='^\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|running)'
re_next_line='\[[^]]+\] \[ALPM\] (upgraded|installed|running|transaction completed)'
q_last_upgrade_first_line=`quote_ere "$last_upgrade_first_line" /`
last_upgrade=`sed -En "/$q_last_upgrade_first_line/,/Running 'pacman/p" /var/log/pacman.log`

script=$(cat <<'SCRIPT'
    use strict;
    use warnings;
    my $re = qr{\[[^]]+\] \[ALPM\] (transaction started|upgraded|installed|transaction completed|running)};
    my $r1;
    my $r = [];
    my $line_found;
    my $extra_output;
    while (<>) {
        if (/$re/) {
            if ($line_found && $extra_output) {
                push @$r, $r1 . $_;
            }
            $r1 = $_;
            $line_found = 1;
            $extra_output = 0;
        } elsif ($line_found) {
            $r1 .= $_;
            $extra_output = 1;
        }
    }
    foreach my $fragment (@$r) {
        print '=' x `tput cols`, "\n";
        print $fragment;
        print "\n";
    }
SCRIPT
)
perl -e "$script" <<< "$last_upgrade"

And this one takes 0.88 sec.

Trilby wrote:

I run into such thinking regularly from those learning to program/script, and the idea behind this question is one that just needs to die once and for all.

I'd call them new school. Let's take Moore's law. Computational power grows more and more. And those who's long since entered the field have a hard time to adapt.

The idea is pretty simple. What matters at the end of the day is whether user will notice the difference in efficiency or not. Also, programming is all about compromises. You can optimize a program to a point, where no noticeable increase in performance can be observed. But will you be able to maintain the program after that? And actually that might be someone else.

That's why I do all that munging. One step at a time. That way it's simpler to swallow (read). Divide and rule. A couple of simpler steps instead of one complex one.

Trilby wrote:

It does something very simple, yet the code is very complex.

So you think that putting my sed script into yours would make the code simpler? I hope you don't.

Offline

#15 2018-07-19 15:33:29

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,530
Website

Re: Simplifying upgrades

x-yuri wrote:

So you think that putting my sed script into yours would make the code simpler? I hope you don't.

No, and I never said anything of the sort.  I said your code is completely unreadable.  You have a huge function to define regular expressions rather than just using them properly in the first place.  You could argue this is just my problem, and perhaps it is - but I don't see anyone else chiming in to comment on your code.

x-yuri wrote:

I'd call them new school. Let's take Moore's law. Computational power grows more and more. And those who's long since entered the field have a hard time to adapt.

I guess then I'm just too old and stupid to adapt, so you should not want my advice anyways.  I gather most other arch users and others on these forums who have not had comments on your code are also just too old and stupid for your taste too.

So have fun being the brilliant young kid who is so much smarter than all of us.

Last edited by Trilby (2018-07-19 15:38:38)

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#16 2018-07-19 17:18:02

x-yuri
Member
Registered: 2013-01-06
Posts: 160

Re: Simplifying upgrades

Trilby wrote:

I said your code is completely unreadable.

But what else can I do? I've updated the code following most of the suggestions. We're basically left with, "The code is very complex. There's just nowhere to start." But I see no way to make it simpler. The perl version must be more readable, you can check it out, particularly in the original post. With supposed output.

Trilby wrote:

You have a huge function to define regular expressions rather than just using them properly in the first place.

A huge sed script? How do I use them properly?

Trilby wrote:

I guess then I'm just too old and stupid to adapt, so you should not want my advice anyways.

I certainly want. I didn't mean to offend you. Let me remind you, I was answering to, "I run into such thinking regularly from those learning to program/script." My line is what one would generally answer to that, isn't it? There are those who has just started to learn, and they just don't know any better. There are those who are in the field for quite a while, and can't overcome their habits.

I hope that's not you. But you do say suspicious things. Anyways, I'd rather switch from personalities to the code.

P.S. I sincerely apologize if I offended anybody. That wasn't meant.

Offline

Pages: 1

Board footer

Powered by FluxBB