pacman sync - slow with cntlm proxy

LordChaos73 · 2019-03-12 10:07:44

Hi,

I have to use cntlm as a proxy at work for my internet connection. Everything works fine, except pacman synchronizing with mirrors. The weird thing is, downloading software from these mirrors is lightning fast, it's just the synchronisation with pacman that's slow:

:: Synchronizing package databases...
 core                                                                              133,0 KiB  18,6M/s 00:00 [################################################################] 100%
error: failed retrieving file 'extra.db' from mirror.ams1.nl.leaseweb.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'extra.db' from mirror.i3d.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'community.db' from mirror.ams1.nl.leaseweb.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds--------------------]   1%
error: failed retrieving file 'community.db' from mirror.i3d.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds---------------------------------]   0%
error: failed retrieving file 'community.db' from mirror.koddos.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'community.db' from archlinux.mirror.pcextreme.nl : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds------------------]   0%
error: failed retrieving file 'community.db' from mirror.neostrada.nl : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed to update community (download library error)
 multilib is up to date
error: failed to synchronize all databases
Error installing repo packages

x1carbon6 :: ~/Downloads » env |grep -i proxy                                                                                                                                  1 ↵
http_proxy=http://127.0.0.1:3128
https_proxy=http://127.0.0.1:3128
ftp_proxy=http://127.0.0.1:3128
no_proxy=127.0.0.1,localhost,*.local,*.[redacted],*.[redacted],*.[redacted],*.[redacted]

Everything else seems to work fine, it's just the pacman syncing that always hangs.
I've tried different mirrors and protocols (http & https) but that doesn't seem to help.

Is pacman using rsync maybe to synchronize databases?

Here's the pacman debug logging:

x1carbon6 :: ~ » sudo pacman -Syy --debug                                                                                                                                      1 ↵
debug: pacman v5.1.3 - libalpm v11.0.3
debug: config: attempting to read file /etc/pacman.conf
debug: config: new section 'options'
debug: config: HoldPkg: pacman
debug: config: HoldPkg: glibc
debug: config: arch: x86_64
debug: config: SigLevel: Required
debug: config: SigLevel: DatabaseOptional
debug: config: LocalFileSigLevel: Optional
debug: config: new section 'core'
debug: config file /etc/pacman.conf, line 76: including /etc/pacman.d/mirrorlist
debug: config: new section 'extra'
debug: config file /etc/pacman.conf, line 79: including /etc/pacman.d/mirrorlist
debug: config: new section 'community'
debug: config file /etc/pacman.conf, line 85: including /etc/pacman.d/mirrorlist
debug: config: new section 'multilib'
debug: config file /etc/pacman.conf, line 94: including /etc/pacman.d/mirrorlist
debug: config: finished parsing /etc/pacman.conf
debug: setup_libalpm called
debug: option 'logfile' = /var/log/pacman.log
debug: option 'gpgdir' = /etc/pacman.d/gnupg/
debug: option 'hookdir' = /etc/pacman.d/hooks/
debug: option 'cachedir' = /var/cache/pacman/pkg/
debug: registering sync database 'core'
debug: database path for tree core set to /var/lib/pacman/sync/core.db
debug: "/var/lib/pacman/sync/core.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/core.db.sig could not be opened
debug: missing optional signature
debug: setting usage of 15 for core repository
debug: adding new server URL to database 'core': https://mirror.ams1.nl.leaseweb.net/archlinux/core/os/x86_64
debug: adding new server URL to database 'core': https://mirror.i3d.net/pub/archlinux/core/os/x86_64
debug: adding new server URL to database 'core': https://mirror.koddos.net/archlinux/core/os/x86_64
debug: adding new server URL to database 'core': https://archlinux.mirror.pcextreme.nl/core/os/x86_64
debug: adding new server URL to database 'core': https://mirror.neostrada.nl/archlinux/core/os/x86_64
debug: registering sync database 'extra'
debug: database path for tree extra set to /var/lib/pacman/sync/extra.db
debug: "/var/lib/pacman/sync/extra.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/extra.db.sig could not be opened
debug: missing optional signature
debug: setting usage of 15 for extra repository
debug: adding new server URL to database 'extra': https://mirror.ams1.nl.leaseweb.net/archlinux/extra/os/x86_64
debug: adding new server URL to database 'extra': https://mirror.i3d.net/pub/archlinux/extra/os/x86_64
debug: adding new server URL to database 'extra': https://mirror.koddos.net/archlinux/extra/os/x86_64
debug: adding new server URL to database 'extra': https://archlinux.mirror.pcextreme.nl/extra/os/x86_64
debug: adding new server URL to database 'extra': https://mirror.neostrada.nl/archlinux/extra/os/x86_64
debug: registering sync database 'community'
debug: database path for tree community set to /var/lib/pacman/sync/community.db
debug: "/var/lib/pacman/sync/community.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/community.db.sig could not be opened
debug: missing optional signature
debug: setting usage of 15 for community repository
debug: adding new server URL to database 'community': https://mirror.ams1.nl.leaseweb.net/archlinux/community/os/x86_64
debug: adding new server URL to database 'community': https://mirror.i3d.net/pub/archlinux/community/os/x86_64
debug: adding new server URL to database 'community': https://mirror.koddos.net/archlinux/community/os/x86_64
debug: adding new server URL to database 'community': https://archlinux.mirror.pcextreme.nl/community/os/x86_64
debug: adding new server URL to database 'community': https://mirror.neostrada.nl/archlinux/community/os/x86_64
debug: registering sync database 'multilib'
debug: database path for tree multilib set to /var/lib/pacman/sync/multilib.db
debug: "/var/lib/pacman/sync/multilib.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/multilib.db.sig could not be opened
debug: missing optional signature
debug: setting usage of 15 for multilib repository
debug: adding new server URL to database 'multilib': https://mirror.ams1.nl.leaseweb.net/archlinux/multilib/os/x86_64
debug: adding new server URL to database 'multilib': https://mirror.i3d.net/pub/archlinux/multilib/os/x86_64
debug: adding new server URL to database 'multilib': https://mirror.koddos.net/archlinux/multilib/os/x86_64
debug: adding new server URL to database 'multilib': https://archlinux.mirror.pcextreme.nl/multilib/os/x86_64
debug: adding new server URL to database 'multilib': https://mirror.neostrada.nl/archlinux/multilib/os/x86_64
:: Synchronizing package databases...
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/core/os/x86_64/core.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/core.db.part (wb)
downloading core.db...
debug: curl returned error 0 from transfer
debug: response code: 200
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/core/os/x86_64/core.db.sig
debug: maxsize: 16384
debug: opened tempfile for download: /var/lib/pacman/sync/core.db.sig.part (wb)
debug: curl returned error 63 from transfer
debug: failed retrieving file 'core.db.sig' from mirror.ams1.nl.leaseweb.net : Maximum file size exceeded
debug: "/var/lib/pacman/sync/core.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/core.db.sig could not be opened
debug: missing optional signature
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/extra/os/x86_64/extra.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/extra.db.part (wb)
downloading extra.db...
debug: curl returned error 28 from transfer
error: failed retrieving file 'extra.db' from mirror.ams1.nl.leaseweb.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: url: https://mirror.i3d.net/pub/archlinux/extra/os/x86_64/extra.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/extra.db.part (wb)
downloading extra.db...
debug: curl returned error 0 from transfer
debug: response code: 200
debug: url: https://mirror.i3d.net/pub/archlinux/extra/os/x86_64/extra.db.sig
debug: maxsize: 16384
debug: opened tempfile for download: /var/lib/pacman/sync/extra.db.sig.part (wb)
debug: curl returned error 0 from transfer
debug: response code: 404
debug: "/var/lib/pacman/sync/extra.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/extra.db.sig could not be opened
debug: missing optional signature
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from mirror.ams1.nl.leaseweb.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: url: https://mirror.i3d.net/pub/archlinux/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from mirror.i3d.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: url: https://mirror.koddos.net/archlinux/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from mirror.koddos.net : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: url: https://archlinux.mirror.pcextreme.nl/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from archlinux.mirror.pcextreme.nl : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: url: https://mirror.neostrada.nl/archlinux/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from mirror.neostrada.nl : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: failed to sync db: download library error
error: failed to update community (download library error)
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/multilib/os/x86_64/multilib.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/multilib.db.part (wb)
downloading multilib.db...
debug: curl returned error 0 from transfer
debug: response code: 200
debug: url: https://mirror.ams1.nl.leaseweb.net/archlinux/multilib/os/x86_64/multilib.db.sig
debug: maxsize: 16384
debug: opened tempfile for download: /var/lib/pacman/sync/multilib.db.sig.part (wb)
debug: curl returned error 63 from transfer
debug: failed retrieving file 'multilib.db.sig' from mirror.ams1.nl.leaseweb.net : Maximum file size exceeded
debug: "/var/lib/pacman/sync/multilib.db.sig" is not readable: No such file or directory
debug: sig path /var/lib/pacman/sync/multilib.db.sig could not be opened
debug: missing optional signature
error: failed to synchronize all databases
debug: unregistering database 'local'
debug: unregistering database 'core'
debug: unregistering database 'extra'
debug: unregistering database 'community'
debug: unregistering database 'multilib'
x1carbon6 :: ~ »

EDIT: added debug logging

Last edited by LordChaos73 (2019-03-12 10:17:48)

carbolymer · 2019-03-15 09:13:45

Is wget reporting also slow downloads from mirrors you mentioned? e.g. can you try:

wget https://mirror.neostrada.nl/archlinux/extra/os/x86_64/extra.db

LordChaos73 · 2019-03-19 09:50:00

carbolymer wrote:

Is wget reporting also slow downloads from mirrors you mentioned? e.g. can you try:
wget https://mirror.neostrada.nl/archlinux/extra/os/x86_64/extra.db

Yeah, it's slow as hell (for every mirror). It's probably a proxy issue.

x1carbon6 :: ~/Downloads » wget https://mirror.neostrada.nl/archlinux/extra/os/x86_64/extra.db
--2019-03-19 10:43:42--  https://mirror.neostrada.nl/archlinux/extra/os/x86_64/extra.db
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [application/x-tar]
Saving to: ‘extra.db’

extra.db                                         [               <=>                                                                            ]   6,61M   643KB/s    in 11s     

2019-03-19 10:43:58 (643 KB/s) - Read error at byte 6926848 (The TLS connection was non-properly terminated.).Retrying.

Slithery · 2019-03-19 09:52:50

Is this only an issue with https mirrors?

LordChaos73 · 2019-03-19 10:15:20

Slithery wrote:

Is this only an issue with https mirrors?

Euhmm no,

debug: url: http://archlinux.mirror.wearetriple.com/community/os/x86_64/community.db
debug: maxsize: 26214400
debug: opened tempfile for download: /var/lib/pacman/sync/community.db.part (wb)
downloading community.db...
debug: curl returned error 28 from transfer
error: failed retrieving file 'community.db' from archlinux.mirror.wearetriple.com : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
debug: failed to sync db: download library error
error: failed to update community (download library error)

I'm in contact with my clients' proxy specialists to see if we can fix this somehow.

EDIT: gave the wrong answer here

Last edited by LordChaos73 (2019-03-19 12:53:21)

Trilby · 2019-03-19 12:39:31

For the time being, just switch to http mirrors. SSL on the line doesn't really add any security benefit for package and package database downloads - it can add a small degree of privacy if you are really worried that an attacker might monitor which packages you are downloading, but that's all the information they could get.

EDIT: nevermind - this post was based on incorrect information that has been revised.

Last edited by Trilby (2019-03-19 13:02:18)

LordChaos73 · 2019-03-19 12:52:37

Trilby wrote:

For the time being, just switch to http mirrors. SSL on the line doesn't really add any security benefit for package and package database downloads - it can add a small degree of privacy if you are really worried that an attacker might monitor which packages you are downloading, but that's all the information they could get.

Yeah, I'm aware of this and you have a valid point. However, as you can see above switching to http doesn't help at all.

Trilby · 2019-03-19 13:03:43

What had you previously shown indicating that http mirrors were fine? Is this a sporadic issue? Or did you test with wget which worked well with http mirrors but pacman isnt? Or where did that now-deleted information come from?

LordChaos73 · 2019-03-19 13:09:14

Trilby wrote:

What had you previously shown indicating that http mirrors were fine? Is this a sporadic issue? Or did you test with wget which worked well with http mirrors but pacman isnt? Or where did that now-deleted information come from?

I did show http mirrors were _not_ fine, but I answered the question if it was an https issue only, incorrectly.
So it's not a sporadic issue and it affects _both_ http and https connections.

Last edited by LordChaos73 (2019-03-19 13:10:34)

Arch Linux

#1 2019-03-12 10:07:44

pacman sync - slow with cntlm proxy

#2 2019-03-15 09:13:45

Re: pacman sync - slow with cntlm proxy

#3 2019-03-19 09:50:00

Re: pacman sync - slow with cntlm proxy

#4 2019-03-19 09:52:50

Re: pacman sync - slow with cntlm proxy

#5 2019-03-19 10:15:20

Re: pacman sync - slow with cntlm proxy

#6 2019-03-19 12:39:31

Re: pacman sync - slow with cntlm proxy

#7 2019-03-19 12:52:37

Re: pacman sync - slow with cntlm proxy

#8 2019-03-19 13:03:43

Re: pacman sync - slow with cntlm proxy

#9 2019-03-19 13:09:14

Re: pacman sync - slow with cntlm proxy

Board footer