Download speed within terminal is extremely slow

Tintin5371 · 2020-09-04 13:11:37

Arch served me really well over the past years but recently when I formatted my computer to install Arch again with AwesomeWM I saw that the download speed within pacman is really slow. This is not just in pacman rather in yay too but I won't take yay into consideration since it is not supported by Arch. Before my format I used to get 2-3 Mbps everytime I used to download using pacman but now the speed is around 200-300 Kbps and sometime fluctuating to 700 Kbps. I know such question has previously been asked and even answered but most of such answers contain people solving the issue after the update but in my case no update actually worked.

This is the behavior that I witnessed. I checked my internet speed using Speedtest and I got

 
Download Speed: 29.57 Mbps 
Upload Speed: 20.73 Mbps

When I checked the same using

$ speedtest-cli --simple

Ping: 18.158 ms
Download: 29.21 Mbit/s
Upload: 9.68 Mbit/s

I checked the same using wget command

$ wget --output-document=/dev/null https://speed.hetzner.de/100MB.bin

and my speed was around 100Kbps.

I configured my network using NetworkManager and it doesn't give me any issues when it comes to downloading from the net or using anyother application like qbittorrent. My kernel version is

$ uname -a
Linux midori 5.8.5-arch1-1 #1 SMP PREEMPT Thu, 27 Aug 2020 18:53:02 +0000 x86_64 GNU/Linux

Information about my network cards

sudo lshw -C network
[sudo] password for tintin: 
  *-network                 
       description: Ethernet interface
       product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: enp2s0
       version: 10
       serial: 8c:16:45:d4:13:c3
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8168 driverversion=8.048.03-NAPI latency=0 link=no multicast=yes port=twisted pair
       resources: irq:128 ioport:3000(size=256) memory:b2204000-b2204fff memory:b2200000-b2203fff
  *-network
       description: Wireless interface
       product: QCA9377 802.11ac Wireless Network Adapter
       vendor: Qualcomm Atheros
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: wlp3s0
       version: 31
       serial: 30:d1:6b:ea:6d:17
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=ath10k_pci driverversion=5.8.5-arch1-1 firmware=WLAN.TF.2.1-00021-QCARMSWP-1 ip=192.168.1.103 latency=0 link=yes multicast=yes wireless=IEEE 802.11
       resources: irq:129 memory:b2000000-b21fffff

So what am I doing wrong in here or am I missing something that is causing this issue? Any sort of information in regards to this would be really appreciated.

Last edited by Tintin5371 (2020-09-04 15:26:09)

CarbonChauvinist · 2020-09-04 13:23:43

Looks like you've checked everything except for the mirror(s) you're using.

Use a tool like reflector to pick better mirrors?

eschwartz · 2020-09-04 13:38:02

Tintin5371 wrote:

This is the behavior that I witnessed. I checked my internet speed using Speedtest and I got
 
Download Speed: 29.57 Mbps 
Upload Speed: 20.73 Mbps
When I checked the same using
$ speedtest-cli --simple

Ping: 18.158 ms
Download: 29.21 Mbit/s
Upload: 9.68 Mbit/s
I checked the same using wget command
$ wget --output-document=/dev/null https://speed.hetzner.de/100MB.bin
and my speed was around 100Kbps.

I"m confused. So you're getting speed tests that report 30mb download speeds but when you try to download speed test files using wget, it's really slow. What does this have to do with pacman?

Tintin5371 · 2020-09-04 15:08:32

To answer @CarbonChauvinist, I already used

reflector

to list out the most speedy servers. Sorry that I forgot to mention that.

Now to answer @eschwartz, it has to do with pacman since this same situation repeats itself with pacman as well. Within pacman I'm getting 100-200 kbps at all time which downloading packages through it. This same situation repeats itself with yay, wget and even git clone though I'm not concerning about that in this post atleast.

Hope I was able to clear the doubts in regards to my issue. If you have any more doubts then please revert back to me

eschwartz · 2020-09-04 15:12:39

Tintin5371 wrote:

[...] it has to do with pacman since [...]. This same situation repeats itself with yay, wget and even git clone though I'm not concerning about that in this post atleast.

Thank you for clarifying that in fact this does NOT have to do with pacman.

pacman is merely one of a number of programs affected by your issue. Your issue is NOT pacman.

Please fix the thread title to more clearly describe your issue.

Trilby · 2020-09-04 15:13:41

Tintin5371 wrote:

I already used `reflector` to list out the most speedy servers.

And that will do exactly nothing for you. You need to replace your mirrorlist with those that reflector lists.

As for comparing speedtest, speedtest-cli, and that wget command output - those are apples and oranges. That data is meaningless. Speedtest will use a different server each time and gives wildly different results on each use. The hetzner test is using the same remote server (notwithstanding any load balancing done at their end) but even there I get wildly ranging results that do not reflect local connection speed, but rather traffic on the wider internet between the local network and remote server.

Last edited by Trilby (2020-09-04 15:17:46)

eschwartz · 2020-09-04 15:15:33

Changing the mirrorlist won't fix "git clone" or "wget".

It also won't fix pacman, because if git clone and wget have issues too, the issue is obviously not the mirrorlist. It's on the user's end.

Trilby · 2020-09-04 15:19:13

eschwartz, please see my edit. I don't even take those "speedtest" results with a grain of salt ... I take them with a beach of the Great Salt Lake: they're meaningless. But pacman running slower does mean something and can be troubleshot (or troubleshooted?)

Last edited by Trilby (2020-09-04 15:19:41)

Tintin5371 · 2020-09-04 15:34:13

Thanks for your clarification @Trilby. Yes I replaced my mirrorlist with the ones listed through reflector. I followed the Wiki honestly on this part.

But as you mentioned the mirrorlist doesn't affect the wget or git clone so what's the thing that I specifically need to look for to solve the issue. Though I was already aware of the fact that git clone or wget doesn't concerns itself with mirrorlist so I didn't mention it previously. I mainly mentioned pacman in the title previously since I thought if I could figure out the reason for soggy speed in that then maybe I would be able to figure out the solution for other commands as well.

Last edited by Tintin5371 (2020-09-04 15:47:16)

Trilby · 2020-09-04 15:45:47

Do git clone and wget operations seem notably slower to you, or are you just using the speedtest metrics as an indicator?

Tintin5371 · 2020-09-04 15:48:50

No no they are actually slow when I try to download through them. Speedtest results are just supplementary to substantiate my claims

seth · 2020-09-04 16:18:11

Let's first see your actual network config:

ip a; ip r

like qbittorrent

Please rule out any sideload - your download speed could also suffer from lagging ACK packages due to high up-load.

WIld guess: "pacman -Qi linux-firmware"

GaKu999 · 2020-09-04 16:34:09

My little grain of salt.

You are comparing single transfers with plain speedtest-cli, it by default uses multiple connections per file...
Attempt ‘speedtest-cli —single —simple’

I for example have quite the variation from normal to single...
Side note my network is trash today so I can’t give reliable results, peak time?
With this pandemic it’s not a surprise...

Another thing to consider is the router, before I had a trashy router that caused a lot of trouble, sometimes it’s the router fault...

And post what @seth asked to nail down the rest.

Tintin5371 · 2020-09-04 17:31:37

In reply to @GaKu999, I don't know why would you comment something which has already been conveyed before in this thread. Your comment has no relevance to my issue in whatsoever. Can you try to read the issue description once again. And please use code tag while mentioning a code.

In reply to @seth,

$ ip a; ip r
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 8c:16:45:d4:13:c3 brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 30:d1:6b:ea:6d:17 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.103/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0
       valid_lft 6169sec preferred_lft 6169sec
    inet6 fe80::f624:d74e:4d1a:eb6a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
default via 192.168.1.1 dev wlp3s0 proto dhcp metric 600 
192.168.1.0/24 dev wlp3s0 proto kernel scope link src 192.168.1.103 metric 600

and

$ pacman -Qi linux-firmware
Name            : linux-firmware
Version         : 20200817.7a30af1-1
Description     : Firmware files for Linux
Architecture    : any
URL             : https://git.kernel.org/?p=linux/kernel/git/firmware/linux-firmware.git;a=summary
Licenses        : GPL2  GPL3  custom
Groups          : None
Provides        : None
Depends On      : None
Optional Deps   : None
Required By     : None
Optional For    : linux
Conflicts With  : None
Replaces        : None
Installed Size  : 562.54 MiB
Packager        : Laurent Carlier <lordheavym@gmail.com>
Build Date      : Thu 20 Aug 2020 10:13:39 AM IST
Install Date    : Sun 30 Aug 2020 04:58:39 PM IST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Last edited by Tintin5371 (2020-09-04 17:35:10)

seth · 2020-09-04 17:36:17

Ok, firmware is installed - so dead end there.
Standard MTU, so that's not it either.

Did you ensure there's no side-traffic (notably bittorrent)?
Can you try the wired interface? Does it yield different results?

Tintin5371 · 2020-09-04 17:50:37

Yes I ensured that there are no other applications running on my machine that could consume a huge amount of data. No qbittorrent or browser based downloads or anything.

As you suggested I also tried to use Ethernet cable to check. Sadly, the result hasn't changed at all. It's the same

GaKu999 · 2020-09-04 18:11:41

The server you used to test with wget is painfully unresponsive, post the cmdline you used with reflector to create your mirrorlist, or the contents of reflector.conf. And also the mirrorlist itself.

Last edited by GaKu999 (2020-09-04 18:16:08)

Tintin5371 · 2020-09-04 18:42:52

As it has already been confirmed within this tread that the mirrorlist isn't at fault so I don't under why would you like to see my reflector.conf or anything in relation to that. And the server that I mentioned won't respond if you change the value at the end of link (like changing 100MB to something like 10MB). Moreover, this was just to show the speed with wget which is the same as that of pacman or git clone

I would really love to provide you any output whatsoever until you can actually establish the reason behind asking so.

seth · 2020-09-04 18:50:28

Did you try to download the testfile w/ the browser (which should be good according to the speedtest run)
Also re-attempt it afterwards w/ wget in case your ISP holds a cache.

Tintin5371 · 2020-09-05 02:21:09

Yeah, I tried to do that. Through browser it was around 1.4 - 1.7 Mbps and through wget it was around 400 Kbps. So the problem still persists

seth · 2020-09-05 06:28:00

But that's a far cry from the previously claimed disparity - and hetzner is MUCH faster.
We can't say "everything normal in the browser but wget is bad" - btw. pacman defaults to curl, so if you didn't change that to wget, it's not even the implementation.

Do you reside in the cheese and chocolate exporting nation suggested by your email address?
Do you have a second system so you can test the LAN speed?

Tintin5371 · 2020-09-05 07:17:58

Yes, I do have a second system but it is out of order at the moment so can't test my speed on that. I tried today again through browser it was 2.3 Mbps but within wget it was around 200 - 300 Kbps. Either way I don't think this issues is going any near to being resolved.

And as far as my email ID goes I simply use since I like their service more than Gmail. I'm in India at the moment

Last edited by Tintin5371 (2020-09-05 07:18:45)

seth · 2020-09-05 07:44:12

In order to resolve the issue we need to understand where it actually is.
We've ruled out the NIC and the test results are kinda all over the place and not really tied to any implementation. Let alone that the pure idea that "using the console makes my internet slow" is nuts.

So let's localize your download that a bit.

wget -O /dev/null http://speedtest.mumbai1.linode.com/100MB-mumbai1.bin

Tintin5371 · 2020-09-05 08:30:31

With that command

wget -O /dev/null http://speedtest.mumbai1.linode.com/100MB-mumbai1.bin

the download speed exponentially increased to 2 Mbps. But the issue persists when downloading with pacman or using git clone. I checked it just now.

And the title is simply a situational description of what is going on which is in no way a speculation or a claim

Last edited by Tintin5371 (2020-09-05 09:21:33)

seth · 2020-09-05 11:25:46

Well, factor 10.
There however seems a routing issue to at least European servers.
The browser downloads might actually indeed somewhat cover this w/ simultaneous connetions (we're still factor 10 below the original speedtest)

a) how fast does the last file download in a browser
b) if you run 5 wget instances at the same time, does the combined speed exceed 2Mbps
c) git clone from where and what's your pacman mirror? (I don't think reflector accounts for traffic shenanigans)

Does India have sth. like the great firewall (ie. that prevents you from learning about Winnie the Pooh in China… just with a different focus)

Arch Linux

#1 2020-09-04 13:11:37

Download speed within terminal is extremely slow

#2 2020-09-04 13:23:43

Re: Download speed within terminal is extremely slow

#3 2020-09-04 13:38:02

Re: Download speed within terminal is extremely slow

#4 2020-09-04 15:08:32

Re: Download speed within terminal is extremely slow

#5 2020-09-04 15:12:39

Re: Download speed within terminal is extremely slow

#6 2020-09-04 15:13:41

Re: Download speed within terminal is extremely slow

#7 2020-09-04 15:15:33

Re: Download speed within terminal is extremely slow

#8 2020-09-04 15:19:13

Re: Download speed within terminal is extremely slow

#9 2020-09-04 15:34:13

Re: Download speed within terminal is extremely slow

#10 2020-09-04 15:45:47

Re: Download speed within terminal is extremely slow

#11 2020-09-04 15:48:50

Re: Download speed within terminal is extremely slow

#12 2020-09-04 16:18:11

Re: Download speed within terminal is extremely slow

#13 2020-09-04 16:34:09

Re: Download speed within terminal is extremely slow

#14 2020-09-04 17:31:37

Re: Download speed within terminal is extremely slow

#15 2020-09-04 17:36:17

Re: Download speed within terminal is extremely slow

#16 2020-09-04 17:50:37

Re: Download speed within terminal is extremely slow

#17 2020-09-04 18:11:41

Re: Download speed within terminal is extremely slow

#18 2020-09-04 18:42:52

Re: Download speed within terminal is extremely slow

#19 2020-09-04 18:50:28

Re: Download speed within terminal is extremely slow

#20 2020-09-05 02:21:09

Re: Download speed within terminal is extremely slow

#21 2020-09-05 06:28:00

Re: Download speed within terminal is extremely slow

#22 2020-09-05 07:17:58

Re: Download speed within terminal is extremely slow

#23 2020-09-05 07:44:12

Re: Download speed within terminal is extremely slow

#24 2020-09-05 08:30:31

Re: Download speed within terminal is extremely slow

#25 2020-09-05 11:25:46

Re: Download speed within terminal is extremely slow

Board footer