[SOLVED] pacman insanely slow

shining · 2008-08-13 20:29:45

ekerazha wrote:

I was replying to zenlord, you should read more carefully.

You are the one who should read more carefully :
http://bbs.archlinux.org/viewtopic.php? … 28#p406228
http://bbs.archlinux.org/viewtopic.php? … 29#p406229

Please go back to the topic now, if you have anything useful to add.

ekerazha · 2008-08-13 20:33:22

shining wrote:

ekerazha wrote:
I was replying to zenlord, you should read more carefully.
You are the one who should read more carefully :
http://bbs.archlinux.org/viewtopic.php? … 28#p406228
http://bbs.archlinux.org/viewtopic.php? … 29#p406229
Please go back to the topic now, if you have anything useful to add.

I was replying the the second part indeed, as there are known issues (slowness) using pacman on XFS... and you should know this.

shining · 2008-08-13 20:34:07

void.pointer wrote:

Okay guys let's calm down. The main issue seems to be a networking issue, as I've already speculated in earlier posts. There's no need to get into a war about file system performance. I appreciate everyone's help, I really do, but I don't want to see anything bad happen! Let's here more suggestions on solving the real problem, if possible. So far you guys are doing great and I really appreciate your help!

There has been a number of suggestions that I didn't see you answering :
1) how many mirrors did you try
2) if the issue happens using both ftp and http mirrors
3) if firefox is slow as well, accessing the same files, for example :
ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz
4) same with wget, how long does it take for resolving :
wget ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz

If 3 and 4 works fine, is pacman still much slower using this mirror?
(it is not recommended to use this mirror but well you already provided a dig output for it, and it shouldn't cause too much load to just download the small core.db.tar.gz file)

5) did you disable ipv6?

edit:
6) try other dns servers like opendns : https://www.opendns.com/start . just put the ip found there in /etc/resolv.conf

Last edited by shining (2008-08-13 20:52:43)

fwojciec · 2008-08-13 20:47:48

I experienced something similar recently, when I changed my ISP and the whole wifi setup, pacman would take forever just to confirm that repos are up to date. I think I solved it by setting my computers to use OpenDNS.

finferflu · 2008-08-13 21:03:43

ekerazha wrote:

I was replying to zenlord, you should read more carefully.
However... the unefficiency of the pacman DB is a date-of-fact, as already discussed and proven many many times (here and on the ML).

Please, let's try to keep this discussion on topic and useful. As it would be very bad to lock this discussion, since there is some issue going on, we would be forced to delete the inappropriate posts.

kakabaratruskia · 2008-08-13 22:10:37

I had a similar problem. Using an http repo fixes it, since it doesn't check if your db is up to date.

void.pointer · 2008-08-13 22:33:33

shining wrote:

There has been a number of suggestions that I didn't see you answering :
1) how many mirrors did you try

I'm using whatever is default on archlinux. I believe there's a full list in /etc/pacman.d/mirrorlist

shining wrote:

2) if the issue happens using both ftp and http mirrors

As I said, I'm using the default. What http mirrors should I use? Is it as simple as going into the mirror list and renaming all 'ftp' to 'http'?

shining wrote:

3) if firefox is slow as well, accessing the same files, for example :
ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz

I can't test this on my linux machine in question because I do not have a desktop environment installed. I don't really want to install one either. I'll try this on my windows machine when I get home. It's on the same LAN, so if this is a DNS issue I would expect the problem to show itself on my windows machine too. I'll let you know ASAP.

::EDIT::
I just tried this on my windows machine and it's pretty quick. It took less than a second to get a download request dialog in firefox when I visited the link you provided.

shining wrote:

4) same with wget, how long does it take for resolving :
wget ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz

[robert@server ~]$ time wget ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz
--2008-08-13 22:24:26--  ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz
           => `core.db.tar.gz'
Resolving ftp.archlinux.org... 209.85.41.145, 209.85.41.144
Connecting to ftp.archlinux.org|209.85.41.145|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /core/os/i686 ... done.
==> SIZE core.db.tar.gz ... 32335
==> PASV ... done.    ==> RETR core.db.tar.gz ... done.
Length: 32335 (32K)

100%[==============================================================>] 32,335       168K/s   in 0.2s

2008-08-13 22:24:37 (168 KB/s) - `core.db.tar.gz' saved [32335]


real    0m11.130s
user    0m0.003s
sys     0m0.007s

Above, the "Resolving ftp.archlinux.org..." is what took the longest. I would say it probably took 9-10 seconds out of the 11 seconds total.

shining wrote:

If 3 and 4 works fine, is pacman still much slower using this mirror?
(it is not recommended to use this mirror but well you already provided a dig output for it, and it shouldn't cause too much load to just download the small core.db.tar.gz file)

I still have to answer #3 later, but I'll answer this one once I have the results. Could you explain what you mean please? I am not sure I understand this question. What do you mean by "this mirror"?

shining wrote:

5) did you disable ipv6?

Yes. I followed the instructions here.

shining wrote:

edit:
6) try other dns servers like opendns : https://www.opendns.com/start . just put the ip found there in /etc/resolv.conf

I wasn't aware I was hosting a DNS server? I'm a bit confused about this suggestion. What is this supposed to help?
::EDIT::
Sorry, I feel stupid. I know what you mean now. Have my machine *use* a different DNS server. Below is what my /etc/resolv.conf looks like. Note it is delegating to my router as the DNS server (Which is what my windows machines do, I believe. My router ultimately knows the real DNS servers). 'router' here is the domain name of my router. It should map to 192.168.10.1.

 # Generated by dhcpcd for interface eth0
 search router
 nameserver 192.168.10.1

I can't manually change this file per your suggestion, since it is auto-generated when I reboot and also when I restart /etc/rc.d/network.
Sorry for not being able to answer some of these questions. Again, I'll answer them when I get home. I'm currently at work and I cannot test some of these things.

Last edited by void.pointer (2008-08-13 23:33:40)

zenlord · 2008-08-13 22:59:07

OK, shoot me if you will. Tested the dig and time wget-commands and all was very well.

So I looked inside the mirrorlist and found out that I had all servers uncommented. Changing this and leaving only 1 open (belnet in Belgium), made the pacman-sync run much smoother.

Sorry to have wasted some of your time - I hope this post will help some other guy once...

Zl.

void.pointer · 2008-08-13 23:31:43

zenlord wrote:

OK, shoot me if you will. Tested the dig and time wget-commands and all was very well.
So I looked inside the mirrorlist and found out that I had all servers uncommented. Changing this and leaving only 1 open (belnet in Belgium), made the pacman-sync run much smoother.
Sorry to have wasted some of your time - I hope this post will help some other guy once...
Zl.

You're not wasting anyone's time. Thanks for helping out. I am in the USA, so I tried commenting out everything that wasn't in USA and it still ran pretty bad. I even commented out all the FTP mirrors and it was still bad. As a last resort I kept only 1 uncommented and no substantial difference.

cactus · 2008-08-14 00:08:32

Questions to answer:
- are downloads slow to start from the windows box you talked about in your first post?
- have you tried disabling ipv6 yet?

void.pointer · 2008-08-14 00:11:40

cactus wrote:

Questions to answer:
- are downloads slow to start from the windows box you talked about in your first post?

Not at all. It's very fast. Takes less than a second to start the download.

cactus wrote:

- have you tried disabling ipv6 yet?

This question was answered in my previous post (The large one) about 2 or 3 up.

fwojciec · 2008-08-14 00:25:00

void.pointer wrote:

I can't manually change this file per your suggestion, since it is auto-generated when I reboot and also when I restart /etc/rc.d/network.

If you switch to netcfg to configure wifi you should be able to specify DNS servers explicitly in the configuration file for the network. Also, you could set Arch system to use static IP and then the resolve.conf file will not be automatically updated each time you connect. Or you can also set DNS globally by editing the settings of your router.

rooloo · 2008-08-14 03:47:51

I did it this morning after reading this thread, and everything (core, extra and community, no other repo's) was up to date, but still it took 36 seconds to complete the -Sy. I have always found this to be rather strange, since pacman itself says everything is up to date, so I guess nothing has to be sync'ed, no?

I had this issue one before as well, nothing would update. pacman -Syy solved it. I believe it was corrupt database, however that happened.

If you have not done so, as the shinning suggested. try running

pacman -Syy

I didn't see where u tried this yet.

void.pointer · 2008-08-14 03:57:22

[robert@server forum]$ time sudo pacman -Syy
:: Synchronizing package databases...
 core                      31.4K   38.3K/s 00:00:01 [##############################] 100%
 extra                    396.5K   60.3K/s 00:00:07 [##############################] 100%
 community                354.0K   76.2K/s 00:00:05 [##############################] 100%
 aurbuild                   0.5K  120.0K/s 00:00:00 [##############################] 100%

real    0m24.569s
user    0m0.263s
sys     0m1.023s

[robert@server forum]$ time sudo pacman -Sy
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 aurbuild is up to date

real    1m30.125s
user    0m0.003s
sys     0m0.023s

Notice the substantial difference in timing between the two.

shining · 2008-08-14 06:55:18

void.pointer wrote:

shining wrote:

4) same with wget, how long does it take for resolving :
wget ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz

[robert@server ~]$ time wget ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz
--2008-08-13 22:24:26--  ftp://ftp.archlinux.org/core/os/i686/core.db.tar.gz
           => `core.db.tar.gz'
Resolving ftp.archlinux.org... 209.85.41.145, 209.85.41.144
Connecting to ftp.archlinux.org|209.85.41.145|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /core/os/i686 ... done.
==> SIZE core.db.tar.gz ... 32335
==> PASV ... done.    ==> RETR core.db.tar.gz ... done.
Length: 32335 (32K)

100%[==============================================================>] 32,335       168K/s   in 0.2s

2008-08-13 22:24:37 (168 KB/s) - `core.db.tar.gz' saved [32335]


real    0m11.130s
user    0m0.003s
sys     0m0.007s

Above, the "Resolving ftp.archlinux.org..." is what took the longest. I would say it probably took 9-10 seconds out of the 11 seconds total.

I would say, let's forget about pacman for a while and concentrate on the wget testcase. 10 seconds for resolving is simply crazy.
And in the same time, "dig ftp.archlinux.org" reports a query of only a few milliseconds? That doesn't make sense.

 # Generated by dhcpcd for interface eth0
 search router
 nameserver 192.168.10.1
I can't manually change this file per your suggestion, since it is auto-generated when I reboot and also when I restart /etc/rc.d/network.

You can disable the auto-generation by adding the -R flag to dhcpcd arguments in /etc/conf.d/dhcpcd
After than you can manually change the file.

shining · 2008-08-14 06:57:40

zenlord wrote:

OK, shoot me if you will. Tested the dig and time wget-commands and all was very well.
So I looked inside the mirrorlist and found out that I had all servers uncommented. Changing this and leaving only 1 open (belnet in Belgium), made the pacman-sync run much smoother.
Sorry to have wasted some of your time - I hope this post will help some other guy once...

Just for clarification, the mirror used is the first one pacman finds, that means the first uncommented one.
So it does not matter how many you have uncommented. What matters is that the first uncommented mirror is fast from where you are.

void.pointer · 2008-08-19 16:12:06

Well, I managed to solve the slow downs I believe. I'll never know for sure what really caused the slow downs, but at least it is faster now. Basically to solve the problem all I did was comment out servers in /etc/pacman.d/mirrorlist one at a time and do a "sudo pacman -Sy" until I found a server that was fast. Now I get the following timings:

[robert@server ~]$ time sudo pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 aurbuild is up to date
:: Starting full system upgrade...
warning: subversion: local (1.5.1-5) is newer than extra (1.4.6-4)
 local database is up to date

real    0m7.834s
user    0m0.197s
sys     0m0.110s

I appreciate everyone's help.

tigrmesh · 2008-08-19 16:33:17

@void.pointer

I'm glad you got this working. Please mark this thread as solved. Thank you.

Arch Linux

#26 2008-08-13 20:29:45

Re: [SOLVED] pacman insanely slow

#27 2008-08-13 20:33:22

Re: [SOLVED] pacman insanely slow

#28 2008-08-13 20:34:07

Re: [SOLVED] pacman insanely slow

#29 2008-08-13 20:47:48

Re: [SOLVED] pacman insanely slow

#30 2008-08-13 21:03:43

Re: [SOLVED] pacman insanely slow

#31 2008-08-13 22:10:37

Re: [SOLVED] pacman insanely slow

#32 2008-08-13 22:33:33

Re: [SOLVED] pacman insanely slow

#33 2008-08-13 22:59:07

Re: [SOLVED] pacman insanely slow

#34 2008-08-13 23:31:43

Re: [SOLVED] pacman insanely slow

#35 2008-08-14 00:08:32

Re: [SOLVED] pacman insanely slow

#36 2008-08-14 00:11:40

Re: [SOLVED] pacman insanely slow

#37 2008-08-14 00:25:00

Re: [SOLVED] pacman insanely slow

#38 2008-08-14 03:47:51

Re: [SOLVED] pacman insanely slow

#39 2008-08-14 03:57:22

Re: [SOLVED] pacman insanely slow

#40 2008-08-14 06:55:18

Re: [SOLVED] pacman insanely slow

#41 2008-08-14 06:57:40

Re: [SOLVED] pacman insanely slow

#42 2008-08-19 16:12:06

Re: [SOLVED] pacman insanely slow

#43 2008-08-19 16:33:17

Re: [SOLVED] pacman insanely slow

Board footer