[SOLVED] pacman insanely slow

void.pointer · 2008-08-13 03:30:18

Hi,

I believe this will speak for itself:

[robert@server ~]$ time sudo pacman -Sy
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 aurbuild is up to date

real    1m56.210s
user    0m0.000s
sys     0m0.013s

2 minutes just for -Sy? What the heck? I noticed that even during a 'wget', hostnames take a pretty long time to resolve. Something is up. Can anyone help out? My DNS servers can't be slow, since I'm able to resolve DNS from another windows machine on my LAN perfectly fine and it resolves quickly.

::EDIT::
Would IPv6 have anything to do with this? Is it enabled by default on an archlinux install? How can I disable it?

Last edited by void.pointer (2008-08-19 16:53:55)

Allan · 2008-08-13 03:37:22

Have you got you hostname in /etc/hosts?

void.pointer · 2008-08-13 03:58:05

Not sure. Here's what my /etc/hosts file looks like:

  1 #
  2 # /etc/hosts: static lookup table for host names
  3 #
  4
  5 #<ip-address>   <hostname.domain.org>   <hostname>
  6 127.0.0.1   localhost.localdomain   localhost
  7
  8
  9 # End of file

What should it look like?

Allan · 2008-08-13 04:00:49

What ever you added in the "HOSTNAME=" line in /etc/rc.conf should be at the end of the uncommented line in that file.

void.pointer · 2008-08-13 04:03:19

Okay I'll try that. In the meantime, could you explain the reason for the weird syntax? According to what I've learned about /etc/hosts, there's only 3 values per line. The first is the IP to map to, second is something I'm not sure about, and the 3rd is the domain name. Why 4? It's a bit confusing. Also, why would this slow things down?

::EDIT::
Just tried this and it didn't help. I have samba installed, and netbios is enabled (So that my LAN computers can access my Linux box by DNS instead of IP). The netbios name happens to be the exact same name as the one specified in rc.conf's HOSTNAME variable. Is this a problem?

Last edited by void.pointer (2008-08-13 04:09:59)

freakcode · 2008-08-13 04:24:41

Practical example. If in rc.conf is:

HOSTNAME="atena"

/etc/hosts should contain:

#
# /etc/hosts: static lookup table for host names
#

#<ip-address>    <hostname.domain.org>    <hostname>
127.0.0.1        localhost.localdomain    localhost
127.0.0.1        atena.homelinux.org    atena

(Reboot after any hostname changes)

void.pointer wrote:

Also, why would this slow things down?

Not having your hostname in /etc/hosts will slowdown because will try to resolve the current hostname everytime.

Netbios having the same name as hostname isn't an issue.

void.pointer · 2008-08-13 04:48:22

Still having no luck. I added the following line to my /etc/hosts:

127.0.0.1    server.mydomain.com    server

Anything else that could be causing the slowness?

freakcode · 2008-08-13 05:58:54

That hostname issue is #1 cause for most. If that didn't solve, try using other mirrors for pacman.

zenlord · 2008-08-13 08:14:32

I too find pacman rather sluggish. I tend to not care if it takes two minutes instead of 30 seconds, but I see it all the time that my CPU never scales up and still uses only 7%, my RAM usage doesn't exceed 23% and still pacman completely locks my pc (starting other programs is a no go). I never found out shy this is happening...

/EDIT: and this sluggishness completely disappears when the downloading and installing starts - so it's only the synching that takes a lot of time (so it cannot be the mirror's fault I guess: I almost always get speeds around 800KB/s)

Zl.

Last edited by zenlord (2008-08-13 08:16:27)

shining · 2008-08-13 08:50:24

zenlord wrote:

I too find pacman rather sluggish. I tend to not care if it takes two minutes instead of 30 seconds, but I see it all the time that my CPU never scales up and still uses only 7%, my RAM usage doesn't exceed 23% and still pacman completely locks my pc (starting other programs is a no go). I never found out shy this is happening...
/EDIT: and this sluggishness completely disappears when the downloading and installing starts - so it's only the syncing that takes a lot of time (so it cannot be the mirror's fault I guess: I almost always get speeds around 800KB/s)
Zl.

Well, high disk usage usually has similar effects. But I find it strange to have -Sy that slow.
First for clarification, is it also slow when the databases are already uptodate? Or only when something is refreshed? You can force the refresh with -Syy.
When nothing is refreshed, the only thing that happens is a ftp/http query, and a stat of the .lastupdate file. Having this taking more than a few seconds is hard to believe.
When it is not uptodate, after these first two steps, the actual download of the database tarball happens, then a recursive rm of the whole database directory (like rm -r), followed by an extraction of the tarball, and finally a removal of the tarball.
So yes, the download + removal + extraction can take some times, but on a normal system, it is just a master of seconds.
Apparently you don't have any network issues and downloading is fine, so you are left with removal + extraction part. So you probably have big filesystem issues. Like heavy fragmentation or slow disk or crap filesystem.

zenlord · 2008-08-13 11:31:40

shining wrote:

Well, high disk usage usually has similar effects. But I find it strange to have -Sy that slow.
First for clarification, is it also slow when the databases are already uptodate? Or only when something is refreshed? You can force the refresh with -Syy.
When nothing is refreshed, the only thing that happens is a ftp/http query, and a stat of the .lastupdate file. Having this taking more than a few seconds is hard to believe.

I did it this morning after reading this thread, and everything (core, extra and community, no other repo's) was up to date, but still it took 36 seconds to complete the -Sy. I have always found this to be rather strange, since pacman itself says everything is up to date, so I guess nothing has to be sync'ed, no?

shining wrote:

Apparently you don't have any network issues and downloading is fine, so you are left with removal + extraction part. So you probably have big filesystem issues. Like heavy fragmentation or slow disk or crap filesystem.

The filesystem is XFS. I will look into using the xfsprogs, but since this installation is only a few months old I wouldn't expect much fragmentation.

THX!

Zl.

shining · 2008-08-13 11:35:02

zenlord wrote:

shining wrote:
Well, high disk usage usually has similar effects. But I find it strange to have -Sy that slow.
First for clarification, is it also slow when the databases are already uptodate? Or only when something is refreshed? You can force the refresh with -Syy.
When nothing is refreshed, the only thing that happens is a ftp/http query, and a stat of the .lastupdate file. Having this taking more than a few seconds is hard to believe.
I did it this morning after reading this thread, and everything (core, extra and community, no other repo's) was up to date, but still it took 36 seconds to complete the -Sy. I have always found this to be rather strange, since pacman itself says everything is up to date, so I guess nothing has to be sync'ed, no?

Indeed, there is nothing to do in that case, just a small remote query and a small stat of a file. It is really nothing.
But if you have some network issues, the first step could maybe take ages.
Does it happen with every mirror? Does it happen with both http and ftp?

pheon · 2008-08-13 12:14:54

Hi,
in order to diagnose the network issues, try the following:
- access the mirror via firefox (or any other browser)
- check latency of an icmp echo request (ping)
- install mtr and run it on the mirror...it'll hopefully show the bottleneck
- add the mirror to your /etc/hosts (this should knock out any dns-related problem)

and pls flush dns cache (/etc/rc.d/nscd restart) after every test

Disabling IPv6 won't harm either

br

void.pointer · 2008-08-13 13:49:44

Just to let you guys know, doing 'pacman -Sy' takes 2 minutes as you saw before in my timings. This is when there is *nothing* to update. It simply says "XXXX is up to date". I'm not sure if it is the mirrors because I added the official mirror for aurbuild and it has the same slowness.

My filesystem is ext3. I'm not an expert on the linux file systems so I just picked one.

shining · 2008-08-13 14:00:31

void.pointer wrote:

Just to let you guys know, doing 'pacman -Sy' takes 2 minutes as you saw before in my timings. This is when there is *nothing* to update. It simply says "XXXX is up to date". I'm not sure if it is the mirrors because I added the official mirror for aurbuild and it has the same slowness.
My filesystem is ext3. I'm not an expert on the linux file systems so I just picked one.

The amount of filesystem operations done when there is nothing to update is ridiculous. And ext3 is fine, it is probably the most widely used.

Please read the two posts above, and follow all suggestions pheon and I made to diagnose your network issues.

And I would still try using other DNS servers just in case, even if they work fine for your windows machines.

Also which network driver do you use?

void.pointer · 2008-08-13 14:23:07

shining wrote:

Also which network driver do you use?

How can I find this information for you?

Also, I cannot use 'mtr' because it says "mtr: unable to get raw sockets.". I have a feeling raw icmp needs to be allowed? Do I do this in /etc/iptables? What lines do I add?

Last edited by void.pointer (2008-08-13 14:40:08)

shining · 2008-08-13 15:03:47

void.pointer wrote:

shining wrote:
Also which network driver do you use?
How can I find this information for you?

Well, first provide the information you are able to provide
If you are using LAN / ethernet, then it should not matter.
You can try using lspci and lshwd to find it (pacman -S lshwd)

void.pointer · 2008-08-13 15:29:38

I am using ethernet/LAN.

I edited my previous post to mention an issue I had with mtr. Have any idea what the problem could be?

shining · 2008-08-13 15:47:23

void.pointer wrote:

I am using ethernet/LAN.
I edited my previous post to mention an issue I had with mtr. Have any idea what the problem could be?

No idea other than trying as root.

void.pointer · 2008-08-13 15:54:15

shining wrote:

void.pointer wrote:
I am using ethernet/LAN.
I edited my previous post to mention an issue I had with mtr. Have any idea what the problem could be?
No idea other than trying as root.

After running this command:

[robert@server ~]$ sudo mtr ftp.archlinux.org

I get the following results. Note that when I ran the command it took about 10 seconds for the status page to appear to display what you will see below. In addition, I thought I would also mention that pacman -Sy seems to go faster when "core", "extra", or any of the other repositories actually have updates to receive. If they have no updates they take substantially longer to complete.

                                         My traceroute  [v0.72]
server (0.0.0.0)                                                               Wed Aug 13 15:51:39 2008
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. L100.VFTTP-44.DLLSTX.verizon-gni.net                      0.0%    33    7.3   9.3   5.4  62.3  10.2
 2. P9-2.LCR-04.DLLSTX.verizon-gni.net                        0.0%    33    6.2   7.3   5.6   8.7   0.8
 3. so-7-0-0-0.PEER-RTR1.DFW80.verizon-gni.net                0.0%    33    7.9   9.7   5.6  61.3  10.3
 4. 64.128.212.137                                            0.0%    33    9.8   7.4   5.6  15.7   2.1
 5. hagg-03-ge-0-0-0-540.hsto.twtelecom.net                   0.0%    33   14.1  14.6  11.8  15.9   1.3
 6. 5e.fc.374a.static.theplanet.com                           0.0%    33   61.1  59.6  58.6  61.1   0.7
 7. dion.freedback.com                                        0.0%    32   61.0  59.5  58.4  61.2   0.7

ekerazha · 2008-08-13 19:41:42

@zenlord
pacman crappy db backend (3 text files for every package to be parsed) + some filesystems (like XFS) = very very slow pacman

phrakture · 2008-08-13 20:11:40

ekerazha wrote:

@zenlord
pacman crappy db backend (3 text files for every package to be parsed) + some filesystems (like XFS) = very very slow pacman

That's simply not the case here, as a cursory look at the code would tell you. -Sy does not read any database entries at all. In fact, it goes through each config section, contacts the server for the "last update time" and compares it to a single file (/var/lib/pacman/sync/<repo name>/.lastupdate).

So what you've claimed is blatantly false and another case of your incessant trolling about the pacman DB.

This is, infact, a network issue, as the OP also states that wget is slow. Last time I checked, wget does not read the "crappy" pacman database

void.pointer - could you try to "dig" a few servers, to see if you DNS lookup is painfully slow (you said this was what wget appeared to hang on)

$ dig archlinux.org

; <<>> DiG 9.5.0-P1 <<>> archlinux.org
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17248
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 4

;; QUESTION SECTION:
;archlinux.org.                 IN      A

;; ANSWER SECTION:
archlinux.org.          37971   IN      A       66.211.213.17

;; AUTHORITY SECTION:
archlinux.org.          37971   IN      NS      ns4.everydns.net.
archlinux.org.          37971   IN      NS      ns2.everydns.net.
archlinux.org.          37971   IN      NS      ns1.everydns.net.
archlinux.org.          37971   IN      NS      ns1.archlinux.org.
archlinux.org.          37971   IN      NS      ns3.everydns.net.

;; ADDITIONAL SECTION:
ns1.archlinux.org.      76309   IN      A       66.211.213.17
ns3.everydns.net.       47709   IN      A       208.96.6.134
ns1.everydns.net.       35773   IN      A       208.76.56.56
ns4.everydns.net.       128926  IN      A       64.158.219.3

;; Query time: 2 msec
;; SERVER: 67.207.128.4#53(67.207.128.4)
;; WHEN: Wed Aug 13 13:11:16 2008
;; MSG SIZE  rcvd: 213

Note the "query time"

void.pointer · 2008-08-13 20:15:09

phrakture wrote:

That's simply not the case here, as a cursory look at the code would tell you. -Sy does not read any database entries at all. In fact, it goes through each config section, contacts the server for the "last update time" and compares it to a single file (/var/lib/pacman/sync/<repo name>/.lastupdate).
So what you've claimed is blatantly false and another case of your incessant trolling about the pacman DB.
This is, infact, a network issue, as the OP also states that wget is slow. Last time I checked, wget does not read the "crappy" pacman database
void.pointer - could you try to "dig" a few servers, to see if you DNS lookup is painfully slow (you said this was what wget appeared to hang on)
<snip>
Note the "query time"

Thanks for helping out phrakture. Below is the result of the dig command on my machine:

[robert@server ~]$ dig archlinux.org

; <<>> DiG 9.5.0-P1 <<>> archlinux.org
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47874
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;archlinux.org.                 IN      A

;; ANSWER SECTION:
archlinux.org.          41951   IN      A       66.211.213.17

;; Query time: 22 msec
;; SERVER: 192.168.10.1#53(192.168.10.1)
;; WHEN: Wed Aug 13 20:12:34 2008
;; MSG SIZE  rcvd: 47

ekerazha · 2008-08-13 20:22:01

phrakture wrote:

ekerazha wrote:
@zenlord
pacman crappy db backend (3 text files for every package to be parsed) + some filesystems (like XFS) = very very slow pacman
That's simply not the case here, as a cursory look at the code would tell you. -Sy does not read any database entries at all. In fact, it goes through each config section, contacts the server for the "last update time" and compares it to a single file (/var/lib/pacman/sync/<repo name>/.lastupdate).
So what you've claimed is blatantly false and another case of your incessant trolling about the pacman DB.
This is, infact, a network issue, as the OP also states that wget is slow. Last time I checked, wget does not read the "crappy" pacman database

I was replying to zenlord, you should read more carefully.

However... the unefficiency of the pacman DB is a date-of-fact, as already discussed and proven many many times (here and on the ML).

Last edited by ekerazha (2008-08-13 20:23:45)

void.pointer · 2008-08-13 20:26:18

ekerazha wrote:

phrakture wrote:
ekerazha wrote:
@zenlord
pacman crappy db backend (3 text files for every package to be parsed) + some filesystems (like XFS) = very very slow pacman
That's simply not the case here, as a cursory look at the code would tell you. -Sy does not read any database entries at all. In fact, it goes through each config section, contacts the server for the "last update time" and compares it to a single file (/var/lib/pacman/sync/<repo name>/.lastupdate).
So what you've claimed is blatantly false and another case of your incessant trolling about the pacman DB.
This is, infact, a network issue, as the OP also states that wget is slow. Last time I checked, wget does not read the "crappy" pacman database
I was replying to zenlord, you should read more carefully.
However... the unefficiency of the pacman DB is a date-of-fact, as already discussed and proven many many times (here and on the ML).

Okay guys let's calm down. The main issue seems to be a networking issue, as I've already speculated in earlier posts. There's no need to get into a war about file system performance. I appreciate everyone's help, I really do, but I don't want to see anything bad happen! Let's here more suggestions on solving the real problem, if possible. So far you guys are doing great and I really appreciate your help!

Arch Linux

#1 2008-08-13 03:30:18

[SOLVED] pacman insanely slow

#2 2008-08-13 03:37:22

Re: [SOLVED] pacman insanely slow

#3 2008-08-13 03:58:05

Re: [SOLVED] pacman insanely slow

#4 2008-08-13 04:00:49

Re: [SOLVED] pacman insanely slow

#5 2008-08-13 04:03:19

Re: [SOLVED] pacman insanely slow

#6 2008-08-13 04:24:41

Re: [SOLVED] pacman insanely slow

#7 2008-08-13 04:48:22

Re: [SOLVED] pacman insanely slow

#8 2008-08-13 05:58:54

Re: [SOLVED] pacman insanely slow

#9 2008-08-13 08:14:32

Re: [SOLVED] pacman insanely slow

#10 2008-08-13 08:50:24

Re: [SOLVED] pacman insanely slow

#11 2008-08-13 11:31:40

Re: [SOLVED] pacman insanely slow

#12 2008-08-13 11:35:02

Re: [SOLVED] pacman insanely slow

#13 2008-08-13 12:14:54

Re: [SOLVED] pacman insanely slow

#14 2008-08-13 13:49:44

Re: [SOLVED] pacman insanely slow

#15 2008-08-13 14:00:31

Re: [SOLVED] pacman insanely slow

#16 2008-08-13 14:23:07

Re: [SOLVED] pacman insanely slow

#17 2008-08-13 15:03:47

Re: [SOLVED] pacman insanely slow

#18 2008-08-13 15:29:38

Re: [SOLVED] pacman insanely slow

#19 2008-08-13 15:47:23

Re: [SOLVED] pacman insanely slow

#20 2008-08-13 15:54:15

Re: [SOLVED] pacman insanely slow

#21 2008-08-13 19:41:42

Re: [SOLVED] pacman insanely slow

#22 2008-08-13 20:11:40

Re: [SOLVED] pacman insanely slow

#23 2008-08-13 20:15:09

Re: [SOLVED] pacman insanely slow

#24 2008-08-13 20:22:01

Re: [SOLVED] pacman insanely slow

#25 2008-08-13 20:26:18

Re: [SOLVED] pacman insanely slow

Board footer