[SOLVED]Pacman very slow to sync local and repository

orion91 · 2008-03-07 18:27:46

Lately pacman is very slow when synchronize local database with repository database.
When i use # pacman -Syu the time between downloads of repo.db.tar.gz is very very slow.
For example this is output of my pacman -Syu command:

[..]
2008-03-07 19:01:02 (197 KB/s) - "extra.db.tar.gz" saved [318995]

--2008-03-07 19:01:47-- ftp://ftp.free.fr/mirrors/ftp.archlinux … .db.tar.gz
=> `community.db.tar.gz'
[..]

After download of extra.db.tar.gz the cursor flashes for long time and pacman write some data to hard disk, only after 45seconds pacman starts to download next db files..

I think that pacman have some problems.. Some ideas?

Last edited by orion91 (2008-03-07 19:21:09)

shining · 2008-03-07 18:34:16

Try pacman-optimize.

orion91 · 2008-03-07 18:38:02

shining wrote:

Try pacman-optimize.

Unfortunately, I have tried this way..
I used the command "pacman-optimize && sync" but i have not had improvements

dyscoria · 2008-03-07 18:44:01

Requires some partitioning to be done, but the ReiserFS filesystem is usually recommended for your /var partition and helps alot with pacman performance (it's better with the small files).

Last edited by dyscoria (2008-03-07 18:44:19)

orion91 · 2008-03-07 18:50:32

dyscoria wrote:

Requires some partitioning to be done, but the ReiserFS filesystem is usually recommended for your /var partition and helps alot with pacman performance (it's better with the small files).

I don't know this, i've filesystem XFS on my root partition..

shining · 2008-03-07 19:12:49

Hm maybe that's why nobody uses XFS
Anyway, the fact that you are using XFS for / doesn't prevent you from doing a /var partition with another filesystem.

dyscoria · 2008-03-07 19:18:09

Ouch, XFS is practically the opposite of ReiserFS (XFS is better for very large files).

You should consider using a separate /var partition using ReiserFS, though i'm not sure what the process of doing so is without a reinstall. I assume there's a way...

orion91 · 2008-03-07 19:20:14

I don't know this about XFS filesystem.
Now i know..

Thanks to all for this informations

ekerazha · 2008-03-08 13:18:50

Using a separate partition for the database of a package manager? Very funny.

orion91, the truth is that pacman is slow and its database implementation is very inefficient, it takes about 50 seconds on my system too. I've suggested an excellent way to improve pacman (sqlite backend), but devs don't like it. So live with it... I think Arch still is the best linux distro out there, also with this poorly engineered package manager.

shining · 2008-03-08 15:48:01

ekerazha wrote:

Using a separate partition for the database of a package manager? Very funny.
orion91, the truth is that pacman is slow and its database implementation is very inefficient, it takes about 50 seconds on my system too. I've suggested an excellent way to improve pacman (sqlite backend), but devs don't like it. So live with it... I think Arch still is the best linux distro out there, also with this poorly engineered package manager.

That's very original, I must admit that you were the first one to suggest this (if we don't count the thousand of ppl that suggested it before you)
When do you stop trolling and start contributing? Oh, that's right, never, because you are too busy. And guess what, we all are.

ekerazha · 2008-03-08 16:56:40

shining wrote:

That's very original, I must admit that you were the first one to suggest this (if we don't count the thousand of ppl that suggested it before you)

I've said "I suggested" not "I was the 1st to suggest". Somebody suggested it before? Well, he was right. Please read what I write.

When do you stop trolling and start contributing? Oh, that's right, never, because you are too busy. And guess what, we all are.

This discussion is already be done... and no, that isn't the real answer (well, it is incomplete). Please read what I wrote. In my experience I can say to you that the biggest trolls are who begin to say "you are trolling" to the other people.

Bye

Last edited by ekerazha (2008-03-08 17:01:21)

fwojciec · 2008-03-08 17:05:54

ekerazha wrote:

(...) it takes about 50 seconds on my system too. (...)

Something must be wrong with your system:

time pacman -Syu
:: Synchronizing package databases...
 testing is up to date
 core is up to date
 extra is up to date
 community is up to date
 filip is up to date
:: Starting full system upgrade...
 local database is up to date

real    0m1.144s
user    0m0.087s
sys     0m0.123s

time pacman -Syyu
:: Synchronizing package databases...
 testing                   24.1K  109.3K/s 00:00:00 [#######################################################################] 100%
 core                      23.6K   95.2K/s 00:00:00 [#######################################################################] 100%
 extra                    304.7K  281.6K/s 00:00:01 [#######################################################################] 100%
 community                340.3K  195.9K/s 00:00:02 [#######################################################################] 100%
 filip                      1.0K   12.8M/s 00:00:00 [#######################################################################] 100%
:: Starting full system upgrade...
 local database is up to date

real    0m6.310s
user    0m0.223s
sys     0m1.883s

Last edited by fwojciec (2008-03-08 17:09:33)

ekerazha · 2008-03-08 17:20:35

Maybe the same thing is wrong on the system of orion91?

No... a short explanation: the database implementation of pacman is very slow, however after having used pacman the first time, the kernel caching begins to play. So the 2nd, 3rd etc. time you use pacman, it could be faster.

synthead · 2008-03-08 17:59:37

I get the same thing. It takes for-EVER to sync pacman the first time, but every other time is better. I was thinking of running "pacman -Sy" in the background as my systems start up so they'll at least be in cache by the time I go to use pacman. But I would even consider this a big problem. I hate to do the old man vs machine, but ubuntu ... you just type in your package in the GUI manager and it pops up right away.

Speaking of, I am a 100% supporter of having a good GUI for pacman as long as pacman is always as easy to use at the command line.

fwojciec · 2008-03-08 18:07:50

ekerazha wrote:

Maybe the same thing is wrong on the system of orion91?
No... a short explanation: the database implementation of pacman is very slow, however after having used pacman the first time, the kernel caching begins to play. So the 2nd, 3rd etc. time you use pacman, it could be faster.

Hmm, I just rebooted and tried again:

time pacman -Syyu
:: Synchronizing package databases...
 testing                   23.4K   72.0K/s 00:00:00 [#######################################################################] 100%
 core                      23.6K  105.1K/s 00:00:00 [#######################################################################] 100%
 extra                    304.7K  279.0K/s 00:00:01 [#######################################################################] 100%
 community                340.7K  271.0K/s 00:00:01 [#######################################################################] 100%
 filip                      1.0K  250.7K/s 00:00:00 [#######################################################################] 100%
:: Starting full system upgrade...
 local database is up to date

real    0m6.318s
user    0m0.280s
sys     0m1.740s

It doesn't seem to make a difference if its the first time after boot or not.

shining · 2008-03-08 18:16:17

fwojciec wrote:

It doesn't seem to make a difference if its the first time after boot or not.

That's because your filesystem and/or hd isn't awfully slow.

ekerazha · 2008-03-08 18:17:42

fwojciec wrote:

ekerazha wrote:
Maybe the same thing is wrong on the system of orion91?
No... a short explanation: the database implementation of pacman is very slow, however after having used pacman the first time, the kernel caching begins to play. So the 2nd, 3rd etc. time you use pacman, it could be faster.
Hmm, I just rebooted and tried again:
time pacman -Syyu
:: Synchronizing package databases...
 testing                   23.4K   72.0K/s 00:00:00 [#######################################################################] 100%
 core                      23.6K  105.1K/s 00:00:00 [#######################################################################] 100%
 extra                    304.7K  279.0K/s 00:00:01 [#######################################################################] 100%
 community                340.7K  271.0K/s 00:00:01 [#######################################################################] 100%
 filip                      1.0K  250.7K/s 00:00:00 [#######################################################################] 100%
:: Starting full system upgrade...
 local database is up to date

real    0m6.318s
user    0m0.280s
sys     0m1.740s
It doesn't seem to make a difference if its the first time after boot or not.

Well... orion91 has this "problem", I confirmed it, synthead confirmed it, if you want you can join #archlinux.it and most people (maybe everybody) will confirm this.

So what can I say? If that is true you have an extremely fast hard disk with a filesystem which is extremely fast managing the small text files of the pacman database.

Last edited by ekerazha (2008-03-08 18:24:37)

ekerazha · 2008-03-08 18:22:59

@fwojciec
However... after the "first time", we have that kernel caching, so this is another strange thing... it should be faster after the first time, while you say you always have about "6 seconds".

fwojciec · 2008-03-08 18:30:16

My desktop (from which the above results come) has a 10000RPM raptor hd and reiserfs so it's not slow. But I think it is primarily the filesystem that's responsible for the good performance. Here is my laptop, with a laptop hd, reiserfs, throttled down to 0.60 GHz I should add -- the first result is directly after booting, the second immediately afterwards (it is faster, but not by a lot):

filip@major_tom ~ $ time pacman -Syyu
:: Synchronizing package databases...
 testing                   23.8K  107.9K/s 00:00:00 [#############################################################] 100%
 core                      23.6K  104.7K/s 00:00:00 [#############################################################] 100%
 extra                    311.4K  201.8K/s 00:00:02 [#############################################################] 100%
 community                345.7K  275.1K/s 00:00:01 [#############################################################] 100%
 filip                      1.6K  168.1K/s 00:00:00 [#############################################################] 100%
:: Starting full system upgrade...
warning: libtorrent: local (0.12.0-1) is newer than community (0.11.9-1)
warning: rtorrent: local (0.8.0-1) is newer than community (0.7.9-1)
 local database is up to date

real    0m8.788s
user    0m0.453s
sys     0m2.896s
filip@major_tom ~ $ time pacman -Syyu
:: Synchronizing package databases...
 testing                   23.8K   78.7K/s 00:00:00 [#############################################################] 100%
 core                      23.6K   95.2K/s 00:00:00 [#############################################################] 100%
 extra                    311.4K  247.1K/s 00:00:01 [#############################################################] 100%
 community                345.7K  232.2K/s 00:00:01 [#############################################################] 100%
 filip                      1.6K    3.9M/s 00:00:00 [#############################################################] 100%
:: Starting full system upgrade...
warning: libtorrent: local (0.12.0-1) is newer than community (0.11.9-1)
warning: rtorrent: local (0.8.0-1) is newer than community (0.7.9-1)
 local database is up to date

real    0m7.241s
user    0m0.473s
sys     0m2.333s

Overall I'd say what really makes a difference is reiserfs. I used to use XFS in the past as well and I can confirm that with XFS pacman sync can be awfully slow.

ekerazha · 2008-03-08 18:37:30

My desktop (from which the above results come) has a 10000RPM raptor hd and reiserfs so it's not slow.

Ehehehe......

I think a package manager should have acceptable (I don't say "blazing fast") performances on every supported filesystem and 50 seconds are not so "acceptable". Do you think is it "normal" that people have to create a dedicated "ReiserFS" partition for the database of a package manager? I don't think so... however I can live with it.

fwojciec · 2008-03-08 18:50:50

ekerazha wrote:

My desktop (from which the above results come) has a 10000RPM raptor hd and reiserfs so it's not slow.
Ehehehe......
I think a package manager should have acceptable (I don't say "blazing fast") performances on every supported filesystem and 50 seconds are not so "acceptable". Do you think is it "normal" that people have to create a dedicated "ReiserFS" partition for the database of a package manager? I don't think so... however I can live with it.

Just in case my desktop was not representative of an average system I have also included the results from my laptop in my previous post -- which is pretty average. And personally I don't have a problem with picking the most appropriate filesystem for a given job -- I just use reiserfs on all my root partitions and it has a perceptible impact on the performance of the system as a whole. XFS, in either case, is a bad choice of a filesystem for the root partition -- and this is speaking from experience... It's likely to be slow not only with pacman db syncing but with other things as well -- try deleting kernel source tree on XFS filesystem, or firefox sources for example -- it takes forever.

shining · 2008-03-08 18:53:13

ekerazha wrote:

I think a package manager should have acceptable (I don't say "blazing fast") performances on every supported filesystem and 50 seconds are not so "acceptable". Do you think is it "normal" that people have to create a dedicated "ReiserFS" partition for the database of a package manager? I don't think so... however I can live with it.

I don't use a dedicated partition, and I have acceptable performance, just like fwojciec, on a 1 year old average laptop.
We must be the two exceptions that confirm your rule, right?

shining · 2008-03-08 18:55:58

fwojciec wrote:

ekerazha wrote:
My desktop (from which the above results come) has a 10000RPM raptor hd and reiserfs so it's not slow.
Ehehehe......
I think a package manager should have acceptable (I don't say "blazing fast") performances on every supported filesystem and 50 seconds are not so "acceptable". Do you think is it "normal" that people have to create a dedicated "ReiserFS" partition for the database of a package manager? I don't think so... however I can live with it.
Just in case my desktop was not representative of an average system I have also included the results from my laptop in my previous post -- which is pretty average. And personally I don't have a problem with picking the most appropriate filesystem for a given job -- I just use reiserfs on all my root partitions and it has a perceptible impact on the performance of the system as a whole. XFS, in either case, is a bad choice of a filesystem for the root partition -- and this is speaking from experience... It's likely to be slow not only with pacman db syncing but with other things as well -- try deleting kernel source tree on XFS filesystem, or firefox sources for example -- it takes forever.

I already tried to explain that to ekerazha a while ago. It's a lost cause.

ekerazha · 2008-03-08 19:32:15

fwojciec wrote:

ekerazha wrote:
My desktop (from which the above results come) has a 10000RPM raptor hd and reiserfs so it's not slow.
Ehehehe......
I think a package manager should have acceptable (I don't say "blazing fast") performances on every supported filesystem and 50 seconds are not so "acceptable". Do you think is it "normal" that people have to create a dedicated "ReiserFS" partition for the database of a package manager? I don't think so... however I can live with it.
Just in case my desktop was not representative of an average system I have also included the results from my laptop in my previous post -- which is pretty average. And personally I don't have a problem with picking the most appropriate filesystem for a given job -- I just use reiserfs on all my root partitions and it has a perceptible impact on the performance of the system as a whole. XFS, in either case, is a bad choice of a filesystem for the root partition -- and this is speaking from experience... It's likely to be slow not only with pacman db syncing but with other things as well -- try deleting kernel source tree on XFS filesystem, or firefox sources for example -- it takes forever.

shining doesn't have anything intelligent to say, so I'll talk to you.

Well... ReiserFS is *very* slow at mounting, so it's not suitable for very large partitions. ReiserFS isn't actively developed anymore. XFS is on average the fastest linux filesystem (ex. http://www.debian-administration.org/articles/388 ), "deleting" is the only action really slow on XFS. Having a dedicated ReiserFS partition for < 10MB is just ridiculous and it could be avoided using a proper backend for pacman. This is my ponderated point of view.

Last edited by ekerazha (2008-03-08 19:45:55)

fwojciec · 2008-03-08 19:45:25

I have two 25gb reiserfs partitions that are being mounted during boot time and I don't register any particular slowness -- my system still boots in about 30-40 seconds and it's not tweaked in any way. For large "storage" type partitions I use ext3 with journal_data. In either case -- none of the things you mention about reiserfs are really deal-breakers for me; each filesystem has its positives and negatives.

If you're so convinced that XFS is better for you then I couldn't really say anything to convince you otherwise -- it is apparently possible, however, to tweak XFS to perform better (similar to reiserfs) with small files. See this article if you're interested -- I've never tried this myself, but it might be worth investigating if you're plan to keep on using XFS for your system partitions.

Arch Linux

#1 2008-03-07 18:27:46

[SOLVED]Pacman very slow to sync local and repository

#2 2008-03-07 18:34:16

Re: [SOLVED]Pacman very slow to sync local and repository

#3 2008-03-07 18:38:02

Re: [SOLVED]Pacman very slow to sync local and repository

#4 2008-03-07 18:44:01

Re: [SOLVED]Pacman very slow to sync local and repository

#5 2008-03-07 18:50:32

Re: [SOLVED]Pacman very slow to sync local and repository

#6 2008-03-07 19:12:49

Re: [SOLVED]Pacman very slow to sync local and repository

#7 2008-03-07 19:18:09

Re: [SOLVED]Pacman very slow to sync local and repository

#8 2008-03-07 19:20:14

Re: [SOLVED]Pacman very slow to sync local and repository

#9 2008-03-08 13:18:50

Re: [SOLVED]Pacman very slow to sync local and repository

#10 2008-03-08 15:48:01

Re: [SOLVED]Pacman very slow to sync local and repository

#11 2008-03-08 16:56:40

Re: [SOLVED]Pacman very slow to sync local and repository

#12 2008-03-08 17:05:54

Re: [SOLVED]Pacman very slow to sync local and repository

#13 2008-03-08 17:20:35

Re: [SOLVED]Pacman very slow to sync local and repository

#14 2008-03-08 17:59:37

Re: [SOLVED]Pacman very slow to sync local and repository

#15 2008-03-08 18:07:50

Re: [SOLVED]Pacman very slow to sync local and repository

#16 2008-03-08 18:16:17

Re: [SOLVED]Pacman very slow to sync local and repository

#17 2008-03-08 18:17:42

Re: [SOLVED]Pacman very slow to sync local and repository

#18 2008-03-08 18:22:59

Re: [SOLVED]Pacman very slow to sync local and repository

#19 2008-03-08 18:30:16

Re: [SOLVED]Pacman very slow to sync local and repository

#20 2008-03-08 18:37:30

Re: [SOLVED]Pacman very slow to sync local and repository

#21 2008-03-08 18:50:50

Re: [SOLVED]Pacman very slow to sync local and repository

#22 2008-03-08 18:53:13

Re: [SOLVED]Pacman very slow to sync local and repository

#23 2008-03-08 18:55:58

Re: [SOLVED]Pacman very slow to sync local and repository

#24 2008-03-08 19:32:15

Re: [SOLVED]Pacman very slow to sync local and repository

#25 2008-03-08 19:45:25

Re: [SOLVED]Pacman very slow to sync local and repository

Board footer