Arch Linux

dyscoria

Member

Registered: 2008-01-10

Posts: 1,007

Re: [SOLVED]Pacman very slow to sync local and repository

Having a dedicated ReiserFS partition for < 10MB is just ridiculous

Well actually, the partition you'd generally set aside for ReiserFS is /var. Doing this has its security benefits aswell (although more so for a server) as log files and anything else in /var won't be able to fill up the root partition.

shining doesn't have anything intelligent to say,

Shining said: I don't use a dedicated partition, and I have acceptable performance, just like fwojciec, on a 1 year old average laptop.
I'd say that's a fairly good point he's making.

XFS is on average the fastest linux filesystem (ex. http://www.debian-administration.org/articles/388 ), "deleting" is the only action really slow on XFS

If there are in fact problems with pacman being slow, then we can assume that it is much worse for those with ageing desktops/laptops. For those people, generally they're not going to have ridiculous amounts of hard drive space. Deleting files suddenly becomes something that happens every other day.

---

flack 2.0.6: menu-driven BASH script to easily tag FLAC files (AUR)
knock-once 1.2: BASH script to easily create/send one-time sequences for knockd (forum/AUR)

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

I have two 25gb reiserfs partitions that are being mounted during boot time and I don't register any particular slowness -- my system still boots in about 30-40 seconds and it's not tweaked in any way.  For large "storage" type partitions I use ext3 with journal_data.  In either case -- none of the things you mention about reiserfs are really deal-breakers for me; each filesystem has its positives and negatives.

If you're so convinced that XFS is better for you then I couldn't really say anything to convince you otherwise -- it is apparently possible, however, to tweak XFS to perform better (similar to reiserfs) with small files.  See this article if you're interested -- I've never tried this myself, but it might be worth investigating if you're plan to keep on using XFS for your system partitions.

I'm already using "noatime, nodiratime" etc.

The point is not about what I am convinced, the point is about facts. Reading articles like that ( http://www.debian-administration.org/articles/388, http://linuxgazette.net/122/TWDT.html#piszcz) and their comments (there are 134 comments to the first article I linked), you can have a clear idea about the situation.

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

shining doesn't have anything intelligent to say,

Shining said: I don't use a dedicated partition, and I have acceptable performance, just like fwojciec, on a 1 year old average laptop.
I'd say that's a fairly good point he's making.

Well... that's OBVIOUS if your root partition is ReiserFS. And you need a dedicated ReiserFS partition if for good reasons (ex. http://www.debian-administration.org/articles/388 ) you use a different filesystem.

If there are in fact problems with pacman being slow, then we can assume that it is much worse for those with ageing desktops/laptops. For those people, generally they're not going to have ridiculous amounts of hard drive space. Deleting files suddenly becomes something that happens every other day.

You have to see if the time you gain with different actions is less or is more than the time you lose deleting files.

Using a sqlite backend for pacman you'll have:
*) always better performances, on every filesystem
*) pacman code would be simpler (because you delegate things to the sqlite library)

Last edited by ekerazha (2008-03-08 20:02:37)

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

Having a dedicated ReiserFS partition for < 10MB is just ridiculous

Well actually, the partition you'd generally set aside for ReiserFS is /var. Doing this has its security benefits aswell (although more so for a server) as log files and anything else in /var won't be able to fill up the root partition.

Yeah... and if you have encrypted filesystems, it's another key to enter for another partition on every boot.

fwojciec

Member

Registered: 2007-05-20

Posts: 1,411

Re: [SOLVED]Pacman very slow to sync local and repository

A while back I also was rather impressed by all these benchmarks that praised the "overall" performance of XFS, and I used XFS based on this, but then I realized that "overall" performance is not the best way to approach this problem -- it's better to choose the appropriate filesystem for particular use pattern.  Real life performance in particular applications is more important than generalized, abstract performance benchmarks.  If I were into multimedia editing and such I'd definitely consider XFS for this application, but in my experience XFS as root filesystem -- for Arch specifically -- is by far not the optimal choice.

As far as your idea with sqlite backend is concerned -- I don't really know enough about this from the developer standpoint, since I'm not a programmer, but I also use tracker which uses sqlite for its database and I remember tracker being out of commission for something like two months recently because some changes to sqlite made it stop working and forced tracker developers to revise their code significantly.  In this sense I can see how making pacman depend on another software package which is not under control of pacman development team can be a bad idea.

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

A while back I also was rather impressed by all these benchmarks that praised the "overall" performance of XFS, and I used XFS based on this, but then I realized that "overall" performance is not the best way to approach this problem -- it's better to choose the appropriate filesystem for particular use pattern.  Real life performance in particular applications is more important than generalized, abstract performance benchmarks.  If I were into multimedia editing and such I'd definitely consider XFS for this application, but in my experience XFS as root filesystem -- for Arch specifically -- is by far not the optimal choice.

I don't use my workstations for a "particular" application. I am a "general purpose" man and I need a filesystem that is fast on average. I could use encrypted filesystems on some systems and having to enter a passphrase for every different partition on every boot could be a pain. I could use 3-4 operating systems, primary partitions can be 4 and on some systems they can boot only before a given cylinder, so it could be a problem to have "yet another partition" (or a dos extended partition to host that partition). It's not so simple.

The question is... are you sure the "right way" is to have a dedicated partition or a dedicated filesystem in order to have acceptable performances with pacman, instead of improving the pacman backend?

As far as your idea with sqlite backend is concerned -- I don't really know enough about this from the developer standpoint, since I'm not a programmer, but I also use tracker which uses sqlite for its database and I remember tracker being out of commission for something like two months recently because some changes to sqlite made it stop working and forced tracker developers to revise their code significantly.  In this sense I can see how making pacman depend on another software package which is not under control of pacman development team can be a bad idea.

If there's a version bump of SQLite or a version bump of a SQLite call (ex. _v2), if you want you can stick with the previous version. SQLite 2 is still there also if SQLite 3 is out. SQLite code is public domain, you can embed it inside your application if you want and use it as "your" code.

Last edited by ekerazha (2008-03-08 20:35:34)

dyscoria

Member

Registered: 2008-01-10

Posts: 1,007

Re: [SOLVED]Pacman very slow to sync local and repository

The question is... are you sure the "right way" is to have a dedicated partition or a dedicated filesystem in order to have acceptable performances with pacman, instead of improving the pacman backend?

Well that depends on whether pacman performance is considered bad or not. It seems that for the majority performance is fine. For the minority that do have problems, one solution is for a separate partition. Use of SQLite is simply uneccessary when considering this scenario.

---

flack 2.0.6: menu-driven BASH script to easily tag FLAC files (AUR)
knock-once 1.2: BASH script to easily create/send one-time sequences for knockd (forum/AUR)

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

The question is... are you sure the "right way" is to have a dedicated partition or a dedicated filesystem in order to have acceptable performances with pacman, instead of improving the pacman backend?

Well that depends on whether pacman performance is considered bad or not. It seems that for the majority performance is fine. For the minority that do have problems, one solution is for a separate partition. Use of SQLite is simply uneccessary when considering this scenario.

Why use a solution that is always slower and is very very slow on some filesystems (and very very slow on the filesystem that is faster on average) when you can use a clean solution that is always faster on every filesystem? I think you should consider multiple scenarios.

Last edited by ekerazha (2008-03-09 02:46:37)

dyscoria

Member

Registered: 2008-01-10

Posts: 1,007

Re: [SOLVED]Pacman very slow to sync local and repository

Why use a solution that is always slower and is very very slow on some filesystems (and very very slow on the filesystem that is faster on average)

A little confused by your wording, but I assume you are talking about the solution of using a ReiserFS partition for /var. This is not as you describe 'always slower', but if you disagree please elaborate as i'm unsure about what you're trying to say here. The fact that using /var on XFS is 'very very slow' is irrelevent, even if it's faster on average, as being faster on average doesn't make it immune to weaknesses when performing certain tasks (hence you'd use ReiserFS for this situation).

You might argue that ReiserFS takes a while to mount, but this is a negligable difference if your /var partition is small, as would be for the normal desktop user. For a server it might be larger, but then you wouldn't be rebooting all that often.

when you can use a clean solution [SQLite] that is always faster on every filesystem?

Is there proof of this? Even so, there other things to consider than purely speed. I'm sure a developer could elaborate on their reasons for not using a different method for pacman, but i'm certain it's not based purely on speed.

---

flack 2.0.6: menu-driven BASH script to easily tag FLAC files (AUR)
knock-once 1.2: BASH script to easily create/send one-time sequences for knockd (forum/AUR)

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

A little confused by your wording, but I assume you are talking about the solution of using a ReiserFS partition for /var. This is not as you describe 'always slower', but if you disagree please elaborate as i'm unsure about what you're trying to say here. The fact that using /var on XFS is 'very very slow' is irrelevent, even if it's faster on average, as being faster on average doesn't make it immune to weaknesses when performing certain tasks (hence you'd use ReiserFS for this situation).

You might argue that ReiserFS takes a while to mount, but this is a negligable difference if your /var partition is small, as would be for the normal desktop user. For a server it might be larger, but then you wouldn't be rebooting all that often.

Yeah... maybe it has some weaknesses, but ReiserFS has weaknesses too, maybe more. There are no "perfect" filesystems, but you can't have a different filesystem for every application. And as I've already explained, having "yet another partition" could be problematic. What I'm saying is that you can have a fast pacman implementation on every filesystem. Yes, you can. Maybe another hypothetic application will continue to be slow (and maybe we could improve that application too). The point is that pacman, with a better backend, could be much faster. Yes, it can, it is possible. The database of pacman is a folder with 3 text files for every package, and pacman needs to scan and parse every package. That's just ridiculous. 

when you can use a clean solution [SQLite] that is always faster on every filesystem?

Is there proof of this? Even so, there other things to consider than purely speed. I'm sure a developer could elaborate on their reasons for not using a different method for pacman, but i'm certain it's not based purely on speed.

Yeah... if you'd use SQLite everyday, you'll know it is much faster than a pacman-like approach. There are also technical reasons for this (it's a binary format etc. etc.). There are also some tests (they only confirm what is already obvious) made by an user and published on pacman-dev. A SQLite approach would be better than the current approach under every aspect and I've already explained this in a detailed way in other occasions.

shining

Pacman Developer

Registered: 2006-05-10

Posts: 2,043

Re: [SOLVED]Pacman very slow to sync local and repository

http://bbs.archlinux.org/viewtopic.php? … 81#p319181

Let's stop arguing... this is dumb. If we're all "so busy" let's go be busy elsewhere where we're not broaching topics that have been touched on throughout my entire stay with ArchLinux and people have still to do anything about

---

pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

drf

Member

From: Milano, Italy

Registered: 2008-01-13

Posts: 113

Re: [SOLVED]Pacman very slow to sync local and repository

Sorry to bump into this thread now and bringing it a little OT, but since the problem arise:

Are Arch Devs interested in having a binary database instead of the text based one? I am asking this question now since I'm developing a pacman frontend, and since in our project we also thought about improving libalpm a bit (especially on that side), we could think about providing a patch for a binary database, being it sqlite or anything else (better).
Before starting to code, anyway, I'd like to hear if there is some interest from the devs. It would be frustrating to work days and days on a patch that will be rejected for lacking of interest. That's why I'm asking this.

tigrmesh

IRC Op

From: Florida, US

Registered: 2007-12-11

Posts: 794

Re: [SOLVED]Pacman very slow to sync local and repository

@drf - Here's a link to the pacman-dev mailing list:  http://archlinux.org/mailman/listinfo/pacman-dev

Here's a link to the pacman bug tracker:  http://bugs.archlinux.org/index.php?pro … 1&do=index.

You can see if this question has come up before and how the devs responded.

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

Sorry shining, but I'm not phrakture.

RedShift

Member

From: Belgium

Registered: 2004-07-16

Posts: 230

Re: [SOLVED]Pacman very slow to sync local and repository

If the sync operation is slow, then your mirrors are slow. Check your mirrors please.

---

:?

ekerazha

Member

Registered: 2007-02-27

Posts: 290

Re: [SOLVED]Pacman very slow to sync local and repository

@drf - Here's a link to the pacman-dev mailing list:  http://archlinux.org/mailman/listinfo/pacman-dev

Here's a link to the pacman bug tracker:  http://bugs.archlinux.org/index.php?pro … 1&do=index.

You can see if this question has come up before and how the devs responded.

A decent alternative to the SQLite based approach (however... SQLite would be better) could be similar to this: http://bugs.archlinux.org/task/8586 directly using the tar.gz file...