Arch Alternatives

sokuban · 2007-02-07 09:14:13

I'll admit to having a similar problem. Though I don't take initiative to fix small problems and end up living with them. I guess I'll copy my /var to somewhere else then put it back on a reiserfs partition and see if it works well.

I could swear I heard rumours that pacman was bad with reiserfs, maybe I hear things backwards. (Don't trust my memory.)

But I don't get this thing about not being inherently flawed. I had to use a lot of fixes to get everything working perfectly and I don't see problem with them. I personally believe that every distro is inherently flawed. Why because if it doesn't work "perfectly" there is an inherent flaw isn't there?

(You should notice my sarcasm, I'm not complaining about flaws.)

onearm · 2007-02-07 09:34:01

No problem here, ext3 using dir_index:

time pacman -Ss pacman

real    0m0.584s
user    0m0.297s
sys    0m0.281s

Before I was using Xfs for /var and that was really slow (I mean even after pacman-optimize and removing all the cache)

lucke · 2007-02-07 09:49:19

Arr, perhaps adding "noatime" to fstab would help a bit?

sjmorgan · 2007-02-07 11:32:47

phrakture wrote:

What are your system specs? Hard drive? hdparm -tT output? File system. Have you tried dir_index-ing the filesystem? What have and haven't you tried. Do you have anything disk related in your dmesg output?

AMD Athlon XP 3000+, 1GB of RAM.

/dev/hda3 / ext3 defaults 0 1
/dev/hda1 /boot ext2 defaults 0 1
/dev/hda2 swap swap defaults 0 0

Kernel command line: root=/dev/hda3 ro
    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
hda: Maxtor 92041U4, ATA DISK drive
hda: max request size: 128KiB
hda: 40020624 sectors (20490 MB) w/512KiB Cache, CHS=39703/16/63, UDMA(66)
hda: cache flushes not supported
 hda: hda1 hda2 hda3
EXT3 FS on hda3, internal journal
Adding 262576k swap on /dev/hda2.  Priority:-1 extents:1 across:262576k

/dev/hda:
 Timing cached reads:   870 MB in  2.00 seconds = 434.75 MB/sec
 Timing buffered disk reads:   74 MB in  3.05 seconds =  24.30 MB/sec

Most people seem to agree that this could be fixed by not using a ton of small files to store the "database". Is there a reason this strategy was used or was it just lack of foresight? I can't see any benefits to it personally, and the downsides should be obvious.

I'll try using dir_index and see if it improves things, but the fact remains that this shouldn't even be a problem.

Last edited by sjmorgan (2007-02-07 11:35:51)

lucke · 2007-02-07 11:42:53

One advantage of such design is that it is simple. Among other things it allows all kinds of scripts and frontends to access the data without much hassle. Recovery from partial corruption is also easier (well, with DB one would just keep a backup DB).

There's been lots of discussion about that in the forums and bugtracker alike. I think there's even a patch (not fully complete?) adding DB support, somewhere in the bugtracker.

wuischke · 2007-02-07 11:49:58

@thread starter: debian testing/unstable would be my choice for rolling release distro, but you're right about the disadvantages.
If 100% newest software is not important, I would take a look at vector and zenwalk.

[tux@arch ~]$ time pacman -Ss pacman
[...]
real 0m55.472s
user 0m0.619s
sys 0m1.653s
[tux@arch ~]$ time pacman -Ss pacman
[...]
real 0m0.735s
user 0m0.385s
sys 0m0.327s

I do have a slow 5400 rpms hard drive and use ext3 without any optimization. You see, it takes quite a while.

My (purely theoretical) approach to this problem:

Problem: slow in reading many files
existing solution: pacman-cage, but risk of data loss

Idea: Disk space should never be a big problem, but a loss is.

It should be possible to have the database twice. One time implemented as it already is.
One time for read-only use in a single file (which will be loaded to RAM only when needed) When installing new packages the single file needs to be updated.
[it's about the same as FS RAM caching, but intentionally done, not automatically as a side effect]

This way searching would be faster, but downside of other implementations would be gone. You still can access the simple text files using scripts, it is faster and you don't have the risk of data loss.

Example usage:
pacman -Ss package
pacman loads the single file into RAM and searches for the package.

pacman -S package
pacman -A package.pkg.tar.gz
pacman updates both, the single file and the original data base.

Now, having two data bases, one should still use the original database for things like pacman -Qi package (as it will be faster), but the single file for everything concerning lots of files.

How do you think about it? [/What points didn't I see in my limited knowledge?]

sjmorgan · 2007-02-07 12:24:12

Can dir_index only be applied when formatting a partition? If that's the case then I'm SOL. I even tried resizing one of my partitions so I could create a new one to put /var on, but seeing as I had to resort to using gparted - probably the application singularly most incapable of performing the task it was designed for - that idea soon went out the window.

EDIT: Never mind, I managed to enable it using tune2fs.

Last edited by sjmorgan (2007-02-07 13:11:37)

F · 2007-02-07 12:27:45

[f|~]% time pacman -Ss pacman
...
pacman -Ss pacman  0.25s user 0.27s system 96% cpu 0.537 total


[f|~]% mount
/dev/sda3 on / type ext3 (rw)                <----- HERE
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /proc/bus/usb type usbfs (rw)
none on /dev/pts type devpts (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda4 on /usr type ext3 (rw)           <----- AND HERE.
/dev/sda1 on /mnt/ntfs-c type ntfs (ro,noexec,nosuid,nodev,umask=000)
/dev/sda5 on /mnt/ntfs-d type ntfs (ro,noexec,nosuid,nodev,umask=000)

Last edited by F (2007-02-07 12:28:22)

clarence · 2007-02-07 12:36:39

<cheekybugger> I like to breathe whilst waiting for pacman </cheekybugger>

Mr Green · 2007-02-07 12:58:21

/var/lib/pacman-db.img on /var/lib/pacman type ext2 (rw,loop=/dev/loop0)

loopy like me

delphinen · 2007-02-07 18:27:17

Arch alternatives? I would say Gentoo or Slackware.

Spider.007 · 2007-02-07 21:19:06

Although most of the discussion in this thread is offtopic; and I think it is a good idea to split it to a separate thread; I looked into the activities pacman does while performing a search; and I see that it opens both depends and desc files. Does anyone have a idea why the depends files are opened? The search doesn't seem to be doing anything with them? It seems like a 50% overhead now

Also, it seems that using specialized tools such as grep would do the job much faster:

[S|7]$ time pacman -sS pacman
current/pacman 2.9.8-4
    A .tar.gz based package manager with dependency support
extra/namcap 1.5.4-1
    A Pacman package analyzer
extra/srcpac 0.4.1-1
    The pacman from-source wrapper

real    0m0.747s
user    0m0.376s
sys     0m0.372s
[S|7]$  time grep -rl pacman .|grep desc
./current/pacman-2.9.8-4/desc
./extra/srcpac-0.4.1-1/desc
./local/pacman-2.9.8-4/desc
./community/jacman-0.3.1-2/desc

real    0m0.294s
user    0m0.000s
sys     0m0.296s

.8 versus .3 seems like an interesting optimalisation to make without much effort?

Last edited by Spider.007 (2007-02-07 21:23:39)

mucknert · 2007-02-07 21:41:14

clarence wrote:

<cheekybugger> I like to breathe whilst waiting for pacman </cheekybugger>

YOU DON'T SAY!

Lone_Wolf · 2007-02-07 22:22:30

sjmorgan wrote:

phrakture wrote:
What are your system specs? Hard drive? hdparm -tT output? File system. Have you tried dir_index-ing the filesystem? What have and haven't you tried. Do you have anything disk related in your dmesg output?
AMD Athlon XP 3000+, 1GB of RAM.
/dev/hda3 / ext3 defaults 0 1
/dev/hda1 /boot ext2 defaults 0 1
/dev/hda2 swap swap defaults 0 0
Kernel command line: root=/dev/hda3 ro
    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
hda: Maxtor 92041U4, ATA DISK drive
hda: max request size: 128KiB
hda: 40020624 sectors (20490 MB) w/512KiB Cache, CHS=39703/16/63, UDMA(66)
hda: cache flushes not supported
 hda: hda1 hda2 hda3
EXT3 FS on hda3, internal journal
Adding 262576k swap on /dev/hda2.  Priority:-1 extents:1 across:262576k
/dev/hda:
 Timing cached reads:   870 MB in  2.00 seconds = 434.75 MB/sec
 Timing buffered disk reads:   74 MB in  3.05 seconds =  24.30 MB/sec
Most people seem to agree that this could be fixed by not using a ton of small files to store the "database". Is there a reason this strategy was used or was it just lack of foresight? I can't see any benefits to it personally, and the downsides should be obvious.
I'll try using dir_index and see if it improves things, but the fact remains that this shouldn't even be a problem.

Features/Specifications:

* Maxtor Model: 92041U4
* Capacity: 20.4GB
* Spindal Speed: 5400 RPM
* Data Buffer: 2 MB
* Average Access Time: 9.5ms
* Latency (average): 5.55 ms
* Form Factor: 3.5''
* Interface: ATA-5/Ultra DMA 66
* Drive Only

Your problem doesn't appear to be pacman, BUT instead the speed of your hdd.
Those drives were standard for P3 systems 5 years ago, and even then they were considered slow.

Misfit138 · 2007-02-08 01:46:40

sjmorgan-

I get 2 seconds from a fresh boot, and 0.5 thereafter. If you would like to know what I did to improve my pacman speed, here it is:

1. I added "noatime" to my / in fstab.
2. I ran the pacman-cage script.
3. I use Server = http://mirror.cs.vt.edu/pub/ArchLinux/current/os/i686 (and /extra/os/i686, and /community/os/i686) in my pacman.d

There is no one who is a bigger knuckleheaded noob at Arch than me, so if I can do it, I am sure you can too.

mutlu_inek · 2007-02-08 04:59:33

Misfit138 wrote:

1. I added "noatime" to my / in fstab.

I see the point. But if /var/lib/pacman is not a separate partition (or a loopback fs), this will effect nearly all of /, something that may not be all that desired. Why not do the following:

sudo chattr -R +A /var/lib/pacman/

This may have to be repeated from time to time, though, since newly added files will not have this attribute.

Last edited by mutlu_inek (2007-02-08 05:00:48)

Snarkout · 2007-02-08 18:51:57

You could setup a cron job to do that every few minutes, I suppose. Does changing the attribs on a single directory really have the same net gain that doing it on an entire partition does? Also, I understand that some folks feel there are security issues to consider, but how often do people look at read-access times? Is there some other issue I'm not taking into account? I honestly haven't used noatime since I started using dir_indexing - I wonder what the gain would be using both...

mutlu_inek · 2007-02-09 00:15:55

Snarkout wrote:

You could setup a cron job to do that every few minutes, I suppose.

Why every few minutes? This needs to be done only after absolutely new programs have been added to pacman and you did a pacman -Sy. Adding something like

alias pacman-upgrade='pacman -Syu && chattr -R +A /var/lib/pacman/'
alias pacman-refresh='pacman -Sy && chattr -R +A /var/lib/pacman/'

to .bashrc would be enough.

Snarkout wrote:

Does changing the attribs on a single directory really have the same net gain that doing it on an entire partition does?

Most linux file systems write to any file that is accessed and change the access time value. This means that what seems a mere reading of files is actually a read-write access. Of course, the more tiny files are read, the more costly this is in regard to 'reading' time. AFAIK it does not matter whether you set the value for the whole partition in /etc/fstab or whether you assign the attribute to the single files. So why lose the feature altogether if it only annoys in regard to pacman?

picpak · 2007-02-09 01:58:26

My problem isn't with searching, it's with installing files. Say for example I do:

root@komputer:$ time pacman -S xpenguins

Targets: xpenguins-2.2-1

Total Package Size:   0.2 MB

Proceed with upgrade? [Y/n] n

real    1m20.960s
user    0m1.256s
sys    0m2.684s

It took nearly a minute and a half for it to ask if I wanted to install and me to say no. I did run pacman-optimize, and I think that's the problem. Any way to revert back?

dolby · 2007-02-09 02:18:57

real    0m4.331s
user    0m0.932s
sys     0m1.612s

on a reiser 3 partition

i really dont get the "bitchin" about pacman's speed made by gentoo users mostly , but in this case also.
pacman works very well and its certain its only going to get better

wuischke · 2007-02-09 04:43:20

This has nothing to do with Gentoo/anyotherdistri nor bitchin' (e.g. I never did more than testing with Gentoo, but am a Ubuntu-convert.), it's a _very_ annoying problem for many user. Most probably it's the user's fault for not having a super fast hdd (yeah, I know mine is pretty slow, but I won't get a new one for my notebook) and most probably not the best fitted file system (but I'll stick with ext3, too).
Imho it's important to find a way to lower the effect of the slow hdd/fs in order to make arch better useable. Once cached pacman is very fast and it's fun to use. But on first run after reboot/some time not using pacman it is awfully slow. And running pacman on boot for caching doesn't seem like a solution for me either as I usually don't boot in the intention to install/... something, but first read mails, check forums,... and after some time maybe use pacman - being uncached again.

clarence · 2007-02-09 05:03:15

I think it's far enough that you may feel annoyed about the first use delay wuischke, I get annoyed with it too. But IMO given the relatively small dev base of Arch, it's a small price to pay for what is to me a great package manager.

Just my 2c.

picpak · 2007-02-09 13:45:45

picpak wrote:

My problem isn't with searching, it's with installing files. Say for example I do:
root@komputer:$ time pacman -S xpenguins

Targets: xpenguins-2.2-1

Total Package Size:   0.2 MB

Proceed with upgrade? [Y/n] n

real    1m20.960s
user    0m1.256s
sys    0m2.684s
It took nearly a minute and a half for it to ask if I wanted to install and me to say no. I did run pacman-optimize, and I think that's the problem. Any way to revert back?

Just ran pacman-cage...check this out!

root@komputer:$ time pacman -S xpenguins

Targets: xpenguins-2.2-1

Total Package Size:   0.2 MB

Proceed with upgrade? [Y/n] n

real    0m11.465s
user    0m1.232s
sys    0m1.984s

Snarkout · 2007-02-09 15:55:46

mutlu_inek wrote:

Why every few minutes? This needs to be done only after absolutely new programs have been added to pacman and you did a pacman -Sy.

OK, every few hours then. IME it's a fairly hitless operation, but your point is a good one. I've always worked in environments where aliasing commands like that is strongly discouraged if it's more than 'rm=rm -i' so I may carry that prejudice with me.

Most linux file systems write to any file that is accessed and change the access time value. This means that what seems a mere reading of files is actually a read-write access. Of course, the more tiny files are read, the more costly this is in regard to 'reading' time. AFAIK it does not matter whether you set the value for the whole partition in /etc/fstab or whether you assign the attribute to the single files. So why lose the feature altogether if it only annoys in regard to pacman?

Is it actually the file itself or the inode? I thought it was the inode. I suppose I should read up on it some.

wuischke · 2007-02-09 15:57:17

clarence wrote:

I think it's far enough that you may feel annoyed about the first use delay wuischke, I get annoyed with it too. But IMO given the relatively small dev base of Arch, it's a small price to pay for what is to me a great package manager.
Just my 2c.

I agree with you, pacman is a really great app as Arch is a great distribution.

When you say the dev base is small, than it is _our_ user's job to help them make Arch and pacman even better than it already is.

I haven't taken a look into the code yet nor am I a good coder (and thus don't know the difficulties faced), but I see easier ways to perform many operations.

Imho a faster way for initial operations of 'pacman -S package':

$ cd /var/lib/pacman
$ ls -d */* | grep bash
current/bash-3.2-1
extra/bash-completion-20060301-6
local/bash-3.2-1

pacman -S bash (executed afterwards) needed a minute, above code about a second. pacman does more than just searching for relevant packages, sure.

Anyway, I (claim I) could write a faster replacement for "pacman -S" basing on above commands. [check if installed, version, dependencies; dl and untar pkg, create/update desc files depends]
File operations I had to do in order to install bash (if i hadn't installed it.):
1. search for bash in local - not found*; search for bash in current - found
2. parse current/bash-x.x.x/depends
3. (check for glibc and readline in local using above code, assume both are installed, check readline version)
4. download and install package
5. create local/bash-x.x.x/files desc depends

*if found, compare versions in directory name

Thanks to the very good layout of the /var/lib/pacman-directory this is possible using solely bash (as slow as it is) and still being faster than native code. (I claim.)
This is not possible for search as descriptions are not searched this way. But operations involving only packages specified by name would be faster by optimizing the operations.

Or am I flawed about this?

Arch Linux

#26 2007-02-07 09:14:13

Re: Arch Alternatives

#27 2007-02-07 09:34:01

Re: Arch Alternatives

#28 2007-02-07 09:49:19

Re: Arch Alternatives

#29 2007-02-07 11:32:47

Re: Arch Alternatives

#30 2007-02-07 11:42:53

Re: Arch Alternatives

#31 2007-02-07 11:49:58

Re: Arch Alternatives

#32 2007-02-07 12:24:12

Re: Arch Alternatives

#33 2007-02-07 12:27:45

Re: Arch Alternatives

#34 2007-02-07 12:36:39

Re: Arch Alternatives

#35 2007-02-07 12:58:21

Re: Arch Alternatives

#36 2007-02-07 18:27:17

Re: Arch Alternatives

#37 2007-02-07 21:19:06

Re: Arch Alternatives

#38 2007-02-07 21:41:14

Re: Arch Alternatives

#39 2007-02-07 22:22:30

Re: Arch Alternatives

#40 2007-02-08 01:46:40

Re: Arch Alternatives

#41 2007-02-08 04:59:33

Re: Arch Alternatives

#42 2007-02-08 18:51:57

Re: Arch Alternatives

#43 2007-02-09 00:15:55

Re: Arch Alternatives

#44 2007-02-09 01:58:26

Re: Arch Alternatives

#45 2007-02-09 02:18:57

Re: Arch Alternatives

#46 2007-02-09 04:43:20

Re: Arch Alternatives

#47 2007-02-09 05:03:15

Re: Arch Alternatives

#48 2007-02-09 13:45:45

Re: Arch Alternatives

#49 2007-02-09 15:55:46

Re: Arch Alternatives

#50 2007-02-09 15:57:17

Re: Arch Alternatives

Board footer