Slow database performance - WHY ? WHY ? WHY ?

MetalheadGautham · 2008-11-23 13:18:01

When I run pacman -Syu, the command takes a long time to execute. When I run pacman -S something, and that something does not exist as a package, the command again takes a long time to execute while searching. So my question is, why does pacman take a long time for entire database search operations, when the number of packages in the is quite limited ?

De Mysteriis · 2008-11-23 13:33:59

I have the same problem since yesterday when I ran pacman-optimize before that everything was alright and pacman was really fast but I needed more. Which turned out to be a really bad idea. The problem is when I first run pacman it's REALLY slow but after that first run everything speeds up and I know the database gets cached but before pacman-optimize it was...well faster. It's not a big issue only little annoying. Sorry for my bad English if there is any.

MetalheadGautham · 2008-11-25 00:36:20

I have been using pacman since I installed archlinux a few months back. And I still have had that speed issue right from the start.

apaige · 2008-11-25 15:09:08

Because it's not a real database: it's thousands of very small files.

test1000 · 2008-11-25 17:20:45

yeah. it's recommended to put your pacman "database" on a partition with ext3 (and iv'e also heard you can use reiser3) if you want speed..

rson451 · 2008-11-25 18:02:25

test1000 wrote:

yeah. it's recommended to put your pacman "database" on a partition with ext3 (and iv'e also heard you can use reiser3) if you want speed..

Reiser is great for small files. I know many people using a reiser partition just for their pacman cache. Personally I just stick with ext3 and I don't see a horrible lag in things.

patroclo7 · 2008-11-25 18:43:13

Reiser for /var is a great speed improvement for pacman: the output of pacman -Ss <foo> is... immediate. xfs and jfs for /var are a very bad idea, ext3 is always a good compromise.

MetalheadGautham · 2008-11-28 01:07:19

My whole / partition is in ext3 and my /home is again ext3.

Anyway, WHY doesn't pacman have a single database file ?
Doesn't arch's goal state that simplicity matters most ?

Last edited by MetalheadGautham (2008-11-28 01:08:25)

rson451 · 2008-11-28 01:39:58

MetalheadGautham wrote:

My whole / partition is in ext3 and my /home is again ext3.
Anyway, WHY doesn't pacman have a single database file ?
Doesn't arch's goal state that simplicity matters most ?

This is bikeshed. Search the forums if you want your answer.

MetalheadGautham · 2008-11-28 07:38:12

^^HOW is this bikeshed ? I am not flaming anybody or asking a stupid question.

sand_man · 2008-11-28 09:49:24

Wouldn't NOT implementing a database be simpler?

BTW: I think that what rson451 means is that you are complaining about something that is quite minimal and almost pointless. If pacman starts running slow, clear your cache.

Last edited by sand_man (2008-11-28 09:52:26)

MetalheadGautham · 2008-11-28 10:13:50

You mean pacman cache ? I clear it every few weeks. But speed does not change.

test1000 · 2008-11-28 13:01:58

if you have ext3 partition the speed should be almost intantenous. so try looking into your HD settings with hdparm or whateva it's called..

MetalheadGautham · 2008-11-29 08:25:37

hdparm gave me this output:

bash-3.2$ hdparm

hdparm - get/set hard disk parameters - version v8.9

Usage:  hdparm  [options] [device] ..

Options:
 -a   get/set fs readahead
 -A   get/set the drive look-ahead flag (0/1)
 -b   get/set bus state (0 == off, 1 == on, 2 == tristate)
 -B   set Advanced Power Management setting (1-255)
 -c   get/set IDE 32-bit IO setting
 -C   check drive power mode status
 -d   get/set using_dma flag
 -D   enable/disable drive defect management
 -E   set cd-rom drive speed
 -f   flush buffer cache for device on exit
 -F   flush drive write cache
 -g   display drive geometry
 -h   display terse usage information
 -H   read temperature from drive (Hitachi only)
 -i   display drive identification
 -I   detailed/current information directly from drive
 -k   get/set keep_settings_over_reset flag (0/1)
 -K   set drive keep_features_over_reset flag (0/1)
 -L   set drive doorlock (0/1) (removable harddisks only)
 -M   get/set acoustic management (0-254, 128: quiet, 254: fast)
 -m   get/set multiple sector count
 -N   get/set max visible number of sectors (HPA) (VERY DANGEROUS)
 -n   get/set ignore-write-errors flag (0/1)
 -p   set PIO mode on IDE interface chipset (0,1,2,3,4,...)
 -P   set drive prefetch count
 -q   change next setting quietly
 -Q   get/set DMA queue_depth (if supported)
 -r   get/set device  readonly flag (DANGEROUS to set)
 -R   obsolete
 -s   set power-up in standby flag (0/1) (DANGEROUS)
 -S   set standby (spindown) timeout
 -t   perform device read timings
 -T   perform cache read timings
 -u   get/set unmaskirq flag (0/1)
 -U   obsolete
 -v   defaults; same as -acdgkmur for IDE drives
 -V   display program version and exit immediately
 -w   perform device reset (DANGEROUS)
 -W   get/set drive write-caching flag (0/1)
 -x   obsolete
 -X   set IDE xfer mode (DANGEROUS)
 -y   put drive in standby mode
 -Y   put drive to sleep
 -Z   disable Seagate auto-powersaving mode
 -z   re-read partition table
 --direct          use O_DIRECT to bypass page cache for timings
 --drq-hsm-error   crash system with a "stuck DRQ" error (VERY DANGEROUS)
 --fibmap          show device extents (and fragmentation) for a file
 --fibmap-sector   show absolute LBA of a specfic sector of a file
 --Istdin          read identify data from stdin as ASCII hex
 --Istdout         write identify data to stdout as ASCII hex
 --make-bad-sector deliberately corrupt a sector directly on the media (VERY DANGEROUS)
 --read-sector     read and dump (in hex) a sector directly from the media
 --security-help   display help for ATA security commands
 --verbose         display extra diagnostics from some commands
 --write-sector    repair/overwrite a (possibly bad) sector directly on the media (VERY DANGEROUS)

Well, I don't know which option to use to check this problem. Help needed.

rson451 · 2008-11-29 18:23:45

MetalheadGautham wrote:

^^HOW is this bikeshed ? I am not flaming anybody or asking a stupid question.

Because this has been discussed many many many times in the past. Search for pacman sqlite for past discussions. I'm not trying to be an ass, just pointing out that this will get nowhere.

ckristi · 2008-11-29 18:48:57

Have you tried:

pacman-optimize

It's supposed to make access to those small files faster.

moljac024 · 2008-11-29 19:55:49

ckristi wrote:

Have you tried:
pacman-optimize
It's supposed to make access to those small files faster.

I ran that once. It broke pacman.

brando56894 · 2008-11-30 06:37:26

same here i just installed arch tonight on my dads dell, everything was working fine (well for the most part) until i optimized the database using shaman. after that downloads would start out fast then slow to a crawl in a few seconds then completely stop. the only way i could get it to download the packages was to use aria2 to download the packages and that was still really slow. any way to fix this since its really annoying that it takes close to a half hour to download about 30mb on a broadband connection and using the fastest pacman mirror.

Last edited by brando56894 (2008-11-30 06:37:41)

Allan · 2008-11-30 06:43:00

brando56894: the database has nothing to do with download speed. That is a network issue.

brando56894 · 2008-11-30 20:01:10

it must be a configuration problem then because its definitely not my connection.

heres my pacman.conf:

#
#
# /etc/pacman.conf
#
# See the pacman manpage for option directives

#
# GENERAL OPTIONS
#
[options]
UseSyslog
NoPassiveFtp
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
HoldPkg     = pacman glibc
XferCommand = /usr/bin/aria2c -s 2 -m 2 -d / -o %o %u

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# Testing is disabled by default.  To enable, uncomment the following
# two lines.  You can add preferred servers immediately after the header,
# and they will be used before the default mirrors.
#[testing]
#Include = /etc/pacman.d/mirrorlist

[core]
Server=ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/i686
# Add your preferred servers here, they will be used first

[extra]
Server=ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/i686
# Add your preferred servers here, they will be used first

[community]
Server=ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/i686
# Add your preferred servers here, they will be used first

# Unstable is disabled by default.  To enable, uncomment the following
# two lines.  You can add preferred servers immediately after the header,
# and they will be used before the default mirrors.
[unstable]
Server=ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/i686

[kdemod-legacy]
Server = http://kdemod.ath.cx/repo/legacy/i686

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#Server = file:///home/custompkgs

and heres my mirrorlist:

#
# Arch Linux repository mirrorlist
#

Server = ftp://mirror.cs.vt.edu/pub/ArchLinux/$repo/os/i686

# United States
#Server = ftp://ftp-linux.cc.gatech.edu/pub/linux/distributions/archlinux/$repo/os/i686
#Server = ftp://ftp.archlinux.org/$repo/os/i686
#Server = ftp://locke.suu.edu/linux/dist/archlinux/$repo/os/i686
#Server = ftp://mirrors.unixheads.org/archlinux/$repo/os/i686
#Server = http://mirrors.easynews.com/linux/archlinux/$repo/os/i686
#Server = ftp://ftp.ibiblio.org/pub/linux/distributions/archlinux/$repo/os/i686
#Server = http://holmes.umflint.edu/archlinux/$repo/os/i686
#Server = http://mirror.neotuli.net/arch/$repo/os/i686

i really love arch and it took me about 8 hours last night to install it on my dads computer (what im on now, i ran into a lot of problems namely with alsa, grub and x) and pacman barely works. I really dont feel like reinstalling arch again. I have arch on my computer up at school and have never had this sort of problem before. this all seemed to happen when i optimized the database after arch wouldnt update anything because it said that all the packages already existed. I performed all the maintainence operations and boom! this happens

i also created a new reiserfs partition for /var since i read that it wasnt a good idea to have /var on a JFS partition (my / and /home are JFS partitions), but that didnt seem to make any difference at all. is there a way to completely wipe out all of pacman and start from scratch? everything else works fine except for pacman.

Last edited by brando56894 (2008-11-30 20:10:22)

SamC · 2008-11-30 20:12:28

@brando56894: How on earth would the database affect download speed? Perhaps it's your choice of filesystem or mirror that's affecting it.

brando56894 · 2008-11-30 21:08:36

I have absolutely no idea, hence the reason why im posting this here.

I dont accept the solution "oh its the mirrors fault" or "its your choice of filesystem" because ive searched the forum and other people have had the same problems, and they keep saying that they've tried multiple mirrors and the same thing keeps happening so its obviously a software error (database/configuration/etc...)

When I try to use shaman now it gives me an error that says "libalpm did not respond" and then something about no changes were made.

IMHO pacman sucks compared to apt. I started out using kubuntu and I loved how fast synaptic/apt was but kubuntu was to generalized to give great performance, so I switched to arch and I love it but pacman fails in comparison to apt

SamC · 2008-11-30 21:15:38

@brando56894: Why not try debian? It's got apt, but doesn't come with so much stuff preinstalled

patroclo7 · 2008-11-30 22:02:04

Yes, many of us have absolutely no problem or slowness with pacman, but if you hate it use an apt-get based distribution (many are much lighter and faster than kubuntu).

Cotton · 2009-02-03 23:06:36

@brando56894: Have you tried the rankmirrors script to identify the fastest mirrors for you?
Enable all the sites in your mirrorlist, then run:

rankmirrors -t /etc/pacman.d/mirrorlist

Put the fastest one at the top of the file.
For me, this resulted in pacman -Syuw completing in under 5 seconds (1.5 GHz CPU, all partitions ext3, well ext4 actually)

Arch Linux

#1 2008-11-23 13:18:01

Slow database performance - WHY ? WHY ? WHY ?

#2 2008-11-23 13:33:59

Re: Slow database performance - WHY ? WHY ? WHY ?

#3 2008-11-25 00:36:20

Re: Slow database performance - WHY ? WHY ? WHY ?

#4 2008-11-25 15:09:08

Re: Slow database performance - WHY ? WHY ? WHY ?

#5 2008-11-25 17:20:45

Re: Slow database performance - WHY ? WHY ? WHY ?

#6 2008-11-25 18:02:25

Re: Slow database performance - WHY ? WHY ? WHY ?

#7 2008-11-25 18:43:13

Re: Slow database performance - WHY ? WHY ? WHY ?

#8 2008-11-28 01:07:19

Re: Slow database performance - WHY ? WHY ? WHY ?

#9 2008-11-28 01:39:58

Re: Slow database performance - WHY ? WHY ? WHY ?

#10 2008-11-28 07:38:12

Re: Slow database performance - WHY ? WHY ? WHY ?

#11 2008-11-28 09:49:24

Re: Slow database performance - WHY ? WHY ? WHY ?

#12 2008-11-28 10:13:50

Re: Slow database performance - WHY ? WHY ? WHY ?

#13 2008-11-28 13:01:58

Re: Slow database performance - WHY ? WHY ? WHY ?

#14 2008-11-29 08:25:37

Re: Slow database performance - WHY ? WHY ? WHY ?

#15 2008-11-29 18:23:45

Re: Slow database performance - WHY ? WHY ? WHY ?

#16 2008-11-29 18:48:57

Re: Slow database performance - WHY ? WHY ? WHY ?

#17 2008-11-29 19:55:49

Re: Slow database performance - WHY ? WHY ? WHY ?

#18 2008-11-30 06:37:26

Re: Slow database performance - WHY ? WHY ? WHY ?

#19 2008-11-30 06:43:00

Re: Slow database performance - WHY ? WHY ? WHY ?

#20 2008-11-30 20:01:10

Re: Slow database performance - WHY ? WHY ? WHY ?

#21 2008-11-30 20:12:28

Re: Slow database performance - WHY ? WHY ? WHY ?

#22 2008-11-30 21:08:36

Re: Slow database performance - WHY ? WHY ? WHY ?

#23 2008-11-30 21:15:38

Re: Slow database performance - WHY ? WHY ? WHY ?

#24 2008-11-30 22:02:04

Re: Slow database performance - WHY ? WHY ? WHY ?

#25 2009-02-03 23:06:36

Re: Slow database performance - WHY ? WHY ? WHY ?

Board footer