Pacman Optimization - Part 2

trapdoor · 2006-05-15 12:19:48

The script really rocks, i just did

pacman -Ss pacman

in < 1 second

L0cutus · 2006-05-15 12:24:56

i have changed fstab from:
/var/lib/pacman.db /var/lib/pacman ext2 loop,defaults 0 0

to:
/var/lib/pacman.db /var/lib/pacman ext2 loop,user,rw 0 0

now pacman -Syu doesn't stop anymore, BUT it want to install every time the same packages 8-O !

brain0 · 2006-05-15 15:14:27

On ext2, it is necessary to perform filesystem checks after unclean umounts to ensure filesystem integrity. This is not possible on a loopback file via fstab, because fsck will be run before the underlying filesystem is mounted read-write. I solved this by disabling the automatic fs check on the pacman db and doing it myself later.

Do this to automount pacman.db properly with caged pacman:

Add this line to your /etc/fstab:

# pacman db
/var/lib/pacman.db              /var/lib/pacman                 ext2    defaults,loop,noauto            0       0

and add this to your /etc/rc.local:

. /etc/rc.conf
. /etc/rc.d/functions

# Check and mount pacman database
stat_busy "Mounting the pacman database"
/sbin/e2fsck -p /var/lib/pacman.db
if [ $? -gt 1 ]; then
   stat_fail
   echo
   echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED"
   echo
else
   mount /var/lib/pacman
   if [ $? -gt 0 ]; then
      stat_fail
      echo
      echo "WARNING: MOUNTING PACMAN DATABASE FAILED"
      echo
   else
      stat_done
   fi
fi

vidal · 2006-05-15 20:14:34

I prefer having the database mounted only when necessary, so I use this small script to run pacman:

#!/bin/bash
#
# /usr/bin/pac
#

# database location
DB=/var/lib/pacman.fs

# program to run after mounting
CM=pacman


if [ `id -un` != "root" ]; then
  echo "You must have root privileges"
  exit 1
fi

if ! [ -e "$DB" ]; then
  echo "'$DB' is not accessible"
  exit 1
fi

if ! grep -q "$DB" /etc/mtab; then
  fsck -a "$DB" > /dev/null 2>&1
  if [ $? -gt 1 ]; then
    echo "Database check failed"
    exit 1
  fi
  mount -o loop -t ext2 "$DB" /var/lib/pacman
  if [ $? -gt 0 ]; then
    echo "Database mount failed"
    exit 1
  fi
fi

"$CM" $*
RET=$?

umount "$DB"

exit $RET

L0cutus · 2006-05-16 08:27:27

vidal wrote:

I prefer having the database mounted only when necessary, so I use this small script to run pacman:
[/code]

There are problem with makepkg if you use this script,
and also with users use of pacman since you cannot
do a simple 'pacman -Ss package' from an user account...

vidal · 2006-05-16 11:43:01

L0cutus wrote:

vidal wrote:
I prefer having the database mounted only when necessary, so I use this small script to run pacman:
[/code]
There are problem with makepkg if you use this script,
and also with users use of pacman since you cannot
do a simple 'pacman -Ss package' from an user account...

You should create a similar script for running makepkg. All you have to do is change the command that is run after mounting the database. I have edited the script to make that easier. Makepkg is supposed to be run as root, so there shouldn't be any more issues.
I can't think of a simple way of making the script useable for a non-privileged user.

brain0 · 2006-05-16 12:02:53

vidal wrote:

All you have to do is change the command that is run after mounting the database. I have edited the script to make that easier. Makepkg is supposed to be run as root, so there shouldn't be any more issues.

Wrong. makepkg is absolutely not supposed to be run as root.

I can't think of a simple way of making the script useable for a non-privileged user.

I can ...

vidal · 2006-05-16 12:34:10

brain0 wrote:

Wrong. makepkg is absolutely not supposed to be run as root.

I know, fakeroot.

I can't think of a simple way of making the script useable for a non-privileged user.
I can ...

Well, I'm all ears.

Thikasabrik · 2006-05-26 17:37:37

I thought I'd add two points to this topic.

1. Those scripts can easily lead to erradication of the pacman db.. use with care! (eg. if one runs pacman uncage without the db mounted... :shock: )

2. If you use something more like...

mkfs.ext2 -O dir_index -b 1024 -i 1024 -m 0 $pacmandb

...to make the dd filesystem you will be able to get away with a much smaller size for the file. Mine is 30mb right now, could be smaller. Since we are dealing with a lot of dirs and tiny files, making the size of an inode as small as possible prevents running up against the inode limit, which is what you are hitting when the db won't fit inside 60mb... To see what I mean compare the output of df -h with df -ih for a pacman.db from the prev. scripts.

kakabaratruskia · 2006-06-25 18:05:17

DAMN!!! I've lost my pacman.db with the fsking pacman-uncage script!!!! please someone help!!!

Thikasabrik · 2006-06-25 19:03:12

Well, I hate to say I told you so... so I'll offer what I can instead
If you have some kind of backup, even if it's from a while back, you can probably work something out. If you know roughly you've installed anew since the backup then you can manually add those packages back into the ./local folder once the backup is restored, using the corresponding pkg.tar.gz files to obtain a file list, which you will need to slightly change the format of (see existing ./local/ package entries). As for upgrades try a trial pacman -Su to see what got upgraded, then replace the ./local entries correspondingly. A pain in the arse, yes, and it does require you to have a backup... If not the same process could potentially work but would be mind-bogglingly tedious. You could always start from an existing db from a system with similar packages.

I suppose theortically a 'recover' mode could be created for pacman: If each package listed a key file, or set of key files, in it's description then, on database damage, pacman could scan the drive for these key files, prompting the user in cases of uncertainty, and restoring the db step by step. However, not messing around with the db too much is probably a more sensible solution..

I think it's a good idea to recommend performing the steps for uncaging, or even caging, manually until someone comes up with a safer script.

kakabaratruskia · 2006-06-25 19:11:47

I'm reintsalling every repo.... I'll later delete the packages I don't need

bud · 2006-07-12 22:15:44

my god, the speed is awesome : DDDD
thx for this!

kozaki · 2006-07-23 09:14:05

Got a strange situation in this location.
Got pacman-cage installed back 10th of April.
As I'm running out of place in "/" partition, I checked what was taking the room here. /var/lib/

drwxr-xr-x 40 root root 1024 Jul 23 01:11 pacman
drwxr-xr-x 38 root root 4096 Apr 10 19:21 pacman.bak
-rw-r--r-- 1 root root 157286400 Apr 10 20:16 pacman.db
drwxr-xr-x 40 root root 4096 Apr 12 00:07 pacman.new

/pacman.bak was created when I first runned pacman-cage ; it makes 180 Mo !
Ever less understandable to me there's a /pacman.new/pacman/ created 07/19/06 (with /bfinch, /brice, /cjdj & /community sub-directories) !?

Made a backup then deleted /pacman.new, and pacman (or yaourt) -Syu work AFAIK

cromo · 2006-10-19 20:40:30

I just caged the pacman and have the same concern as kozaki has - I got three pacman dirs now in /var/lib: pacman, pacman.bak and pacman.new. The latter one was created after the first pacman -Syu after caging. Why is it like this? Also, the dir sizes are weird as well. pacman.bak is only 7MB, so is pacman and pacman.new, but pacman.db is 170. But I am pretty sure that I checked a while ago and it was pacman.bak that was 180MB and pacman.db was only 20MB??? I am getting scared of this thing :?

OK. I think I get it now. I run pacman-optimize after caging, which obviously was unsuccessful. I am pretty sure kozaki has done the same thing. pacman-optimize creates pacman.new and also does same directory moving, thus these weird directory structures, kozaki.

Zanton · 2006-10-25 19:57:56

I think I made a mistake : I did not read the previous post on pacman optimization, and I just ran this pacman-uncage script. Now my pacman doesn't work at all : no update via -Syu, and if I try to build a package from AUR, the dependancies are completely f****** up.
So, to be sure, did I do wrong, and if yes, is it possible to fix the mess I deserved ?

kozaki · 2006-10-26 17:43:29

Zanton > I guess the answer shows up on first post in this topic. Yes it is recoverable, _if_ you made a backup before (especially pacman/local as the rest can be rebuild).

as for /var/lib/pacman.new/pacman/... it is re-created whatsoever !?!

18    /var/lib/pacman
5    /var/lib/slocate
1    /var/lib/openldap
91    /var/lib/pacman.new

My /var/lib/pacman.db got totally corrupted. And I have had 4 total freezes last 8 days, all of them when running pacman -Syu...

Anyway I (made a backup &) unmounted it and runned fsck.ext2 -fv on it, which printed over 1000 error/fix messages on /var/lib/pacman.db !
2nd fsck didn't show any error or warning so I remounted it and consider it as ready-to-serve as it does not show no more error messages.

Zanton · 2006-10-26 20:47:21

I didn't make a backup, so all is lost now ? By the way, I can't find in the first post where it says it can be recovered
I just noticed I only ran pacman-uncage, not pacman-cage. Is there any hope ?

Thanks for helping

kozaki · 2006-11-04 18:15:35

I got pacman-cage from [archlinux.fr] repo & deleted fstab pacman.db entry & added 'pacmandb' in rc.conf DAEMONS's line.
Worked nice last few days.

Now I'm stuck again. I'll report with details, as it might be of interest for you guys that made pacman-cage, or users like me.
See :
$ ls -lth /var/lib/pacman/local/

total 0

$ ls -lth /var/lib/pacman.new/local/

total 2,7M
drwxr-xr-x 2 root root 4,0K oct 23 14:40 kdewebdev-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:39 kdesdk-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:21 kdepim-3.5.4-2
drwxr-xr-x 2 root root 4,0K oct 23 14:20 libmal-0.40-2
drwxr-xr-x 2 root root 4,0K oct 23 14:20 pilot-link-0.11.8-4
drwxr-xr-x 2 root root 4,0K oct 23 14:20 kdelibs-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:18 kdebase-3.5.4-6
drwxr-xr-x 2 root root 4,0K oct 23 14:18 eject-2.1.5-1
drwxr-xr-x 2 root root 4,0K oct 23 14:18 libraw1394-1.2.0-1
drwxr-xr-x 2 root root 4,0K oct 23 14:16 mtools-3.9.10-1
drwxr-xr-x 2 root root 4,0K oct 23 14:08 gnokii-0.6.12-1
drwxr-xr-x 2 root root 4,0K oct 23 13:50 kernel26-2.6.18.1-2
drwxr-xr-x 2 root root 4,0K oct 22 23:34 versionpkg-0.7.1-1
drwxr-xr-x 2 root root 4,0K oct 22 23:34 yaourt-0.4.7-1
drwxr-xr-x 2 root root 4,0K oct 22 23:34 opera-snapshot-466-20061020

$ ls -lth /var/lib/pacman.new/pacman/local/

total 3,3M
drwxr-xr-x 2 root root 4,0K nov 3 14:25 rxvt-unicode-8.0-1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 gimp-ufraw-0.10-2
drwxr-xr-x 2 root root 4,0K nov 3 14:25 amarok-base-mysqlfree-1.4.4-1.1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 amarok-engine-xine-1.4.4-1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 pacman-cage-2.9.8-2
drwxr-xr-x 2 root root 4,0K nov 3 14:25 firefox-i18n-2.0-3
drwxr-xr-x 2 root root 4,0K nov 3 00:14 4l-1.0r6-1
drwxr-xr-x 2 root root 4,0K nov 3 00:14 lightscribe-1.4.113.1-1
drwxr-xr-x 2 root root 4,0K nov 2 21:11 firefox2-fr-2.0-1
drwxr-xr-x 2 root root 4,0K nov 1 22:28 pacman-cage-2.9.8-1
drwxr-xr-x 2 root root 4,0K nov 1 21:58 man-pages-fr-2.39.0-1
drwxr-xr-x 2 root root 4,0K nov 1 15:04 kqemu-1.3.0pre9-1

WTH does it came from ?
Fortunetly there's _no_ package in /var/lib/pacman.new/local/ that doesn't show in /var/lib/pacman.new/pacman/local/. Basically those are the same but 2nd shows packages I installed last 10 days or so.
So I backed all things up and moved this to /var/lib/pacman/local & things are under back... well, for the moment.

Still got this warning which I dunno how to deal with, since it does not show elsewhere in this bbs (neither on Google):
config: line 6: all directives must belong to a section each time I run pacman or yaourt as a user or as root.

Now I want to launch pacman-uncage

# pacman-uncage
pacman-uncage: Pacmandb must be mounted

So pacman.db was not mounted last time I rebooted.
See what pacmandb's daemon from wain say :
# /etc/rc.d/pacmandb restart

:: Unmounting the pacman database [FAIL]
:: Mounting the pacman database [BUSY]
/var/lib/pacman.db: INCONSISTENCE INATTENDUE ; EXÉCUTEZ fsck MANUELLEMENT. (i.e., sans options -a ou -p)
[FAIL]
WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED

That's right. pacman.db got totally corrupted, even when I had no crash nor power failure for days. # /sbin/e2fsck -v /var/lib/pacman.db

e2fsck 1.39 (29-May-2006)
/var/lib/pacman.db contient un système de fichiers comportant des erreurs, vérification forcée.
Passe 1 : vérification des i-noeuds, des blocs et des tailles
Passe 2 : vérification de la structure des répertoires
l'entrée « 855resolution-0.4-3 » dans /??? (24338) est un lien vers le répertoire /kozaki_repo (24339).
Effacer<y>? yes (pour oui)
l'entrée « 915resolution-0.5.2-2 » dans /??? (24338) a un type de fichier incorrect (était 2, devrait être 1).
Réparer<y>? yes (pour oui)
...[hundreds of line)...

kozaki · 2006-11-04 18:26:05

First run didn't do it as pacman.new/ existed (so it may be where pacman.new came from, or rather pacman-cage, as I never run pacman-uncage before!)
# pacman-uncage

==> md5sum'ing the old database...
==> copying pacman.db contents back, note: the time needed to get a brew is now.
mkdir: ne peut créer le répertoire `/var/lib/pacman.new': Le fichier existe.
==> unmounting old dbroot and moving new one in
rmdir: /var/lib/pacman: Le répertoire n'est pas vide.
==> md5sum'ing the new database...
==> checking integrity...
pacman-uncage: integrity check FAILED, reverting to old database

But it erased pacman.new/ and 2nd run finished succesfully : # pacman-uncage

==> md5sum'ing the old database...
==> copying pacman.db contents back, note: the time needed to get a brew is now.
==> unmounting old dbroot and moving new one in
==> md5sum'ing the new database...
==> checking integrity...
==> Removing old pacman.db

+ it went fine : ls -lth pacman/local/

total 2,7M
drwxr-xr-x 2 root root 4,0K nov 4 00:29 gaim-svn-17333-1
drwxr-xr-x 2 root root 4,0K nov 4 00:28 xvidcore-1.1.2-1
drwxr-xr-x 2 root root 4,0K nov 4 00:28 opera-snapshot-478-20061103
drwxr-xr-x 2 root root 4,0K nov 3 14:25 rxvt-unicode-8.0-1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 gimp-ufraw-0.10-2
drwxr-xr-x 2 root root 4,0K nov 3 14:25 amarok-base-mysqlfree-1.4.4-1.1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 amarok-engine-xine-1.4.4-1
drwxr-xr-x 2 root root 4,0K nov 3 14:25 pacman-cage-2.9.8-2
drwxr-xr-x 2 root root 4,0K nov 3 14:25 firefox-i18n-2.0-3
drwxr-xr-x 2 root root 4,0K nov 3 00:14 4l-1.0r6-1
drwxr-xr-x 2 root root 4,0K nov 3 00:14 lightscribe-1.4.113.1-

at last I have _one_ pacman/local/ directory
I'll wait a bit before I run pacman-cage again, but I'll definitivelly run it again, now wain has improved it with 'pacmandb' check at boot time.

funkyou · 2006-12-06 11:52:25

I tried pacman-cage and it worked for some days... But suddenly my system freezed during a -Suy and the db was fragged after this... Lucky i had a db backup...

I dont know whats causing this, maybe someone alse has experienced this and has a solution, as i like the caged pacman for its speed...

sh__ · 2006-12-06 18:00:56

funkyou wrote:

I tried pacman-cage and it worked for some days... But suddenly my system freezed during a -Suy and the db was fragged after this... Lucky i had a db backup...

This has happened a couple of times to me too during pacman -Syu. Emergency sync with AltGr + SysRq + S usually (= when you are lucky) unfreezes the system. If you had a complete lockup, you need to run e2fsck on the caged filesystem afterwards.

I don't have an idea what causes it. Are you using laptop-mode-tools?

funkyou · 2006-12-06 19:34:33

sh__ wrote:

I don't have an idea what causes it. Are you using laptop-mode-tools?

No, its a desktop machine... The only app i use for powersaving etc is athcool...

caust1c · 2006-12-08 17:19:07

i've got the same problem.
I'm using the beyond kernel, and got some lockups too
thinking of switching back to normal pacman :E

kozaki · 2006-12-18 08:54:27

I have had no more prob since installed improved pacman-cage from archlinux.fr repo. So I recommand anyone wanting to use a more robust pacman-cage try this now wain has improved it with 'pacmandb' check at boot time (add it to your DAEMONS line in /etc/rc.conf).

Arch Linux

#26 2006-05-15 12:19:48

Re: Pacman Optimization - Part 2

#27 2006-05-15 12:24:56

Re: Pacman Optimization - Part 2

#28 2006-05-15 15:14:27

Re: Pacman Optimization - Part 2

#29 2006-05-15 20:14:34

Re: Pacman Optimization - Part 2

#30 2006-05-16 08:27:27

Re: Pacman Optimization - Part 2

#31 2006-05-16 11:43:01

Re: Pacman Optimization - Part 2

#32 2006-05-16 12:02:53

Re: Pacman Optimization - Part 2

#33 2006-05-16 12:34:10

Re: Pacman Optimization - Part 2

#34 2006-05-26 17:37:37

Re: Pacman Optimization - Part 2

#35 2006-06-25 18:05:17

Re: Pacman Optimization - Part 2

#36 2006-06-25 19:03:12

Re: Pacman Optimization - Part 2

#37 2006-06-25 19:11:47

Re: Pacman Optimization - Part 2

#38 2006-07-12 22:15:44

Re: Pacman Optimization - Part 2

#39 2006-07-23 09:14:05

Re: Pacman Optimization - Part 2

#40 2006-10-19 20:40:30

Re: Pacman Optimization - Part 2

#41 2006-10-25 19:57:56

Re: Pacman Optimization - Part 2

#42 2006-10-26 17:43:29

Re: Pacman Optimization - Part 2

#43 2006-10-26 20:47:21

Re: Pacman Optimization - Part 2

#44 2006-11-04 18:15:35

Re: Pacman Optimization - Part 2

#45 2006-11-04 18:26:05

Re: Pacman Optimization - Part 2

#46 2006-12-06 11:52:25

Re: Pacman Optimization - Part 2

#47 2006-12-06 18:00:56

Re: Pacman Optimization - Part 2

#48 2006-12-06 19:34:33

Re: Pacman Optimization - Part 2

#49 2006-12-08 17:19:07

Re: Pacman Optimization - Part 2

#50 2006-12-18 08:54:27

Re: Pacman Optimization - Part 2

Board footer