You are not logged in.

#26 2006-05-15 12:19:48

trapdoor
Member
Registered: 2005-03-27
Posts: 82
Website

Re: Pacman Optimization - Part 2

The script really rocks, i just did

pacman -Ss pacman

in < 1 second

Offline

#27 2006-05-15 12:24:56

L0cutus
Member
Registered: 2005-11-17
Posts: 24

Re: Pacman Optimization - Part 2

i have changed fstab from:
/var/lib/pacman.db     /var/lib/pacman ext2       loop,defaults      0 0

to:
/var/lib/pacman.db     /var/lib/pacman ext2       loop,user,rw      0 0

now pacman -Syu doesn't stop anymore, BUT it want to install every time the same packages 8-O !

Offline

#28 2006-05-15 15:14:27

brain0
Developer
From: Aachen - Germany
Registered: 2005-01-03
Posts: 1,382

Re: Pacman Optimization - Part 2

On ext2, it is necessary to perform filesystem checks after unclean umounts to ensure filesystem integrity. This is not possible on a loopback file via fstab, because fsck will be run before the underlying filesystem is mounted read-write. I solved this by disabling the automatic fs check on the pacman db and doing it myself later.

Do this to automount pacman.db properly with caged pacman:

Add this line to your /etc/fstab:

# pacman db
/var/lib/pacman.db              /var/lib/pacman                 ext2    defaults,loop,noauto            0       0

and add this to your /etc/rc.local:

. /etc/rc.conf
. /etc/rc.d/functions

# Check and mount pacman database
stat_busy "Mounting the pacman database"
/sbin/e2fsck -p /var/lib/pacman.db
if [ $? -gt 1 ]; then
   stat_fail
   echo
   echo "WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED"
   echo
else
   mount /var/lib/pacman
   if [ $? -gt 0 ]; then
      stat_fail
      echo
      echo "WARNING: MOUNTING PACMAN DATABASE FAILED"
      echo
   else
      stat_done
   fi
fi

Offline

#29 2006-05-15 20:14:34

vidal
Member
Registered: 2006-04-03
Posts: 12

Re: Pacman Optimization - Part 2

I prefer having the database mounted only when necessary, so I use this small script to run pacman:

#!/bin/bash
#
# /usr/bin/pac
#

# database location
DB=/var/lib/pacman.fs

# program to run after mounting
CM=pacman


if [ `id -un` != "root" ]; then
  echo "You must have root privileges"
  exit 1
fi

if ! [ -e "$DB" ]; then
  echo "'$DB' is not accessible"
  exit 1
fi

if ! grep -q "$DB" /etc/mtab; then
  fsck -a "$DB" > /dev/null 2>&1
  if [ $? -gt 1 ]; then
    echo "Database check failed"
    exit 1
  fi
  mount -o loop -t ext2 "$DB" /var/lib/pacman
  if [ $? -gt 0 ]; then
    echo "Database mount failed"
    exit 1
  fi
fi

"$CM" $*
RET=$?

umount "$DB"

exit $RET

Offline

#30 2006-05-16 08:27:27

L0cutus
Member
Registered: 2005-11-17
Posts: 24

Re: Pacman Optimization - Part 2

vidal wrote:

I prefer having the database mounted only when necessary, so I use this small script to run pacman:
[/code]

There are problem with makepkg if you use this script,
and also with users use of pacman since you cannot
do a simple 'pacman -Ss package' from an user account...

Offline

#31 2006-05-16 11:43:01

vidal
Member
Registered: 2006-04-03
Posts: 12

Re: Pacman Optimization - Part 2

L0cutus wrote:
vidal wrote:

I prefer having the database mounted only when necessary, so I use this small script to run pacman:
[/code]

There are problem with makepkg if you use this script,
and also with users use of pacman since you cannot
do a simple 'pacman -Ss package' from an user account...

You should create a similar script for running makepkg. All you have to do is change the command that is run after mounting the database. I have edited the script to make that easier. Makepkg is supposed to be run as root, so there shouldn't be any more issues.
I can't think of a simple way of making the script useable for a non-privileged user.

Offline

#32 2006-05-16 12:02:53

brain0
Developer
From: Aachen - Germany
Registered: 2005-01-03
Posts: 1,382

Re: Pacman Optimization - Part 2

vidal wrote:

All you have to do is change the command that is run after mounting the database. I have edited the script to make that easier. Makepkg is supposed to be run as root, so there shouldn't be any more issues.

Wrong. makepkg is absolutely not supposed to be run as root.

I can't think of a simple way of making the script useable for a non-privileged user.

I can ...

Offline

#33 2006-05-16 12:34:10

vidal
Member
Registered: 2006-04-03
Posts: 12

Re: Pacman Optimization - Part 2

brain0 wrote:

Wrong. makepkg is absolutely not supposed to be run as root.

I know, fakeroot.

I can't think of a simple way of making the script useable for a non-privileged user.

I can ...

Well, I'm all ears.

Offline

#34 2006-05-26 17:37:37

Thikasabrik
Member
Registered: 2004-02-23
Posts: 92

Re: Pacman Optimization - Part 2

I thought I'd add two points to this topic.

1. Those scripts can easily lead to erradication of the pacman db.. use with care! (eg. if one runs pacman uncage without the db mounted... :shock: )

2. If you use something more like...

mkfs.ext2 -O dir_index -b 1024 -i 1024 -m 0 $pacmandb

...to make the dd filesystem you will be able to get away with a much smaller size for the file. Mine is 30mb right now, could be smaller. Since we are dealing with a lot of dirs and tiny files, making the size of an inode as small as possible prevents running up against the inode limit, which is what you are hitting when the db won't fit inside 60mb... To see what I mean compare the output of df -h with df -ih for a pacman.db from the prev. scripts.

Offline

#35 2006-06-25 18:05:17

kakabaratruskia
Member
From: Santiago, Chile
Registered: 2003-08-24
Posts: 596

Re: Pacman Optimization - Part 2

DAMN!!! I've lost my pacman.db with the fsking pacman-uncage script!!!! please someone help!!!


And where were all the sportsmen who always pulled you though?
They're all resting down in Cornwall
writing up their memoirs for a paper-back edition
of the Boy Scout Manual.

Offline

#36 2006-06-25 19:03:12

Thikasabrik
Member
Registered: 2004-02-23
Posts: 92

Re: Pacman Optimization - Part 2

Well, I hate to say I told you so... so I'll offer what I can instead  wink
If you have some kind of backup, even if it's from a while back, you can probably work something out. If you know roughly you've installed anew since the backup then you can manually add those packages back into the ./local folder once the backup is restored, using the corresponding pkg.tar.gz files to obtain a file list, which you will need to slightly change the format of (see existing ./local/ package entries). As for upgrades try a trial pacman -Su to see what got upgraded, then replace the ./local entries correspondingly. A pain in the arse, yes, and it does require you to have a backup... If not the same process could potentially work but would be mind-bogglingly tedious. You could always start from an existing db from a system with similar packages.

I suppose theortically a 'recover' mode could be created for pacman: If each package listed a key file, or set of key files, in it's description then, on database damage, pacman could scan the drive for these key files, prompting the user in cases of uncertainty, and restoring the db step by step. However, not messing around with the db too much is probably a more sensible solution..

I think it's a good idea to recommend performing the steps for uncaging, or even caging, manually until someone comes up with a safer script.

Offline

#37 2006-06-25 19:11:47

kakabaratruskia
Member
From: Santiago, Chile
Registered: 2003-08-24
Posts: 596

Re: Pacman Optimization - Part 2

I'm reintsalling every repo.... I'll later delete the packages I don't need


And where were all the sportsmen who always pulled you though?
They're all resting down in Cornwall
writing up their memoirs for a paper-back edition
of the Boy Scout Manual.

Offline

#38 2006-07-12 22:15:44

bud
Member
From: swe
Registered: 2006-07-12
Posts: 74

Re: Pacman Optimization - Part 2

my god, the speed is awesome : DDDD
thx for this!


Hello, I am normal!

Offline

#39 2006-07-23 09:14:05

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Pacman Optimization - Part 2

Got a strange situation in this location.
Got pacman-cage installed back 10th of April.
As I'm running out of place in "/" partition, I checked what was taking the room here. /var/lib/

drwxr-xr-x 40 root root         1024 Jul 23 01:11 pacman
drwxr-xr-x 38 root root         4096 Apr 10 19:21 pacman.bak
-rw-r--r--  1 root root    157286400 Apr 10 20:16 pacman.db
drwxr-xr-x 40 root root         4096 Apr 12 00:07 pacman.new

/pacman.bak was created when I first runned pacman-cage ; it makes  180 Mo !
Ever less understandable to me there's a /pacman.new/pacman/ created 07/19/06 (with /bfinch, /brice, /cjdj & /community sub-directories) !?

Made a backup then deleted /pacman.new, and pacman (or yaourt) -Syu work AFAIK


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#40 2006-10-19 20:40:30

cromo
Member
From: Czestochowa, Poland
Registered: 2006-09-20
Posts: 87

Re: Pacman Optimization - Part 2

I just caged the pacman and have the same concern as kozaki has - I got three pacman dirs now in /var/lib: pacman, pacman.bak and pacman.new. The latter one was created after the first pacman -Syu after caging. Why is it like this? Also, the dir sizes are weird as well. pacman.bak is only 7MB, so is pacman and pacman.new, but pacman.db is 170. But I am pretty sure that I checked a while ago and it was pacman.bak that was 180MB and pacman.db was only 20MB??? I am getting scared of this thing  :?

OK. I think I get it now. I run pacman-optimize after caging, which obviously was unsuccessful. I am pretty sure kozaki has done the same thing. pacman-optimize creates pacman.new and also does same directory moving, thus these weird directory structures, kozaki.

Offline

#41 2006-10-25 19:57:56

Zanton
Member
Registered: 2005-12-06
Posts: 80

Re: Pacman Optimization - Part 2

I think I made a mistake : I did not read the previous post on pacman optimization, and I just ran this pacman-uncage script. Now my pacman doesn't work at all : no update via -Syu, and if I try to build a package from AUR, the dependancies are completely f****** up.
So, to be sure, did I do wrong, and if yes, is it possible to fix the mess I deserved ?

Offline

#42 2006-10-26 17:43:29

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Pacman Optimization - Part 2

Zanton  > I guess the answer shows up on first post in this topic. Yes it is recoverable, _if_ you made a backup before (especially pacman/local as the rest can be rebuild).

as for /var/lib/pacman.new/pacman/... it is re-created whatsoever !?!

18    /var/lib/pacman
5    /var/lib/slocate
1    /var/lib/openldap
91    /var/lib/pacman.new

My /var/lib/pacman.db got totally corrupted. And I have had 4 total freezes last 8 days, all of them when running pacman -Syu... a032.gif

Anyway I (made a backup &) unmounted it and runned fsck.ext2 -fv on it, which printed over 1000 error/fix messages on /var/lib/pacman.db !
2nd fsck didn't show any error or warning so I remounted it and consider it as ready-to-serve as it does not show no more error messages.


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#43 2006-10-26 20:47:21

Zanton
Member
Registered: 2005-12-06
Posts: 80

Re: Pacman Optimization - Part 2

I didn't make a backup, so all is lost now ? By the way, I can't find in the first post where it says it can be recovered hmm
I just noticed I only ran pacman-uncage, not pacman-cage. Is there any hope ?

Thanks for helping

Offline

#44 2006-11-04 18:15:35

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Pacman Optimization - Part 2

I got pacman-cage from [archlinux.fr] repo & deleted fstab pacman.db entry & added 'pacmandb' in rc.conf DAEMONS's line.
Worked nice last few days.

Now I'm stuck again. I'll report with details, as it might be of interest for you guys that made pacman-cage, or users like me.
See :
$ ls -lth /var/lib/pacman/local/

total 0

$ ls -lth /var/lib/pacman.new/local/

total 2,7M
drwxr-xr-x 2 root root 4,0K oct 23 14:40 kdewebdev-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:39 kdesdk-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:21 kdepim-3.5.4-2
drwxr-xr-x 2 root root 4,0K oct 23 14:20 libmal-0.40-2
drwxr-xr-x 2 root root 4,0K oct 23 14:20 pilot-link-0.11.8-4
drwxr-xr-x 2 root root 4,0K oct 23 14:20 kdelibs-3.5.4-1
drwxr-xr-x 2 root root 4,0K oct 23 14:18 kdebase-3.5.4-6
drwxr-xr-x 2 root root 4,0K oct 23 14:18 eject-2.1.5-1
drwxr-xr-x 2 root root 4,0K oct 23 14:18 libraw1394-1.2.0-1
drwxr-xr-x 2 root root 4,0K oct 23 14:16 mtools-3.9.10-1
drwxr-xr-x 2 root root 4,0K oct 23 14:08 gnokii-0.6.12-1
drwxr-xr-x 2 root root 4,0K oct 23 13:50 kernel26-2.6.18.1-2
drwxr-xr-x 2 root root 4,0K oct 22 23:34 versionpkg-0.7.1-1
drwxr-xr-x 2 root root 4,0K oct 22 23:34 yaourt-0.4.7-1
drwxr-xr-x 2 root root 4,0K oct 22 23:34 opera-snapshot-466-20061020

$ ls -lth /var/lib/pacman.new/pacman/local/

total 3,3M
drwxr-xr-x 2 root root 4,0K nov  3 14:25 rxvt-unicode-8.0-1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 gimp-ufraw-0.10-2
drwxr-xr-x 2 root root 4,0K nov  3 14:25 amarok-base-mysqlfree-1.4.4-1.1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 amarok-engine-xine-1.4.4-1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 pacman-cage-2.9.8-2
drwxr-xr-x 2 root root 4,0K nov  3 14:25 firefox-i18n-2.0-3
drwxr-xr-x 2 root root 4,0K nov  3 00:14 4l-1.0r6-1
drwxr-xr-x 2 root root 4,0K nov  3 00:14 lightscribe-1.4.113.1-1
drwxr-xr-x 2 root root 4,0K nov  2 21:11 firefox2-fr-2.0-1
drwxr-xr-x 2 root root 4,0K nov  1 22:28 pacman-cage-2.9.8-1
drwxr-xr-x 2 root root 4,0K nov  1 21:58 man-pages-fr-2.39.0-1
drwxr-xr-x 2 root root 4,0K nov  1 15:04 kqemu-1.3.0pre9-1

WTH does it came from ?
Fortunetly there's _no_ package in /var/lib/pacman.new/local/ that doesn't show in /var/lib/pacman.new/pacman/local/. Basically those are the same but 2nd shows packages I installed last 10 days or so.
So I backed all things up and moved this to /var/lib/pacman/local & things are under back... well, for the moment.

Still got this warning which I dunno how to deal with, since it does not show elsewhere in this bbs (neither on Google):
config: line 6: all directives must belong to a section each time I run pacman or yaourt as a user or as root.

Now I want to launch pacman-uncage

# pacman-uncage
pacman-uncage: Pacmandb must be mounted

So pacman.db was not mounted last time I rebooted.
See what pacmandb's daemon from wain say :
# /etc/rc.d/pacmandb restart

:: Unmounting the pacman database                                                                                                                                                                                                       [FAIL]
:: Mounting the pacman database                                                                                                                                                                                                         [BUSY]

/var/lib/pacman.db: INCONSISTENCE INATTENDUE ; EXÉCUTEZ fsck MANUELLEMENT.        (i.e., sans options -a ou -p)
                                                                                                                                                                                                                                        [FAIL]

WARNING: PACMAN DATABASE FILESYSTEM CHECK FAILED, NOT MOUNTED

That's right. pacman.db got totally corrupted, even when I had no crash nor power failure for days. # /sbin/e2fsck -v /var/lib/pacman.db

e2fsck 1.39 (29-May-2006)
/var/lib/pacman.db contient un système de fichiers comportant des erreurs, vérification forcée.
Passe 1 : vérification des i-noeuds, des blocs et des tailles
Passe 2 : vérification de la structure des répertoires
l'entrée « 855resolution-0.4-3 » dans /??? (24338) est un lien vers le répertoire /kozaki_repo (24339).
Effacer<y>? yes (pour oui)
l'entrée « 915resolution-0.5.2-2 » dans /??? (24338) a un type de fichier incorrect (était 2, devrait être 1).
Réparer<y>? yes (pour oui)
...[hundreds of line)...


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#45 2006-11-04 18:26:05

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Pacman Optimization - Part 2

First run didn't do it as pacman.new/ existed (so it may be where pacman.new came from, or rather pacman-cage, as I never run pacman-uncage before!)
# pacman-uncage

==> md5sum'ing the old database...
==> copying pacman.db contents back, note: the time needed to get a brew is now.
mkdir: ne peut créer le répertoire `/var/lib/pacman.new': Le fichier existe.
==> unmounting old dbroot and moving new one in
rmdir: /var/lib/pacman: Le répertoire n'est pas vide.
==> md5sum'ing the new database...
==> checking integrity...
pacman-uncage: integrity check FAILED, reverting to old database

But it erased pacman.new/ and 2nd run finished succesfully : # pacman-uncage

==> md5sum'ing the old database...
==> copying pacman.db contents back, note: the time needed to get a brew is now.
==> unmounting old dbroot and moving new one in
==> md5sum'ing the new database...
==> checking integrity...
==> Removing old pacman.db

+ it went fine : ls -lth pacman/local/

total 2,7M
drwxr-xr-x 2 root root 4,0K nov  4 00:29 gaim-svn-17333-1
drwxr-xr-x 2 root root 4,0K nov  4 00:28 xvidcore-1.1.2-1
drwxr-xr-x 2 root root 4,0K nov  4 00:28 opera-snapshot-478-20061103
drwxr-xr-x 2 root root 4,0K nov  3 14:25 rxvt-unicode-8.0-1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 gimp-ufraw-0.10-2
drwxr-xr-x 2 root root 4,0K nov  3 14:25 amarok-base-mysqlfree-1.4.4-1.1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 amarok-engine-xine-1.4.4-1
drwxr-xr-x 2 root root 4,0K nov  3 14:25 pacman-cage-2.9.8-2
drwxr-xr-x 2 root root 4,0K nov  3 14:25 firefox-i18n-2.0-3
drwxr-xr-x 2 root root 4,0K nov  3 00:14 4l-1.0r6-1
drwxr-xr-x 2 root root 4,0K nov  3 00:14 lightscribe-1.4.113.1-

at last I have _one_ pacman/local/ directory smile
I'll wait a bit before I run pacman-cage again, but I'll definitivelly run it again, now wain has improved it with 'pacmandb' check at boot time.


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

#46 2006-12-06 11:52:25

funkyou
Member
From: Berlin, DE
Registered: 2006-03-19
Posts: 848
Website

Re: Pacman Optimization - Part 2

I tried pacman-cage and it worked for some days... But suddenly my system freezed during a -Suy and the db was fragged after this... Lucky i had a db backup...

I dont know whats causing this, maybe someone alse has experienced this and has a solution, as i like the caged pacman for its speed...


want a modular and tweaked KDE for arch? try kdemod

Offline

#47 2006-12-06 18:00:56

sh__
Member
Registered: 2005-07-19
Posts: 272

Re: Pacman Optimization - Part 2

funkyou wrote:

I tried pacman-cage and it worked for some days... But suddenly my system freezed during a -Suy and the db was fragged after this... Lucky i had a db backup...

This has happened a couple of times to me too during pacman -Syu. Emergency sync with AltGr + SysRq + S usually (= when you are lucky) unfreezes the system.  If you had a complete lockup, you need to run e2fsck on the caged filesystem afterwards.

I don't have an idea what causes it. Are you using laptop-mode-tools?

Offline

#48 2006-12-06 19:34:33

funkyou
Member
From: Berlin, DE
Registered: 2006-03-19
Posts: 848
Website

Re: Pacman Optimization - Part 2

sh__ wrote:

I don't have an idea what causes it. Are you using laptop-mode-tools?

No, its a desktop machine... The only app i use for powersaving etc is athcool...


want a modular and tweaked KDE for arch? try kdemod

Offline

#49 2006-12-08 17:19:07

caust1c
Member
Registered: 2005-12-02
Posts: 56

Re: Pacman Optimization - Part 2

i've got the same problem.
I'm using the beyond kernel, and got some lockups too hmm
thinking of switching back to normal pacman :E

Offline

#50 2006-12-18 08:54:27

kozaki
Member
From: London >. < Paris
Registered: 2005-06-13
Posts: 671
Website

Re: Pacman Optimization - Part 2

I have had no more prob since installed improved pacman-cage from archlinux.fr repo. So I recommand anyone wanting to use a more robust pacman-cage try this now wain has improved it with 'pacmandb' check at boot time (add it to your DAEMONS line in /etc/rc.conf).


Seeded last month: Arch 50 gig, derivatives 1 gig
Desktop @3.3GHz 8 gig RAM, linux-ck
laptop #1 Atom 2 gig RAM, Arch linux stock i686 (6H w/ 6yrs old battery smile) #2: ARM Tegra K1, 4 gig RAM, ChrOS
Atom Z520 2 gig RAM, OMV (Debian 7) kernel 3.16 bpo on SDHC | PGP Key: 0xFF0157D9

Offline

Board footer

Powered by FluxBB