/var/cache/pacman/pkg "management"

jeff story · 2010-12-20 05:54:27

What I have learned / done so far:

1) After getting bit by a recent nvidia driver bug https://bbs.archlinux.org/viewtopic.php?id=106945 that resulted in an Xorg starting failure , I

a) For the second time, was screwed from a simple downgrade solution by my brutal method of managing my /var/cache/pacman/pkg directory size by, deleting based on file size, largest to 2mb files get deleted, and "hope for the best" plus an occasional pacman -Sc command.

b) Temporarily loaded the nouveau driver to be disappointed with instability issues. I got lucky getting Xorg back a few days later with nvidia after restoring it and a pacman -Syu upgrade.

c) Installed a second, complete and separate, fresh Arch install in addition to my original install as a backup on my main computer. This time, I created a 20GB partition to mount as /var/cache/pacman/pkg !!!

All was good until I realized what a monster I had in the making when I'd have to "manage" a full 20GB pacman pkg partition down the road.

So a few quick Google searches were in order. I found this thread in the forum https://bbs.archlinux.org/viewtopic.php?id=9104 that contained a python script that would possibly manage my /var/cache/pacman/pkg directory with the finess I was looking for.

That thread led me to an AUR http://aur.archlinux.org/packages.php?ID=37572 which I wasn't sure I wanted to install, due to my lack of complete understanding of "what and where" would be installed to manage something as critical as the pacman pkg cache. I thought it may be better in my case to just learn how to use the script.

So after a bit more googling on python scripts, I did the following

1) created a directory on my desktop called packman-pkg-cache-cleaner

2) created a file in it called pacman.cache.cleaner

3) copied and pasted the updated script from https://wiki.archlinux.org/index.php/CacheClean into it

4) removed the "env " from the first line of the script

5) changed the permissions to allow anyone to execute it

6) opened a terminal in the directory and after a few tries, figured out the following command

[jeff@Arch2009p2 packman-pkg-cache-cleaner]$ sudo ./pacman.cache.cleaner 2

then it asked and gave me this info

Password: 
File 'lshw-B.02.15-1-x86_64.pkg.tar.xz' doesn't match package pattern!

Then I used a file browser to see what was left in my /var/cache/pacman/pkg directory.
It appears to have 2 of every package, so I'm thinking it worked!!!

During all this, I also found this from an Arch forum post. http://arm.konnichi.com/search/index.ph … ommunity=1
Its an archive of older arch packages. Possibly an alternative to all the above. No idea of the "safety" etc... Perhaps someone who knows more can add.......
EDIT 1 Never mind....ARM architecture packages only.
EDIT 2 My error. I was thinking ARM http://en.wikipedia.org/wiki/ARM_architecture
Arch ARM= Arch Rollback Machine. I just learned of this.

My questions are:

1) have I made any errors other than running a script I absolutely don't understand

2) Would some of the above info be useful to supplement the wiki https://wiki.archlinux.org/index.php/CacheClean

3) would anyone who knows more about python scripting care to explain any additional info here so I could also add it to the wiki?

Last edited by jeff story (2010-12-20 08:21:24)

karol · 2010-12-20 06:38:10

I have no idea what the hell you're talking about. ARM is safe and it provides old packages for 32 and 64-bit. What your initial problem was? Not having old version of some package in your cache + it's not available anymore from Arch servers? If so, please read the wiki:
https://wiki.archlinux.org/index.php/Do … g_Packages
https://wiki.archlinux.org/index.php/Do … ckages#ARM

Mr Green · 2010-12-20 06:49:27

There are many threads on the rolling back/downgrading issue on the forum. While it can solve some problems short term, many old packages will not sometimes not meet current dependencies.

Or Stop certain packages loading

Major system packages such as xorg/nvidia/kernel do sometimes cause problems, but that is what makes Arch bleeding edge.

You are of course welcome to adapt your script and share with others, that is what the community is about.

For my own part I would have thought a system snapshot would have been a better way to go.

MrG

Last edited by Mr Green (2010-12-20 06:49:50)

jeff story · 2010-12-20 07:10:57

Ok I see I missed out some essential info..... Arch ARM

I somehow missed the whole Arch Rollback Machine thing! During my search for a method to manage my pacman pkg cache file, I stumbled upon the ARM URL, and thought it could be something to research in the future, so I bookmarked it.

When righting my post, I just thought it might be worth mentioning. After posting and proof reading, I somehow mistakenly came to the conclusion that it was a "users" private ARM computer architecture package repo.......

Sorry if I offended anyone. It now appears that the Arch ARM system is probably a better option than trying to "manage" your own /var/cache/pacman/pkg directory to serve the same purpose.

So now I'm wondering how long has the ARM been around, and what is the use of the script and info I found if the ARM has always been available?

Last edited by jeff story (2010-12-20 08:17:32)

karol · 2010-12-20 07:15:59

You don't *have to* use ARM. People write scripts / apps that do sth the way *they* want.
That python script is 5 years old, ARM https://bbs.archlinux.org/viewtopic.php?id=53665 may not be that old.

Last edited by karol (2010-12-20 07:16:51)

jeff story · 2010-12-20 07:35:23

@ Mr Green

I have no idea how to create a python script, I just found it, and figured out how to use it.
See:
https://bbs.archlinux.org/viewtopic.php?id=9104
http://aur.archlinux.org/packages.php?ID=37572
https://wiki.archlinux.org/index.php/CacheClean

There isn't much info on the CacheClean wiki for us Linux newbs.... I was trying to share what I had figured out. And yea. well aware of cutting edge issues.
Also, this was intended for strictly a short term solution, as I explained with my problem, it was resolved after a few days and an upgrade.

@ Karol,

Hey.......chill.

Try to remember that not everyone has read everything available on every wiki subject and is at your level.......
Meant no disrespect in any way questioning the ARM. AND... as I posted and asked for additional input on it, was obviously not aware of all the details.

Last edited by jeff story (2010-12-20 07:38:34)

jeff story · 2010-12-20 08:29:48

The original script from 5 years ago:

EDITED deleted by Jeff - see Karol's diff post

Script from the wiki
https://wiki.archlinux.org/index.php/CacheClean

#!/usr/bin/env python
"""cache_clean - a simple python script to clean up the /var/cache/pacman/pkg directory.
More versatile than 'pacman -Sc' in that you can select how many old versions
to keep.
Usage: cache_clean {-p} {-v} <# of copies to keep>
  # of copies to keep - (required) how many generations of each package to keep
  -p - (optional) preview what would be deleted; forces verbose (-v) mode.
  -v - (optional) show deleted packages."""
# Note that the determination of package age is done by simply looking at the date-time
# modified stamp on the file. There is just enough variation in the way package version
# is done that I thought this would be simpler & just as good.
# Also note that you must be root to run this script.

import getopt
import os
import re
import sys

# helper function to get tuple of (file dtm, file name, file sz)    
def fn_stats(fn):
  s = os.stat(fn)
  return (s[8], fn, s[6])

# cleanup does the actual pkg file deletion
def cleanup(run_list):
  # strictly speaking only the first two of these globals need to be listed.
  global n_deleted, bytes_deleted, opt_verbose, opt_preview, n_to_keep
  # return if the run_list is too short
  #print run_list
  #return
  if len(run_list) <= n_to_keep: return
  # Build list of tuples (date-time file modified, file name, file size)
  dtm_list = [fn_stats(tfn) for tfn in run_list]
  # Sort the list by date-time
  dtm_list.sort()
  # Build list of the filenames to delete (all but last n_to_keep).
  # <showing_off>
  #kill_list = [tfn[1] for tfn in dtm_list[:-n_to_keep]]
  #bytes_deleted = sum(x[2] for x in dtm_list[:-n_to_keep])
  # </showing_off>
  kill_list = []
  for x in dtm_list[:-n_to_keep]:
    kill_list.append(x[1])
    bytes_deleted += x[2]
  if opt_verbose: print (kill_list)
  n_deleted += len(kill_list)
  # and finally delete (if not in preview mode)
  if not opt_preview:
    for dfn in kill_list:
      os.unlink(dfn)

######################################################################
# mainline processing

# process command line options
try:
  opts, pargs = getopt.getopt(sys.argv[1:], 'vp')
  opt_dict = dict(opts)
  opt_preview = '-p' in opt_dict
  opt_verbose = '-v' in opt_dict
  if opt_preview: opt_verbose = True
  if len(pargs) == 1:
    n_to_keep = pargs[0]
  else:
    raise getopt.GetoptError("missing required argument.")
  try:
    n_to_keep = int(n_to_keep)
    if n_to_keep <= 0: raise ValueError
  except ValueError as e:
    raise getopt.GetoptError("# of copies to keep must be numeric > 0!")
except getopt.GetoptError as msg:
  print ("Error:",msg,"\n",__doc__)
  sys.exit(1)

# change to the pkg directory & get a sorted list of its contents
os.chdir('/var/cache/pacman/pkg')
pkg_fns = os.listdir('.')
pkg_fns.sort()

# Pattern to use to extract the package name from the tar file name:
# for pkg e.g. 'gnome-common-2.8.0-1-i686.pkg.tar.gz' group(1) is 'gnome-common'.
bpat = re.compile(r'^(.+)-\d[^-]+-.+?(-i686|-x86_64|-any)?\.pkg\.tar\.(gz|bz2|xz)(\.aria2)?$')

n_deleted = 0
bytes_deleted = 0
pkg_base_nm = ''
# now look for "runs" of the same package name differing only in version info.
for run_end in range(len(pkg_fns)):
  fn = pkg_fns[run_end]

  # make sure we skip directories
  if os.path.isdir(fn) == False:
    mo = bpat.match(fn) # test for a match of the package name pattern
    if mo:
      #print >>sys.stdout, "Processing file '" + fn + "'", mo.lastindex
      tbase = mo.group(1) # gets the 'base' package name
      # include the architecture in the name if it's present
      if mo.lastindex > 1:
        if mo.group(2) is not None:
            tbase += mo.group(2)
      #print 'tbase: ',tbase,'  ',mo.lastindex
      # is it a new base name, i.e. the start of a new run?
      if tbase != pkg_base_nm: # if so then process the prior run
        if pkg_base_nm != '':
          cleanup(pkg_fns[run_start:run_end])
        pkg_base_nm = tbase # & setup for the new run
        run_start = run_end
    else:
      print ("File '"+fn+"' doesn't match package pattern!", file=sys.stderr)
  else:
    print >>sys.stdout, "skipping directory '"+fn+"'!"

# catch the final run of the list
run_end += 1
cleanup(pkg_fns[run_start:run_end])

if opt_verbose:
  if opt_preview:
    print ("Preview mode (no files deleted):"),
  print (n_deleted,"files deleted,",bytes_deleted/1000,"kbytes.")

Last edited by jeff story (2010-12-20 09:30:50)

karol · 2010-12-20 08:39:40

And here's the diff for easier viewing

[karol@black ~]$ diff -Naur old new
--- old    2010-12-20 08:37:21.000000000 +0000
+++ new    2010-12-20 08:37:33.000000000 +0000
@@ -1,8 +1,8 @@
 #!/usr/bin/env python
-"""pkg_clean - a simple python script to clean up the /var/cache/pacman/pkg directory.
+"""cache_clean - a simple python script to clean up the /var/cache/pacman/pkg directory.
 More versatile than 'pacman -Sc' in that you can select how many old versions
 to keep.
-Usage: pkg_clean {-p} {-v} <# of copies to keep>
+Usage: cache_clean {-p} {-v} <# of copies to keep>
   # of copies to keep - (required) how many generations of each package to keep
   -p - (optional) preview what would be deleted; forces verbose (-v) mode.
   -v - (optional) show deleted packages."""
@@ -16,20 +16,35 @@
 import re
 import sys
 
+# helper function to get tuple of (file dtm, file name, file sz)    
+def fn_stats(fn):
+  s = os.stat(fn)
+  return (s[8], fn, s[6])
+
 # cleanup does the actual pkg file deletion
 def cleanup(run_list):
-  # strictly speaking only the first of these globals needs to be listed.
-  global n_deleted, opt_verbose, opt_preview, n_to_keep
+  # strictly speaking only the first two of these globals need to be listed.
+  global n_deleted, bytes_deleted, opt_verbose, opt_preview, n_to_keep
   # return if the run_list is too short
+  #print run_list
+  #return
   if len(run_list) <= n_to_keep: return
-  # Build list of tuples (date-time file modified, file name)
-  dtm_list = [(os.stat(tfn)[8], tfn) for tfn in run_list]
+  # Build list of tuples (date-time file modified, file name, file size)
+  dtm_list = [fn_stats(tfn) for tfn in run_list]
   # Sort the list by date-time
   dtm_list.sort()
   # Build list of the filenames to delete (all but last n_to_keep).
-  kill_list = [tfn[1] for tfn in dtm_list[:-n_to_keep]]
-  if opt_verbose: print kill_list
+  # <showing_off>
+  #kill_list = [tfn[1] for tfn in dtm_list[:-n_to_keep]]
+  #bytes_deleted = sum(x[2] for x in dtm_list[:-n_to_keep])
+  # </showing_off>
+  kill_list = []
+  for x in dtm_list[:-n_to_keep]:
+    kill_list.append(x[1])
+    bytes_deleted += x[2]
+  if opt_verbose: print (kill_list)
   n_deleted += len(kill_list)
+  # and finally delete (if not in preview mode)
   if not opt_preview:
     for dfn in kill_list:
       os.unlink(dfn)
@@ -41,8 +56,8 @@
 try:
   opts, pargs = getopt.getopt(sys.argv[1:], 'vp')
   opt_dict = dict(opts)
-  opt_preview = opt_dict.has_key('-p')
-  opt_verbose = opt_dict.has_key('-v')
+  opt_preview = '-p' in opt_dict
+  opt_verbose = '-v' in opt_dict
   if opt_preview: opt_verbose = True
   if len(pargs) == 1:
     n_to_keep = pargs[0]
@@ -51,10 +66,10 @@
   try:
     n_to_keep = int(n_to_keep)
     if n_to_keep <= 0: raise ValueError
-  except ValueError, e:
+  except ValueError as e:
     raise getopt.GetoptError("# of copies to keep must be numeric > 0!")
-except getopt.GetoptError, msg:
-  print "Error:",msg,"n",__doc__
+except getopt.GetoptError as msg:
+  print ("Error:",msg,"\n",__doc__)
   sys.exit(1)
 
 # change to the pkg directory & get a sorted list of its contents
@@ -62,28 +77,45 @@
 pkg_fns = os.listdir('.')
 pkg_fns.sort()
 
-# for pkg e.g. 'gnome-common-2.8.0-1.pkg.tar.gz' group(1) is 'gnome-common'.
-bpat = re.compile(r'(.+)-[^-]+-.+.pkg.tar.gz$')
+# Pattern to use to extract the package name from the tar file name:
+# for pkg e.g. 'gnome-common-2.8.0-1-i686.pkg.tar.gz' group(1) is 'gnome-common'.
+bpat = re.compile(r'^(.+)-\d[^-]+-.+?(-i686|-x86_64|-any)?\.pkg\.tar\.(gz|bz2|xz)(\.aria2)?$')
+
 n_deleted = 0
+bytes_deleted = 0
 pkg_base_nm = ''
-# now look for "runs" of the same package differing only in version info.
+# now look for "runs" of the same package name differing only in version info.
 for run_end in range(len(pkg_fns)):
   fn = pkg_fns[run_end]
-  mo = bpat.match(fn) # test for a match of the package name pattern
-  if mo:
-    tbase = mo.group(1) # gets the 'base' package name
-    # is it a new base name, i.e. the start of a new run?
-    if tbase != pkg_base_nm: # if so then process the prior run
-      if pkg_base_nm != '':
-        cleanup(pkg_fns[run_start:run_end])
-      pkg_base_nm = tbase # & setup for the new run
-      run_start = run_end
+
+  # make sure we skip directories
+  if os.path.isdir(fn) == False:
+    mo = bpat.match(fn) # test for a match of the package name pattern
+    if mo:
+      #print >>sys.stdout, "Processing file '" + fn + "'", mo.lastindex
+      tbase = mo.group(1) # gets the 'base' package name
+      # include the architecture in the name if it's present
+      if mo.lastindex > 1:
+        if mo.group(2) is not None:
+            tbase += mo.group(2)
+      #print 'tbase: ',tbase,'  ',mo.lastindex
+      # is it a new base name, i.e. the start of a new run?
+      if tbase != pkg_base_nm: # if so then process the prior run
+        if pkg_base_nm != '':
+          cleanup(pkg_fns[run_start:run_end])
+        pkg_base_nm = tbase # & setup for the new run
+        run_start = run_end
+    else:
+      print ("File '"+fn+"' doesn't match package pattern!", file=sys.stderr)
   else:
-    print >>sys.stderr, "File '"+fn+"' doesn't match package pattern!"
+    print >>sys.stdout, "skipping directory '"+fn+"'!"
 
 # catch the final run of the list
 run_end += 1
 cleanup(pkg_fns[run_start:run_end])
 
 if opt_verbose:
-  print n_deleted,"files deleted."
+  if opt_preview:
+    print ("Preview mode (no files deleted):"),
+  print (n_deleted,"files deleted,",bytes_deleted/1000,"kbytes.")

@ jeff story
What was the point of your last post?

jeff story · 2010-12-20 08:56:25

Just hypothetically for a moment,

If this script could be modified to save "only" the just prior to currently installed version of all packages, it seems this would give a quick and simple downgrade solution to most (all?) upgrade breakage issues?

Also, if after every pacman -Syu the script could be automated to run and managed the pacman pkg directory as per above, this would also result in a minimal pkg directory size?

Last edited by jeff story (2010-12-20 08:57:52)

karol · 2010-12-20 09:01:03

How about you run 'pacman -Sc' and then 'pacman -Syu'?
If any packages got updated you still have the version you had before the update [1] in the cache but not older versions because you removed them with 'pacman -Sc'.

[1] which may not be the previous version of the package - some packages get updated quite often / some people update once in a blue moon:
You have a package foo-1.0.
It got updated to foo-1.1 but you haven't updated your Arch, so you still have foo-1.0.
It got updated again to foo-1.2 and you updated your Arch, so now you have foo-1.2. Unfortunately it has a nasty bug and you need to downgrade. You can pick foo-1.0 from you cache or download foo-1.1 from e.g. ARM.

Last edited by karol (2010-12-20 09:05:14)

jeff story · 2010-12-20 09:06:16

@ karol

Hhhmmm.....

I was thinking that probably not everyone who has the ability to understand python would take the time to follow all the links and read every bit of my long post. They may however, take the time to just glance at the script if it's available, and add info like you did to showthe diff.

Thanks!!

karol · 2010-12-20 09:08:47

jeff story wrote:

@ karol
Hhhmmm.....
I was thinking that probably not everyone who has the ability to understand python would take the time to follow all the links and read every bit of my long post. They may however, take the time to just glance at the script if it's available, and add info like you did to showthe diff.
Thanks!!

OK, now I understand. But I don't think the old script is of much value - the version in the wiki is the one you should be using.

jeff story · 2010-12-20 09:24:52

OK,

You brought up something I didn't think of. Some upgrades would or could skip a version # of a package. That would leave my hypothetical idea above dead in the water if you needed the "skipped" version to downgrade to, to restore a working system? Then it would have to be ARM repo to the rescue!

With the ARM having many older than current versions available..... I guess this script s kinda redundant to it on some levels...

I'll just delete the original version of the script since you posted the upgrades (differences) to the original.

Last edited by jeff story (2010-12-22 09:58:49)

karol · 2010-12-20 10:41:34

You can try https://github.com/pbrisbin/scripts/blo … /downgrade

[karol@black ~]$ /home/karol/docs/code/downgrade xorg-server

 The following packages are available in your cache:
    1    local    xorg-server-1.9.2-2-i686.pkg.tar.xz [installed]

    please choose a version, type s to [s]earch A.R.M.: s

 The following packages are available from the A.R.M.:
    1    extra    xorg-server-1.9.2-2-i686.pkg.tar.xz [installed]
    2    extra    xorg-server-1.9.2-1-i686.pkg.tar.xz
    3    extra    xorg-server-1.9.0-1-i686.pkg.tar.xz
    4    extra    xorg-server-1.8.1.902-1-i686.pkg.tar.xz
    5    extra    xorg-server-1.8.1-1-i686.pkg.tar.xz
    6    extra    xorg-server-1.7.6-3-i686.pkg.tar.xz
    7    extra    xorg-server-1.7.5.902-1-i686.pkg.tar.xz
    8    extra    xorg-server-1.7.5.901-2-i686.pkg.tar.xz
    9    extra    xorg-server-1.7.5.901-1-i686.pkg.tar.xz
    10    extra    xorg-server-1.7.5-1-i686.pkg.tar.gz
    11    extra    xorg-server-1.7.4.901-1-i686.pkg.tar.gz
    12    extra    xorg-server-1.7.3.902-1-i686.pkg.tar.gz
    13    extra    xorg-server-1.7.3.901-1-i686.pkg.tar.gz
    14    extra    xorg-server-1.7.3-1-i686.pkg.tar.gz
    15    extra    xorg-server-1.7.2-2-i686.pkg.tar.gz
    16    extra    xorg-server-1.7.2-1-i686.pkg.tar.gz
    17    extra    xorg-server-1.7.1.902-1-i686.pkg.tar.gz
    18    extra    xorg-server-1.7.1.901-2-i686.pkg.tar.gz
    19    extra    xorg-server-1.7.1.901-1-i686.pkg.tar.gz
    20    extra    xorg-server-1.7.1-1-i686.pkg.tar.gz

    please choose a version, type q to [q]uit: q

Arch Linux

#1 2010-12-20 05:54:27

/var/cache/pacman/pkg "management"

#2 2010-12-20 06:38:10

Re: /var/cache/pacman/pkg "management"

#3 2010-12-20 06:49:27

Re: /var/cache/pacman/pkg "management"

#4 2010-12-20 07:10:57

Re: /var/cache/pacman/pkg "management"

#5 2010-12-20 07:15:59

Re: /var/cache/pacman/pkg "management"

#6 2010-12-20 07:35:23

Re: /var/cache/pacman/pkg "management"

#7 2010-12-20 08:29:48

Re: /var/cache/pacman/pkg "management"

#8 2010-12-20 08:39:40

Re: /var/cache/pacman/pkg "management"

#9 2010-12-20 08:56:25

Re: /var/cache/pacman/pkg "management"

#10 2010-12-20 09:01:03

Re: /var/cache/pacman/pkg "management"

#11 2010-12-20 09:06:16

Re: /var/cache/pacman/pkg "management"

#12 2010-12-20 09:08:47

Re: /var/cache/pacman/pkg "management"

#13 2010-12-20 09:24:52

Re: /var/cache/pacman/pkg "management"

#14 2010-12-20 10:41:34

Re: /var/cache/pacman/pkg "management"

Board footer