armh: pacman history and ARM helper

Xyne · 2012-07-21 06:24:01

Project page: http://xyne.archlinux.ca/projects/armh

Summary:
It can search the ARM for old packages. It can also determine previously installed package sets and download those packages to a given directory. The project page contains some example output. Displaying changes in the package set for arbitrary time points is useful even without the ARM packages (e.g. to see what packages have changed in the last x days, or since the middle of January).

Questions, comments, etc. are welcome as always.

Last edited by Xyne (2012-12-21 20:21:01)

mrman · 2012-07-21 15:19:51

I look forward to using this.

mrman · 2012-07-29 01:38:43

ImportError: No module named pycman

pycman doesn't appear to exist anymore.
https://bbs.archlinux.org/viewtopic.php?id=13679

Edit: pyalpm provides pycman

Last edited by mrman (2012-07-29 01:53:20)

Xyne · 2012-07-29 01:44:23

"pycman" is a module provided by the pyalpm package.

Xyne · 2012-07-29 04:31:12

*bump*
It has been packaged now:
http://xyne.archlinux.ca/projects/armh/ (available in [xyne-any])
https://aur.archlinux.org/packages.php?ID=61298

I have also added options for colorizing and tabulating the output of "--show changes". You can find an example on the project page.

Xyne · 2012-12-21 20:24:12

New option: --trace <pkgname> [<pkgname>...]
It will show the pacman.log history for the given packages, with support for the "--color" option. There's an example on the project page.

Opanos · 2013-04-07 12:55:22

After recent pacman's update, i get two warnings

/usr/lib/python3.3/site-packages/pycman/config.py:130: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'Color'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))
/usr/lib/python3.3/site-packages/pycman/config.py:125: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'LocalFileSigLevel'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))

All though armh works like expected.

Xyne · 2013-04-07 13:20:06

Those are pyalpm errors. You should report them on the bugtracker.

The errors should not affect armh's operation.

karol · 2013-05-21 19:34:27

I've stumbled upon an issue when using armh to search for 'foobillard':

$ armh --search foobillard
downloading http://arm.konnichi.com/pool/packages/ to /tmp/ARM-karol/packages.htm
downloading http://arm.konnichi.com/pool/community/ to /tmp/ARM-karol/community.htm

Nothing?

$ grep foobillard /tmp/ARM-karol/packages.htm | grep i686
      <a href="foobillard++-3.41beta-1-i686.pkg.tar.xz">foobillard++-3.41bet..&gt;</a> 03-Nov-2011 02:01   23M  
      <a href="foobillard++-3.41beta-1-i686.pkg.tar.xz.sig">foobillard++-3.41bet..&gt;</a> 03-Nov-2011 02:01  287   
      <a href="foobillard++-3.42beta-2-i686.pkg.tar.xz">foobillard++-3.42bet..&gt;</a> 05-Jan-2012 14:01   34M  
      <a href="foobillard++-3.42beta-2-i686.pkg.tar.xz.sig">foobillard++-3.42bet..&gt;</a> 05-Jan-2012 14:01  287   
      <a href="foobillard++-3.42beta-3-i686.pkg.tar.xz">foobillard++-3.42bet..&gt;</a> 02-Feb-2012 12:16   34M  
      <a href="foobillard++-3.42beta-3-i686.pkg.tar.xz.sig">foobillard++-3.42bet..&gt;</a> 02-Feb-2012 12:16  287   
      <a href="foobillard++-3.42beta-4-i686.pkg.tar.xz">foobillard++-3.42bet..&gt;</a> 15-Oct-2012 10:59   34M  
      <a href="foobillard++-3.42beta-4-i686.pkg.tar.xz.sig">foobillard++-3.42bet..&gt;</a> 15-Oct-2012 10:59  287   
      <a href="foobillard++-3.4beta-1-i686.pkg.tar.xz">foobillard++-3.4beta..&gt;</a> 15-Sep-2011 20:41   22M  
      <a href="foobillard++-3.4beta-1-i686.pkg.tar.xz.sig">foobillard++-3.4beta..&gt;</a> 15-Sep-2011 20:41  287

Using the ARM websearch http://arm.konnichi.com/search/index.ph … ommunity=1 I get

extra:
foobillard-3.0a-3-i686.pkg.tar.gz
foobillard-3.0a-4-i686.pkg.tar.gz
foobillard-3.0a-5-i686.pkg.tar.xz
foobillard-3.0a-6-i686.pkg.tar.xz
foobillard++-3.41beta-1-i686.pkg.tar.xz
foobillard++-3.41beta-1-i686.pkg.tar.xz.sig
foobillard++-3.42beta-2-i686.pkg.tar.xz
foobillard++-3.42beta-2-i686.pkg.tar.xz.sig
foobillard++-3.42beta-3-i686.pkg.tar.xz
foobillard++-3.42beta-3-i686.pkg.tar.xz.sig
foobillard++-3.42beta-4-i686.pkg.tar.xz
foobillard++-3.42beta-4-i686.pkg.tar.xz.sig
foobillard++-3.4beta-1-i686.pkg.tar.xz
foobillard++-3.4beta-1-i686.pkg.tar.xz.sig

Searching for 'foobillard++' seems to work:

$ armh --search foobillard++
[2011-11-03 03:01] foobillard++ 3.41beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-i686.pkg.tar.xz
[2011-11-03 03:01] foobillard++ 3.41beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-i686.pkg.tar.xz.sig
[2012-01-05 15:01] foobillard++ 3.42beta-2 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-i686.pkg.tar.xz
[2012-01-05 15:01] foobillard++ 3.42beta-2 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-i686.pkg.tar.xz.sig
[2012-02-02 13:16] foobillard++ 3.42beta-3 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-i686.pkg.tar.xz
[2012-02-02 13:16] foobillard++ 3.42beta-3 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-i686.pkg.tar.xz.sig
[2012-10-15 12:59] foobillard++ 3.42beta-4 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-i686.pkg.tar.xz
[2012-10-15 12:59] foobillard++ 3.42beta-4 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-i686.pkg.tar.xz.sig
[2011-09-15 22:41] foobillard++ 3.4beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-i686.pkg.tar.xz
[2011-09-15 22:41] foobillard++ 3.4beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-i686.pkg.tar.xz.sig

Why doesn't armh print any matches for 'foobillard'?

BTW, http://xyne.archlinux.ca/projects/armh/ has '$armh' and even '$arm' instead of '$ armh' in a couple places.

Xyne · 2013-05-21 20:03:35

The match was done by a simple "if name in pkgnames" where pkgnames were the supplied search terms. I have updated it to interpret search terms as Python regular expressions.

Thanks for pointing out typos on the project page. I'll correct them now.

karol · 2013-05-22 02:46:55

1. It still doesn't show foobillard - does it work for you?
The problem may not be with pattern matching, as packages.htm doesn't seem to mention foobillard at all (see my previous post).
I don't speak python, so I see just a lot of whitespace in /usr/bin/armh ;P

2. Would it be possible to sort packages by date?

$ armh --search firefox$
...
[2013-05-06 06:41] firefox 20.0.1-5 http://arm.konnichi.com/pool/packages/firefox-20.0.1-5-i686.pkg.tar.xz
[2013-05-06 06:51] firefox 20.0.1-5 http://arm.konnichi.com/pool/packages/firefox-20.0.1-5-i686.pkg.tar.xz.sig
[2013-05-14 12:13] firefox 21.0-1 http://arm.konnichi.com/pool/packages/firefox-21.0-1-i686.pkg.tar.xz
[2013-05-14 12:22] firefox 21.0-1 http://arm.konnichi.com/pool/packages/firefox-21.0-1-i686.pkg.tar.xz.sig
[2010-09-16 01:04] firefox 3.6.10-1 http://arm.konnichi.com/pool/packages/firefox-3.6.10-1-i686.pkg.tar.xz
[2010-10-19 13:27] firefox 3.6.11-1 http://arm.konnichi.com/pool/packages/firefox-3.6.11-1-i686.pkg.tar.xz
...

$ armh --search foobillard
[2011-11-03 03:01] foobillard++ 3.41beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-i686.pkg.tar.xz
[2011-11-03 03:01] foobillard++ 3.41beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-i686.pkg.tar.xz.sig
[2012-01-05 15:01] foobillard++ 3.42beta-2 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-i686.pkg.tar.xz
[2012-01-05 15:01] foobillard++ 3.42beta-2 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-i686.pkg.tar.xz.sig
[2012-02-02 13:16] foobillard++ 3.42beta-3 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-i686.pkg.tar.xz
[2012-02-02 13:16] foobillard++ 3.42beta-3 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-i686.pkg.tar.xz.sig
[2012-10-15 12:59] foobillard++ 3.42beta-4 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-i686.pkg.tar.xz
[2012-10-15 12:59] foobillard++ 3.42beta-4 http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-i686.pkg.tar.xz.sig
[2011-09-15 22:41] foobillard++ 3.4beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-i686.pkg.tar.xz
[2011-09-15 22:41] foobillard++ 3.4beta-1 http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-i686.pkg.tar.xz.sig

3. What does '--sigs' do?
4. How does '--ignore' work? 'armh --search foo --ignore bar' still prints bar.
5. Have you contacted ARM maintainer through this e-mail https://bbs.archlinux.org/profile.php?id=41072 ?

Edit:
The oldest packages in http://arm.konnichi.com/pool/packages/ are

      mlocate-0.22.4-2-i68..> 16-Aug-2010 23:18   67K  
      mlocate-0.22.4-2-x86..> 16-Aug-2010 23:19   70K  
      devtools-0.9.8-1-any..> 16-Aug-2010 23:36  7.7K  
      pixman-0.18.4-1-i686..> 17-Aug-2010 00:05  134K  
      pixman-0.18.4-1-x86_..> 17-Aug-2010 00:05  138K  
      libxvmc-1.0.6-1-i686..> 17-Aug-2010 00:10   18K  
      libxvmc-1.0.6-1-x86_..> 17-Aug-2010 00:10   19K 
...

foobillard is in http://arm.konnichi.com/extra/os/i686/

...
      foobillard++-3.4beta..> 15-Sep-2011 20:41   22M  
      foobillard++-3.4beta..> 15-Sep-2011 20:41  287   
      foobillard-3.0a-3-i6..> 09-Nov-2008 03:22  1.0M  
      foobillard-3.0a-4-i6..> 30-Jan-2010 11:48  1.0M  
      foobillard-3.0a-5-i6..> 27-Feb-2010 02:09  1.0M  
      foobillard-3.0a-6-i6..> 10-Jul-2010 19:29  1.0M  
      foomatic-db-1:4.0.15..> 06-Apr-2012 03:12   19M  
      foomatic-db-1:4.0.15..> 06-Apr-2012 03:12  287 
...

https://bbs.archlinux.org/viewtopic.php … 62#p819662

The last updates are from 16th August. I think Ibiblio server stopped syncing and ARM mirrors that server.

Not sure if downloading core, extra etc. for any, i686 and x86_64 instead of http://arm.konnichi.com/pool/packages/ would help.

Last edited by karol (2013-05-22 03:10:30)

Xyne · 2013-05-22 07:34:01

1

$ armh --search foobillard
[2011-11-03 02:07] foobillard++ 3.41beta-1 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-x86_64.pkg.tar.xz[/url]
[2011-11-03 02:07] foobillard++ 3.41beta-1 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.41beta-1-x86_64.pkg.tar.xz.sig[/url]
[2012-01-05 14:09] foobillard++ 3.42beta-2 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-x86_64.pkg.tar.xz[/url]
[2012-01-05 14:09] foobillard++ 3.42beta-2 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-2-x86_64.pkg.tar.xz.sig[/url]
[2012-02-02 12:25] foobillard++ 3.42beta-3 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-x86_64.pkg.tar.xz[/url]
[2012-02-02 12:25] foobillard++ 3.42beta-3 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-3-x86_64.pkg.tar.xz.sig[/url]
[2012-10-15 11:07] foobillard++ 3.42beta-4 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-x86_64.pkg.tar.xz[/url]
[2012-10-15 11:07] foobillard++ 3.42beta-4 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.42beta-4-x86_64.pkg.tar.xz.sig[/url]
[2011-09-15 20:47] foobillard++ 3.4beta-1 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-x86_64.pkg.tar.xz[/url]
[2011-09-15 20:47] foobillard++ 3.4beta-1 [url]http://arm.konnichi.com/pool/packages/foobillard++-3.4beta-1-x86_64.pkg.tar.xz.sig[/url]

Are you sure that you have a complete and up-to-date index file? (armh -y)

2

Results are returned in the order they appear in the index files be default. I have now added a sort option (name, version, date).

3

When downloading packages it will retrieve matching signature files. This lets you create a local database with signed packages.

4

Given that the previous behavior treated search arguments as package names that were matched naïvely, it made no sense to use the ignore option with a search, i.e. "find foo but don't show me foo". Now that search terms are regular expressions, I have added support for the ignore option.

5

Nope, not yet.

Edit

armh isn't a museum tour guide. I doubt that anyone really needs to retrieve packages from 3 years ago. If I can get some sort of custom interface set up on the armh server then that may change, but I don't want to pull in even more index files (#repos * #architectures).

The updated version will be up in the next hour or so.

karol · 2013-05-22 12:28:18

1. I should have removed this point after I made the edit. I get the same results as you do, the issue was that 'foobillard' (not 'foobillard++') was missing. Now I know why and I can always use the ARM search box if I need some really ancient software.
2. Oh yessss, it works!
3. Ah, I get it now.
4. Thanks :-)
5. Maybe he can re-add the very old packages to http://arm.konnichi.com/pool/packages/ ;P

Xyne · 2013-05-22 17:47:21

5. Old packages lack the architecture in the package name, so they can't be mixed in a single directory due to name collisions.

karol · 2013-05-22 17:58:10

Xyne wrote:

5. Old packages lack the architecture in the package name, so they can't be mixed in a single directory due to name collisions.

I don't understand what you mean: http://arm.konnichi.com/search/index.ph … ommunity=1 :

foobillard-3.0a-3-i686.pkg.tar.gz
foobillard-3.0a-4-i686.pkg.tar.gz
foobillard-3.0a-5-i686.pkg.tar.xz
foobillard-3.0a-6-i686.pkg.tar.xz
foobillard++-3.41beta-1-i686.pkg.tar.xz
foobillard++-3.41beta-1-i686.pkg.tar.xz.sig
foobillard++-3.42beta-2-i686.pkg.tar.xz
foobillard++-3.42beta-2-i686.pkg.tar.xz.sig
foobillard++-3.42beta-3-i686.pkg.tar.xz
foobillard++-3.42beta-3-i686.pkg.tar.xz.sig
foobillard++-3.42beta-4-i686.pkg.tar.xz
foobillard++-3.42beta-4-i686.pkg.tar.xz.sig
foobillard++-3.4beta-1-i686.pkg.tar.xz
foobillard++-3.4beta-1-i686.pkg.tar.xz.sig

Can you give some examples?

Xyne · 2013-05-22 19:30:34

I don't remember when the change was made, but prior to some version of Pacman packages were named foo-2.3-1-pkg.tar.gz instead of e.g. foo-2.3-1-i686.pkg.tar.gz, so you could not keep packages for different architectures in the same directory, which is what the pool directories do.

They changed that after I joined but I don't remember when. Obviously it was before the versions of foobillard++ that you have posted above.

karol · 2013-05-22 20:45:57

I didn't know that. You're right, I found http://arm.konnichi.com/extra/os/i686/l … pkg.tar.gz
It's theoretically possible that Arch may revert to this naming scheme if it's going to drop the 32-bit ;P

Thank you very much for all this information and wonderful applications.

karol · 2013-11-18 23:00:04

Do you plan to update it to support the new ARM?

Xyne · 2013-11-20 22:18:26

I've been meaning to check the status of development in the ARM thread. I see that Seblu has created a new version. I had a quick look, but there is no way to get full package lists at the moment without scraping the site. I have sent Seblu an email to ask if he could provide a full list (in JSON format). If so, I will work out the details with him and update armh accordingly.

Note that armh is still useful for parsing the pacman log file (e.g. to determine packages installed at a given time).

seblu · 2013-11-21 02:01:16

I added a packages hierarchy to be able to easily list/find a package by its name. It was a several time asked feature.
I agree that JSON output would be the best, but I have no dynamic code on this vhost at the moment. I would prefer moving the base to another host before doing that.

I updated the doc in our wiki.

Xyne · 2013-11-21 02:15:28

That still requires recursive scraping of the HTML to find all packages. Would it be possible to generate a single index page that lists everything?

How are you synchronizing packages btw? My guess would be with a cron job, in which case that cron job could easily generate a static list each time the package pool changes, no?

Arch Linux

#1 2012-07-21 06:24:01

armh: pacman history and ARM helper

#2 2012-07-21 15:19:51

Re: armh: pacman history and ARM helper

#3 2012-07-29 01:38:43

Re: armh: pacman history and ARM helper

#4 2012-07-29 01:44:23

Re: armh: pacman history and ARM helper

#5 2012-07-29 04:31:12

Re: armh: pacman history and ARM helper

#6 2012-12-21 20:24:12

Re: armh: pacman history and ARM helper

#7 2013-04-07 12:55:22

Re: armh: pacman history and ARM helper

#8 2013-04-07 13:20:06

Re: armh: pacman history and ARM helper

#9 2013-05-21 19:34:27

Re: armh: pacman history and ARM helper

#10 2013-05-21 20:03:35

Re: armh: pacman history and ARM helper

#11 2013-05-22 02:46:55

Re: armh: pacman history and ARM helper

#12 2013-05-22 07:34:01

Re: armh: pacman history and ARM helper

1

2

3

4

5

Edit

#13 2013-05-22 12:28:18

Re: armh: pacman history and ARM helper

#14 2013-05-22 17:47:21

Re: armh: pacman history and ARM helper

#15 2013-05-22 17:58:10

Re: armh: pacman history and ARM helper

#16 2013-05-22 19:30:34

Re: armh: pacman history and ARM helper

#17 2013-05-22 20:45:57

Re: armh: pacman history and ARM helper

#18 2013-11-18 23:00:04

Re: armh: pacman history and ARM helper

#19 2013-11-20 22:18:26

Re: armh: pacman history and ARM helper

#20 2013-11-21 02:01:16

Re: armh: pacman history and ARM helper

#21 2013-11-21 02:15:28

Re: armh: pacman history and ARM helper

Board footer