PacStats revamped! (Statistical charts about pacman activity)

encelo · 2010-04-08 23:42:42

Hi guys,
I'm pretty sure no one remembers this really old post.
I was presenting to the world my project PacStats, it was neat and cute but also the classic toy project that dies after some days.
Well, I don't know why but I'm taking a look at it again after years and I have got some nice ideas that I would be glad to implement.

Project description wrote:

PacStats is able to analyze the log and the lib directory of the ArchLinux package manager (pacman) and generate statistical charts.
The GUI is programmed in Python with PyGTK, the internal database is based on SQLite and the charts are made with Matplotlib.

This time I'm more serious, the project has a page on Google Code (where you can get the sources, report an issue, etc.), Ohloh (for interesting coding statistics), freshmeat and is already on AUR.
Enjoy your pacman charts!

Last edited by encelo (2010-04-23 01:42:49)

encelo · 2010-04-12 07:51:59

Many new changes have been made these days to the sources!
First of all the home of the project now show two new screenshots: the first one is about the new database information window, while in the second you can see the preferences one.
Another change for end users is the addition of two new charts, one plots the number of transactions per day this month, while the other let you know how many packages haven't been updated in the last few months.

encelo · 2010-04-19 23:00:33

A lot of new commits have been pushed to the repository.
Having a look at the updated screenshots can reveal new charts, but for the most important enhancement you have to run the program: as a matter of fact database creation and charts plotting are way faster than before!

encelo · 2010-04-22 00:10:14

Be sure to check the last commit, the initial parsing process is now almost instantaneous!
There are also two new charts to play with, and some of you may be glad to read that the program now recognize very old (>3 years) timestamp formats.

karol · 2010-04-22 00:33:49

"The database is out of sync. Do you want to update now?"
I get this every time - do I have to somehow save the db?

Edit: I'll better check the last commit first ...

Last edited by karol (2010-04-22 00:35:36)

encelo · 2010-04-22 00:38:35

karol wrote:

"The database is out of sync. Do you want to update now?"
I get this every time - do I have to somehow save the db?
Edit: I'll better check the last commit first ...

The database is automatically saved.
Yes, I suggest you update to the last changeset because a couple of bugs related with the startup check have been corrected lately.

karol · 2010-04-22 00:41:36

Yeah, it's fixed :-) And the initial parsing is lightning fast indeed.

encelo · 2010-04-22 00:46:00

karol wrote:

Yeah, it's fixed

I'm glad to see that everything works correctly for you, but keep on testing it, so that a 0.1 version could be released soon.

karol wrote:

And the initial parsing is lightning fast indeed.

Look at the benchmarks on my system:
BEFORE

Parsing the log "/var/log/pacman.log"
Parsed 29481 lines in 34.430000 seconds
Parsing the lib "/var/lib/pacman/local"
Parsed 1030 packages in 2.110000 seconds

AFTER

Parsing the log "/var/log/pacman.log"
Parsed 29481 lines in 0.690000 seconds
Parsing the lib "/var/lib/pacman/local"
Parsed 1030 packages in 0.270000 seconds

Something, that I am ashamed of, was the cause for the big slowdown in the previous revision.

Last edited by encelo (2010-04-22 00:51:28)

encelo · 2010-04-26 22:14:39

Got some news for the project, the last revision introduces four new charts!
You can now learn the most adopted licenses, the length of package description and the update frequency for some of them.

wido · 2010-05-13 11:56:12

Hello,
problem:

i accept a message "The database is empty. Do you want to build it now?", program crash:

[img]$ pacstats
"transactions" table exists already
"packages" table exists already
/usr/lib/python2.6/site-packages/matplotlib/backends/backend_gtk.py:621: DeprecationWarning: Use the new widget gtk.Tooltip
self.tooltips = gtk.Tooltips()
Gtk-Message: (for origin information, set GTK_DEBUG): failed to retrieve property `GtkTreeView::odd-row-color' of type `GdkColor' from rc file value "((GString*) 0x1fc9d60)" of type `GString'
Parsing the log "/var/log/pacman.log"
Parsed 6418 lines in 0.080000 seconds
Traceback (most recent call last):
File "/usr/bin/pacstats", line 38, in <module>
app = app.Application(share_dir)
File "/usr/lib/python2.6/site-packages/PacStats/app.py", line 48, in __init__
self.main_win = main_win.Main_Window(self)
File "/usr/lib/python2.6/site-packages/PacStats/main_win.py", line 95, in __init__
self.check_update()
File "/usr/lib/python2.6/site-packages/PacStats/main_win.py", line 129, in check_update
self.on_update_db_activate(None)
File "/usr/lib/python2.6/site-packages/PacStats/main_win.py", line 221, in on_update_db_activate
self._libparser.parse()
File "/usr/lib/python2.6/site-packages/PacStats/libparser.py", line 42, in parse
locale.setlocale(locale.LC_ALL, 'en_US.UTF8')
File "/usr/lib/python2.6/locale.py", line 513, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting[/img]

archlinux x86_64, testing, xorg 1.8 and french.

encelo · 2010-05-15 01:06:40

wido wrote:

Hello,
problem:
i accept a message "The database is empty. Do you want to build it now?", program crash:
[...]
  File "/usr/lib/python2.6/site-packages/PacStats/libparser.py", line 42, in parse
    locale.setlocale(locale.LC_ALL, 'en_US.UTF8')
[...]
archlinux x86_64, testing, xorg 1.8 and french.

First of all, thank you for the report.
As you see the problem is caused by the fact that the application is trying to set an English locale that you don't have installed on your workstation.
But why am I forcing an English locale in the pacman library parser?

Well, actually not many users are going to run that part of the code, that is because it is triggered by very old timestamps.
They have got a format that is easier to let Python parse using the datetime module.
The application is assuming that the old versions of pacman that were writing those old packages timestamps were also all forced to use an English locale, but I'm not sure this is always the case.
Maybe pacman itself was using locale settings to write those strings, and maybe your old package timestamps have French month names and date format.

Look into /var/lib/pacman/local and try to find which packages have the %BUILDDATE% or %INSTALLDATE% fields of their desc file not containing an epoch number and report me the timestamps string.
You could also try to simply comment out line 42 in libparser.py to see if timestamps are correctly parsed with your current locale.

Spider.007 · 2010-05-15 11:57:05

I like it! I miss one obvious feature though; a graph of transactions over the entire period my system has been running. Also, wouldn't it be more logical to use the red color for removals, and yellow for synchronizations?

encelo · 2010-05-15 12:56:35

Spider.007 wrote:

I like it! I miss one obvious feature though; a graph of transactions over the entire period my system has been running.

How do you think to implement something like this? Maybe a histogram for ever year?
Because I've got more than five years of logs and plotting dozens of mothly histograms won't be very good, unless I find a way to plot a lot of data but zoom on the last period of time.

Spider.007 wrote:

Also, wouldn't it be more logical to use the red color for removals, and yellow for synchronizations?

And maybe green for installations.
Yeah, actually it makes sense to me.

Spider.007 · 2010-05-15 14:06:12

encelo wrote:

Spider.007 wrote:
I like it! I miss one obvious feature though; a graph of transactions over the entire period my system has been running.
How do you think to implement something like this? Maybe a histogram for ever year?
Because I've got more than five years of logs and plotting dozens of mothly histograms won't be very good, unless I find a way to plot a lot of data but zoom on the last period of time.

I've got more than five years of history too, but I don't see what would go wrong if you would plot all of them? Zooming is already supported, so you can just zoom-out by default. Here is a quick mockup of what I think it could look like:

Ogion · 2010-05-15 14:20:06

Ooh nice.
This kind of programs (this one; also pacgraph) always brings me to re-look at what i have installed and if i need it etc..

Ogion

encelo · 2010-05-15 15:07:31

Spider.007 wrote:

I've got more than five years of history too, but I don't see what would go wrong if you would plot all of them? Zooming is already supported, so you can just zoom-out by default.

Actually what I really have got in mind for a next major version is to have a custom GUI for every chart, so that you would click on the Transactions chart and then choose the granularity (daily, monthly and so on) and the time period.

Ogion wrote:

This kind of programs (this one; also pacgraph) always brings me to re-look at what i have installed and if i need it etc..

In the future it will be more useful than now, I intend to parse also dependency data between packages.

wido · 2010-05-16 14:12:11

I add :

/etc/profile:

# Locale
LANG=fr_FR.UTF-8 
LC_CTYPE=fr_FR.UTF-8
LC_ALL=fr_FR.UTF-8

/etc/locale.gen:

en_US.UTF-8 UTF-8 
fr_FR.UTF-8 UTF-8  
fr_FR ISO-8859-1  
fr_FR@euro ISO-8859-15

locale-gen and source /etc/profile

a program work now

encelo · 2010-05-17 03:40:17

wido wrote:

I add :
[...]
a program work now

Ok, but that's a dirty hack.
Could you please email me a compressed archive of your /var/lib/pacman/local directory?
I would then try to fix the issue once and for all.

Arch Linux

#1 2010-04-08 23:42:42

PacStats revamped! (Statistical charts about pacman activity)

#2 2010-04-12 07:51:59

Re: PacStats revamped! (Statistical charts about pacman activity)

#3 2010-04-19 23:00:33

Re: PacStats revamped! (Statistical charts about pacman activity)

#4 2010-04-22 00:10:14

Re: PacStats revamped! (Statistical charts about pacman activity)

#5 2010-04-22 00:33:49

Re: PacStats revamped! (Statistical charts about pacman activity)

#6 2010-04-22 00:38:35

Re: PacStats revamped! (Statistical charts about pacman activity)

#7 2010-04-22 00:41:36

Re: PacStats revamped! (Statistical charts about pacman activity)

#8 2010-04-22 00:46:00

Re: PacStats revamped! (Statistical charts about pacman activity)

#9 2010-04-26 22:14:39

Re: PacStats revamped! (Statistical charts about pacman activity)

#10 2010-05-13 11:56:12

Re: PacStats revamped! (Statistical charts about pacman activity)

#11 2010-05-15 01:06:40

Re: PacStats revamped! (Statistical charts about pacman activity)

#12 2010-05-15 11:57:05

Re: PacStats revamped! (Statistical charts about pacman activity)

#13 2010-05-15 12:56:35

Re: PacStats revamped! (Statistical charts about pacman activity)

#14 2010-05-15 14:06:12

Re: PacStats revamped! (Statistical charts about pacman activity)

#15 2010-05-15 14:20:06

Re: PacStats revamped! (Statistical charts about pacman activity)

#16 2010-05-15 15:07:31

Re: PacStats revamped! (Statistical charts about pacman activity)

#17 2010-05-16 14:12:11

Re: PacStats revamped! (Statistical charts about pacman activity)

#18 2010-05-17 03:40:17

Re: PacStats revamped! (Statistical charts about pacman activity)

Board footer