pacmanlog2gource - visualize package upgrade history

matthiaskrgr · 2012-04-03 15:05:39

Hi everyone,
I made a script which converts a copy of the pacman log /var/log/pacman.log into a different format so you can visualize it using the program "gource".

Visulization looks like this:
(Version 1.9.0, pacmanlog2gource -lmf)

Video by baronmax: http://www.youtube.com/watch?v=lCBjzC78t4o (Version 1.8.3)

The small object floating around the screen is the "root" user.
Yellow rays indicate the package has been upgraded, green indicate installation and red ones removal.
"pacmanlog2gource -g" will visualize the log right after it has been converted.
You can get rid of package names appending " -m".

There are a few more options to tweak the way how the visualization is done, check "pacmanlog2gource -h" for more information.

PKGBUILDS:
pacmanlog2gource and pacmanlog2gource-git are in AUR.
Alternatively you can clone/fork/whatever it via Github

Feedback is welcome

Regards,
Matthias

Last edited by matthiaskrgr (2012-05-26 14:55:50)

wolfcore · 2012-04-04 03:14:30

For some reason, it fails to run on awesome... the screen just flashes, but the program seems to terminate immediately :0

matthiaskrgr · 2012-04-04 05:54:25

Mmmmh interesting. I'm using awesome, too, and it works fine here.
Does gource "crash" (giving some crash-like output), or does it give the same output you get when you run "gource -h" ?
Or is there no output at all?

wolfcore · 2012-04-04 14:32:39

It gives the output of gource -h

matthiaskrgr · 2012-04-04 14:44:57

Ok, pacmanlog2gource probably doesn't find the log it is supposed to generate.
What is the output of
ls ~/.pacmanlog2gource/
?
It should be
./ ../ pacman_gource_pie.log pacman_gource_tree.log pacman_now.log version

Does pacmanlog2gource generate any suspicious output while running?
Can you remove the .pacmanlog2gource directory (hidden directory in your home folder), re-run pacmanlog2gource, and upload the log (pacmanlog2gource.log) somewhere, so I can see if there are any warnings/errors?
pacmanlog2gource -c >& pacmanlog2gource.log

Stebalien · 2012-04-04 15:09:22

Works here.

wolfcore · 2012-04-04 15:37:57

matthiaskrgr wrote:

Ok, pacmanlog2gource probably doesn't find the log it is supposed to generate.
What is the output of
ls ~/.pacmanlog2gource/
?
It should be
./ ../ pacman_gource_pie.log pacman_gource_tree.log pacman_now.log version

Yes, all these files are here.

matthiaskrgr wrote:

Does pacmanlog2gource generate any suspicious output while running?

It only generates the help output.

matthiaskrgr wrote:

Can you remove the .pacmanlog2gource directory (hidden directory in your home folder), re-run pacmanlog2gource, and upload the log (pacmanlog2gource.log) somewhere, so I can see if there are any warnings/errors?
pacmanlog2gource -c >& pacmanlog2gource.log

After running

pacmanlog2gource

It now works by running

pacmanlog2gource -g

What does the -g flag do?-- I couldn't find it in the --help output.

Also, is the program called pacmanlog2gource, or gource?

Anyways, I did uploaded the log http://sprunge.us/hSRJ

pacmanlog2gource -g now works. It may have worked since the beginning, but I thought the command to run was just 'gource'

Your program looks awesome, by the way!

matthiaskrgr · 2012-04-04 15:54:53

wolfcore wrote:

matthiaskrgr wrote:
Ok, pacmanlog2gource probably doesn't find the log it is supposed to generate.
What is the output of
ls ~/.pacmanlog2gource/
?
It should be
./ ../ pacman_gource_pie.log pacman_gource_tree.log pacman_now.log version
Yes, all these files are here.
matthiaskrgr wrote:
Does pacmanlog2gource generate any suspicious output while running?
It only generates the help output.
matthiaskrgr wrote:
Can you remove the .pacmanlog2gource directory (hidden directory in your home folder), re-run pacmanlog2gource, and upload the log (pacmanlog2gource.log) somewhere, so I can see if there are any warnings/errors?
pacmanlog2gource -c >& pacmanlog2gource.log
After running
pacmanlog2gource
It now works by running
pacmanlog2gource -g
What does the -g flag do?-- I couldn't find it in the --help output.

Try "-h" instead of "--help"
"-g" first updates the log and then starts "gource" to visualize the log.

Also, is the program called pacmanlog2gource, or gource?

My script is called "pacmanlog2gource".
"gource" is a seperate program.
My script mostly creates the log which "gource" can read. The "-g" option starts "gource" after the log has been generated.

Anyways, I did uploaded the log http://sprunge.us/hSRJ

Looks fine.

pacmanlog2gource -g now works. It may have worked since the beginning, but I thought the command to run was just 'gource'

Explained above.

Your program looks awesome, by the way!

Thanks!

baronmax · 2012-05-16 16:03:14

This may be the coolest thing I've ever seen in terms of "updates". Being able to actually VISUALIZE how your system is updated is really amazing.

Thanks for this, it really made my day! Very easy to use and very easy to make the videos to upload to youtube. Works flawlessly!

Here's a youtube video of my newest Arch system's updates (only a few weeks worth) using pacmanlog2gource:

http://www.youtube.com/watch?v=WOU404fu … e=youtu.be
It's an intel Sandy Bridge 2600K with a *relatively* bare system (probably not bare compared to many others, but it's bare for me). It's running AWN + Compiz as WM and "only" (haha) 799 packages total.

I'm definitely anxious to see how this looks after a year, or more. Now that I am done distro-hopping (Arch is my $HOME), I will be able to see how this looks after many, many more updates.

Again, thank you so much Matthias, this is incredibly cool.

matthiaskrgr · 2012-05-16 21:26:23

Hehe, thanks.

Note that you can hide file names using the -m argument, this will look much prettier in my opinion.

If you want to render a video in a higher resolution, you can get the command which is run via pacmanlog2gource -i (also check out -h) and replace -1280x720 with -1920x1080 for example.

I just released 1.8.1 which will display the log in shorter time while make the movement of the root-user less hectical. The amount of cpu threads ffmpeg is run is chosen a little more wisely and some command line arguments are improved slighly.

For the record, I'm using arch since end of august 2011, my pacmanlog2gource log has 8376 lines, the original log 24686 lines (I update my system several times a day ). The files are found in .pacmanlog2gource in your home directory.

Regards, Matthias

Rolinh · 2012-05-21 06:17:34

I've just tested it (under Awesome too ) and it works just like a charm.
Great job.

However, why don't you use a repository for your project? This way, it would be easier for people to contribute and follow development process.

matthiaskrgr · 2012-05-21 15:58:07

Mmh, I'll think about it.

I already do use a repo for deving, but mostly to see what changes I did in case I break something.
I make a lot of very small commits and commit messages aren't really worth reading...

matthiaskrgr · 2012-05-22 13:41:46

Rolinh wrote:

However, why don't you use a repository for your project? This way, it would be easier for people to contribute and follow development process.

https://github.com/matthiaskrgr/pacmanlog2gource

Rolinh · 2012-05-22 16:42:44

PS: sorry, I found this comment in v0.5 quite funny considering what was above (fi fi fi...)

# yes, the code above sucks

Last edited by Rolinh (2012-05-22 16:42:58)

matthiaskrgr · 2012-05-22 22:12:52

Heh.

Performance was horrible back then.

I just let version 0.5 run with my current pacman.log.

0.5: 40 minutes 12 seconds
current git: 2 minutes 28 seconds

Awebb · 2012-05-22 23:00:33

It looks cool. I don't know if I can see any relevant information, but a script that makes a logfile look that cool is good.

matthiaskrgr · 2012-05-24 01:22:08

The script is mostly a toy, it doesn't have any great meaning but converting the log and makeing gource visualize it.

I just released 1.8.5

pacmanlog2gource -i does now show if we hide filenames or not, prints the version of gource, arguments passed to pacmanlog2gource and a link to this thread and the github page.
more package subcategories have been added: libreoffice/extensions, lib/32, gambas/[2, 3], several for kde, vim, texlive
I also added a check which gets a checksum from a part (log generation) of the script itself and saves this checksum, so we can see if the part has changed or not (due to an update for example). If an older checksum and the new checksum mismatch (indicating that log-generation process has changed), we regenerate the entire log.

While googling "pacmanlog2gource" out of curiosity, I found this
http://www.firstpost.com/topic/organiza … 30-30.html which was quite amusing.

Regards, Matthias

Rolinh · 2012-05-24 08:42:59

Hi,

I think the PKGBUILD should be rewritten to fetch data from github instead of including them in the tarball.
I can do this and also write a -git version if you want.

EDIT: just a simple question: why don't you use a switch case statement in makelog() function instead of the huge amount of elif?

Last edited by Rolinh (2012-05-24 09:19:47)

matthiaskrgr · 2012-05-24 09:49:09

Rolinh wrote:

Hi,
I think the PKGBUILD should be rewritten to fetch data from github instead of including them in the tarball.
I can do this and also write a -git version if you want.

Done

EDIT: just a simple question: why don't you use a switch case statement in makelog() function instead of the huge amount of elif?

Mmmh, I'll have a look at it.

Rolinh · 2012-05-24 10:01:53

This was quick

matthiaskrgr · 2012-05-24 17:59:24

matthiaskrgr wrote:

Rolinh wrote:
Hi,
I think the PKGBUILD should be rewritten to fetch data from github instead of including them in the tarball.
I can do this and also write a -git version if you want.
Done
EDIT: just a simple question: why don't you use a switch case statement in makelog() function instead of the huge amount of elif?
Mmmh, I'll have a look at it.

Fixed in latest master.

Rolinh · 2012-05-24 19:06:13

This was quick (bis)

Much more readable. Did you notice any change in performance?

matthiaskrgr · 2012-05-24 19:21:45

1.8.5: ~67.40 lines per s
git: ~67.97 lines per s (case stuff)
(for 9194 lines)

I don't think that this can be called a big improvement
However, I was able to do a few minor code simplifications while implemeting the case checks.

matthiaskrgr · 2012-05-26 14:41:32

1.9.0 has been released:
Implement -l and -L to shown an arch linux icon/logo in the lower right corner of the screen, if we don't have the images locally, try downloading them
Create a lockfile so we don't get problems if we run pacmanlog2gource twice at the same time
Use defined colors for package groups
Fix bug where logfiles were deleted due to change of logfile generation although we used -n to keep the log from being updated
Show when stoudt of ffmpeg starts/stops
Increase bloom (--bloom-multiplier) from 1.0 (default) to 1.3
Other code changes (make use of case instead of if if if etc...)

Regards, Matthias

matthiaskrgr · 2012-06-24 11:47:26

2.0.0 has been released:

Speed up the script by usig "let" instead of "expr" for calculations.
Check if /var/log/pacman.log exists
Display purging efficiency
Implement -q quiet mode which is faster than default but doesn't estimate when log generation will be done
Use function to get better size (bytes) values.

Kind regards, Matthias

Arch Linux

#1 2012-04-03 15:05:39

pacmanlog2gource - visualize package upgrade history

#2 2012-04-04 03:14:30

Re: pacmanlog2gource - visualize package upgrade history

#3 2012-04-04 05:54:25

Re: pacmanlog2gource - visualize package upgrade history

#4 2012-04-04 14:32:39

Re: pacmanlog2gource - visualize package upgrade history

#5 2012-04-04 14:44:57

Re: pacmanlog2gource - visualize package upgrade history

#6 2012-04-04 15:09:22

Re: pacmanlog2gource - visualize package upgrade history

#7 2012-04-04 15:37:57

Re: pacmanlog2gource - visualize package upgrade history

#8 2012-04-04 15:54:53

Re: pacmanlog2gource - visualize package upgrade history

#9 2012-05-16 16:03:14

Re: pacmanlog2gource - visualize package upgrade history

#10 2012-05-16 21:26:23

Re: pacmanlog2gource - visualize package upgrade history

#11 2012-05-21 06:17:34

Re: pacmanlog2gource - visualize package upgrade history

#12 2012-05-21 15:58:07

Re: pacmanlog2gource - visualize package upgrade history

#13 2012-05-22 13:41:46

Re: pacmanlog2gource - visualize package upgrade history

#14 2012-05-22 16:42:44

Re: pacmanlog2gource - visualize package upgrade history

#15 2012-05-22 22:12:52

Re: pacmanlog2gource - visualize package upgrade history

#16 2012-05-22 23:00:33

Re: pacmanlog2gource - visualize package upgrade history

#17 2012-05-24 01:22:08

Re: pacmanlog2gource - visualize package upgrade history

#18 2012-05-24 08:42:59

Re: pacmanlog2gource - visualize package upgrade history

#19 2012-05-24 09:49:09

Re: pacmanlog2gource - visualize package upgrade history

#20 2012-05-24 10:01:53

Re: pacmanlog2gource - visualize package upgrade history

#21 2012-05-24 17:59:24

Re: pacmanlog2gource - visualize package upgrade history

#22 2012-05-24 19:06:13

Re: pacmanlog2gource - visualize package upgrade history

#23 2012-05-24 19:21:45

Re: pacmanlog2gource - visualize package upgrade history

#24 2012-05-26 14:41:32

Re: pacmanlog2gource - visualize package upgrade history

#25 2012-06-24 11:47:26

Re: pacmanlog2gource - visualize package upgrade history

Board footer