You are not logged in.

#126 2013-02-20 16:23:44

schultzter
Member
From: Montreal, QC
Registered: 2012-01-18
Posts: 15
Website

Re: pkgstats round two: take your vote and help improving Arch

Where I work we often use the boot disk's UUID or the first network card's MAC address to identify a machine. That's not perfect, they can be played with too, but they are typically more stable than a machine's IP address.

Is the data anonymized before it is sent? Or only once it arrives on the server? Is it sent over a secure connection? Just some questions that come to mind, I'll try to look at the code later.


Headed for the second star to the right and straight on 'til morning...

  Schultzter

Offline

#127 2013-05-09 21:01:45

maggie
Member
Registered: 2011-02-12
Posts: 255

Re: pkgstats round two: take your vote and help improving Arch

So it runs /etc/cron.weekly/pkgstats. Does that mean it runs Sunday at 00:00? What happens if my machine is switched off then (laptop)?

Offline

#128 2013-05-10 07:43:50

x33a
Forum Fellow
Registered: 2009-08-15
Posts: 4,587

Re: pkgstats round two: take your vote and help improving Arch

@ maggie, Anacron?

Basically it depends on which cron implementation you are using. But most usually have mechanisms in place to run *missed jobs*.

Offline

#129 2016-01-14 22:14:34

glider
Member
Registered: 2016-01-14
Posts: 6

Re: pkgstats round two: take your vote and help improving Arch

@schultzter you miss some important points which was mentioned by @Pierre

Pierre wrote:

Not at all.

See:
1) I don't want track individual users for privacy reasons.
2) Everything that is sent by pkgstats can easily manipulated by the user without us noticing. So any idea based on sending data is flawed.
3) The IP hash is only used to prevent too easy flooding; not to track users or make the stats any more accurate.
4) There is no way to get exact values, but over time if more and more people use pkgstats some single variations (e.g. when someone sends garbage) wont matter.

Offline

#130 2016-01-15 04:52:14

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,728

Re: pkgstats round two: take your vote and help improving Arch

He glider, Welcome to Arch Linux.

Be aware that you just responded to a post that is coming up on three years old.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#131 2016-09-26 19:28:34

SirCmpwn
Member
Registered: 2013-09-18
Posts: 89

Re: pkgstats round two: take your vote and help improving Arch

Sorry to necropost, but this looks like the right place to ask.

Is there anywhere to get the raw data from pkgstats? The web page is a bit hard to analyze.

Offline

#132 2016-11-28 08:56:43

psycho_tea_drinker
Member
From: West Sussex, United Kingdom
Registered: 2013-07-02
Posts: 41

Re: pkgstats round two: take your vote and help improving Arch

SirCmpwn wrote:

Sorry to necropost, but this looks like the right place to ask.

Is there anywhere to get the raw data from pkgstats? The web page is a bit hard to analyze.

Yes!!! big_smile I recently made a project which scrapes the site and outputs the result as a JSON file. (Have updated the wiki at: https://wiki.archlinux.org/index.php/pkgstats with a link to the project.)

You will need to install Haskell, alternatively I've got the JSON file hosted here: http://trycatchchris.co.uk/files/packagestatistics.json. I could look into getting this dockerized, or even hosted and updated accordingly.

Do you have any project in mind for the data?

Last edited by psycho_tea_drinker (2016-11-28 09:01:29)

Offline

#133 2017-08-30 07:19:02

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: pkgstats round two: take your vote and help improving Arch

I did a little polish of the stats page. I might extract it into its own service some day. There is a JSON export now:
* https://www.archlinux.de/statistics
* https://www.archlinux.de/statistics/package
* https://www.archlinux.de/statistics/package.json
* https://www.archlinux.de/statistics/module.json

Expect some things to change though.

Offline

#134 2017-08-30 11:14:31

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,751
Website

Re: pkgstats round two: take your vote and help improving Arch

Pierre wrote:

I did a little polish of the stats page. I might extract it into its own service some day. There is a JSON export now:
* https://www.archlinux.de/statistics/

This page is 404. sad


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#135 2017-08-30 11:17:30

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: pkgstats round two: take your vote and help improving Arch

Yep, fixed that. Remove the trailing /.

Offline

#136 2017-09-01 00:57:30

Xavion
Member
From: Australia
Registered: 2010-03-13
Posts: 38

Re: pkgstats round two: take your vote and help improving Arch

@Pierre
With the "package.json" file:
1) How do we convert the 'count' to a percentage?  Is there a maximum value somewhere, or do we just use 'filesystem' (count = 29901) as the reference point?
2) The file is about 2.1 MiB in size.  Can you also provide a "package.json.tar.bz2"?  This would get it down to about 308 KiB.

Last edited by Xavion (2017-09-01 03:12:33)

Offline

#137 2017-09-01 10:05:43

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: pkgstats round two: take your vote and help improving Arch

It's already gzip'ed and about 380KByte. But this is just a complete dump. Probably not that useful. I plan to decouple this API from the rest of the website and start improving on it. E.g. it would be great to asko for stats of a single package.

I still need to figure out what is going on with the numbers. Using some package as base sounds fine to me for now.

Offline

#138 2017-09-01 21:19:12

Xavion
Member
From: Australia
Registered: 2010-03-13
Posts: 38

Re: pkgstats round two: take your vote and help improving Arch

Pierre wrote:

It's already gzip'ed and about 380KByte.

What's the URL?  I want to start using it right away.

Pierre wrote:

But this is just a complete dump. Probably not that useful.

How often is the dump updated (e.g. daily or weekly)?

Offline

#139 2017-12-27 14:09:36

ua4000
Member
Registered: 2015-10-14
Posts: 395

Re: pkgstats round two: take your vote and help improving Arch

archlinux32.org project also offers pkgstats package and seems to report it's summary also to https://pkgstats.archlinux.de/post

The data can be distinguish by arch="$(uname -m)" -  is there a way on https://pkgstats.archlinux.de/package to filter for i686 usage ?
Or should archlinux32.org project don't provide the current pkgstats package and implement a fork with another post server ?

Thanks.

Offline

#140 2017-12-27 14:15:00

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: pkgstats round two: take your vote and help improving Arch

ua4000 wrote:

is there a way on https://pkgstats.archlinux.de/package to filter for i686 usage?

What do you mean filter for it?  In old data maybe, but in current data: yes all i686 is already filtered out!


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#141 2017-12-28 08:42:32

ua4000
Member
Registered: 2015-10-14
Posts: 395

Re: pkgstats round two: take your vote and help improving Arch

filtered = discarded - i686 statistics are now not available anymore ?

I rewrite my question: does it make sense to use and run pkgstats package (same as on archlinux.org) in archlinux32 ?

Offline

#142 2017-12-28 12:14:18

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: pkgstats round two: take your vote and help improving Arch

I suppose it does collect the architecture along with everything else, and right now it does still have i686 included.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#143 2018-01-22 06:48:11

Jristz
Member
From: America/Santiago
Registered: 2011-06-11
Posts: 1,022

Re: pkgstats round two: take your vote and help improving Arch

Trilby wrote:

I suppose it does collect the architecture along with everything else, and right now it does still have i686 included.

I'm more concerned that still on archlinux.de and not on archlinux.org and the move to .org was due like since it conception I remember


Well, I suppose that this is somekind of signature, no?

Offline

#144 2018-07-30 17:23:14

jens1o
Member
Registered: 2018-07-30
Posts: 5

Re: pkgstats round two: take your vote and help improving Arch

Hi, I wonder where I can see the sourcecode of the server-side processing? The link in post #2 is dead currently?

Offline

#145 2018-08-30 18:05:00

yan12125
Member
Registered: 2017-11-01
Posts: 36

Re: pkgstats round two: take your vote and help improving Arch

I got "400 bad request" if I run pkgstats on my server. Any idea what's going on?

The output of pkgstats -s: https://cfp.vim-cn.com/cbfmx

The culprit might be the empty modules= line. The server is a Linode VM, which uses the kernel provided by Linode. Seems all modules are linked into the kernel itself -

  • There's only an empty file modules.dep in /lib/modules/$(uname -r)

  • `lsmod` shows nothing

By the way, the value for mirror is incorrect due to a change in pacman 5.1. I applied a fix similar to https://lists.archlinux.org/pipermail/a … 04720.html and it works now.

Offline

#146 2019-03-15 08:29:14

iMike
Member
From: Sweden
Registered: 2019-02-14
Posts: 6

Re: pkgstats round two: take your vote and help improving Arch

Nice project.  Just checked it out, but pages for package and module statistics https://pkgstats.archlinux.{de,org}/{package,module} don't show any stats.  The German fun stats page does though.  Is the project still alive?

Offline

#147 2019-03-15 09:16:13

yan12125
Member
Registered: 2017-11-01
Posts: 36

Re: pkgstats round two: take your vote and help improving Arch

iMike wrote:

Nice project.  Just checked it out, but pages for package and module statistics https://pkgstats.archlinux.{de,org}/{package,module} don't show any stats.  The German fun stats page does though.  Is the project still alive?

https://pkgstats.archlinux.de/{package,module} still works fine! Tested with aur/firefox-beta-bin 66.0rc2-1.

Offline

#148 2019-03-15 12:16:13

iMike
Member
From: Sweden
Registered: 2019-02-14
Posts: 6

Re: pkgstats round two: take your vote and help improving Arch

My bad.  Can confirm it's working!  ("No script" was blocking.)

Offline

#149 2019-06-22 08:25:20

Pierre
Developer
From: Bonn
Registered: 2004-07-05
Posts: 1,964
Website

Re: pkgstats round two: take your vote and help improving Arch

Just an update:
* pkgstats 2.4 no longer collects module data
* I improved the website https://pkgstats.archlinux.de/ and added some documentation
* An initial version of an API is available and documented at https://pkgstats.archlinux.de/api/doc

Offline

#150 2019-06-22 09:23:50

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,751
Website

Re: pkgstats round two: take your vote and help improving Arch

I think the fun stats for DE usage may have a bug -- 63% of informant machines have KDE/plasma, but only 34% have kwin 31% have plasma-desktop installed according to the pkg stats.

It's nice to see GNOME getting utterly trounced, but unfortunately I don't think it's accurate. wink

EDIT: plasma-desktop is probably the better indicator

Last edited by WorMzy (2019-06-22 09:28:56)


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

Board footer

Powered by FluxBB