Additional meta-data for Pacman?

ids1024 · 2015-01-02 04:07:48

progandy wrote:

ids1024 wrote:
As for the query language, adding regex support would be easy, and could help. I will work on it soon.
I don't believe regex is necessary. I'd like to search for e.g " ( format=png or format=jpg ) and gui=yes"

Regex support could be useful too, but I can't think of a use case at the moment. As for what you're describing... you wouldn't happen to know how to implement that? I have been trying but I can't figure it out. I think a recursive function should be involved, but I can't figure out how to do it. It really should be simple... but my programming experience is somewhat limited.

Edit: With much Googling (well, DuckDuckGo actually) I have learned that something called a recursive-descent parser should be able to do it, although it may not be the best method. I can probably manage that, but not today.

Last edited by ids1024 (2015-01-02 05:03:01)

jernst · 2015-01-02 05:03:56

The easiest thing would if python had some kind of suitable parser ready to use. (I don't really know python, so I don't know whether it has but this seems to have some thoughts on this: http://stackoverflow.com/questions/1113 … ee-fashion )

Once you have the expression parsed, you walk down the parse tree, evaluate the leaves, and join them as you walk back up. That indeed would be recursive.

ids1024 · 2015-01-02 06:25:36

Ok, I think I have it working. The syntax is now in this format:

$ ./communitymeta "group=base-devel and group=base"

ids1024 · 2015-01-02 23:16:57

I have made more changes, and the client seems to be in good shape now. Many more features could be added in the future, but it seems to have the necessary core functionality working well.

I think I will start working on some metadata so this can actually be useful.

Edit: I have noticed another feature that could be useful. I am current writing basic metadata for sxiv, which uses imlib2 to handle metadata. It would make sense to get the format parameter from imlib2.

Edit2: To support what I mentioned, I added support for things like "format $imlib2.format". Additional values can be added in addition to the parameter reference.

Last edited by ids1024 (2015-01-03 00:23:42)

ids1024 · 2015-01-03 00:41:11

What metadata is really needed is a categorization system. Like in the AUR, but with many more categories, and possibly a hierarchy. There would by categories for "terminal emulator", "web browser", "window manager", "video editor", etc. This is a feature that is really lacking, and something that is useful enough to be worth integrating into pacman.

It seems that this should be one of the "core fields" I mentioned earlier. It makes sense to have both a general category like "network" (one of the AUR categories) and a specific one like "web browser". There would have to be some official (although community members could add to it) list of categories, and it would be important to have sane names for them.

Any thoughts on this?

karol · 2015-01-03 05:03:16

http://pacnet.archlinux.pl/ ?

ids1024 · 2015-01-03 05:08:07

karol wrote:

http://pacnet.archlinux.pl/ ?

Yes, that should do. I will try to integrate pacnet into the sync function so that their categories can be used with this program.

Edit: I have added such functionality, providing a "category" field.
Edit2: Although pacnet has not been updated since 2012... perhaps it would be better to import the data into communitymeta. It should be helpful either way.
Edit3: Actually, the most recent changes are 2014. However, there are only 3 in the year.

Last edited by ids1024 (2015-01-03 05:40:54)

ids1024 · 2015-01-03 21:50:48

I have added more functionality. Communitymeta syncs with pacnet for categories, and also retrieves file lists so it can be used like pkgfile. Regex support can be enabled with -r or --regex. Here are some examples of these features:

Find all packages with "python" in their descriptions that are hosted on github:
$ communitymeta -r 'url=.*github and desc=.*python'
Find all packages in extra with names starting with "python":
$ communitymeta -r 'name=python and repo=extra'
List packages in pacnet's "app-pacman" category:
$ communitymeta category=app-pacman
Find packages containing files with "pkgfile" in their paths:
./communitymeta -r 'file=.*pkgfile'

I think this tool could be useful even without the community metadata. I don't think there is anything else that allows queries like this of package data, and certainly none that also allow finding files (like pkgfile) and categories from pacnet.

Leonid.I · 2015-01-03 23:16:29

ids1024 wrote:

I have added more functionality. Communitymeta syncs with pacnet for categories, and also retrieves file lists so it can be used like pkgfile. Regex support can be enabled with -r or --regex. Here are some examples of these features:
Find all packages with "python" in their descriptions that are hosted on github:
$ communitymeta -r 'url=.*github and desc=.*python'
Find all packages in extra with names starting with "python":
$ communitymeta -r 'name=python and repo=extra'
List packages in pacnet's "app-pacman" category:
$ communitymeta category=app-pacman
Find packages containing files with "pkgfile" in their paths:
./communitymeta -r 'file=.*pkgfile'
I think this tool could be useful even without the community metadata. I don't think there is anything else that allows queries like this of package data, and certainly none that also allow finding files (like pkgfile) and categories from pacnet.

Not to ruin on your parade... but I think you are falling into the same trap as systemd (hostnamed actually) and Fedora devs. Namely, a desire to classify everything.

Just look at the discussion about Chassis values in hostnamectl of the on-going discussion on fedora-devel about which software to include in Gnome-software center. The main issue with having fields is that their contents should be agreed upon, i.e. there should be certain coordination among packagers and an overseeing body. The latter is trivial; the former just never works.

To the point: what is wrong with the package description? It is free-form, yes, but fortunately pacman -Ss understands regexp. So,

$ pacman -Ss "imap[A-Za-z0-9-]server"
extra/dovecot 2.2.15-1
    An IMAP and POP3 server written with security primarily in mind
extra/imap 2007f-4
    An IMAP/POP server

will void any need for protocol= and whatnot. For more complicated queries, there is always grep.

TL;DR: If you feel that some packages have an inaccurate description, then file a bugreport. But from my perspective, metadata should be kept to a minimum...

ids1024 · 2015-01-03 23:26:35

@Leonid.I The description field could probably handle many cases, yes. Perhaps the idea of community provided metadata is unrealistic or not particularly useful. Even in that case, however, I think the tool I have developed could be useful. None of the queries in my previous post use the community metadata; they all use standard metadata or the categories from pacnet.

That said, I think there is value to additional metadata. The description might not be particularly useful for finding what opens a specific file format, for example.

If the community metadata idea doesn't work, I will remove that portion of the script and just have it as a tool for more advanced queries of existing data.

fsckd · 2015-01-04 02:12:25

how about community provided tags instead? then imap related stuff can be tagged with imap. could also tag with mtu or mua or whatever else.

ids1024 · 2015-01-04 03:35:22

fsckd wrote:

how about community provided tags instead? then imap related stuff can be tagged with imap. could also tag with mtu or mua or whatever else.

That's a good idea; thanks for the suggestion. Even at this point, the metadata seems over-complicated. It is unclear what to name fields, what fields to have etc. Tags should simplify things and make it more realistic. Certainly community provided tags will work if there is enough interest in the community. It seems like tags would require less central coordination, which is good for a community project like this. I also don't see many disadvantages to using tags instead.

I should be changing communitymeta for this soon. I will make the file format a simple list of tags, one per line. There will also be support for comments, since those could be useful.

Edit: I have made such a change. It uses the format "tag=imap", since communitymeta also supports native pacman metadata.

Last edited by ids1024 (2015-01-04 06:36:19)

ids1024 · 2015-01-17 17:58:30

I have posted communitytags (the new name for communitymeta) in Community Contributions.

https://bbs.archlinux.org/viewtopic.php?pid=1493955

Arch Linux

#26 2015-01-02 04:07:48

Re: Additional meta-data for Pacman?

#27 2015-01-02 05:03:56

Re: Additional meta-data for Pacman?

#28 2015-01-02 06:25:36

Re: Additional meta-data for Pacman?

#29 2015-01-02 23:16:57

Re: Additional meta-data for Pacman?

#30 2015-01-03 00:41:11

Re: Additional meta-data for Pacman?

#31 2015-01-03 05:03:16

Re: Additional meta-data for Pacman?

#32 2015-01-03 05:08:07

Re: Additional meta-data for Pacman?

#33 2015-01-03 21:50:48

Re: Additional meta-data for Pacman?

#34 2015-01-03 23:16:29

Re: Additional meta-data for Pacman?

#35 2015-01-03 23:26:35

Re: Additional meta-data for Pacman?

#36 2015-01-04 02:12:25

Re: Additional meta-data for Pacman?

#37 2015-01-04 03:35:22

Re: Additional meta-data for Pacman?

#38 2015-01-17 17:58:30

Re: Additional meta-data for Pacman?

Board footer