Communitytags - Allow packages to be tagged by the community

ids1024 · 2015-01-17 17:53:03

Communitytags was started as a response to this thread. This tool allows tags to be added to packages, so that users can search for packages by tag. Here is a basic example:

$ communitytags tag=suckless
9base
dmenu
dwm
slock
surf
swarp
sxiv
tabbed
wmii
wmname
xssstate

Tags are stored in a github repository as files in a simple format. A file exists for each package, and each line is a tag. Blank lines or comments starting with # will be ignored.

To submit tags, you can ask for collaborator access to the github repo. At the moment, I will grant anyone such access, but this will change if it becomes a problem. It may be a good idea to move to a different method of hosting the tags in the future, but this is how it works for now. To ask to be added as a collaborator, make your request on the Github issue I created for this purpose.

The query tool can be found in this repository. It also supports some standard package metadata and Pacnet provided categories. By adding the -r option to communitytags, the search will have regex support. Here are some examples:

Find packages tagged suckless:
$ communitytags tag=suckless
Find packages tagged foo or bar but not baz:
$ communitytags '(tag=foo or tag=bar) and not tag=baz'
Find packages hosted on suckless.org missing the suckless tag:
$ communitytags -r 'url=.*suckless.org and not tag=suckless'
Find all packages with "python" in their descriptions that are hosted on github:
$ communitytags -r 'url=.*github and desc=.*python'
Find all packages in extra with names starting with "python":
$ communitytags -r 'name=python and repo=extra'
List packages in pacnet's "app-pacman" category:
$ communitytags category=app-pacman
Find packages containing files with "pkgfile" in their paths:
$ communitytags -r 'file=.*pkgfile'

As you can see, communitytags can be used for various queries that do not involve tags. Even though there are currently few tags, it can be useful for some advanced queries.

jernst · 2015-01-17 20:18:56

Excellent!!

thiagowfx · 2015-01-17 20:59:18

It isn't in the AUR yet?
- lazy one here wants to test it

ids1024 · 2015-01-17 23:49:20

I have created an AUR package:

https://aur.archlinux.org/packages/communitytags-git/

thiagowfx · 2015-01-23 03:46:36

Thanks, I'm bookmarking it. I'll test it and provide some feedback later.

thiagowfx · 2015-01-23 20:56:03

I guess I'm with some problem:

thiago@archpad ..tytags-git/src/archcommunitytags-tools (git)-[master] % communitytags tag=suckless
Metadata directory does not exist: running initial sync
Cloning into '/home/thiago/.communitytags'...
remote: Counting objects: 31, done.
remote: Total 31 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (31/31), 2.61 KiB | 0 bytes/s, done.
Resolving deltas: 100% (3/3), done.
Checking connectivity... done.
Getting file lists from sync dbs...
Traceback (most recent call last):
  File "/usr/lib/python3.4/tarfile.py", line 1620, in gzopen
    t = cls.taropen(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1596, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1470, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python3.4/tarfile.py", line 2249, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/usr/lib/python3.4/tarfile.py", line 1082, in fromtarfile
    buf = tarfile.fileobj.read(BLOCKSIZE)
  File "/usr/lib/python3.4/gzip.py", line 365, in read
    if not self._read(readsize):
  File "/usr/lib/python3.4/gzip.py", line 433, in _read
    if not self._read_gzip_header():
  File "/usr/lib/python3.4/gzip.py", line 297, in _read_gzip_header
    raise OSError('Not a gzipped file')
OSError: Not a gzipped file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/communitytags", line 145, in <module>
    sync()
  File "/usr/bin/communitytags", line 82, in sync
    tar = tarfile.open(mode='r:gz', fileobj=files)
  File "/usr/lib/python3.4/tarfile.py", line 1566, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1624, in gzopen
    raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file
communitytags tag=suckless  9.17s user 0.92s system 39% cpu 25.457 total
1 thiago@archpad ..tytags-git/src/archcommunitytags-tools (git)-[master] % communitytags tag=suckless                                                                  :(
Traceback (most recent call last):
  File "/usr/bin/communitytags", line 158, in <module>
    for i in matches:
  File "/usr/bin/communitytags", line 156, in <genexpr>
    matches = (package for package, pacmeta in packages
  File "/usr/bin/communitytags", line 36, in parseMetadata
    with open(tagdir + '.pacnet') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/thiago/.communitytags/.pacnet'

Edit: this was a result of running communitytags from your AUR package, but the same error occurs by running it directly.

Last edited by thiagowfx (2015-01-23 20:56:53)

ids1024 · 2015-01-23 21:48:48

@thiagowfx I don't know what is causing that issue. I have pushed a commit to make the syncing more verbose. Try it again and post the output.

thiagowfx · 2015-01-24 00:00:21

thiago@archpad ~/aur/communitytags-git % communitytags tag=suckless
Metadata directory does not exist: running initial sync
Cloning into '/home/thiago/.communitytags'...
remote: Counting objects: 31, done.
remote: Total 31 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (31/31), done.
Resolving deltas: 100% (3/3), done.
Checking connectivity... done.
Getting file lists from sync dbs...
http://archlinux.c3sl.ufpr.br/testing/os/x86_64/testing.files
http://archlinux.c3sl.ufpr.br/core/os/x86_64/core.files
http://archlinux.c3sl.ufpr.br/extra/os/x86_64/extra.files
http://archlinux.c3sl.ufpr.br/community-testing/os/x86_64/community-testing.files
http://archlinux.c3sl.ufpr.br/community/os/x86_64/community.files
http://archlinux.c3sl.ufpr.br/multilib-testing/os/x86_64/multilib-testing.files
http://archlinux.c3sl.ufpr.br/multilib/os/x86_64/multilib.files
http://bohoomil.com/repo/x86_64/infinality-bundle.files
http://bohoomil.com/repo/multilib/x86_64/infinality-bundle-multilib.files
http://bohoomil.com/repo/fonts/infinality-bundle-fonts.files
https://deadbeef.ninja/blackarch/blackarch/os/x86_64/blackarch.files
http://www.lasca.ic.unicamp.br/pub/archassault/archassault/os/x86_64/archassault.files
Traceback (most recent call last):
  File "/usr/lib/python3.4/tarfile.py", line 1620, in gzopen
    t = cls.taropen(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1596, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1470, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python3.4/tarfile.py", line 2249, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/usr/lib/python3.4/tarfile.py", line 1082, in fromtarfile
    buf = tarfile.fileobj.read(BLOCKSIZE)
  File "/usr/lib/python3.4/gzip.py", line 365, in read
    if not self._read(readsize):
  File "/usr/lib/python3.4/gzip.py", line 433, in _read
    if not self._read_gzip_header():
  File "/usr/lib/python3.4/gzip.py", line 297, in _read_gzip_header
    raise OSError('Not a gzipped file')
OSError: Not a gzipped file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/communitytags", line 146, in <module>
    sync()
  File "/usr/bin/communitytags", line 83, in sync
    tar = tarfile.open(mode='r:gz', fileobj=files)
  File "/usr/lib/python3.4/tarfile.py", line 1566, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python3.4/tarfile.py", line 1624, in gzopen
    raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file
communitytags tag=suckless  7.79s user 0.64s system 31% cpu 26.350 total
1 thiago@archpad ~/aur/communitytags-git % communitytags tag=suckless                                                                                                  :(
Traceback (most recent call last):
  File "/usr/bin/communitytags", line 159, in <module>
    for i in matches:
  File "/usr/bin/communitytags", line 157, in <genexpr>
    matches = (package for package, pacmeta in packages
  File "/usr/bin/communitytags", line 36, in parseMetadata
    with open(tagdir + '.pacnet') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/thiago/.communitytags/.pacnet'

After removing (commenting out) archassault from my pacman.conf, it worked okay.

1 thiago@archpad ~/aur/communitytags-git % rm -rf ~/.communitytags                                                                                                :(
thiago@archpad ~/aur/communitytags-git % communitytags tag=suckless  
Metadata directory does not exist: running initial sync
Cloning into '/home/thiago/.communitytags'...
remote: Counting objects: 31, done.
remote: Total 31 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (31/31), done.
Resolving deltas: 100% (3/3), done.
Checking connectivity... done.
Getting file lists from sync dbs...
http://archlinux.c3sl.ufpr.br/testing/os/x86_64/testing.files
http://archlinux.c3sl.ufpr.br/core/os/x86_64/core.files
http://archlinux.c3sl.ufpr.br/extra/os/x86_64/extra.files
http://archlinux.c3sl.ufpr.br/community-testing/os/x86_64/community-testing.files
http://archlinux.c3sl.ufpr.br/community/os/x86_64/community.files
http://archlinux.c3sl.ufpr.br/multilib-testing/os/x86_64/multilib-testing.files
http://archlinux.c3sl.ufpr.br/multilib/os/x86_64/multilib.files
http://bohoomil.com/repo/x86_64/infinality-bundle.files
http://bohoomil.com/repo/multilib/x86_64/infinality-bundle-multilib.files
http://bohoomil.com/repo/fonts/infinality-bundle-fonts.files
https://deadbeef.ninja/blackarch/blackarch/os/x86_64/blackarch.files
http://xyne.archlinux.ca/repos/xyne/xyne-any.files
Syncing with pacnet...
9base
dmenu
dwm
slock
surf
swarp
sxiv
tabbed
wmii
wmname
xssstate
communitytags tag=suckless  12.94s user 1.01s system 38% cpu 36.067 total

ids1024 · 2015-01-24 00:03:51

@thiagowfx You beat me by a few seconds. I had just found what I think is the problem when you posted. Try with the latest commit.

Edit: It looks like I was right. That repo uses xz compression for the .files instead of gzip.
Edit2: Also, you could have just run --sync instead of deleting the .communitytags.

Last edited by ids1024 (2015-01-24 00:09:20)

thiagowfx · 2015-01-26 13:40:27

Okay, it works flawlessly now.

Is there any way to retrieve a list of all tags with a counter? Something such as

suckless 8
gui 2

for example.

Also, if you want to improve it, after passing a -h flag, I think that

positional arguments:
  query

doesn't help much. What do you think about documenting the queries?

Of course, those are just suggestions.

ids1024 · 2015-01-26 16:53:13

I could add a command for listing tags like that. I also agree that the documentation is lacking, since the only documentation at the moment is the first post to this thread and the source code itself.

ids1024 · 2015-01-26 18:02:40

I have added a man page providing more documentation and a --tags command listing the tags with a counter.

Arch Linux

#1 2015-01-17 17:53:03

Communitytags - Allow packages to be tagged by the community

#2 2015-01-17 20:18:56

Re: Communitytags - Allow packages to be tagged by the community

#3 2015-01-17 20:59:18

Re: Communitytags - Allow packages to be tagged by the community

#4 2015-01-17 23:49:20

Re: Communitytags - Allow packages to be tagged by the community

#5 2015-01-23 03:46:36

Re: Communitytags - Allow packages to be tagged by the community

#6 2015-01-23 20:56:03

Re: Communitytags - Allow packages to be tagged by the community

#7 2015-01-23 21:48:48

Re: Communitytags - Allow packages to be tagged by the community

#8 2015-01-24 00:00:21

Re: Communitytags - Allow packages to be tagged by the community

#9 2015-01-24 00:03:51

Re: Communitytags - Allow packages to be tagged by the community

#10 2015-01-26 13:40:27

Re: Communitytags - Allow packages to be tagged by the community

#11 2015-01-26 16:53:13

Re: Communitytags - Allow packages to be tagged by the community

#12 2015-01-26 18:02:40

Re: Communitytags - Allow packages to be tagged by the community

Board footer