What's the deal with python-magic?

danielquinn · 2018-09-02 23:19:12

I'm trying to write a PKGBUILD for a Python package I wrote and I've run up against what appears to be a total mess in the magic namespace that's making this task rather complicated:

This package does some stuff with mimetypes, so I had it require the file-magic module.
- The version on PyPI for this module is 0.4.0
- The API looks like this: magic.detect_from_filename(path)
Arch has a package called python-magic in the community repository that, as far as I can tell is actually this same module, though the version numbers are totally different.
- It has a current version of 5.30-2
- The API is identical to file-magic==0.4.0 above: magic.detec_from_filename(path)
There is also a module on PyPI called python-magic.
- Its version number is also drastically different from what we see in the community repo: 0.4.15
- It's API is totally different however and looks like this: magic.from_file(path).

I managed to write a PKGINFO file with depends=("python-magic"). It installed just fine, but when I ran my module I got:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 578, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 895, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (file-magic 0.3.0 (/usr/lib/python3.7/site-packages), Requirement.parse('file-magic>=0.4.0'), {'aletheia'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/aletheia", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3105, in <module>
    @_call_aside
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3089, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3118, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 580, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 593, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 781, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'file-magic>=0.4.0' distribution was not found and is required by aletheia

It appears my setup.py has injected a check for requirements and it's finding that file-magic isn't available... even though if you import magic the API is identical.

I honestly don't know what to do here, but I'm guessing that I'm not the first person to have run into this problem so I'm asking here. What's the right way to work around this? What do other packages do?

The package in question is here.

eschwartz · 2018-09-03 00:08:56

file-magic is the *official* bindings provided by the developers of "file".

It currently uses the version pinned to the file release, which is *not* the same version as the PyPI release... which is confusing, yes.

Now, python-magic also provides "import magic" but using a different pkgname and a different API to boot! But that's an unofficial set of bindings, as evidenced by the warning on the project page: https://github.com/ahupp/python-magic#name-conflict

To add to the confusion, there is also https://pypi.org/project/magic/ -- which you'd think would be the natural name for this, right? But no, it's a dead project with a vanished author, and the code doesn't even exist. The PyPI entry is completely bitrotted.
I've actually created https://bugs.astron.com/view.php?id=15 to see if the file-magic package can acquire that namespace.

...

Yes, it's all a mess. But you do need file-magic to come from community/python-magic, not python-magic-ahupp from the AUR.
Or depend in setup.py, on python-magic instead of file-magic, and depend on python-magic-ahupp from the AUR.

danielquinn · 2018-09-03 09:11:09

I'm sorry, but I'm still confused. Could you perhaps spell it out a bit more for me? Given that I control the Python package in question, which module should I make it depend on, the best-practise if you will? What should I have in my setup.py, and what should I have in my PKGINFO?

eschwartz · 2018-09-03 12:31:29

You should use file-magic in your setup.py and python-magic in your PKGBUILD/.PKGINFO

This is because you want to rely on packages in the official repositories rather than requiring users to install an additional AUR package.

... Also you should fix the checkdepends on pytest ==> python-pytest. Currently they don't matter as you don't declare a check() function...

danielquinn · 2018-09-03 20:14:21

Thank you so much for your help. I was stumped on the whole mess of different "magics" out there and your explanation really helped clear things up.

The problem in my case went a bit further though. It turns out that Arch's file-magic is version 0.3.0 while my project had a dependency on 0.4.0. So, even though I was using file-magic in my module and Arch was using file-magic internally (called python-magic because REASONS), my code was still barfing with that backtrace because of the version mismatch. It never occurred to me that python-magic-5.30-2 == file-magic-0.3.0 and not 0.4.0. What a headache.

Anyway, my package is up on the AUR now and it appears to be working. Thanks again!

Arch Linux

#1 2018-09-02 23:19:12

What's the deal with python-magic?

#2 2018-09-03 00:08:56

Re: What's the deal with python-magic?

#3 2018-09-03 09:11:09

Re: What's the deal with python-magic?

#4 2018-09-03 12:31:29

Re: What's the deal with python-magic?

#5 2018-09-03 20:14:21

Re: What's the deal with python-magic?

Board footer