jupyter metapackage in setup.py causes `DistributionNotFound`

leuko · 2020-12-05 17:24:27

My PKGBUILD for `python-nbgrader`:

# Maintainer: leuko <leuko_aydos_de>
pkgname=python-nbgrader-git
provides=('python-nbgrader')
conflicts=('python-nbgrader')
pkgver=r2892.3aa5af2a
pkgrel=1
pkgdesc="A system for assigning and grading notebooks"
arch=(any)
url="https://github.com/jupyter/nbgrader"
license=(BSD)
depends=(
	python
	python-sqlalchemy
	python-dateutil
	python-traitlets
	python-tornado
	python-requests
	python-jsonschema
	python-alembic
	python-rapidfuzz
	jupyter
	jupyter-notebook
	jupyter-nbconvert
	jupyter-nbformat
	python-jupyter_core
	python-jupyter_client
)
makedepends=("python-setuptools" "git")
source=("git+https://github.com/jupyter/nbgrader")
md5sums=('SKIP')
pkgver() {
        cd "$srcdir/nbgrader"
        printf "r%s.%s" "$(git rev-list --count HEAD)" "$(git rev-parse --short HEAD)"
}
build() {
        cd "$srcdir/nbgrader"
	#sed -Ei 's/^(\s*)("jupyter")/\1#\2/' setup.py
	python setup.py build
}
package() {
        cd "$srcdir/nbgrader"
	python setup.py install --prefix=/usr --root="$pkgdir" --optimize=1 --skip-build
	install -Dm644 LICENSE "$pkgdir/usr/share/licenses/$pkgname/LICENSE"

Without the `sed` line, the package creation and installation are successful, but I get the following error when I run the program:

$ nbgrader
pkg_resources.DistributionNotFound: The 'jupyter' distribution was not found and is required by nbgrader

When I remove "jupyter" from `install_requires` array in `setup.py` of `python-nbgrader` then there is no error and everything is fine, but I think this only a workaround.

On Archlinux `jupyter` is a metapackage and does not populate any files in `/usr/lib/python3.9/site-packages/jupyter`. I think that could be the reason why I get this error. Is this true?

I am new to `setup_tools`. Is it normal to include metapackages in `install_requires`? Or should every metapackage install at least one file in `site-packages` dir?

PS: Additonally I would be grateful for any styling comments on my PKGBUILD.

Lone_Wolf · 2020-12-06 16:26:06

python-setuptools knows nothing about archlinux packages (meta , split or any other type) .

You have looked at https://wiki.archlinux.org/index.php/Py … guidelines ?

The aur python-nbgrader comments inidcate you have the same issue with the stable version .

$ nbgrader
pkg_resources.DistributionNotFound: The 'jupyter' distribution was not found and is required by nbgrader

Maybe you can find the lines in nbgrader command that trigger that warning and post them ?
That way we'd know how/where nbgrader looks for "the jupyter distribution"

Last edited by Lone_Wolf (2020-12-06 16:31:00)

loqs · 2020-12-06 22:10:55

Lone_wolf from the autogenerated /usr/bin/nbgrader entrypoint using an also autogenerated /usr/lib/python3.9/site-packages/nbgrader-0.6.1-py3.9.egg-info/requires.txt both built by python setuptools?

Lone_Wolf · 2020-12-06 23:36:19

I was hoping for some custom logic by nbgrader, but you're right loqs. It's python-setuptools that reports this.

$ nbgrader 
Traceback (most recent call last):
  File "/usr/bin/nbgrader", line 33, in <module>
    sys.exit(load_entry_point('nbgrader==0.7.0.dev0', 'console_scripts', 'nbgrader')())
  File "/usr/bin/nbgrader", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/nbgrader/apps/__init__.py", line 1, in <module>
    from .baseapp import NbGrader
  File "/usr/lib/python3.9/site-packages/nbgrader/apps/baseapp.py", line 19, in <module>
    from nbgrader.exchange import ExchangeFactory
  File "/usr/lib/python3.9/site-packages/nbgrader/exchange/__init__.py", line 2, in <module>
    from nbgrader.exchange.abc import (Exchange, ExchangeError, ExchangeCollect, ExchangeFetch, ExchangeFetchAssignment,
  File "/usr/lib/python3.9/site-packages/nbgrader/exchange/abc/__init__.py", line 1, in <module>
    from .exchange import ExchangeError, Exchange
  File "/usr/lib/python3.9/site-packages/nbgrader/exchange/abc/exchange.py", line 10, in <module>
    from nbgrader.coursedir import CourseDirectory
  File "/usr/lib/python3.9/site-packages/nbgrader/coursedir.py", line 9, in <module>
    from .utils import full_split, parse_utc
  File "/usr/lib/python3.9/site-packages/nbgrader/utils.py", line 14, in <module>
    from setuptools.archive_util import unpack_archive
  File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 16, in <module>
    import setuptools.version
  File "/usr/lib/python3.9/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3239, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 770, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jupyter' distribution was not found and is required by nbgrader
$

it seems the question now is :
How does python-setuptools determine if jupyter is installed and what archlinux package satisfies that requirement ?

eschwartz · 2020-12-06 23:56:30

It is looking for a PEP 376 compatible module, e.g. from PyPI, e.g. pip-installable, named "jupyter", providing a suitable dist-info installation record in your python site-packages directory.

This is NOT the same as a pacman-installable *.pkg.tar.zst providing a suitable .PKGINFO installation record converted, in your pacman /var/lib/pacman/local directory, into a desc file.

Admittedly https://pypi.org/project/jupyter/ does exist, but it does nothing -- it is a metapackage too. No one would ever bother installing it. Don't depend on it either.

leuko · 2020-12-07 22:33:23

According to the comment in AUR [1] `community/jupyter` lacks the `jupyter.dist-info` directory. I have to file a request there.

Thank you for your elaborate feedback!

https://aur.archlinux.org/packages/python-nbgrader

eschwartz · 2020-12-07 22:36:25

leuko wrote:

`community/jupyter` lacks the `jupyter.dist-info` directory. I have to file a request there.

It's not lacking it, there isn't supposed to be one in a pacman meta-package, I don't see reason to file a request.

Last edited by eschwartz (2020-12-07 22:38:41)

loqs · 2020-12-07 22:36:28

You were correct in removing jupyter from the install_requires in setup.py, no change is needed to the jupyter package.
Edit:
https://bugs.archlinux.org/task/68884

Last edited by loqs (2020-12-07 23:47:25)

leuko · 2020-12-08 18:48:10

eschwartz wrote:

...
Admittedly https://pypi.org/project/jupyter/ does exist, but it does nothing -- it is a metapackage too. No one would ever bother installing it. Don't depend on it either.

I am confused. According to https://github.com/jupyter/nbgrader/blo … tup.py#L92 other Python packages can depend on Jupyter, or is this usage wrong?

eschwartz · 2020-12-08 18:55:27

nbgrader should likely not do this. Especially since half of the metapackage dependencies are hardcoded into nbgrader's dependencies again.

loqs · 2020-12-08 19:12:13

Slightly off topic

makedepends=("python-setuptools" "git")

python-setuptools should be in depends instead of or as well as in makedepends as it is being used at run time for dependency checking e.t.c..

Lone_Wolf · 2020-12-08 20:20:53

nbgrader already lists jupyter_core and jupyter_client as required,

Archlinux packages python-jupyter_client and python-jupyter_core both include folders under usr/lib/python3.9/site-packages/ that should satisfy python-setuptools / PEP 376.

Last edited by Lone_Wolf (2020-12-08 20:21:17)

flying sheep · 2020-12-08 22:38:17

It’s irrelevant if we believe that nbgrader should do this or not. Nothing prevents it or other python distributions from depending on “jupyter”.

“jupyter” is a regular “distribution” that comes with metadata (check the wheel file), Python has no concept of “meta packages”.

Therefore a package can choose to depend on it (even if that’s probably not the best idea) and community/jupyter should include /usr/lib/python3.x/site-packages/{jupyter.py,jupyter-1.0.0.dist-info/*}

Last edited by flying sheep (2020-12-08 22:39:35)

loqs · 2020-12-08 22:43:29

The PKGBUILD depends on jupyter so that guarantees jupyter being available, the checking provide by python setuptools is then not required and removing jupyter from the install_requires in setup.py during prepare() resolves the issue.

Last edited by loqs (2020-12-08 22:44:04)

Arch Linux

#1 2020-12-05 17:24:27

jupyter metapackage in setup.py causes `DistributionNotFound`

#2 2020-12-06 16:26:06

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#3 2020-12-06 22:10:55

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#4 2020-12-06 23:36:19

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#5 2020-12-06 23:56:30

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#6 2020-12-07 22:33:23

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#7 2020-12-07 22:36:25

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#8 2020-12-07 22:36:28

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#9 2020-12-08 18:48:10

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#10 2020-12-08 18:55:27

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#11 2020-12-08 19:12:13

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#12 2020-12-08 20:20:53

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#13 2020-12-08 22:38:17

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

#14 2020-12-08 22:43:29

Re: jupyter metapackage in setup.py causes `DistributionNotFound`

Board footer