Sharing system site packages across different python versions?

__cmatt · 2020-01-30 23:55:16

Hi everyone,

I've got a simple question: can you share python packages installed from the official repositories (e.g. python-tensorflow) to alternative python versions installed in /usr/local/bin/python3.x?

I can perfectly understand how a python package optimised for Arch is only available only for the python that the system uses. I usually go for the lazy way and install everything via pip on virtual environments for things where optimisation doesn't really make any difference. Whereas for tensorflow stuff I've been using the system python (which is fine I guess if you're still using environments). Now I was just curious if you could bind these packages to alternative installs for cases when a certain python version is wanted...

schard · 2020-01-31 08:17:04

Python packages will generally not support arbitrary python versions.
E.g. I recently upgraded a python software project that I maintain to use the new walrus operator introduced in python 3.8.
So it will not be compatible with any lower python versions.
Furthermore the units' locations are defined within the package's file tree. So you at least would need to repackage those packages to install the respective units into the proper python library path.

__cmatt · 2020-01-31 10:06:32

Oh nice! I wasn't aware of this new walrus operator.
Yes, that makes perfect sense. So I imagine that with some effort you would be able to manually install these packages; I'm just seeing that this is detailed in the site.py file in under the python directory, so there's my answer.
Thanks for the quick reply!

EDIT: Do you see any noticeable performance improvements compared to packages installed with pip?

Last edited by __cmatt (2020-01-31 10:23:08)

schard · 2020-01-31 10:33:55

I rarely install packages using pip on my systems, since I need many of them system-wide.
I have no benmchmark comparing the performance of packages installed via pip and pacman at hand.

__cmatt · 2020-01-31 13:14:39

I understand.
Cheers for your help!

eschwartz · 2020-02-04 08:27:43

pkgfile -l python-tensorflow| grep '\.so$'

None of these compiled C extensions will work on any version of python other than some version of 3.8, which they were initially compiled for. You would need to recompile them anyway.

schard wrote:

Python packages will generally not support arbitrary python versions.
E.g. I recently upgraded a python software project that I maintain to use the new walrus operator introduced in python 3.8.
So it will not be compatible with any lower python versions.

This itself is not usually a problem, as most widely distributed modules try to stay compatible with most or all non-EOL versions of python.

Furthermore the units' locations are defined within the package's file tree. So you at least would need to repackage those packages to install the respective units into the proper python library path.

export PYTHONPATH=/usr/lib/python3.8/site-packages
/usr/local/bin/python3.7

This would break specifically in the case of compiled C extensions, and anything that requires python 3.8 due to the use of walruses. But it would tend to work for everything else, albeit without .pyc loading. (If you ran python as root, you would end up with __pycache__/*.cpython-37.pyc files untracked by pacman, for use with python 3.7.)

Arch Linux

#1 2020-01-30 23:55:16

Sharing system site packages across different python versions?

#2 2020-01-31 08:17:04

Re: Sharing system site packages across different python versions?

#3 2020-01-31 10:06:32

Re: Sharing system site packages across different python versions?

#4 2020-01-31 10:33:55

Re: Sharing system site packages across different python versions?

#5 2020-01-31 13:14:39

Re: Sharing system site packages across different python versions?

#6 2020-02-04 08:27:43

Re: Sharing system site packages across different python versions?

Board footer