You are not logged in.

#1 2020-01-30 23:55:16

__cmatt
Member
Registered: 2019-12-15
Posts: 3

Sharing system site packages across different python versions?

Hi everyone,

I've got a simple question: can you share python packages installed from the official repositories (e.g. python-tensorflow) to alternative python versions installed in /usr/local/bin/python3.x?

I can perfectly understand how a python package optimised for Arch is only available only for the python that the system uses. I usually go for the lazy way and install everything via pip on virtual environments for things where optimisation doesn't really make any difference. Whereas for tensorflow stuff I've been using the system python (which is fine I guess if you're still using environments). Now I was just curious if you could bind these packages to alternative installs for cases when a certain python version is wanted...

Offline

#2 2020-01-31 08:17:04

schard
Member
From: Hannover
Registered: 2016-05-06
Posts: 811
Website

Re: Sharing system site packages across different python versions?

Python packages will generally not support arbitrary python versions.
E.g. I recently upgraded a python software project that I maintain to use the new walrus operator introduced in python 3.8.
So it will not be compatible with any lower python versions.
Furthermore the units' locations are defined within the package's file tree. So you at least would need to repackage those packages to install the respective units into the proper python library path.

Offline

#3 2020-01-31 10:06:32

__cmatt
Member
Registered: 2019-12-15
Posts: 3

Re: Sharing system site packages across different python versions?

Oh nice! I wasn't aware of this new walrus operator.
Yes, that makes perfect sense. So I imagine that with some effort you would be able to manually install these packages; I'm just seeing that this is detailed in the site.py file in under the python directory, so there's my answer.
Thanks for the quick reply! smile

EDIT: Do you see any noticeable performance improvements compared to packages installed with pip?

Last edited by __cmatt (2020-01-31 10:23:08)

Offline

#4 2020-01-31 10:33:55

schard
Member
From: Hannover
Registered: 2016-05-06
Posts: 811
Website

Re: Sharing system site packages across different python versions?

I rarely install packages using pip on my systems, since I need many of them system-wide.
I have no benmchmark comparing the performance of packages installed via pip and pacman at hand.

Offline

#5 2020-01-31 13:14:39

__cmatt
Member
Registered: 2019-12-15
Posts: 3

Re: Sharing system site packages across different python versions?

I understand.
Cheers for your help!

Offline

#6 2020-02-04 08:27:43

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Sharing system site packages across different python versions?

pkgfile -l python-tensorflow| grep '\.so$'

None of these compiled C extensions will work on any version of python other than some version of 3.8, which they were initially compiled for. You would need to recompile them anyway.

schard wrote:

Python packages will generally not support arbitrary python versions.
E.g. I recently upgraded a python software project that I maintain to use the new walrus operator introduced in python 3.8.
So it will not be compatible with any lower python versions.

This itself is not usually a problem, as most widely distributed modules try to stay compatible with most or all non-EOL versions of python.

Furthermore the units' locations are defined within the package's file tree. So you at least would need to repackage those packages to install the respective units into the proper python library path.

export PYTHONPATH=/usr/lib/python3.8/site-packages
/usr/local/bin/python3.7

This would break specifically in the case of compiled C extensions, and anything that requires python 3.8 due to the use of walruses. But it would tend to work for everything else, albeit without .pyc loading. (If you ran python as root, you would end up with __pycache__/*.cpython-37.pyc files untracked by pacman, for use with python 3.7.)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB