Arch Linux

elitepleb

Member

Registered: 2020-12-03

Posts: 3

FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

After a pacman -Syu and python upgrading from 3.8.6-1 -> 3.9.0-1, python can't find any of it's modules.

Pacman log https://pastebin.com/zijVMKwL
Python help('modules') outputs https://pastebin.com/6yxrGUCV

pip ends with ModuleNotFoundError: No module named 'pip._vendor.progress'
qutebrowser ends with importlib.metadata.PackageNotFoundError: qutebrowser

I'm lost for what to do or how to fix this and there doesn't seem to be any prior occurrence of this.
I tried turning it on and off again, reinstalling pip and python, but none of that helped.
How the hell could this have happened?

EDIT:
Getting pip from not pacman and reinstalling all the packages at least makes some of them work
1. curl -sS https://bootstrap.pypa.io/get-pip.py | sudo python
2. pip install $(pip freeze | sed 's/==.*//g')
3. pikaur -S $( pacman -Q | cut -d ' ' -f 1)
I had to reinstall pikaur manually from source for it to work.

Last edited by elitepleb (2020-12-03 09:39:57)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,361

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

That's a completely horrifying idea.

Do NOT use pip as root to factually replace system python packages. That's what lead to the issue in the first place and will lead to issues in the future. Remove this entire mess and install the packages as needed from the repos and if you must use pip use it as your user or in a virtualenv.

Or be consequent in the other direction remove the python package (... or replace it with a mock package to satisfy deps) and handle everything via your own pip.

flyingscorpio

Member

Registered: 2020-06-04

Posts: 34

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

I have the same problem, my modules are still in ~/.local/lib/python3.8/site-packages.
Since yesterday's python update in now also have ~/.local/lib/python3.9/site-packages.

Can I just remove ~/.local/lib/python3.8/site-packages ?
installing pip from pypa.io doesn't seem clean to me, I'd rather keep pacman's python-pip...

elitepleb

Member

Registered: 2020-12-03

Posts: 3

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

That's a completely horrifying idea.

Do NOT use pip as root to factually replace system python packages. That's what lead to the issue in the first place and will lead to issues in the future. Remove this entire mess and install the packages as needed from the repos and if you must use pip use it as your user or in a virtualenv.

Or be consequent in the other direction remove the python package (... or replace it with a mock package to satisfy deps) and handle everything via your own pip.

Yes, sudo pip is a horrid idea, good thing i didn't say to do it.
line 2. does a normal user pip reinstall. no sudo there.

I was just lost as to what to do, guess now i can handle it better

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,361

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

The question you have to ask yourself is why that is even populated in the first place. Do you knowingly use pip for projects or is this a result of following internet guides for installing certain packages? If the packages are in the repo/the AUR you should generally prefer those and there shouldn't be a reason for ~/.local/lib/python anything to even exist.

If it does exist like that, you should probably prefer venvs to have a environment that doesn't affect the normal user session to the point that system packages fail to work. It's likely going to take a while for the pip based modules to be available for python 3.9 but I'm admittedly not that much of an expert here as I've never had use for manually installing something with pip as all the packages I've had to work with were available in the repos/AUR so far.

elitepleb

Member

Registered: 2020-12-03

Posts: 3

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

The question you have to ask yourself is why that is even populated in the first place. Do you knowingly use pip for projects or is this a result of following internet guides for installing certain packages? If the packages are in the repo/the AUR you should generally prefer those and there shouldn't be a reason for ~/.local/lib/python anything to even exist.

If it does exist like that, you should probably prefer venvs to have a environment that doesn't affect the normal user session to the point that system packages fail to work. It's likely going to take a while for the pip based modules to be available for python 3.9 but I'm admittedly not that much of an expert here as I've never had use for manually installing something with pip as all the packages I've had to work with were available in the repos/AUR so far.

IDK man, python's module situation is messed up.
If anything, the pacman python packages are the cause of all of this, because they still expect and look for files in the 3.8 folder, when the system is running 3.9.

I'm just dealing with the fallout of updating my system with pacman -Syu and seeing programs that rely on python like qutebrowser fail.
I guess this is just a result of a pure rolling release distro like arch, when you are right on the edge of updating one thing, and not having everything else update in time.
This should solve itself in some time, we just need to wait for the maintainers.

Of course i use virtual envs for python projects, but some python packages like youtube-dl pretend to be normal programs and trouble comes again.
Python always had a messy system history, i guess i was just unfortunate enough to catch it this update cycle.

flyingscorpio

Member

Registered: 2020-06-04

Posts: 34

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

Yes just manually installed the packages in my .local/lib with pacman, the ones I don't recognize are dependencies and just a couple aren't in the repositories.
This may just have been the occasion to clean that up I guess !

sshaikh

Member

Registered: 2019-07-03

Posts: 64

Re: FIXED: Python 3.8.6-1 -> 3.9.0-1 update lost all it's modules

The same thing happened to me but only for AUR libraries - discussion and solution for me here: https://bbs.archlinux.org/viewtopic.php?pid=1941160

tldr:

1. If python is updated, then all dependant libraries have to be updated.
2. Maintainers of official python packages coordinate with this release and will poke their PKGBUILDs to force rebuild official packages, which you will get at the same time as the python upgrade.
3. There is no such mechanism for AUR packages and PKGBUILDs will look the same either side of the update.
4. Therefore you must force the rebuild of all AUR python libraries.

HTH

Last edited by sshaikh (2020-12-03 12:57:57)