What's the story on pip's externally-managed-environment error?

lfitzgerald · 2023-06-27 01:25:34

I guess Arch adopted https://peps.python.org/pep-0668/ because trying to pip install things now gives:

$ pip install .
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.

    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

However, pip already does not install packages to the system location. On Arch, installing to /usr/lib/python3.11 requires sudo. Without sudo, the packages are installed to the user directory where none of this alleged conflict with pacman is applicable.

$ pip install --break-system-packages .
Defaulting to user installation because normal site-packages is not writeable

So what's the deal here? Is there a way to get pip to recognize that I am in fact not trying to install system-wide (without passing --user everytime)? Where was this change implemented anyways?

V1del · 2023-06-27 07:18:03

In the python package: https://gitlab.archlinux.org/archlinux/ … de37301c5d

As for PIP not recognizing it's ran as a user and would manipulate the user instance potentially a bug in pip, but if --user would be your modus operandi anyway, alias that so it's implicitly added?

FWIW depending on what exactly, changes that your user level but non-virtualized packages can have system level consequences (because you e.g. get used to using a certain tool that either looks up the local lib instead of the global one or is started from the local location instead of the global one) still make the unvirtualized usage of PIP not a thing to recommend.

Last edited by V1del (2023-06-27 07:20:53)

progandy · 2023-06-27 08:04:01

The reason the user location is blocked by this is mentioned in the PEP:

This may pose a critical problem for the integrity of distros, which often have package-management tools that are themselves written in Python. For example, it’s possible to unintentionally break Fedora’s dnf command with a pip install command, making it hard to recover.
This applies both to system-wide installs (sudo pip install) as well as user home directory installs (pip install --user), since packages in either location show up on the sys.path of /usr/bin/python3.

lfitzgerald · 2023-06-28 17:51:55

V1del wrote:

because you e.g. get used to using a certain tool that either looks up the local lib instead of the global one or is started from the local location instead of the global one

"being used to a tool that then stops working" sounds a lot like what's happening here with pip, tbh.

Your suggestion about `--user` doesn't work, because you still get the slap in the face from pip. You need to alias `pip --user --break-system-packages` (just for other people reading the thread). But what a terrible thing to force users to do. They could at least have made it a config option so people don't have to monkey around with shell aliases.

Seems a bit like this is a bad decision from Python. But why did Arch decide to go along with it? Just because it's a PEP?

progandy wrote:

The reason the user location is blocked by this is mentioned in the PEP:
This may pose a critical problem for the integrity of distros, which often have package-management tools that are themselves written in Python. For example, it’s possible to unintentionally break Fedora’s dnf command with a pip install command, making it hard to recover.
This applies both to system-wide installs (sudo pip install) as well as user home directory installs (pip install --user), since packages in either location show up on the sys.path of /usr/bin/python3.

But why does Arch care about Fedora bugs? Pacman is not written in Python unlike Fedora.

Lone_Wolf · 2023-06-29 09:25:58

https://wiki.archlinux.org/title/Arch_Linux#Simplicity wrote:

It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes:

Following official Python Enhancememt Proprosals aka PEPs matches that very well.

progandy · 2023-06-29 09:48:36

Lone_Wolf wrote:

Following official Python Enhancememt Proprosals aka PEPs matches that very well.

Arch did not even implement the PEP, that has been done by upstream python, only the externally-managed flag was enabled, restoring --user would require an additional patch and changing upstream behavior.

If you really want to permanently disable the block on your own system, then you could disable the externally-managed configuration by telling pacman to skip installing usr/lib/python3.11/EXTERNALLY-MANAGED
Then you are responsible to make sure no --user package breaks any globally installed package and never call pip as root.

Last edited by progandy (2023-06-29 09:51:34)

lfitzgerald · 2023-06-30 23:03:08

Perfect! After running

sudo rm /usr/lib/python3.11/EXTERNALLY-MANAGED

everything works normally again.

How do I prevent pacman from installing it again?

Lone_Wolf · 2023-07-01 11:02:21

pacman.conf has a NoExtract option that can prevent that, see its manpage .

Using NoExtract has a huge potential of causing issues, make sure to document carefully what you use it for and why.
(mentioning it when asking for help with python related issues is also a good idea)

Last edited by Lone_Wolf (2023-07-01 11:02:53)

ugjka · 2023-07-01 13:13:08

i adopted pipx

grat · 2023-08-22 17:05:51

So, having read through this thread, two questions spring to mind:

1) Why don't the OS's like fedora, red hat, etc., use a virtual environment to protect their system packages?

2) What's the official way of managing all of those packages you've already installed via pip, and can no longer update/remove because of PEP-668?

Trilby · 2023-08-22 18:17:35

I think this is a great idea from python - and about f***ing time. As noted file conflicts with pacman packages are just one concern that this addresses. Using a --user flag or installing to a user's home directory avoids file conflicts, but not API conflicts with the system python. And this kind of API conflict has been the source of countless problems for users coming to this forum. In the end we have to help them clean up the mess they made and tell them to stop foolishly using pip to install python packages.

It's about time the tool itself took some responsibility and told users not to do something stupid.

And if it's not stupid for your use case, or if it is and you're okay with that, there is a flag you can add to get that to happen. But making it less likely for simple mistaken usage to cause significant system-wide problems is great. I might take back some of the nasty things I say about python users if / when this PEP gets broad implementation.

grat wrote:

2) What's the official way of managing all of those packages you've already installed via pip, and can no longer update/remove because of PEP-668?

You want the offical way of managing something you installed in a non-official way? This change in pip's behavior does not stop you from managing those packages just the way you would before - you just need to acknowledge that either A) you know you're doing something dumb, or 2) you know your use is an exception to the rule that this would be generally dumb to do. If neither of those apply, you probably shouldn't have those python packages installed.

Last edited by Trilby (2023-08-30 12:58:40)

lfitzgerald · 2023-08-22 19:49:30

grat wrote:

1) Why don't the OS's like fedora, red hat, etc., use a virtual environment to protect their system packages?

Those distros lag behind upstream more than arch, so maybe it's on the way.

From what I can tell, the arch perspective on this is: "idiots keep asking stupid python questions on the forum, so now we can't have nice things". Other distros don't have this community management problem with their forums, it seems to be an arch thing that forum regulars get an aneurysm from seeing a noob thread, and solve it by mutilating their distro instead of just ignoring threads they don't like.

Thank God they left an escape hatch in. Gives me a bit more time as I migrate to saner distro.

grat wrote:

2) What's the official way of managing all of those packages you've already installed via pip, and can no longer update/remove because of PEP-668?

That sounds like a one time thing, so just use the flag the error says that one time. And I guess install them the proper way from now on.

Or just delete the file and ignore this stupidity like I do.

Last edited by lfitzgerald (2023-08-22 19:50:49)

iTrooz · 2023-08-28 21:22:21

grat wrote:

Those distros lag behind upstream more than arch, so maybe it's on the way.

Virtual environments have been a thing for a while, IIRC

Does anyone have a more likely answer for 1) ? (Including for ArchLinux, as there wasn't really an answer for it)

Lone_Wolf · 2023-08-30 08:39:52

1) Why don't the OS's a virtual environment to protect their system packages?

a virtual environment can isolate applications from the OS, but who's in control of what is downloaded / run in that environment ?

iTrooz · 2023-08-30 10:06:56

I'm not sure what you mean
In this case, the virtual environment would isolate the system's applications (and libraries) from the user space
In would be controlled by pacman directly, same as now.

Lone_Wolf · 2023-08-30 10:25:18

I may have interpreted the question differently then you .

For clarity :
Is the idea to
- run system pacakges in a virtual environment to protect them from user changes ?

- run python user apps always in a virtual environment

- none of the above , please explain

?

iTrooz · 2023-08-30 10:27:34

Lone_Wolf wrote:

- run system packages in a virtual environment to protect them from user changes ?

this

Last edited by iTrooz (2023-08-30 10:27:48)

Trilby · 2023-08-30 13:02:16

Then they wouldn't be system packages and couldn't use other system packages as dependencies but would have to bundle everything they needed resulting in many versions of common python dependencies being installed in different places. Basically that'd be reinventing scrap and fudgepaks ... Freudian slip, I mean snap and flatpaks.

iTrooz · 2023-08-31 15:30:43

Then they wouldn't be system packages

Why ?

AFAIK, a virtual environment has the same structure as a "regular" python installation, so why couldn't pacman install files in a system virtual environment (e.g. `/usr/lib/system-python` ?), the same way it is currently installing files in `/usr/lib/python3.11` ?

Trilby · 2023-08-31 15:40:52

With all packages installed to the same virtual environment? Perhaps that could work, but why? You'd then not be able to run / use any of those installed packages until you explicitly enter / activate that environment. And if you had that environment activated by default at boot up / login, then you're just back to square one anyways (but with layers of complexity for no purpose).

AsmundEr · 2023-09-15 06:51:25

So I get that users actually being able to install arbitrary software on their machines will at some times cause problems.
But solving this by restricting the ability of users to install arbitrary software on their machines is antithetical to the very spirit and essence of Linux!

In any case, I cannot see that PEP668 is a solution for anything but a very narrow subset of workflows.
For instance if I want to install this package to use it as a command-line tool: https://pypi.org/project/wg-meshconf/
How am I supposed to achieve this while following PEP668, in a way that does not cause exactly the same problems as if I am using "pip install --user wg-meshconf" ?

Furthermore, I have been "sudo pip install"-ing on more than a dozen Arch boxes for over a decade, and I can count on one hand the number of issues this has caused me.
It has never resulted in a broken system, at worst I've had to do a "pip uninstall" to address file conflict found by pacman during a system upgrade.

Finally, a better fix than removing the "EXTERNALLY_MANAGED" file is to use the following config option:

# in ~/.config/pip/pip.conf
[global]
break-system-packages = true

Last edited by AsmundEr (2023-09-15 06:56:12)

Lone_Wolf · 2023-09-15 08:20:14

AsmundEr wrote:

In any case, I cannot see that PEP668 is a solution for anything but a very narrow subset of workflows.

Judging on threads related to pip , anaconda and other 'python made easy' installers/managers threads on this forum the percentage of people seeking python help here that use those workflows is very high.

It is possible to combine several package managers (that includes manual installs) on one system without severe conflicts IF the overseeing manager knows what they are doing.
Several of the posters in this thread could pull that off (some may even be already doing that) but the majority of people get in trouble trying it.

You appear to be one of those that can and are doing that.

Last edited by Lone_Wolf (2023-09-15 08:21:06)

Trilby · 2023-09-15 12:38:09

AsmundEr wrote:

How am I supposed to achieve this...
Finally, a better fix than removing the "EXTERNALLY_MANAGED" file is to use the following config option...

So you are asking about how to acheive something that you feel has been "restricted", yet in the same thread you reference two different ways of achieving it while there was yet another way to achieve it already very early in this thread?

Nothing is "restricted" - there's just a cautionary note now given by a tool with the very real capacity to do damage if users don't know what they are doing.

bulletmark · 2023-09-16 01:13:12

AsmundEr wrote:

How am I supposed to achieve this while following PEP668, in a way that does not cause exactly the same problems as if I am using "pip install --user wg-meshconf" ?

You type:

pipx install wg-meshconf

I'll admit, I used to use pip to occasionally install some obscure python apps which were not packaged in the AUR (and like you I believe I fully understand what I am doing) but since this externally-managed environment lockout I bothered to start using pipx and realised I should have been using it all along.

gururise · 2023-09-16 17:04:53

How would one use pip with CONDA? I am no longer able to pip install -r requirements.txt from within a CONDA environment without running into this error.

Arch Linux

#1 2023-06-27 01:25:34

What's the story on pip's externally-managed-environment error?

#2 2023-06-27 07:18:03

Re: What's the story on pip's externally-managed-environment error?

#3 2023-06-27 08:04:01

Re: What's the story on pip's externally-managed-environment error?

#4 2023-06-28 17:51:55

Re: What's the story on pip's externally-managed-environment error?

#5 2023-06-29 09:25:58

Re: What's the story on pip's externally-managed-environment error?

#6 2023-06-29 09:48:36

Re: What's the story on pip's externally-managed-environment error?

#7 2023-06-30 23:03:08

Re: What's the story on pip's externally-managed-environment error?

#8 2023-07-01 11:02:21

Re: What's the story on pip's externally-managed-environment error?

#9 2023-07-01 13:13:08

Re: What's the story on pip's externally-managed-environment error?

#10 2023-08-22 17:05:51

Re: What's the story on pip's externally-managed-environment error?

#11 2023-08-22 18:17:35

Re: What's the story on pip's externally-managed-environment error?

#12 2023-08-22 19:49:30

Re: What's the story on pip's externally-managed-environment error?

#13 2023-08-28 21:22:21

Re: What's the story on pip's externally-managed-environment error?

#14 2023-08-30 08:39:52

Re: What's the story on pip's externally-managed-environment error?

#15 2023-08-30 10:06:56

Re: What's the story on pip's externally-managed-environment error?

#16 2023-08-30 10:25:18

Re: What's the story on pip's externally-managed-environment error?

#17 2023-08-30 10:27:34

Re: What's the story on pip's externally-managed-environment error?

#18 2023-08-30 13:02:16

Re: What's the story on pip's externally-managed-environment error?

#19 2023-08-31 15:30:43

Re: What's the story on pip's externally-managed-environment error?

#20 2023-08-31 15:40:52

Re: What's the story on pip's externally-managed-environment error?

#21 2023-09-15 06:51:25

Re: What's the story on pip's externally-managed-environment error?

#22 2023-09-15 08:20:14

Re: What's the story on pip's externally-managed-environment error?

#23 2023-09-15 12:38:09

Re: What's the story on pip's externally-managed-environment error?

#24 2023-09-16 01:13:12

Re: What's the story on pip's externally-managed-environment error?

#25 2023-09-16 17:04:53

Re: What's the story on pip's externally-managed-environment error?

Board footer