What's the story on pip's externally-managed-environment error?

sweet_tea · 2023-09-16 21:13:21

I personally really dislike this change. Pipx is a great tool, but not every python package is a complete CLI tool, you cant really install libraries with pipx... and not every Pypi package is in the repo or the AUR. Resolving that is a nightmare when pip and arch mix packages when your needed package only exists on Pypi (from both user and system). I installed python-pillow via pacman and Python cant even resolve where it is.

So this leaves creating Venv's in random directories and having to track them to call them from outside of that environment. I like to use Python as a secondary scripting language to something like Bash. I generally write them, put them in the same directory somewhere, throw a shebang at the top and run them as if they were any regular system tool. So now I have to write a custom wrapper script just to achieve this behavior while avoiding conflicts with packages.

I'm glad more people are just using Pipx and it's nice that this adds a lot of resilience for a user who's not using python that much but this has become somewhat of a nightmare for me. I'm trying to find a better solution then just using a crap load of virtual environments all over the place. So far I've just been using Go-lang since it auto-magically handles these dependencies and the binaries run anywhere.

bulletmark · 2023-09-16 23:22:54

sweet_tea wrote:

So this leaves creating Venv's in random directories and having to track them to call them from outside of that environment. I like to use Python as a secondary scripting language to something like Bash. I generally write them, put them in the same directory somewhere, throw a shebang at the top and run them as if they were any regular system tool. So now I have to write a custom wrapper script just to achieve this behavior while avoiding conflicts with packages.

I create very many such utility "scripts" as well and don't get what you are doing that is unique. If a utility requires a package then 99% of them are packaged in the Arch or the AUR so just pacman install that system package and put your utility in your $PATH, e.g. `~/.local/bin/`. If it is a more sophisticated utility that requires some obscure packages not in Arch or the AUR, then whip up a tiny pyproject.toml (or setup.py if you are old school) in your app source dir which defines those dependencies etc, and then `pipx install .` I often do `pipx install -e .` instead to be able to conveniently edit the installed code if I am still developing it. Again, I used to do those two commands with `pip install --user` but now (since Arch imposed the externally managed environment lockout) I simply use `pipx install`.

bulletmark · 2023-09-17 00:03:23

sweet_tea wrote:

I installed python-pillow via pacman and Python cant even resolve where it is.

BTW, not sure what error you mean here but you can give your venvs access to 3rd party pacman (i.e. system) installed packages by adding `--system-site-packages` to your pipx install line, just like you can on `python -m venv`.

Last edited by bulletmark (2023-09-17 00:04:14)

samm0 · 2023-10-02 21:22:16

gururise wrote:

How would one use pip with CONDA? I am no longer able to pip install -r requirements.txt from within a CONDA environment without running into this error.

That's because your conda env doesn't have pip, so it's using the system-wide one. just do `conda install pip` and you should be good to go.

(Just ran into this issue and eventually figured it out on my own and wanted to post this somewhere in case others need)

Last edited by samm0 (2023-10-02 21:23:22)

dr1fter · 2023-10-14 10:27:43

First of all, thanks for sharing the option to add `break-system-packages true` to ~/.config/pip, which does seem like the best approach for those who prefer to continue installing (python) system packages using pip.

I was / am somewhat surprised to see such a change being rolled-out to Archlinux, considering Archlinux typically targets users that know what they do, and do not want to be "bossed around" / protected from doing stupid things. Truth be told, I am even somewhat irritated about that.

For my personal setup, I have been doing python development for over five years on my machine, and have thus far always installed dependencies as system package / system-wide (I tend to always run everything against "latest-and-greatest" dependencies / update frequently). I never saw issues (except occasional changes I had to do for my own code from incompatible upgrades of used dependencies; which I consider okay and part of the task of software maintenance).

andrewxdev · 2024-01-07 04:48:17

[b]$ pip install --user black[/b]
error: [b]externally-managed-environment[/b]

× [b]This environment is externally managed[/b]
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
   
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using '[b]python -m venv path/to/venv[/b]'.
    Then use [b]path/to/venv/bin/python[/b] and [b]path/to/venv/bin/pip[/b].
   
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use '[b]pipx install xyz[/b]', which will manage a
    virtual environment for you. Make sure you have [b]python-pipx[/b]
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing [b]--break-system-packages[/b].
hint: See [b]PEP 668[/b] for the detailed specification.

Resolving the "externally-managed-environment" error I'm encountering when attempting to install a Python package using pip on Arch Linux, which suggests using pacman for system-wide installations, creating a virtual environment for non-Arch-packaged Python packages, or using pipx for non-Arch packaged Python applications? This error is related to the implementation of PEP 668 in Arch Linux.

PEP 668, titled "Graceful Cooperation Between External and Python Package Managers", aims to resolve conflicts between Python's packaging tools (like pip) and system package managers (like pacman in Arch Linux). This PEP proposes a mechanism for external package managers to indicate that a Python environment is being externally managed, suggesting that Python packaging tools should not modify it without explicit user consent. The intention is to prevent scenarios where changes made by pip could conflict with or overwrite changes made by the system package manager.

This leads to an "externally-managed-environment" error when using pip for package installation, serving as a caution to maintain system integrity and avoid potential conflicts between pip-installed and pacman-managed packages.

To install Python packages on Arch Linux you have several options:

1. Using Arch Linux's Package Manager (pacman):

Before using pip, check if the package is available in the Arch Linux repositories, as this is the recommended way to install software in Arch Linux. For the Black formatter, you can install it using pacman:

sudo pacman -S python-black

This will install Black as a system-wide package managed by pacman.

2. Using Python's Package Installer (pip) in a Virtual Environment:

If the package is not available in the Arch repositories or you need a specific version, it's advisable to use a virtual environment. This prevents conflicts with system packages and adheres to the best practices recommended by PEP 668.

1. Create a Virtual Environment:

First, create a virtual environment. You can do this in your project directory:

python -m venv venv

This command creates a new virtual environment in the venv directory within your current directory.

2. Activate the Virtual Environment:

Before installing packages with pip, you need to activate the virtual environment:

source venv/bin/activate

On Windows, the command is venv\Scripts\activate.

3. Install Black Using pip:

Now you can safely install Black using pip without affecting the system-wide Python packages:

pip install black

3. Using pipenv for Managing Packages and Environments:

pipenv is a tool that automatically creates and manages a virtual environment for your projects. It is a higher-level tool than pip and offers an easy and consistent way to handle project dependencies.

1. Install pipenv:

Install pipenv using pacman:

sudo pacman -S python-pipenv

2. Initialize a Project with pipenv:

In your project directory, you can initialize a pipenv environment and install Black simultaneously:

pipenv install black

This command will create a Pipfile and a Pipfile.lock to manage dependencies and will install Black in an isolated environment.

3. Activate the pipenv Environment:

To activate the pipenv environment, use:

pipenv shell

This will switch to the virtual environment where Black is installed.

4. Using pipx for Python Applications:

For Python applications and CLI tools like Black, pipx is a great option. It automatically manages each application in its own isolated environment.

1. Install pipx (if not already installed):

sudo pacman -S pipx
   pipx ensurepath

2. Install Black using pipx:

pipx install black

With pipx, you can use Black globally without interfering with system-managed Python packages.

Last edited by andrewxdev (2024-01-09 05:08:03)

seth · 2024-01-07 08:47:12

1. Please don't post PSAs
2. hatless parroting the wiki
3. trying to use markdown, https://bbs.archlinux.org/help.php#bbcode

Last edited by seth (2024-01-07 15:12:47)

Trilby · 2024-01-07 14:28:58

seth wrote:

2. parroting ...

Ahem ...

Arch Linux

#26 2023-09-16 21:13:21

Re: What's the story on pip's externally-managed-environment error?

#27 2023-09-16 23:22:54

Re: What's the story on pip's externally-managed-environment error?

#28 2023-09-17 00:03:23

Re: What's the story on pip's externally-managed-environment error?

#29 2023-10-02 21:22:16

Re: What's the story on pip's externally-managed-environment error?

#30 2023-10-14 10:27:43

Re: What's the story on pip's externally-managed-environment error?

#31 2024-01-07 04:48:17

Re: What's the story on pip's externally-managed-environment error?

#32 2024-01-07 08:47:12

Re: What's the story on pip's externally-managed-environment error?

#33 2024-01-07 14:28:58

Re: What's the story on pip's externally-managed-environment error?

Board footer