You are not logged in.

#1 2022-09-28 17:42:40

Möbius14
Member
From: /pol/
Registered: 2019-07-24
Posts: 5

Script solving the "... exists in filesystem" errors in Pacman

A handy script to solve all '<some-file> exists in filesystem' errors one sometimes get when updating with pacman -Syu.
The affected packages are updated one by one by calling pacman -S --overwrite <package>.
At the end it runs the regular pacman -Syu command to update all the remaining packages.
Warning: It uses the --noconfirm flag for every pacman call, so use with care or run as an interactive script.

#!/bin/bash
packages=$(unbuffer sudo pacman -Syu --noconfirm | tee /dev/tty | sed '/^.* exists in filesystem$/!d' | sed 's/\([0-9A-Z \[?;]\+m\?\)\?\([a-zA-Z0-9\-\.\_]\+\): .*$/\2/' | uniq)
if [ ! -z "$packages" ]; then
    for line in $packages; do
        echo -e "\n"
        echo -n "Overwrite $line? [Y/n]" && REPLY=$(sed 1q)
        if [[ $REPLY == [y] ||  $REPLY == [Y] || $REPLY == [yY][eE][sS] || -z $REPLY ]]; then
            echo -n "   Update and overwrite $line..."
            sudo pacman -S "$line" --noconfirm --overwrite "*" &> /dev/null
            echo "  Done!"
        else
            echo -e "\nAborted by user"
            exit 0
        fi
    done
    sudo pacman -Syu --noconfirm
fi

Last edited by Möbius14 (2022-09-28 18:39:18)


>imagine not using arch

Offline

#2 2022-09-28 17:57:53

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,449
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

PSA to anyone who finds this thread, there are a lot of problems with this script and it should not be used.

First, your script does a bunch of useless stuff - not least of which is the whole idea of the loop is completely pointless if you are going to blindly overwrite everything regardless ... why loop packages one by one?  Second, admittedly a pet peeve of mine, you misuse sed multiple times in a pipeline.  More importantly, you ignore / hide all relevant output from pacman (edit: okay, you use tee, so the output is visible, but you still cannot stop / respond to any warnings).  And finally, all this wasted effort is to solve a problem that should not exist in the first place in a properly maintained system ... and worse yet, it obscures the source of the problem likely creating more issues - if nothing else this maintains the existence of the problem if there are packages that (should) conflict with each other.  This is functionally no different from running `pacman -Syu --noconfirm --overwrite "*"` right off the bat (which is also a horrible idea).

Fix the original problem, don't brush it under the rug to fester and grow worse over time.

Last edited by Trilby (2022-09-28 18:15:20)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2022-09-28 18:44:20

seth
Member
Registered: 2012-09-03
Posts: 50,009

Re: Script solving the "... exists in filesystem" errors in Pacman

Everthing Tribly said, plus some radom finds:

sed '/^.* exists in filesystem$/!d'

Fails on localization.

packages=$(unbuffer …

https://archlinux.org/packages/extra/x86_64/expect/

echo -n "   Update and overwrite $line..." | dotacat

https://aur.archlinux.org/packages/dotacat - "Like lolcat, but faster. Written in Rust"

<screamingintoacushion>

echo -e "\nAborted by user"
exit 0

At this point you're setup for partial updates.

echo -n "Overwrite $line? [Y/n]" && REPLY=$(sed 1q)]

"man read" / read --help (on bash)
Also: you can either overwrite all file for a package or none?
Since this runs at the tail of a failed pacman run, how does the user know which files he's gonna override by answering yes here?

echo -e "\n"

Simply "echo" will do…

Online

#4 2022-09-28 18:44:20

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

I suppose these kind of problems are caused when people install something with pip or npm *globally* using our pesky pest *sudo*. Then when they try to install some package with pacman that may pull in some python or nodejs stuff on their own, they may find that the files already exist in the file system that according to pacman shouldn't be there and are essentially not *tracked*. When that happens, I can assure you that you are in a big mess...

So just don't do that. I've never needed to use --overwrite flag because I keep all my crap in $HOME. Something like this:

export PATH=${PATH}:${HOME}/bin
export PATH=${PATH}:${HOME}/.local/bin
export PATH=${PATH}:${HOME}/go/bin
export PATH=${PATH}:${HOME}/perl5/bin
export PATH=${PATH}:${HOME}/.npm-packages/bin
export PATH=${PATH}:${HOME}/.gem/ruby/2.7.0/bin/

Whether it is Python or NPM or whatever new fancy stuff, just look up documentation on how you can install your packages locally in your home directory

If you terribly need something *globally* on your system, then learn how to package it with makepkg and install it with pacman

Resources:
Install npm packages globally without sudo on macOS and Linux
Python/Virtual environment
Install and use pip in a local directory without root/sudo access.
Arch package guidelines

Last edited by ugjka (2022-09-28 19:32:36)


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Online

#5 2022-09-28 19:13:19

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Script solving the "... exists in filesystem" errors in Pacman

This script is a terrible idea...

Möbius14 wrote:

A handy script to solve all '<some-file> exists in filesystem' errors one sometimes get when updating with pacman -Syu.

These errors never happen on a properly maintained Arch system. If you get these errors then it means that you've being doing something wrong and should fix the underlying problem rather than just steamrolling a fix without understanding why you broke your system in the first place.


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#6 2022-09-28 19:16:43

Möbius14
Member
From: /pol/
Registered: 2019-07-24
Posts: 5

Re: Script solving the "... exists in filesystem" errors in Pacman

Trilby wrote:

why loop packages one by one?

In order to show which packages are affected and give you a chance to abort if you come over a package that you want to fix by other means.

Trilby wrote:

but you still cannot stop / respond to any warnings

the idea is to use this when when pacman aborts by this error above, so responding to any other warnings is moot.
Also I clearly warned in the OP that it uses --noconfirm and therefore should be used with care.

Trilby wrote:

This is functionally no different from running `pacman -Syu --noconfirm --overwrite "*"`

False, that way it just overwrites anything without prompting each time.

Trilby wrote:

a problem that should not exist in the first place in a properly maintained system

Absolutely correct. Personally, I have one machine where I build a lot of experimental packages from various sources (aur, pip, git, etc.), which makes this error not so unlikely.
For example when building the aur package fails and then try pip to get it running, and when that fails, build from source from github. Usually it's the latter that then causes this errors.
I'm considering adding more features, like showing the files in question. Although I doubt that would give any more insight in what caused the problem in the first place.
What would you suggest? I mean "this problem shouldn't happen in the first place" isn't helpful when it happens anyway.



Whether it is Python or NPM or whatever new fancy stuff, just look up documentation on how you can install your packages locally in your home directory

Installing them locally wouldn't be a problem. Also, I'm aware that pacman should be used only for python packages that are system required, and the rest locally. However, that simply isn't an option and would cause more problems than occasionally sweeping with --overwrite. Ideally, I'd have all python packages installed with pip, as it offers a much larger selection.

Last edited by Möbius14 (2022-09-28 19:33:12)


>imagine not using arch

Offline

#7 2022-09-28 19:26:43

seth
Member
Registered: 2012-09-03
Posts: 50,009

Re: Script solving the "... exists in filesystem" errors in Pacman

What would you suggest? I mean "this problem shouldn't happen in the first place" isn't helpful when it happens anyway.

Figure why it happens specifically and then terminate that cause.

In your case

For example when building the aur package fails and then try pip to get it running, and when that fails, build from source from github.

Fix the AUR or use a PKGBUILD for the git and most certainly see https://wiki.archlinux.org/title/Python … management

When installing packages using pip, it is recommended to use a virtual environment to prevent conflicts with system packages in /usr. Alternatively, pip install --user can be used to install packages into the user scheme instead of /usr. pipx and Conda integrate environment management into their workflows.

Online

#8 2022-09-28 19:26:49

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,449
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

Möbius14 wrote:

What would you suggest? I mean "this problem shouldn't happen in the first place" isn't helpful when it happens anyway.

It only "happens anyway" because you've already done something wrong.  In 12 years of using arch linux (almost) exclusively, I don't recall ever having to use the overwrite flag (to be fair, it didn't exist for a fair portion of that time, but neither do I recall using it's predecessor --force outside of the manual dev-prescribed interventions that required it).  Ugjka notes also not having used this flag, and I suspect the other responders to this thread also don't ever have a need for it.

So what would I suggest: fix the underlying problem which appears to be running `make install` (or equivalent) as root.  You can certainly build source from github and even install it.  Just don't install it as root, or at very least do not install it with PREFIX=/usr (use PREFIX=/usr/local if absolutely necessary, but I'd just stick with a home directory location myself like PREFIX=~/.local).


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2022-09-28 19:38:40

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

Möbius14 wrote:

Whether it is Python or NPM or whatever new fancy stuff, just look up documentation on how you can install your packages locally in your home directory

Installing them locally wouldn't be a problem. Also, I'm aware that pacman should be used only for python packages that are system required, and the rest locally. However, that simply isn't an option and would cause more problems than occasionally sweeping with --overwrite. Ideally, I'd have all python packages installed with pip, as it offers a much larger selection.

Well, you are essentially advocating for a perpetually broken system because you are too lazy to do the right thing


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Online

#10 2022-09-28 19:42:42

Möbius14
Member
From: /pol/
Registered: 2019-07-24
Posts: 5

Re: Script solving the "... exists in filesystem" errors in Pacman

Trilby wrote:

(use PREFIX=/usr/local if absolutely necessary, but I'd just stick with a home directory location myself like PREFIX=~/.local).

Thanks. Does this add any available commands to bash or do I need to create an alias into the local folder?

@seth
see my reply to ugjka (last post)


>imagine not using arch

Offline

#11 2022-09-28 19:56:36

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

I remembered now that I had this "exists in filesystem" *error* once when installing something from the AUR. The problem was that a conflicting package was not listed in the PKGBUILD's conflicts array! And the SOLUTION was DEFINITELY NOT to use the --overwrite flag but to remove to conflicting package from my system and then proceed with the installation.

If I had chosen to --overwrite, what would happen is that, I would have 2 packages on my system that would claim the ownership of some of the same files and when each one of those packages would get updates they would overwrite the other package's files. A big mess that's about to blow up in your face one day, if you ask me...

Last edited by ugjka (2022-09-28 20:20:50)


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Online

#12 2022-09-28 20:02:32

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Script solving the "... exists in filesystem" errors in Pacman

Möbius14 wrote:

Does this add any available commands to bash or do I need to create an alias into the local folder?

echo $PATH

No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#13 2022-09-28 20:05:54

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,449
Website

Re: Script solving the "... exists in filesystem" errors in Pacman

Möbius14 wrote:

Does this add any available commands to bash or do I need to create an alias into the local folder?

I'm not really sure what you're asking here.  If you build software with a specified PREFIX, the software is installed under that PREFIX.  If you plan on doing so, you may want to add the relevant $PREFIX/bin to your $PATH.  For example, I have the following in my shell profile (not shell rc):

export PATH=$PATH:$HOME/.local/bin

In my case, ~/.local/bin/ is pretty much just my own scripts.  For your use case, you may prefer that the local installs take precedence over system binaries (e.g., if you want to rebuild from source some tool that is already installed), in which case you might use this instead:

export PATH=$HOME/.local/bin:$PATH

And all of this is assuming you use $HOME/.local as the PREFIX for any of this locally built software (which is done with the PREFIX variable for make, or --prefix= for an autotools configure script).

Last edited by Trilby (2022-09-28 20:07:55)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#14 2022-09-29 01:57:22

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Script solving the "... exists in filesystem" errors in Pacman

Möbius14 wrote:
Trilby wrote:

why loop packages one by one?

Trilby wrote:

a problem that should not exist in the first place in a properly maintained system

Absolutely correct. Personally, I have one machine where I build a lot of experimental packages from various sources (aur, pip, git, etc.), which makes this error not so unlikely.
For example when building the aur package fails and then try pip to get it running, and when that fails, build from source from github. Usually it's the latter that then causes this errors.
I'm considering adding more features, like showing the files in question. Although I doubt that would give any more insight in what caused the problem in the first place.
What would you suggest? I mean "this problem shouldn't happen in the first place" isn't helpful when it happens anyway.



Whether it is Python or NPM or whatever new fancy stuff, just look up documentation on how you can install your packages locally in your home directory

Installing them locally wouldn't be a problem. Also, I'm aware that pacman should be used only for python packages that are system required, and the rest locally. However, that simply isn't an option and would cause more problems than occasionally sweeping with --overwrite. Ideally, I'd have all python packages installed with pip, as it offers a much larger selection.

That's why you use virtualenvs or similar (e.g. conda). You don't even need an AUR package then. Those are specific tools to solve the problem you are describing. Randomly installing things system-wide is NOT a solution.

Treat your system (the root-restricted parts of it especially) as sacrosanct. This is good security, but also good for maintainability even if you totally ignore the security benefits. "This simply isn't an option" is demonstrably false in just about every situation, and for those where it doesn't seem false, you can create a thread to ask how those should be handled. Even booting up a docker image or VM for those situations is better than `sudo make install` or `sudo pip install`.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

Board footer

Powered by FluxBB