[SOLVED] What is the correct way to install AUR packages from a list?

phaedo7 · 2021-01-06 18:38:13

Hello

I aim to install my AUR packages on a different system. I made a list of the packages installed using 'yay' this way:

pacman -Qqem > pkglist_aur.txt

I tried installing these packages from the list on a new machine, but it doesnt install anything. I did it this way:

yay -S --needed - < pkglist_aur.txt

Alternatively, I tried this:

yay -S  --needed $(cat pkglist_aur.txt | xargs)

However the above thing breaks as soon as one package in the list throws error. I want something that just continues to the next package in the list and ignore the error. My first thought to avoid this was:

yay -S --needed $(cat pkglist_aur.txt | xargs) || true

But this didnt work for some reason. Finally some thing that worked for me is this:

for word in $(cat pkglist_aur.txt); do yay -S -needed $word || true; done

I believe there must be a better and smarter way to do this. What am I getting wrong here ?

Last edited by phaedo7 (2021-01-08 18:20:09)

Slithery · 2021-01-06 18:41:13

Why not just copy the already built packages from your cache to the other machine and install them using pacman?

phaedo7 · 2021-01-06 18:48:11

This package list I create here is from my machine from a older state. Now my cache has a lot of other packages that I dont need on the new machine. Also I would like to "yay" packages separate from the "pacman" ones.

Awebb · 2021-01-07 04:28:33

I'm not sure what you think this || true is supposed to do here. The for loop in the end looks fine if you're fine with manually checking the output for packages that didn't build properly.

Trilby · 2021-01-07 04:37:32

There's also a useless use of cat and subshell there:

while read pkg; do yay -S --needed $pkg; done < pkglist_aur.txt

Last edited by Trilby (2021-01-07 04:37:46)

respiranto · 2021-01-07 19:21:40

Trilby wrote:

There's also a useless use of cat and subshell there:
while read pkg; do yay -S --needed $pkg; done < pkglist_aur.txt

In bash, one may also write:

for p in $(< pkglist_aur.txt); do yay -S --needed $p; done

This should not spawn a subshell either, while maintaining the order of objects.

eschwartz · 2021-01-08 02:24:57

respiranto wrote:

In bash, one may also write:
for p in $(< pkglist_aur.txt); do yay -S --needed $p; done
This should not spawn a subshell either, while maintaining the order of objects.

No, one may not (or should not) write this. Why are you taking a working solution and trying to propose "alternative ways of also writing it" in bash? If you're going to do so, you should understand the complex programming language you're discussing.

Specifically, please read this: https://mywiki.wooledge.org/DontReadLinesWithFor

Moreover, this was *already* suggested, with trivial differences, above -- Trilby rightly described it as wrong, and proposed the correct, efficient way to do it.

(Incidentally, I'm fairly positive $(<file) does fork a new subshell, and is faster than cat but slower than read. It retains all the BROKENNESS of for i in $(cmd) regardless.)

Last edited by eschwartz (2021-01-08 02:30:28)

phaedo7 · 2021-01-08 11:18:49

Trilby wrote:

There's also a useless use of cat and subshell there:
while read pkg; do yay -S --needed $pkg; done < pkglist_aur.txt

This still breaks if one of the package from the list was not successfully installed

Trilby · 2021-01-08 14:25:35

No it doesnt "break". It continues on with the next package in the list as requested. It is a horrible idea for many reasons, but it meets the stated goal.

phaedo7 · 2021-01-08 14:50:18

Trilby wrote:

No it doesnt "break". It continues on with the next package in the list as requested. It is a horrible idea for many reasons, but it meets the stated goal.

I tried it bunch of times now, it simply exists with EOF

seth · 2021-01-08 15:00:45

Do you maybe somehow "set -o errexit"? (What would explain the otherwise stupid "|| true")

phaedo7 · 2021-01-08 15:13:02

seth wrote:

Do you maybe somehow "set -o errexit"? (What would explain the otherwise stupid "|| true")

The idea behind "|| true" in the for loop is, for each iteration, either the command succeeds or exits with true so that it can move to the next iteration. And no, I havent set "errexit" anywhere

seth · 2021-01-08 15:18:11

That's not a thing. Except if you did set errexit.

while read pkg; do
   yay -S --needed $pkg
done < pkglist_aur.txt

will read every line of pkglist_aur.txt into the $pkg variable and run a new yay instance with it. If yay fails, the loop will just continue w/ the next line - except if errexit.

Post the actual file, your complete IO w/ the failure and the output of "set -o"

Edit: or if eschwartz knows another shell option that impacts this ;-)

Last edited by seth (2021-01-08 15:19:27)

phaedo7 · 2021-01-08 15:38:02

seth wrote:

That's not a thing. Except if you did set errexit.
while read pkg; do
   yay -S --needed $pkg
done < pkglist_aur.txt
will read every line of pkglist_aur.txt into the $pkg variable and run a new yay instance with it. If yay fails, the loop will just continue w/ the next line - except if errexit.
Post the actual file, your complete IO w/ the failure and the output of "set -o"
Edit: or if eschwartz knows another shell option that impacts this ;-)

My pkglist_aur.txt

ttf-arvo
libxft-bgra
mkinitcpio-openswap
aksjdhkasjdhaksdh
rofi-dmenu
ttf-font-awesome-4
picom-tryone-git

Output after I run

while read pkg; do yay -S --needed $pkg; done < pkglist_aur.txt

:: There are 3 providers available for ttf-arvo:

:: Repository AUR

    1) ttf-arvo 2) ttf-google-fonts-typewolf



Enter a number (default=1):  -> invalid number: libxft-bgra



Enter a number (default=1): EOF

 -> Could not find all required packages:

	ttf-arvo (Target)

Stangely this time it didnt even go past first package. The 4th package in the list is just gibberish which I added to see if it goes past that. And my set -o :

allexport      	off
braceexpand    	on
emacs          	on
errexit        	off
errtrace       	off
functrace      	off
hashall        	on
histexpand     	on
history        	on
ignoreeof      	off
interactive-comments	on
keyword        	off
monitor        	on
noclobber      	off
noexec         	off
noglob         	off
nolog          	off
notify         	off
nounset        	off
onecmd         	off
physical       	off
pipefail       	off
posix          	off
privileged     	off
verbose        	off
vi             	off
xtrace         	off

Last edited by phaedo7 (2021-01-08 15:42:34)

Trilby · 2021-01-08 16:00:40

Yeah, it's waiting for input - this is one of several reasons why this is a horrible idea from the start. But this has nothing to do with the shell code in the script, this is just yay waiting for a response. But I have no idea how yay (doesn't) work.

EDIT: given that stdin is redirected, yay may not be able to get a keyboard response, and/or it gets effectively nonsense/gibberish from the input file. You're trying to run an interactive process non-interactively - there is no magic coding to get this done. You could get closer to success with makepkg (with the approriate flags) but it's still a bad idea.

Last edited by Trilby (2021-01-08 16:03:27)

phaedo7 · 2021-01-08 16:08:54

Trilby wrote:

Yeah, it's waiting for input - this is one of several reasons why this is a horrible idea from the start. But this has nothing to do with the shell code in the script, this is just yay waiting for a response. But I have no idea how yay (doesn't) work.
EDIT: given that stdin is redirected, yay may not be able to get a keyboard response, and/or it gets effectively nonsense/gibberish from the input file. You're trying to run an interactive process non-interactively - there is no magic coding to get this done. You could get closer to success with makepkg (with the approriate flags) but it's still a bad idea.

That makes sense now. But then I would think, --noconfirm should work. But that works even worse. Anyway, I would just stick to my "for" script for time being.

P.S: I accidentally reported your comment instead of replying to it. How can I take my report back ?

Trilby · 2021-01-08 16:11:16

You could also direct input to yay from /dev/tty within the loop. But that is getting messy. --noconfirm would not be sufficeint as that just means you don't need to confirm anything (hence the name) - it can't answer questions about multiple options.

seth · 2021-01-08 16:18:21

exec 9<pkglist_aur.txt
while read -u9 pkg; do yay -S --needed $pkg; done

phaedo7 · 2021-01-08 18:19:45

seth wrote:

exec 9<pkglist_aur.txt
while read -u9 pkg; do yay -S --needed $pkg; done

Works like a charm. Marking this solved. So if I get it correctly, simply using a file descriptor solves the issue ?

zpg443 · 2021-01-08 18:22:31

phaedo7 wrote:

My pkglist_aur.txt

ttf-arvo
libxft-bgra
mkinitcpio-openswap
aksjdhkasjdhaksdh
rofi-dmenu
ttf-font-awesome-4
picom-tryone-git

With just seven programs, you cannot just makepkg them the old fashioned way?

phaedo7 · 2021-01-08 18:27:36

They are way more than that. This was just an example

eschwartz · 2021-01-08 19:11:53

If yay consumes stdin, one solution is using a different file descriptor as seth proposed. Another would be to read everything upfront into an array:

mapfile -t allpkgs < pkglist_aur.txt

for pkg in $"{allpkgs[@]}"; do
    yay -S "$pkg"
done

This solves the inherent word-splitting, globbing, and "empty lines" problems of

for pkg in $(<pkglist_aur.txt)

(because mapfile, also known as readarray, splits lines without processing them other than removing the chosen delimiter) but does not solve the potential problem "holding the entire array in memory does not scale to big files".

Last edited by eschwartz (2021-01-08 19:13:32)

respiranto · 2021-01-08 19:59:52

eschwartz wrote:

respiranto wrote:
In bash, one may also write:
for p in $(< pkglist_aur.txt); do yay -S --needed $p; done
This should not spawn a subshell either, while maintaining the order of objects.
No, one may not (or should not) write this. Why are you taking a working solution and trying to propose "alternative ways of also writing it" in bash? If you're going to do so, you should understand the complex programming language you're discussing.

Given the criticism of the original for loop, I wanted to give another solution that is closer to the original one, while--so I thought--being immune to that criticism.
The for and while loop solutions have different structure, and I believe there is generally a value in both syntaxes (though the semantics also vary).
I definitely like your solution with `mapfile', given later.

eschwartz wrote:

Specifically, please read this: https://mywiki.wooledge.org/DontReadLinesWithFor

Thanks for the link!
I did not think of glob expansion, nor did I realise that `$(< file)' needs to work differently to `< file', due to obligatory expansion, and therefore read all input to memory.

Concerning the reading of words instead of lines, I consider this actually useful--in general. Is there a straightforward way to avoid the issues with `for ... $(< file)' (and ideally the newfound one with `while read ...'), when word splitting is desired?

eschwartz · 2021-01-08 21:30:59

Use comments to indicate intent to future developers/users.

# deliberately split on all whitespace, but don't do globbing
set -f
allpkgs=($(<pkglist_aur.txt))
set +f

for pkg in $"{allpkgs[@]}"; do
    yay -S "$pkg"
done

Alad · 2021-01-08 21:33:21

phaedo7 wrote:

They are way more than that. This was just an example

If there are "way more", why are you recompiling them for every system/install you want them on?

https://wiki.archlinux.org/index.php/Pa … repository

Arch Linux

#1 2021-01-06 18:38:13

[SOLVED] What is the correct way to install AUR packages from a list?

#2 2021-01-06 18:41:13

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#3 2021-01-06 18:48:11

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#4 2021-01-07 04:28:33

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#5 2021-01-07 04:37:32

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#6 2021-01-07 19:21:40

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#7 2021-01-08 02:24:57

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#8 2021-01-08 11:18:49

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#9 2021-01-08 14:25:35

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#10 2021-01-08 14:50:18

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#11 2021-01-08 15:00:45

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#12 2021-01-08 15:13:02

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#13 2021-01-08 15:18:11

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#14 2021-01-08 15:38:02

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#15 2021-01-08 16:00:40

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#16 2021-01-08 16:08:54

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#17 2021-01-08 16:11:16

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#18 2021-01-08 16:18:21

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#19 2021-01-08 18:19:45

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#20 2021-01-08 18:22:31

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#21 2021-01-08 18:27:36

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#22 2021-01-08 19:11:53

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#23 2021-01-08 19:59:52

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#24 2021-01-08 21:30:59

Re: [SOLVED] What is the correct way to install AUR packages from a list?

#25 2021-01-08 21:33:21

Re: [SOLVED] What is the correct way to install AUR packages from a list?

Board footer