Automatic installation of packages from both official repos and AUR

tk83 · 2015-01-28 03:09:32

I'm trying to migrate over from Ubuntu where I have some custom scripts that basically install all the packages I need from lists using apt-get commands. I can get this working properly in Arch with packages from the official repositories using pacman, but can't get it working with AUR packages (unfortunately needed for many applications such as Spotify, Taskcoach etc.) using yaourt.

The problem appears to be that yaourt will try to re-build and re-install all AUR packages whenever it's called, even if those packages are already installed and working. For example, for my 'desktop applications' module my script does the following:

PACKAGELIST="amule audacity banshee bleachbit calibre deluge digikam dosbox easytag thunderbird-enigmail gimp gimp-help-en gnome-disk-utility gthumb firefox handbrake kdenlive kipi-plugins kdenetwork-krdc libreoffice-fresh libreoffice-fresh-en-GB kdeedu-marble multibit onboard openshot pavucontrol picard pidgin pidgin-otr soundkonverter spotify taskcoach thunderbird unison variety vlc wine wireshark-qt xsane thunderbird-lightning-bin"
pacman -Qi "$PACKAGELIST" > /dev/null 2>&1 || su - tim -c "yaourt -S --needed --noconfirm $PACKAGELIST" # using https://wiki.archlinux.org/index.php/Yaourt here to get AUR packages too. yaourt needs to run as a non-root user who has password-less sudo access

The first time this is called it works as intended, the problem is on the second and subsequent calls yaourt will completely re-build & re-install the already-installed AUR packages (e.g Spotify, taskcoach). I've checked the man pages but haven't seen an option to change this.

fukawi2 · 2015-01-28 03:50:12

Moving to Programming & Scripting

Head_on_a_Stick · 2015-01-28 07:29:34

Can you change your script to use makepkg instead?
https://wiki.archlinux.org/index.php/Ar … g_packages

tk83 · 2015-01-28 07:33:59

It would be quite a bit of work to automate the whole download file/tar xf/makepkg/pacman -U process - anyway wouldn't this just be duplicating yaourt?
Plus it takes away the seamless-ness of yaourt - meaning I'd have to manually separate out the packages that come from AUR from those that don't in my lists.

Last edited by tk83 (2015-01-28 07:34:19)

drcouzelis · 2015-01-28 15:37:27

I'm confused... Isn't yaourt re-installing those packages because you are telling it to re-install those packages?

Why are you installing the same list of packages multiple timel instead of doing an update?

clfarron4 · 2015-01-28 17:43:49

Doesn't the following update the repositories, check if there are updates in the main repositories and then check if there are updates to AUR packages:

yaourt -Syua

Then you'd only have to work out which packages you need to install the first time and then update away?

Or are you trying to do this for specific packages?

Last edited by clfarron4 (2015-01-28 17:44:35)

tk83 · 2015-01-29 01:35:23

drcouzelis wrote:

I'm confused... Isn't yaourt re-installing those packages because you are telling it to re-install those packages?
Why are you installing the same list of packages multiple timel instead of doing an update?

I'm writing a script to apply the packages I want installed (plus conf files etc.) automatically - a bit like Puppet. I already have this for Ubuntu. So the script will be run multiple times, and shouldn't re-install packages already installed.

I've actually got it working more-or-less (I think the 'less' is simply that some of the packages I wanted to install from AUR are broken builds). Here's my script and an example 'packages.list' file:

function installifnotinstalled () {
	# Calls package manager to install one or more package(s), if they're not already installed
	if [ $DISTRIBUTION = "opensuse" ] ; then
                rpm -q "$1" > /dev/null 2>&1 || zypper -n in "$1"
	elif [ $DISTRIBUTION = 'arch' ]; then
                PACKAGELIST="$@"
                [ -n "$PACKAGELIST" -a "$PACKAGE" != "''" ] && pacman -Q "$PACKAGELIST" > /dev/null 2>&1 || pacman -S --needed --noconfirm $PACKAGELIST 
        elif [ "$DISTRIBUTION" = "ubuntu" -o "$DISTRIBUTION" = "debian" ] ; then
		export DEBIAN_FRONTEND=noninteractive # stop annoying prompts
                dpkg -s "$@" > /dev/null 2>&1 || apt-get -qq -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install "$@"
        fi
}
export -f installifnotinstalled

function installpackagelist () {
        # Installs packages listed in a packages.list file where equivalent packages for each Linux distro are listed in different columns
        # first column is Ubuntu, second is Debian, third is Arch Linux, fourth is Arch AUR, fifth is comments
        # Format example 1 (made up example where logwatch is called log-watch in Arch): logwatch logwatch log-watch # this is a comment
        # Format example 2 (there's an up-to-date Calibre package for Arch but not for Debian and Ubuntu): '' '' calibre
        if [ "$DISTRIBUTION" = "ubuntu" ]; then
                installifnotinstalled `awk '{ print $1 }' packages.list  | tail -n +3 - | xargs`
        elif [ "$DISTRIBUTION" = "debian" ]; then
                installifnotinstalled `awk '{ print $2 }' packages.list  | tail -n +3 - | xargs`
        elif [ "$DISTRIBUTION" = "arch" ]; then
                installifnotinstalled `awk '{ print $3 }' packages.list  | tail -n +3 - | xargs`
                for PACKAGE in `awk '{ print $4 }' packages.list  | tail -n +3 -`; do # deal with Arch AUR packages one-by-one
                    if [ -n "$PACKAGE" -a "$PACKAGE" != "''" ]; then
                        pacman -Q "$PACKAGE" > /dev/null 2>&1 || su - tim -c "yaourt -S --needed --noconfirm $PACKAGE" # using https://wiki.archlinux.org/index.php/Yaourt here to get AUR packages. yaourt needs to run as a non-root user who has password-less sudo access
                    fi
                done
        else
            echo "ERROR installpackagelist: Distribution not supported"
        fi
}
export -f installpackagelist

ubuntu debian arch arch_aur comments
---------------------
amule amule amule
audacity audacity audacity
banshee banshee banshee
bleachbit bleachbit bleachbit
'' '' calibre '' # On Ubuntu & Debian install Calibre using a script
deluge deluge deluge
digikam digikam digikam
digikam-doc digikam-doc '' '' # On Arch digikam includes docs
dosbox dosbox dosbox
dvdstyler dvdstyler '' dvdstyler # Arch AUR
easytag easytag easytag
enigmail enigmail '' '' # thunderbird-enigmail is in Arch AUR but doesn't build as of 29 Jan 2015
gimp gimp gimp
gimp-help-en gimp-help-en gimp-help-en
gnome-disk-utility gnome-disk-utility gnome-disk-utility
google-chrome-stable google-chrome-stable '' '' # No Chrome package on Arch
gthumb gthumb gthumb
filebot filebot '' filebot
firefox firefox firefox
handbrake-cli '' '' '' # On Arch handbrake package includes this
handbrake-gtk handbrake-gtk handbrake
kamoso kamoso '' kamoso # Arch AUR
kdenlive kdenlive kdenlive
kipi-plugins kipi-plugins kipi-plugins
krdc krdc kdenetwork-krdc
libreoffice libreoffice libreoffice-fresh
libreoffice-kde libreoffice-kde '' '' # On Arch not available
libreoffice-help-en-gb libreoffice-help-en-gb libreoffice-fresh-en-GB
marble marble kdeedu-marble
'' '' multibit '' # On Ubuntu & Debian this is installed by script
onboard onboard onboard
openshot openshot openshot
pavucontrol pavucontrol pavucontrol
pepperflashplugin-nonfree pepperflashplugin-nonfree '' '' # not available on Arch
picard picard picard
pidgin pidgin pidgin
pidgin-awayonlock pidgin-awayonlock '' '' # not available on Arch
pidgin-otr pidgin-otr pidgin-otr
plasma-widget-cwp plasma-widget-cwp '' '' # not available on Arch
sflphone-kde sflphone-kde '' '' # sflphone-kde is in Arch AUR but doesn't build as of 29 Jan 2015
soundkonverter soundkonverter soundkonverter
spideroak spideroak '' '' # not available on Arch
spotify-client spotify-client '' spotify # Arch AUR
stopwatch stopwatch '' '' # not available on Arch
taskcoach taskcoach '' taskcoach # Arch AUR
thunderbird icedove thunderbird
unison-gtk unison-gtk unison
usb-creator-kde '' '' '' # not available on Arch or Debian
variety '' variety
vlc vlc vlc
wine wine wine
wireshark wireshark wireshark-qt
xsane xsane xsane
xul-ext-gdata-provider '' '' '' # not available on Arch or Debian
xul-ext-lightning iceowl-extension '' thunderbird-lightning-bin # Arch AUR

And then in my script to install all 'desktop applications' I just call the function 'installpackagelist'. Hope this helps someone and also makes it clearer what I was trying to do.

Last edited by tk83 (2015-01-29 01:35:54)

Awebb · 2015-01-29 09:04:13

I need to automate this. I need to scrape the new posts for 'yaourt' and auto post about learning how the AUR works first, doing it manually for a couple of days and then build some scripts around a lightweight helper like cower.

The wiki has one problem: It tells the user to use a 3rd party repo to install yaourt, so no newcommer will ever install anything the way it's supposed to be installed, mot even yaourt, not even once.

My first go with Arch was like this: Install Arch, play around a bit, miss a few packages, read the AUR article, add yaourt from the fr repo and try to build some packages. Out of 10 packages I tried to build, only one worked. I thought, if this is Arch, then I don't want it. Yet another distro, that promises user content and fails. A year later I learned, that the AUR stuff was fine, I just happened to dip in a transitional phase between pacman update and yaourt catching up. Had I built yaourt from the PKGBUILD, nothing would have happened.

Alad · 2015-01-29 09:38:04

The wiki has one problem: It tells the user to use a 3rd party repo to install yaourt, so no newcommer will ever install anything the way it's supposed to be installed, mot even yaourt, not even once.

It's a single line after a full paragraph encouraging people to install it the "right" way... the problem is people are like electricity, they will always take the path of least resistance, no matter efforts otherwise.

Last edited by Alad (2015-01-29 09:38:46)

tk83 · 2015-01-29 10:03:13

I installed yaourt and one other dependency from AUR the Right Way - using makepkg/pacman -U. During my testing of these scripts I also did it the manual way for a few other packages to get a grip on how it works.
But at the end of the day I prefer automation once I've understood it enough to see how it works.

drcouzelis · 2015-01-29 13:49:22

I still don't quite understand the situation in which you are using this script, but it still kind of seems like the wrong tool for the job...

tk83 wrote:

I'm writing a script to apply the packages I want installed (plus conf files etc.) automatically - a bit like Puppet. I already have this for Ubuntu.

How about Puppet for Arch Linux?

EDIT: How many AUR packages are you installing? If you are using a server / client setup to use Puppet, you can create packages from the AUR, host them your server, then use pacman from Puppet to install them on all of the clients.

Last edited by drcouzelis (2015-01-29 13:51:11)

\hbar · 2015-01-29 14:05:32

What about

to_install=`comm -13 <(pacman -Qq) <(echo -e $list_of_packages)`
yaourt -S $to_install

or thereabouts?

tk83 · 2015-01-29 23:23:46

drcouzelis wrote:

How about Puppet for Arch Linux?
EDIT: How many AUR packages are you installing? If you are using a server / client setup to use Puppet, you can create packages from the AUR, host them your server, then use pacman from Puppet to install them on all of the clients.

Originally these scripts were just for installing packages and configuring things on my laptop. But as I've added more Linux machines (media centre PC, desktop, Chromebook etc.) the scripts have grown and been refactored several times. If I was starting from scratch with it now I'd probably use Puppet, and in fact I'm considering switching to Puppet.

Arch Linux

#1 2015-01-28 03:09:32

Automatic installation of packages from both official repos and AUR

#2 2015-01-28 03:50:12

Re: Automatic installation of packages from both official repos and AUR

#3 2015-01-28 07:29:34

Re: Automatic installation of packages from both official repos and AUR

#4 2015-01-28 07:33:59

Re: Automatic installation of packages from both official repos and AUR

#5 2015-01-28 15:37:27

Re: Automatic installation of packages from both official repos and AUR

#6 2015-01-28 17:43:49

Re: Automatic installation of packages from both official repos and AUR

#7 2015-01-29 01:35:23

Re: Automatic installation of packages from both official repos and AUR

#8 2015-01-29 09:04:13

Re: Automatic installation of packages from both official repos and AUR

#9 2015-01-29 09:38:04

Re: Automatic installation of packages from both official repos and AUR

#10 2015-01-29 10:03:13

Re: Automatic installation of packages from both official repos and AUR

#11 2015-01-29 13:49:22

Re: Automatic installation of packages from both official repos and AUR

#12 2015-01-29 14:05:32

Re: Automatic installation of packages from both official repos and AUR

#13 2015-01-29 23:23:46

Re: Automatic installation of packages from both official repos and AUR

Board footer