yaourt, expected?

jugs · 2009-10-18 11:29:46

Hello,

I have been using AUR for more important packages that I feel should be optimized and compiled on my system (firefox-pgo, mplayer, etc.), but I've been looking for an easier way to check the AUR packages (that I already have installed) for updates, and automagically download, makepkg, and update them. I've installed yaourt in hopes that it could take care of this for me..

However, I have run into (percieved?) issues. Running 'yaourt -Sybu - -aur' gives me:

:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
archlinuxfr is up to date
==> Packages to build from sources:
mplayer 29411-3
Proceed with compilation and installation ? [Y/n]y
Source Targets: mplayer live-media
Proceed with upgrade? [Y/n] y
==> Building mplayer from sources
mplayer was not found on abs repos.archlinux.org
==> Building live-media from sources
live-media was not found on abs repos.archlinux.org

I don't think that yaourt is even looking in the AUR, and also why is this extra live-media dep existing now, as I don't see it being a dependency on AUR for mplayer..?

Please advise.

Allan · 2009-10-18 11:38:59

Never used yaourt so this could be wrong....

The -b flag builds all updates from source. mplayer in [extra] requires live-media.

knedlyk · 2009-10-18 11:40:21

yaourt uses http://repos.archlinux.org/viewvc.cgi to search through packages. It is down for a present moment, so we have to wait for a while...

jugs · 2009-10-18 13:04:33

Thanks for your quick replies

knedlyk wrote:

yaourt uses http://repos.archlinux.org/viewvc.cgi to search through packages. It is down for a present moment, so we have to wait for a while...

I will re-test once this goes back up, and hopefully all will be well! Cheers!

Garns · 2009-10-18 14:06:33

jugs wrote:

Thanks for your quick replies
knedlyk wrote:
yaourt uses http://repos.archlinux.org/viewvc.cgi to search through packages. It is down for a present moment, so we have to wait for a while...
I will re-test once this goes back up, and hopefully all will be well! Cheers!

Since viewvc is down due to a switch to websvn you might have to wait for a long time.

jugs · 2009-10-18 15:08:15

Garns wrote:

Since viewvc is down due to a switch to websvn you might have to wait for a long time.

Thanks for that Garns, I won't hold my breath in the meantime
Cheers!

brain0 · 2009-10-18 21:57:08

That is simply bad design. Yaourt shouldn't try to pull the PKGBUILDs for building official from source via the web interface, as it is slow.

Instead, it should use abs (which is based on rsync) or subversion directly.

legolas558 · 2009-10-20 08:47:48

ok, a patch coming

ava1ar · 2009-10-20 09:19:25

brain0 wrote:

Instead, it should use abs (which is based on rsync) or subversion directly.

I created a patch for yaourt that uses abs as a back-end for accessing ABS while getting sources. If you interested please find details here: http://aur.archlinux.org/packages.php?ID=31320

Last edited by ava1ar (2009-10-20 14:06:48)

pshr · 2009-10-20 10:39:12

Thanks ava1ar !

legolas558 · 2009-10-20 14:48:39

I have written a different patch which does not require ABS to be installed and uses the new WebSVN, this is more similar to the current yaourt mechanism

--- yaourt-original    2009-10-20 10:59:54.000000000 +0200
+++ /usr/bin/yaourt    2009-10-20 11:00:28.000000000 +0200
@@ -32,7 +32,8 @@
 AUR_URL="http://aur.archlinux.org/packages.php?setlang=en&do_Search=SeB=nd&L=2&C=0&PP=100&K="
 AUR_URL3="http://aur.archlinux.org/packages.php?setlang=en&ID="
 ABS_URL="http://archlinux.org/packages/search/?category=all&limit=99000"
-ABS_REPOS_URL="http://repos.archlinux.org/viewvc.cgi"
+#ABS_REPOS_URL="http://repos.archlinux.org/viewvc.cgi"
+ABS_REPOS_URL="http://repos.archlinux.org/wsvn/packages"
 [ -z "$LC_ALL" ] && export LC_ALL=$LANG
 
 ###################################

--- abs.sh-original    2009-10-20 10:53:50.000000000 +0200
+++ /usr/lib/yaourt/abs.sh    2009-10-20 16:45:01.000000000 +0200
@@ -44,9 +44,9 @@
     #if [ "$repository" = "testing" ]; then
     #           repository="all"
     #fi
-    
+
     # Manage specific Community and Testing packages
-    if [ "$repository" = "community" ]; then 
+    if [ "$repository" = "community" ]; then
         # Grab link to download pkgbuild from AUR Community
         [ "$MAJOR" != "getpkgbuild" ] && msg $(eval_gettext 'Searching Community AUR page for $PKG')
         aurid=`findaurid "$PKG"`
@@ -67,23 +67,15 @@
         # Grab link to download pkgbuild from new repos.archlinux.org
         source /etc/makepkg.conf
         [ -z "$CARCH" ] && CARCH="i686"
-        wget -q "${ABS_REPOS_URL}/$PKG/repos/" -O - > "$YAOURTTMPDIR/page.tmp"
-        if [ $? -ne 0 ] || [ ! -s "$YAOURTTMPDIR/page.tmp" ]; then
-            echo $(eval_gettext '$PKG was not found on abs repos.archlinux.org'); manage_error 1 || continue
-        fi
-        repos=( `grep "name=.*i686" "$YAOURTTMPDIR/page.tmp" | awk -F "\"" '{print $2}'` )
-        # if package exists in testing branch and in current branch, select the right url
-        if [ ${#repos[@]} -gt 1 -a $USETESTING -eq 1 ]; then
-            url="$ABS_REPOS_URL/$PKG/repos/${repos[1]}/"
-        else
-            url="$ABS_REPOS_URL/$PKG/repos/${repos[0]}/"
-        fi
+        ## legolas558: full path at WebSVN
+        url="${ABS_REPOS_URL}/$PKG/repos/${repository}-${CARCH}/"
     fi
 
     # Download Files on SVN package page
     wget -q "$url" -O "$YAOURTTMPDIR/page.tmp"
     manage_error $? || continue
-    files=( `grep "name=.*href=\"/viewvc.cgi/" "$YAOURTTMPDIR/page.tmp" | awk -F "\"" '{print $2}'`)
+    files=( `sed -nr 's/(.*)href="?([^ ">]*).*/\2\n\1/; T; P; D;' "$YAOURTTMPDIR/page.tmp" | grep op=dl \
+        | grep -v isdir=1 | sed -e 's/\&/\&/g' `)
     if [ ${#files[@]} -eq 0 ]; then echo "No file found for $PKG"; manage_error 1 || continue; fi
     echo
     if [ "$MAJOR" != "getpkgbuild" ]; then
@@ -98,12 +90,11 @@
 
     for file in ${files[@]}; do
         echo -e "   ${COL_BLUE}-> ${NO_COLOR}${COL_BOLD}$(eval_gettext 'Downloading ${file} in build dir')${NO_COLOR}"
-        if [ "$repository" = "community" ]; then
-            eval $INENGLISH wget --tries=3 --waitretry=3 --no-check-certificate "$ABS_REPOS_URL/community/$category/$PKG/$file?root=community\&view=co" -O $file
-        else
-            eval $INENGLISH wget --tries=3 --waitretry=3 --no-check-certificate "${url}${file}?view=co" -O $file
-        fi
+        pfname=$( echo $file | awk -F[/?] '{ print $(NF-1) }' )
+        ## legolas558: do not use eval since will exit the script
+        $INENGLISH wget --tries=3 --waitretry=3 --no-check-certificate -O "$pfname" "http://repos.archlinux.org${file}"
     done
+    unset pfname
 
     [ "$MAJOR" = "getpkgbuild" ] && return 0
 
@@ -114,15 +105,15 @@
     else
         runasroot=0
     fi
-    
+
     readPKGBUILD
     if [ -z "$pkgname" ]; then
                echo $(eval_gettext 'Unable to read PKGBUILD for $PKG')
         manage_error 1 || continue
     fi
-    
+
     msg "$pkgname $pkgver-$pkgrel $([ "$branchtags" = "TESTING" ] && echo -e "$COL_BLINK[TESTING]")"
-    
+
     # Customise PKGBUILD
     [ $CUSTOMIZEPKGINSTALLED -eq 1 ] && customizepkg --modify

You will need to apply it to /usr/bin/yaourt and /usr/lib/yaourt/abs.sh. I will propose it to the yaourt mantainers.

ava1ar · 2009-10-20 15:24:52

Good work, but personally I prefer to use rsync than the websvn
It will great if yaourt developers include your patch, but looks like they are not very active now...

knedlyk · 2009-10-20 18:46:43

legolas558: What version of yaourt do you have? I tried to apply your patch to yaourt 0.9.2.6-1, and got this:
patch -p0 < yaourt-patch
patching file /usr/bin/yaourt
patching file /usr/lib/yaourt/abs.sh
Hunk #1 FAILED at 44.
Hunk #2 FAILED at 67.
Hunk #3 FAILED at 90.
Hunk #4 FAILED at 105.
4 out of 4 hunks FAILED -- saving rejects to file /usr/lib/yaourt/abs.sh.rej

Gen2ly · 2009-10-20 19:31:57

woah, nice job ava1ar.

legolas558 · 2009-10-20 20:19:45

ava1ar wrote:

Good work, but personally I prefer to use rsync than the websvn
It will great if yaourt developers include your patch, but looks like they are not very active now...

Yeah problem is that adding ABS is more like forking the current yaourt (hence your yaourt-abs package name is correct), while I was just trying to patch the current yaourt. Anyway I realized you had posted yaourt-abs after writing the patch

knedlyk · 2009-10-20 23:48:03

legolas558 wrote:

You will need to apply it to /usr/bin/yaourt and /usr/lib/yaourt/abs.sh. I will propose it to the yaourt mantainers.

I had to apply your patch manually, see my previous post. It works!

legolas558 · 2009-10-21 00:51:09

knedlyk wrote:

legolas558 wrote:
You will need to apply it to /usr/bin/yaourt and /usr/lib/yaourt/abs.sh. I will propose it to the yaourt mantainers.
I had to apply your patch manually, see my previous post. It works!

I am sorry but I tried the patch with the stock yaourt and it works for me..don't know why it failed in your case, maybe you have a modified yaourt?

knedlyk · 2009-10-21 08:30:42

No, I have yaourt installed from http://repo.archlinux.fr/i686. Probably your editor entered some invisible characters... Look in your patch, what changes did you made in line 44? And same in line 114?

@@ -44,9 +44,9 @@
     #if [ "$repository" = "testing" ]; then
     #           repository="all"
     #fi
-    
+
     # Manage specific Community and Testing packages
-    if [ "$repository" = "community" ]; then 
+    if [ "$repository" = "community" ]; then
         # Grab link to download pkgbuild from AUR Community
         [ "$MAJOR" != "getpkgbuild" ] && msg $(eval_gettext 'Searching Community AUR page for $PKG')
         aurid=`findaurid "$PKG"`

Let somebody try to apply this patch independently, just to make sure whether everything is alright with this patch.

Allan · 2009-10-21 08:34:23

Looks like the trailing whitespace was stripped.

ava1ar · 2009-10-21 08:56:59

legolas558 wrote:

ava1ar wrote:
Good work, but personally I prefer to use rsync than the websvn
It will great if yaourt developers include your patch, but looks like they are not very active now...
Yeah problem is that adding ABS is more like forking the current yaourt (hence your yaourt-abs package name is correct), while I was just trying to patch the current yaourt. Anyway I realized you had posted yaourt-abs after writing the patch

Yes, you are right. My idea was since yaourt is a wrapper for pacman when working with binary packages, why not use it as a wrapper for abs while working with ABS packages also.
Another point of doing fork - a number of gaps, that yaourt already has prevents it to be included in community. Insecurity is one of the biggest one among them. I wan't to make yaourt as much as possible transparent for secutiry verification. And one of the way to achieve this - not to reinvent the wheel and shift all available actions onto trusted system utilities, keeping yaourt mostly as an advanced wrapper for them (with some additional functionality that was originally implemented for yaourt - like AUR support).
Next patch I want to implement - disable building packages as a root (not it is just a warning about this) - it will make yaourt much more secure while working with untrusted packages from AUR (I will think over how to keep an option to turn this prohibition off for exceptional cases, but will definitely turn it on by default).

Last edited by ava1ar (2009-10-21 09:00:51)

legolas558 · 2009-10-21 11:08:40

ava1ar: good! please go on, I would also cleanup the shell scripts mess and use a more advanced scripting language to do the job, but anyway if you can make the shell scripts glue work it's OK (although some caching mechanism should be implemented sooner or later).

The best would be to have a 'download bucket' working with curl so that many files can be fetched from internet or a local directory or CD/DVD media.

Also one thing to implement natively is in my opinion the 'customizepkg' feature. That script is an excellent idea, but rarely works - I have actually made some changes to it to have it work (patch at bottom of this post).
Furthermore, customizepkg does not seem to work with source packages because the dependency is checked in a later stage.

--- /usr/bin/customizepkg    2009-10-20 03:32:52.000000000 +0200
+++ /usr/bin/customizepkg     2009-10-20 16:40:54.000000000 +0200
@@ -50,21 +50,23 @@
     grep --invert-match "\(^#\|^$\)" $configfile |
     while read line; do
         unset action context pattern value
-        action=`echo $line | awk -F "#" '{print $1}'`
-        context=`echo $line | awk -F "#" '{print $2}'`
-        pattern=`echo $line | awk -F "#" '{print $3}'`
-        #echo "action=$action , context=$context  , pattern=$pattern"
+        ## get all 3 fields
+        action=$(echo $line | awk -F# '{ print $1 }')
+        context=$(echo $line | awk -F# '{ print $2 }')
+        pattern="$(echo $line | awk -F# '{ print $3 }')"
+        echo "action=$action , context=$context  , pattern=$pattern"
         case $action in
             remove|replace)
                 value=`echo $line | awk -F [^#]# '{print $4}'`
                 echo "=> removes/replaces '$pattern' by '$value' in $context"
                 if [ "$action" = "replace" -a "$context" != "global" ]; then
-                    value=" '$(echo $value | tr -d "\'")'"
+                    value="$(echo $value | tr -d "\'")"
                 fi
                 if [ "$context" != "global" ]; then
                     pattern="$pattern[<>=]*[a-z0-9.]*"
                 fi
                 if [ "$context" = "global" ]; then
+                    echo "sed -i \"s/$pattern/$value/g\" \"$scriptfile\""
                     sed -i "s/$pattern/$value/g" "$scriptfile"
                 else
                     sed -i "/^$context=/,/)$/ s/[[:space:]]*[']*$pattern[']*/$value/g" "$scriptfile"
@@ -115,7 +117,8 @@
     echo "PKGBUILD not found"
     exit 1
 fi
-source ./PKGBUILD || exit 1
+## legolas558: do not source the PKBUILD because it could contain errors
+#source ./PKGBUILD
 
 if [ ! -r "$CONFIGDIR/$pkgname" ]; then
     echo "no configuration found for $pkgname in $CONFIGDIR/"
@@ -123,11 +126,17 @@
 fi
 #TODO: PKGBUILD + file.install
 cp ./PKGBUILD ./PKGBUILD.custom
+echo "Now modifying PKGBUILD"
 modify_file "$CONFIGDIR/$pkgname" "./PKGBUILD" "./PKGBUILD.custom" || exit 1
+## modified by legolas558
+CPKGTMP=$(mktemp customizepkg-$USER-diff.XXXXXXXX)
+echo -e "The following customizepkg rules were applied:\n----\n$(<$CONFIGDIR/$pkgname)\n----\nChanges:" > $CPKGTMP
+diff -u ./PKGBUILD ./PKGBUILD.custom >> $CPKGTMP
+less $CPKGTMP
+rm -f $CPKGTMP
 if [ $MODIFY -eq 1 ]; then
     cp ./PKGBUILD ./PKGBUILD.original
     cp ./PKGBUILD.custom ./PKGBUILD
 fi
 
-
 exit 0

legolas558 · 2009-10-21 11:14:04

Allan wrote:

Looks like the trailing whitespace was stripped.

Yep, I have uploaded the patches here:

http://www.iragan.com/linux/

gav616 · 2009-11-24 12:31:14

thank you, 'yaourt-abs' works sooper!

Arch Linux

#1 2009-10-18 11:29:46

yaourt, expected?

#2 2009-10-18 11:38:59

Re: yaourt, expected?

#3 2009-10-18 11:40:21

Re: yaourt, expected?

#4 2009-10-18 13:04:33

Re: yaourt, expected?

#5 2009-10-18 14:06:33

Re: yaourt, expected?

#6 2009-10-18 15:08:15

Re: yaourt, expected?

#7 2009-10-18 21:57:08

Re: yaourt, expected?

#8 2009-10-20 08:47:48

Re: yaourt, expected?

#9 2009-10-20 09:19:25

Re: yaourt, expected?

#10 2009-10-20 10:39:12

Re: yaourt, expected?

#11 2009-10-20 14:48:39

Re: yaourt, expected?

#12 2009-10-20 15:24:52

Re: yaourt, expected?

#13 2009-10-20 18:46:43

Re: yaourt, expected?

#14 2009-10-20 19:31:57

Re: yaourt, expected?

#15 2009-10-20 20:19:45

Re: yaourt, expected?

#16 2009-10-20 23:48:03

Re: yaourt, expected?

#17 2009-10-21 00:51:09

Re: yaourt, expected?

#18 2009-10-21 08:30:42

Re: yaourt, expected?

#19 2009-10-21 08:34:23

Re: yaourt, expected?

#20 2009-10-21 08:56:59

Re: yaourt, expected?

#21 2009-10-21 11:08:40

Re: yaourt, expected?

#22 2009-10-21 11:14:04

Re: yaourt, expected?

#23 2009-11-24 12:31:14

Re: yaourt, expected?

Board footer