pacgem - Ruby gem installer using pacman

minad · 2011-02-07 09:36:50

Hi,

I created a pacman wrapper program which allows direct installation of rubygems. It is based on gem2arch which seems to be used to generate most of the ruby-* packages in AUR.

The program works as follows:
1. Resolve gem dependencies
2. Create PKGBUILD
3. Generate package using makepkg
4. Check package with namcap
5. Optionally install package with sudo pacman -U

Features:

Gem dependency resolution
Fixing the $pkgdir after gem install (fix some rights, remove native build leftovers, move man pages to the right place)
Automatic dependency resolution for shared objects
- Rubygems do not specify the native library dependencies
- Using readelf/pacman -Qo the dependencies can be detected
- The detected dependencies are added to the $depends variable
namcap checking of the created package

I would be interested in some feedback and testing. I tested it with ruby 1.9.2 and some ruby packages (haml, rack, unicorn). The goal is to get the package quality to an acceptable level (comparable to hand written PKGBUILD). There are nearly no complaints of namcap for the packages I tested.

Daniel

gbrunoro · 2011-02-07 13:39:53

seems nice, going to try this later on and post my impression

petelewis · 2011-03-22 19:31:09

Nice! This is excellent.

I'm installing a bunch of gems now to test it. So far it mostly seems fine. I saw the occasional:

/usr/bin/makepkg: eval: line 1279: unexpected EOF while looking for matching `)'

flying past though, and also several namcap warnings about dependencies being included but not required. I can provide more details if that would help.

Thanks!

petelewis · 2011-03-22 19:35:24

One issue with the shotgun gem, it seems:

% sudo pacman -U ruby-shotgun/ruby-shotgun-0.9-1-any.pkg.tar.xz
resolving dependencies...
looking for inter-conflicts...

Targets (1): ruby-shotgun-0.9-1

Total Download Size:    0.00 MB
Total Installed Size:   0.15 MB

Proceed with installation? [Y/n] 
(1/1) checking package integrity                                                                [########################################################] 100%
(1/1) checking for file conflicts                                                               [########################################################] 100%
error: failed to commit transaction (conflicting files)
ruby-shotgun: /usr/share/man/index.txt.gz exists in filesystem

hbekel · 2011-03-22 21:32:14

Very nice, thanks!

Here are some complaints namcap had with "bond":

$ pacgem bond
...
==> Checking ruby-bond-0.4.0-1-x86_64.pkg.tar.xz with namcap...
ruby-bond W: Dependency included and not needed ('ruby')
ruby-bond W: No ELF files and not an "any" package
ruby-bond E: Missing custom license directory (usr/share/licenses/ruby-bond)

Do you plan to create "any" packages and/or handle MIT licenses in the future? The bond gem includes a LICENSE.txt, but afaik there's no standard filename for licenses in gems... maybe the most common cases like /(LICENSE|COPYING)(\.txt)?/ could be looked for?

minad · 2011-03-22 21:46:14

@petelewis: I fixed it just now in version 0.9.3.

@hbekel:
1. Some of the complaints of namcap can be mostly ignored because they are invalid. Ruby is a dependency for example.
2. Normally the package should be an any package if there are no shared objects in it. Seems to be a problem here. I will investigate this.
3. Licenses are currently not handled but wouldn't be a problem to do it when I have time etc. Patches are also welcome

Daniel

Update: Concering 2 - pacgem detects if there are extensions in the gem (directory ext). In the bond gem there is such a directory but it doesn't build anything. I would consider this a broken gem. So not a pacgem issue.

Last edited by minad (2011-03-22 21:51:05)

petelewis · 2011-03-22 22:06:48

Thanks minad :-)

One more thing - it seems to have trouble installing packages automatically. Should I run it with sudo or should it automatically invoke sudo to ask for the required privileges?

E.g:

 % pacgem shotgun                                                                                              22:03
==> Resolving gems...
  -> (New) shotgun-0.9: reloading rack development server
  -> (New) rack-1.2.2: a modular Ruby webserver interface
==> Making package: ruby-rack 1.2.2-1 (Tue Mar 22 22:03:49 GMT 2011)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving Sources...
  -> Found rack-1.2.2.gem
==> Validating source files with sha256sums...
    rack-1.2.2.gem ... Passed
==> Extracting Sources...
==> Entering fakeroot environment...
==> Starting build()...
==> Installing gem...
Successfully installed rack-1.2.2
1 gem installed
==> Fixing gem installation...
==> Tidying install...
  -> Purging other files...
  -> Compressing man and info pages...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Removing empty directories...
==> Creating package...
  -> Generating .PKGINFO file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: ruby-rack 1.2.2-1 (Tue Mar 22 22:03:52 GMT 2011)
==> Checking ruby-rack-1.2.2-1-any.pkg.tar.xz with namcap...
which: no rackup in (/home/pete/bin:/home/pete/uni/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/share/java/apache-ant/bin:/usr/lib/perl5/vendor_perl/bin:/usr/bin/core_perl:/opt/qt/bin)
ruby-rack W: Referenced library 'rackup' is an uninstalled dependency
==> Installing ruby-rack-1.2.2-1-any.pkg.tar.xz with pacman...
==> ERROR: can't convert nil into String
  -> Use --trace for backtrace.

When I add --trace, I get the following when it tries to install:

==> Installing ruby-rack-1.2.2-1-any.pkg.tar.xz with pacman...
/usr/bin/pacgem:288:in `exists?': can't convert nil into String (TypeError)
        from /usr/bin/pacgem:288:in `installpkg'
        from /usr/bin/pacgem:204:in `block in install'
        from /usr/bin/pacgem:195:in `chdir'
        from /usr/bin/pacgem:195:in `install'
        from /usr/bin/pacgem:311:in `block in run'
        from /usr/bin/pacgem:311:in `each'
        from /usr/bin/pacgem:311:in `run'
        from /usr/bin/pacgem:434:in `block in process'
        from /usr/bin/pacgem:419:in `chdir'
        from /usr/bin/pacgem:419:in `process'
        from /usr/bin/pacgem:361:in `run'
        from /usr/bin/pacgem:502:in `<main>'

Thanks again.

Edit: I should have mentioned that using -c and then installing separately works fine though.

Last edited by petelewis (2011-03-22 22:08:41)

minad · 2011-03-22 22:14:34

Ok thanks for the report. I will look into it.

Can you give me your output of pacman -v?

It tries sudo and falls back to su -c btw. to install
Daniel

Last edited by minad (2011-03-22 22:15:29)

petelewis · 2011-03-22 22:25:45

minad wrote:

Can you give me your output of pacman -v?

% pacman -V

 .--.                  Pacman v3.5.0 - libalpm v6.0.0
/ _.-' .-.  .-.  .-.   Copyright (C) 2006-2011 Pacman Development Team
\  '-. '-'  '-'  '-'   Copyright (C) 2002-2006 Judd Vinet
 '--'
                       This program may be freely redistributed under
                       the terms of the GNU General Public License.

...yes I'm running [testing]. Should have mentioned that, sorry :-)

minad · 2011-03-22 23:05:17

-v (lowercase)

hbekel · 2011-03-23 00:07:23

Here's a patch that tries to find and install custom licenses. Not sure if this is the smartest way to do it, or the best place. It probably won't match all possible cases, but it should at least match the more common ones.

--- pacgem    2011-03-23 00:54:34.348361803 +0100
+++ pacgem-patched    2011-03-23 00:53:55.685028471 +0100
@@ -535,6 +535,46 @@
 
   # Sometimes there are files which are not world readable. Fix this.
   find $pkgdir -type f '!' -perm '-004' -exec chmod o+r -- '{}' ';'
+
+  # Try to find and install custom license  
+  shopt -s nocasematch
+  for l in "${license[@]}"; do
+    custom_license=1
+
+    # Check if this a common license
+    for common in /usr/share/licenses/common/*; do
+
+      # Strip the path and regexp escape dots
+      common="${common/\/usr\/share\/licenses\/common/}" 
+      common="${common/./\.}" 
+
+      # Some gems use sth like "LGPL-2.1" while pacman calls it "LGPL2.1",
+      # so check both variants...
+      for variant in "$common" "${common/-/}"; do
+        if [[ "$l" =~ "$variant" ]]; then
+          custom_license=0
+          break
+        fi
+      done
+    done
+
+    if ((custom_license)); then
+      # Check for a variety of possible license file names and install
+      # the first match into the appropriate license dir...
+      for candidate in \
+        "$l" "${l}.txt" \
+        "${l}-COPYING" "${l}-COPYING.txt" "${l}-LICENSE" "${l}-LICENSE.txt" \
+        "COPYING" "COPYING.txt" "LICENSE" "LICENSE.txt"; do    
+    
+    candidate="$_gemdir/gems/$_gemname-$pkgver/$candidate"
+    if [[ -f "$candidate" ]]; then
+      install -D -m644 "$candidate" "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
+      break;
+    fi
+      done
+    fi
+  done
+  shopt -u nocasematch
 }
 
 _gem_cleanext() {

(patch against pacgem-git 0.9.3, commit 4e13c2404577accdd098)

Btw, I've done some ruby dependency tracking a while ago using a slightly different method, maybe it's of interest to you: https://bbs.archlinux.org/viewtopic.php?id=67680

Last edited by hbekel (2011-03-23 00:15:15)

minad · 2011-03-23 03:19:25

@hbekel: Thank you very much! I merged your patch with some minor modifications. Please take a look at my git repo.
@petelewis: I am using 3.4.3. Please try pacgem-git. It might be fixed.

Daniel

hbekel · 2011-03-23 08:14:19

Oh yes, that's cleaner. I forgot you can have a plain variable and an array of the same name in bash, and I felt unhappy about just using $l

One thing: shouldn't the message be issued only when we actually find a custom license? Otherwise, if the user happens to know that the gem uses a common license, he might wonder why pacgem says "Installing license". Maybe msg "Installing $license license..." directly before the actual "install" command?

Oh, and it looks like you left an "echo" in when testing for candidates.

In any case, it feels great to be able to use pacgem and not be tempted to just "screw it" and use gem directly. Thanks again!

minad · 2011-03-23 15:10:47

I reworked the license code a bit. Most of the work is now done during package build creation in ruby. This allows to add a "custom:" prefix to the license array and multiple custom license files. It searches the gemspec.files array for license candidates and creates a _licensefile variable in the pkgbuild.

@petelewis: Tried the newest version for git?

petelewis · 2011-03-23 19:29:23

minad wrote:

@petelewis: Tried the newest version for git?

Actually, just upgrading to pacman 3.5.1 fixed it!

So - confirming that the automatic installation problem is fixed with both 0.9.3 and pacgem-git, with pacman 3.5.1.

Thanks! :-D

minad · 2011-03-23 20:21:52

petelewis wrote:

Actually, just upgrading to pacman 3.5.1 fixed it!
So - confirming that the automatic installation problem is fixed with both 0.9.3 and pacgem-git, with pacman 3.5.1.

Thank you for testing it!

Stalafin · 2011-03-28 15:47:12

I am trying to install the ical2rem.rb: http://www.offensivethinking.org/projects/ical2rem/

As I understand it, there is no gem available on rubygems.org. However, via his git repository the author provides a .gemspec file, which can be used to build a ruby gem.

Is there a way I can use pacgem to create a package from a local gem on my computer? Not surprisingly, giving pacgem the path to that local gem did not work:

$ pacgem ical2rem.rb-0.2.0.gem 
==> Resolving gems...
==> ERROR: Gem ical2rem.rb-0.2.0.gem (>= 0, runtime) not found
  -> Use --trace for backtrace.

minad · 2011-03-28 21:40:00

Stalafin wrote:

Is there a way I can use pacgem to create a package from a local gem on my computer?

This is not supported currently but it is definitely on my wishlist and todolist

Stalafin · 2011-03-29 09:20:43

Cool, that's great. I am absolutely looking forward to that! :-)

igndenok · 2011-03-29 21:36:26

Just wondering if you have a plan to add 'search' feature in the future ?
Thanks for your package it helps me alot

minad · 2011-03-30 11:26:02

igndenok wrote:

Just wondering if you have a plan to add 'search' feature in the future ?

No, I didn't plan to add one. But it might be a nice addition. I think about it. Please create issues/feature requests at https://github.com/minad/pacgem

Stalafin · 2011-04-01 13:56:24

Haha, I just saw on your github that apparently you were also interested in ical2rem, even before wrote in this thread. :-)

Since I am really not that much of a ruby user, could you suggest how to best get this running on my machine? Is it possible to install the gem for my user only so that I do not mess aroud /usr/ ?

petelewis · 2011-06-03 17:47:43

Hi,

I'm running into a bit of trouble with pacgem at the moment - it keeps wanting to install an updated version of rake (0.9.1), since it's newer than the one that ships with the ruby package. Unfortunately, every other gem is trying to pull this in as a dependency but obviously I can't install it (since it clashes with ruby itself).

Is anyone else finding this?

Any advice?

Thanks.

petelewis · 2011-06-03 21:38:05

Doing a little more investigating, it seems that pacgem wants to update already installed gems every time I try to install a new one. Take hoe as an example. On a clean install (i.e. just with Arch's ruby package, no pacgem-generated ones installed):

 % pacgem hoe
==> Resolving gems...
  -> (New) hoe-2.9.4: Hoe is a rake/rubygems helper for project Rakefiles
  -> (New) rake-0.9.1: Ruby based make-like utility.
==> Making package: ruby-rake 0.9.1-1 (Fri Jun  3 22:30:24 BST 2011)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving Sources...
  -> Found rake-0.9.1.gem
==> Validating source files with sha256sums...
    rake-0.9.1.gem ... Passed
==> Extracting Sources...
==> Entering fakeroot environment...
==> Starting build()...
==> Installing gem...
Successfully installed rake-0.9.1
1 gem installed
==> Installing man pages...
==> Fixing gem installation...
==> Tidying install...
  -> Purging other files...
  -> Compressing man and info pages...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Removing empty directories...
==> Creating package...
  -> Generating .PKGINFO file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: ruby-rake 0.9.1-1 (Fri Jun  3 22:27:30 BST 2011)
==> Checking ruby-rake-0.9.1-1-any.pkg.tar.xz with namcap...
==> Installing ruby-rake-0.9.1-1-any.pkg.tar.xz with pacman...
Password: 
resolving dependencies...
looking for inter-conflicts...

Targets (1): ruby-rake-0.9.1-1

Total Download Size:    0.00 MB
Total Installed Size:   0.93 MB

Proceed with installation? [Y/n] 
(1/1) checking package integrity                                                                [########################################################] 100%
(1/1) checking for file conflicts                                                               [########################################################] 100%
error: failed to commit transaction (conflicting files)
ruby-rake: /usr/bin/rake exists in filesystem
==> Making package: ruby-hoe 2.9.4-1 (Fri Jun  3 22:27:47 BST 2011)
==> Checking runtime dependencies...
==> Missing Dependencies:
  -> ruby-rake>=0.8.7
==> Checking buildtime dependencies...
==> ERROR: Could not resolve all dependencies.
==> ERROR: makepkg ruby-hoe failed
  -> Use --trace for backtrace.

It seems to me that there's no need for this, since hoe only requires rake >= 0.8.7, and

% rake --version
rake, version 0.8.7

from the standard ruby package.

Should pacgem just recognise that the dependency is already met and skip straight to building hoe?

Thanks,

Pete.

Arch Linux

#1 2011-02-07 09:36:50

pacgem - Ruby gem installer using pacman

#2 2011-02-07 13:39:53

Re: pacgem - Ruby gem installer using pacman

#3 2011-03-22 19:31:09

Re: pacgem - Ruby gem installer using pacman

#4 2011-03-22 19:35:24

Re: pacgem - Ruby gem installer using pacman

#5 2011-03-22 21:32:14

Re: pacgem - Ruby gem installer using pacman

#6 2011-03-22 21:46:14

Re: pacgem - Ruby gem installer using pacman

#7 2011-03-22 22:06:48

Re: pacgem - Ruby gem installer using pacman

#8 2011-03-22 22:14:34

Re: pacgem - Ruby gem installer using pacman

#9 2011-03-22 22:25:45

Re: pacgem - Ruby gem installer using pacman

#10 2011-03-22 23:05:17

Re: pacgem - Ruby gem installer using pacman

#11 2011-03-23 00:07:23

Re: pacgem - Ruby gem installer using pacman

#12 2011-03-23 03:19:25

Re: pacgem - Ruby gem installer using pacman

#13 2011-03-23 08:14:19

Re: pacgem - Ruby gem installer using pacman

#14 2011-03-23 15:10:47

Re: pacgem - Ruby gem installer using pacman

#15 2011-03-23 19:29:23

Re: pacgem - Ruby gem installer using pacman

#16 2011-03-23 20:21:52

Re: pacgem - Ruby gem installer using pacman

#17 2011-03-28 15:47:12

Re: pacgem - Ruby gem installer using pacman

#18 2011-03-28 21:40:00

Re: pacgem - Ruby gem installer using pacman

#19 2011-03-29 09:20:43

Re: pacgem - Ruby gem installer using pacman

#20 2011-03-29 21:36:26

Re: pacgem - Ruby gem installer using pacman

#21 2011-03-30 11:26:02

Re: pacgem - Ruby gem installer using pacman

#22 2011-04-01 13:56:24

Re: pacgem - Ruby gem installer using pacman

#23 2011-06-03 17:47:43

Re: pacgem - Ruby gem installer using pacman

#24 2011-06-03 21:38:05

Re: pacgem - Ruby gem installer using pacman

Board footer