Pip2Arch - PyPi packages to Arch PKGBUILDs

Blue Peppers · 2010-11-15 00:57:16

Pip2Arch is a simple tool to convert PyPi packages (the python equivalent of CPAN) into Arch PKGBUILDS.

You can get the code here: https://github.com/bluepeppers/pip2arch

Run

pip2arch.py --help

for help, usage should be simple enough.

Currently does not track dependencies, due to a problem with PyPi(?). Patches very welcome.

I use pip2arch for django servers, where I want to use obscure django packages that are not usually in the aur, but don't want to write a PKGBUILD for them. Using pip2arch, I just have to pay attention to the dependencies.

Thanks,

Laurie

Last edited by Blue Peppers (2010-11-15 00:59:33)

EnvoyRising · 2010-11-16 07:07:50

Obviously not as ideal as automatically tracking dependencies, but what about adding a switch to feed it dependencies?

pip2arch.py -d dep1, dep2>=1.4, dep3

I have to wake up early in the morning, else I'd submit a proof of concept patch.

Blue Peppers · 2010-11-17 00:01:05

Added:

pip2arch.py -d django-core django-templating

will create the PKGBUILD for the hypothetical django-templating package, with django-core as a dependency.

EnvoyRising · 2010-11-17 09:05:20

Couple of patches

1st patch adds an -m option to allow for makedepends:

pip2arch.py -m git some_package

will create a some_package with a make dependency of git

2nd patch adds search functionality to pip2arch:

pip2arch.py -s mathematics
bessy
bidict
IntPy
isodate
munkres
NodeBox
PGAPy
qombinatorics
SciMath
scipy
tabular
TANGO Project - ALGENCAN

makedeps.patch:

--- pip2arch.py    2010-11-17 02:16:41.380000528 -0600
+++ pip2arch.py.new    2010-11-17 02:07:33.536667195 -0600
@@ -13,7 +13,8 @@
 pkgrel=1
 pkgdesc="{pkg.description}"
 url="{pkg.url}"
-depends=('{pkg.pyversion}' {depends})
+depends=('{pkg.pyversion}'{depends})
+makedepends=({makedepends})
 license=('{pkg.license}')
 arch=('any')
 source=('{pkg.download_url}')
@@ -33,6 +34,7 @@
     logging.info('Creating Server Proxy object')
     client = xmlrpclib.ServerProxy('http://pypi.python.org/pypi')
     depends = []
+    makedepends = []
     
     def get_package(self, name, outname, version=None):
         if version is None:
@@ -47,7 +49,7 @@
         raw_urls = self.client.release_urls(name, version)
         logging.info('Got release_urls from PiPy')
         if not len(raw_urls) and len(data):
-            raise LackOfInformation('PyPi did not return the neccisary information to create the PKGBUILD')
+            raise LackOfInformation('PyPi did not return the necessary information to create the PKGBUILD')
         elif len(data) and len(raw_urls):
             urls = {}
             for url in raw_urls:
@@ -79,7 +81,7 @@
             self.url = data.get('home_page', '')
             self.license = data['license']
         except KeyError:
-            raise pip2archException('PiPy did not return needed information')
+            raise pip2archException('Pypi did not return needed information')
         logging.info('Parsed other data')
     
     def choose_version(self, versions):
@@ -95,10 +97,14 @@
             
     def add_depends(self, depends):
         self.depends += depends
+
+    def add_makedepends(self, makedepends):
+        self.makedepends += makedepends
     
     def render(self):
-        depends = '\'' + '\' \''.join(d for d in self.depends) + '\'' if self.depends else ''
-        return BLANK_PKGBUILD.format(pkg=self, date=datetime.date.today(), depends=depends)
+        depends = ' \'' + '\' \''.join(d for d in self.depends) + '\'' if self.depends else ''
+        makedepends = '\'' + '\' \''.join(d for d in self.makedepends) + '\'' if self.makedepends else ''
+        return BLANK_PKGBUILD.format(pkg=self, date=datetime.date.today(), depends=depends, makedepends=makedepends)
 
 
 if __name__ == '__main__':
@@ -113,6 +119,7 @@
                         default=open('PKGBUILD', 'w'),
                         help='The file to output the generated PKGBUILD to')
     parser.add_argument('-d', '--dependencies', dest='depends', action='append')
+    parser.add_argument('-m', '--make-dependencies', dest='makedepends', action='append')
     parser.add_argument('-n', '--output-package-name', dest='outname', action='store', default=None,
                         help='The name of the package that pip2arch will generate')
     
@@ -125,6 +132,10 @@
         sys.exit('ERROR: {0}'.format(e))
     if args.depends:
         p.add_depends(args.depends)
+    
+    if args.makedepends:
+        p.add_makedepends(args.makedepends)
+
     print "Got package information"
     args.outfile.write(p.render())
-    print "Written PKGBUILD"
\ No newline at end of file
+    print "Written PKGBUILD"

search.patch:

--- pip2arch.py.old    2010-11-17 02:17:43.440000529 -0600
+++ pip2arch.py.new    2010-11-17 02:57:03.863333862 -0600
@@ -34,7 +34,13 @@
     client = xmlrpclib.ServerProxy('http://pypi.python.org/pypi')
     depends = []
     
-    def get_package(self, name, outname, version=None):
+    def get_package(self, name, outname, version=None, search=False):
+        if search:
+            results = self.client.search({'description': '%s' % name[1:]})
+            for result in results:
+                print result['name']
+            sys.exit(1)
+
         if version is None:
             versions = self.client.package_releases(name)
             version = self.choose_version(versions)
@@ -113,6 +119,7 @@
                         default=open('PKGBUILD', 'w'),
                         help='The file to output the generated PKGBUILD to')
     parser.add_argument('-d', '--dependencies', dest='depends', action='append')
+    parser.add_argument('-s', '--search', dest='search', action='store_true')
     parser.add_argument('-n', '--output-package-name', dest='outname', action='store', default=None,
                         help='The name of the package that pip2arch will generate')
     
@@ -120,11 +127,11 @@
     
     p = Package()
     try:
-        p.get_package(name=args.pkgname, version=args.version, outname=args.outname or args.pkgname)
+        p.get_package(name=args.pkgname, version=args.version, outname=args.outname or args.pkgname, search=args.search)
     except pip2archException as e:
         sys.exit('ERROR: {0}'.format(e))
     if args.depends:
         p.add_depends(args.depends)
     print "Got package information"
     args.outfile.write(p.render())
-    print "Written PKGBUILD"
\ No newline at end of file
+    print "Written PKGBUILD"

Blue Peppers · 2010-11-17 18:29:17

I've added them. I didn't commit the patches directly, due to them being slightly out of line with the current HEAD of the repo, but I've put your username in the commit message

EnvoyRising · 2010-11-18 08:10:15

Another patch:
- removed str() type-casting in search function-- term is always a string anyways.
- Changed 'Written PKGBUILD' to 'PKGBUILD written'.
- Search is now interactive. Each result lists a number next to it. Typing the number creates a PKGBUILD for the corresponding result.

Interactive search only accepts one number, but it should be relatively trivial to adapt the below patch so that it accepts more than one entry.

--- pip2arch.py    2010-11-18 01:03:49.916667195 -0600
+++ pip2arch.py.new    2010-11-18 02:05:19.086667196 -0600
@@ -1,4 +1,5 @@
 #!/usr/bin/python2
+
 import sys
 import xmlrpclib
 import datetime
@@ -88,11 +89,17 @@
             raise pip2archException('PiPy did not return needed information')
         logging.info('Parsed other data')
         
-    def search(self, term):
-        results = self.client.search({'description': str(term[1:])})
-        for result in results:
-            print ' - '.join((result['name'], result['summary']))
-        #If no results
+    def search(self, term, version=None):
+        results = self.client.search({'description': term[1:]})
+        for i, result in enumerate(results):
+            print "{0}) {1} - {1}".format(i+1, result['name'], result['summary'])
+
+        selection = raw_input('==> Type the number of the module you would '
+                              'like to build a PKGBUILD for:\n'
+                              '==> ') 
+        self.get_package(name=results[int(selection)-1]['name'], version=version,
+                         outname=results[int(selection)]['name'])
+
         if not results:
             print 'No results found'
     
@@ -144,17 +151,16 @@
     p = Package()
     
     if args.search:
-        p.search(args.pkgname)
-        sys.exit(0)
-    
-    try:
-        p.get_package(name=args.pkgname, version=args.version, outname=args.outname or args.pkgname)
-    except pip2archException as e:
-        sys.exit('ERROR: {0}'.format(e))
+        p.search(args.pkgname, version=args.version)
+    else:
+        try:
+            p.get_package(name=args.pkgname, version=args.version, outname=args.outname or args.pkgname)
+        except pip2archException as e:
+            sys.exit('ERROR: {0}'.format(e))
     if args.depends:
         p.add_depends(args.depends)
     if args.makedepends:
         p.add_makedepends(args.makedepends)
     print "Got package information"
     args.outfile.write(p.render())
-    print "Written PKGBUILD"
+    print "PKGBUILD written"

Blue Peppers · 2010-11-18 17:02:16

Done. Have you considered getting a github account? It would make sending patches much easier.

EnvoyRising · 2010-11-18 17:46:26

I have one but everything seems so foreign, lol. I guess I'm just too used to gitorious. That, and I suppose I wasn't expecting to be contributing as much...

Plus, the patch was a quick prototype lacking any sort of error handling, so I figured you might want to doctor it up and/or add multiple selection, to it.

edit:
BTW, my username on github (and everywhere but here, apparently) is aspidites, not EnvoyRisign.

Last edited by EnvoyRising (2010-11-18 17:51:04)

Dieter@be · 2011-03-08 14:47:21

I'll try this out. I have created a bunch of python packages myself on AUR and now I'm getting tired of it
is there a specific reason why we cannot have the dependencies automatically from the pypi page?

Last edited by Dieter@be (2011-03-08 14:47:35)

Mineo · 2011-03-09 12:14:02

The distutils documentation doesn't even mention the various requires options which is bad enough. It also doesn't like dependencies with hypens in them - calling the setup.py will just fail.

Dieter@be · 2011-03-09 13:12:37

Hmm, I was thinking about it differently. I thought the problem was automatically writing dependencies in PKGBUILDS, given a specific pypi. This doesn't have anything to do with calling setup.py, or am I missing something?

Mineo · 2011-03-09 14:58:34

The workflow for uploading a package to pypi is through the setup.py.

Blue Peppers · 2011-03-09 17:27:14

Dieter@be wrote:

Hmm, I was thinking about it differently. I thought the problem was automatically writing dependencies in PKGBUILDS, given a specific pypi. This doesn't have anything to do with calling setup.py, or am I missing something?

A projects dependencies are stored in the setup.py file. PyPi does not seem to parse this beyond looking at the categories that the package fits in. This means that the xmlrpc response that Pip2Arch gets from the PyPi server, does not contain any information on dependencies. To have a semi accurate list of dependencies, we would have to do what pip actually does, which is download the package, untar/unzip it, then parse the setup.py file. However, as Pip is rather modular, this might not be such a bothersome task. So there is hope.

In fact, I might have a go at it this weekend.

stephane · 2011-03-09 19:05:19

I suggest to separate build and package functions with something like :

build() {{
cd $srcdir/{pkg.name}-{pkg.version}
{pkg.pyversion} setup.py build
}}

package() {{
cd $srcdir/{pkg.name}-{pkg.version}
{pkg.pyversion} setup.py install --root="$pkgdir" --skip-build --optimize=1 {pkg.setup_args}
}}

Also, most pkg installed that way will requier "makedepends=('python-distribute')" (or python2-distribute for python2).

Mineo · 2011-03-09 19:41:47

Blue Peppers wrote:

A projects dependencies are stored in the setup.py file. PyPi does not seem to parse this beyond looking at the categories that the package fits in. This means that the xmlrpc response that Pip2Arch gets from the PyPi server, does not contain any information on dependencies. To have a semi accurate list of dependencies, we would have to do what pip actually does, which is download the package, untar/unzip it, then parse the setup.py file.

Oh, PyPi does return the requires-stuff, it's just nobody uses it in their setup.py in the first place.

Last edited by Mineo (2011-03-09 19:44:25)

Blue Peppers · 2011-03-09 20:43:57

Mineo wrote:

Oh, PyPi does return the requires-stuff, it's just nobody uses it in their setup.py in the first place.

Except that Pip gets it right (mostly). So if we use pip, then I'll concider it true to specification.

@stephane, will do.

Daenyth · 2011-03-10 19:57:38

I might like to include this tool in the pkgtools package -- see here: https://github.com/Daenyth/pkgtools/issues/#issue/27

Let me know if that's ok

Blue Peppers · 2011-03-10 21:47:46

Sure, go for it.

Arch Linux

#1 2010-11-15 00:57:16

Pip2Arch - PyPi packages to Arch PKGBUILDs

#2 2010-11-16 07:07:50

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#3 2010-11-17 00:01:05

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#4 2010-11-17 09:05:20

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#5 2010-11-17 18:29:17

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#6 2010-11-18 08:10:15

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#7 2010-11-18 17:02:16

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#8 2010-11-18 17:46:26

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#9 2011-03-08 14:47:21

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#10 2011-03-09 12:14:02

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#11 2011-03-09 13:12:37

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#12 2011-03-09 14:58:34

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#13 2011-03-09 17:27:14

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#14 2011-03-09 19:05:19

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#15 2011-03-09 19:41:47

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#16 2011-03-09 20:43:57

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#17 2011-03-10 19:57:38

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

#18 2011-03-10 21:47:46

Re: Pip2Arch - PyPi packages to Arch PKGBUILDs

Board footer