pm2ml: generate metalinks for pacman download

Xyne · 2012-03-20 18:11:33

pm2ml (pacman2metalink) is basically a spiritual successor to pacman2aria2. It generates fully-compliant (afaik) metalinks for pacman download operations. Use them to download files faster via parallel and segmented downloads. It's also useful if you want to download files on a different computer.

The package also includes ppl, which is basically a successor of powerpill-light, As with powerpill-light, ppl is just a simple Bash script that uses pm2ml. Open it up, see how it works, and then write your own script to suit your needs.

Things pm2ml does that pacman2aria2 doesn't:
* generates metalinks
* includes AUR source tarballs
* includes database files
* includes database and package signatures

As always, questions, comments and feedback are welcome.

Xyne · 2012-03-21 10:24:02

I've moved the Aria2 options from ppl to /etc/ppl.conf. I'm still trying to find the idea defaults. If anyone has suggestions, let me know.

sausageandeggs · 2012-03-21 16:16:43

Thanks for a neat little script Xyne. Are you going to add dependency handling to the pm2ml script. I know It's easy enough to add it to one of the bash scripts but I just wondered if you plan on adding it to the python script.

Xyne · 2012-03-22 02:57:38

Dependencies should now be included. I have also added an option to disable dependency resolution.

sausageandeggs · 2012-03-23 10:07:07

Brilliant thanks a lot Xyne

sewa2k · 2012-03-26 01:02:55

how can i use it?

Xyne · 2012-03-26 09:47:21

sewa2k wrote:

how can i use it?

However you want.

Take a look at the included "ppl*" scripts.

If you just want to do a full system upgrade:

pplsyu

If you want to install the base-devel group:

ppls base-devel

If you want to create a metalink for the base group and matching signature files:

pm2ml -s base > base.meta4

To download those packages on another system:

aria2c -M base.meta4 ...

sewa2k · 2012-03-26 11:19:05

Xyne wrote:

sewa2k wrote:
how can i use it?
However you want.

Thanks a lot!, im having an issue btw when im doing pplsyu

[root@phage sewa]# pplsyu 
:: Sincronizando las bases de datos de paquetes...
 core está actualizado
 extra está actualizado
 community está actualizado
 multilib está actualizado
 repo-ck está actualizado
Traceback (most recent call last):
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 88, in retrieve
    mirror['last_sync'] = timegm(strptime(mirror['last_sync'], MirrorStatus.PARSE_TIME_FORMAT))
  File "/usr/lib/python3.2/_strptime.py", line 482, in _strptime_time
    tt = _strptime(data_string, format)[0]
  File "/usr/lib/python3.2/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pm2ml", line 573, in <module>
    main(sys.argv[1:])
  File "/usr/bin/pm2ml", line 523, in main
    mirrorstatus, mirrors = process_options(reflector_options)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 360, in process_options
    mirrors = ms.get_mirrors()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 113, in get_mirrors
    return self.get_obj()['urls']
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 106, in get_obj
    self.retrieve()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 96, in retrieve
    raise MirrorStatusError('failed to retrieve mirror data: %s' % (e))
Reflector.MirrorStatusError: "failed to retrieve mirror data: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'"
Excepcion capturada
Exception: [metalink_helper.cc:130] errorCode=20 Could not parse Metalink XML document.

"esta actualizado" means "it's up to date"

Thanks in advance

UPDATE: i've solved it by reinstalling reflector

Last edited by sewa2k (2012-03-26 11:34:19)

step-2 · 2012-08-20 21:16:51

great script

thanks a lot xyne

mwknowles92 · 2012-08-21 23:29:46

Typing "ppl --help" gets an exception.

Exception: [metalink_helper.cc:130] errorCode=20 Could not parse Metalink XML document.

Xyne · 2012-08-24 00:23:51

@mwknowles92
It now supports "--help", but note that is it not a full application, only a simple wrapper script provided as an example.

graysky · 2012-10-12 19:10:18

@Xyne - I think I'm misunderstanding how to use reflector with pplsyu. Shouldn't the reflector call happen before calling `pacman -Sy` since we want the sync to happen with the most up-to-date mirror?

Here is /usr/bin/pplsyu as supplied:

#!/bin/bash
set -e
cd /

# This is just an example of how pm2ml can be used in your own scripts.
# This will run the equivalent of "pacman -Syu [pkgnames]"

# Arguments for aria2c.
aria2_args=(
  --metalink-file=-
  --conf-path=/etc/ppl.conf
)
# sync databases
pacman -Sy
#pm2ml -ysd var/lib/pacman/sync | aria2c "${aria2_args[@]}"
# download packages
pm2ml -no var/cache/pacman/pkg -u "$@" -r -p http -l 50 | aria2c "${aria2_args[@]}"
# invoke pacman
pacman -Su "$@"

Here is what I am thinking:

--- pplsyu	2012-10-12 15:09:35.205546758 -0400
+++ pplsyu	2012-10-12 15:09:17.679294570 -0400
@@ -10,6 +10,9 @@
   --metalink-file=-
   --conf-path=/etc/ppl.conf
 )
+# find up-to-date mirror
+reflector -c "United States" -a 1 -f 3 --sort rate --save /etc/pacman.d/mirrorlist
+
 # sync databases
 pacman -Sy
 #pm2ml -ysd var/lib/pacman/sync | aria2c "${aria2_args[@]}"

Xyne · 2012-10-13 22:42:51

That's intentional. I have never used an on-the-fly generated mirrorlist to update the database in any of my update scripts (powerpill/bauerbill/pm2ml/parisync/etc). This will probably change once the official database are finally signed, but until then I leave the choice of which mirror to trust up to the user. Packages can be pulled in from random mirrors because they are checked against the database. Even before the advent of package signing I felt that the checksums provided sufficient reliability.

That said, the ppl* scripts are just examples for deriving your own scripts. I myself use parisync most of the time.

Arch Linux

#1 2012-03-20 18:11:33

pm2ml: generate metalinks for pacman download

#2 2012-03-21 10:24:02

Re: pm2ml: generate metalinks for pacman download

#3 2012-03-21 16:16:43

Re: pm2ml: generate metalinks for pacman download

#4 2012-03-22 02:57:38

Re: pm2ml: generate metalinks for pacman download

#5 2012-03-23 10:07:07

Re: pm2ml: generate metalinks for pacman download

#6 2012-03-26 01:02:55

Re: pm2ml: generate metalinks for pacman download

#7 2012-03-26 09:47:21

Re: pm2ml: generate metalinks for pacman download

#8 2012-03-26 11:19:05

Re: pm2ml: generate metalinks for pacman download

#9 2012-08-20 21:16:51

Re: pm2ml: generate metalinks for pacman download

#10 2012-08-21 23:29:46

Re: pm2ml: generate metalinks for pacman download

#11 2012-08-24 00:23:51

Re: pm2ml: generate metalinks for pacman download

#12 2012-10-12 19:10:18

Re: pm2ml: generate metalinks for pacman download

#13 2012-10-13 22:42:51

Re: pm2ml: generate metalinks for pacman download

Board footer