You are not logged in.

#1 2012-03-20 18:11:33

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

pm2ml: generate metalinks for pacman download

pm2ml (pacman2metalink) is basically a spiritual successor to pacman2aria2. It generates fully-compliant (afaik) metalinks for pacman download operations. Use them to download files faster via parallel and segmented downloads. It's also useful if you want to download files on a different computer.

The package also includes ppl, which is basically a successor of powerpill-light, As with powerpill-light, ppl is just a simple Bash script that uses pm2ml. Open it up, see how it works, and then write your own script to suit your needs.

Things pm2ml does that pacman2aria2 doesn't:
* generates metalinks
* includes AUR source tarballs
* includes database files
* includes database and package signatures

As always, questions, comments and feedback are welcome.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#2 2012-03-21 10:24:02

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pm2ml: generate metalinks for pacman download

I've moved the Aria2 options from ppl to /etc/ppl.conf. I'm still trying to find the idea defaults. If anyone has suggestions, let me know.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#3 2012-03-21 16:16:43

sausageandeggs
Member
Registered: 2009-12-05
Posts: 66

Re: pm2ml: generate metalinks for pacman download

Thanks for a neat little script Xyne. Are you going to add dependency handling to the pm2ml script. I know It's easy enough to add it to one of the bash scripts but I just wondered if you plan on adding it to the python script.

Offline

#4 2012-03-22 02:57:38

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pm2ml: generate metalinks for pacman download

Dependencies should now be included. I have also added an option to disable dependency resolution.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#5 2012-03-23 10:07:07

sausageandeggs
Member
Registered: 2009-12-05
Posts: 66

Re: pm2ml: generate metalinks for pacman download

Brilliant thanks a lot Xyne

Offline

#6 2012-03-26 01:02:55

sewa2k
Member
Registered: 2012-02-26
Posts: 9

Re: pm2ml: generate metalinks for pacman download

how can i use it?

Offline

#7 2012-03-26 09:47:21

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pm2ml: generate metalinks for pacman download

sewa2k wrote:

how can i use it?

However you want. tongue


Take a look at the included "ppl*" scripts.


If you just want to do a full system upgrade:

pplsyu

If you want to install the base-devel group:

ppls base-devel

If you want to create a metalink for the base group and matching signature files:

pm2ml -s base > base.meta4

To download those packages on another system:

aria2c -M base.meta4 ...

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#8 2012-03-26 11:19:05

sewa2k
Member
Registered: 2012-02-26
Posts: 9

Re: pm2ml: generate metalinks for pacman download

Xyne wrote:
sewa2k wrote:

how can i use it?

However you want. tongue

Thanks a lot!, im having an issue btw when im doing pplsyu

[root@phage sewa]# pplsyu 
:: Sincronizando las bases de datos de paquetes...
 core está actualizado
 extra está actualizado
 community está actualizado
 multilib está actualizado
 repo-ck está actualizado
Traceback (most recent call last):
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 88, in retrieve
    mirror['last_sync'] = timegm(strptime(mirror['last_sync'], MirrorStatus.PARSE_TIME_FORMAT))
  File "/usr/lib/python3.2/_strptime.py", line 482, in _strptime_time
    tt = _strptime(data_string, format)[0]
  File "/usr/lib/python3.2/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pm2ml", line 573, in <module>
    main(sys.argv[1:])
  File "/usr/bin/pm2ml", line 523, in main
    mirrorstatus, mirrors = process_options(reflector_options)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 360, in process_options
    mirrors = ms.get_mirrors()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 113, in get_mirrors
    return self.get_obj()['urls']
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 106, in get_obj
    self.retrieve()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 96, in retrieve
    raise MirrorStatusError('failed to retrieve mirror data: %s' % (e))
Reflector.MirrorStatusError: "failed to retrieve mirror data: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'"
Excepcion capturada
Exception: [metalink_helper.cc:130] errorCode=20 Could not parse Metalink XML document.

"esta actualizado" means "it's up to date"

Thanks in advance

UPDATE: i've solved it by reinstalling reflector

Last edited by sewa2k (2012-03-26 11:34:19)

Offline

#9 2012-08-20 21:16:51

step-2
Member
Registered: 2012-05-08
Posts: 118

Re: pm2ml: generate metalinks for pacman download

great script

thanks a lot xyne

Offline

#10 2012-08-21 23:29:46

mwknowles92
Member
Registered: 2010-08-28
Posts: 19

Re: pm2ml: generate metalinks for pacman download

Typing "ppl --help" gets an exception.

Exception: [metalink_helper.cc:130] errorCode=20 Could not parse Metalink XML document.

Offline

#11 2012-08-24 00:23:51

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pm2ml: generate metalinks for pacman download

@mwknowles92
It now supports "--help", but note that is it not a full application, only a simple wrapper script provided as an example.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#12 2012-10-12 19:10:18

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: pm2ml: generate metalinks for pacman download

@Xyne - I think I'm misunderstanding how to use reflector with pplsyu.  Shouldn't the reflector call happen before calling `pacman -Sy` since we want the sync to happen with the most up-to-date mirror?

Here is /usr/bin/pplsyu as supplied:

#!/bin/bash
set -e
cd /

# This is just an example of how pm2ml can be used in your own scripts.
# This will run the equivalent of "pacman -Syu [pkgnames]"

# Arguments for aria2c.
aria2_args=(
  --metalink-file=-
  --conf-path=/etc/ppl.conf
)
# sync databases
pacman -Sy
#pm2ml -ysd var/lib/pacman/sync | aria2c "${aria2_args[@]}"
# download packages
pm2ml -no var/cache/pacman/pkg -u "$@" -r -p http -l 50 | aria2c "${aria2_args[@]}"
# invoke pacman
pacman -Su "$@"

Here is what I am thinking:

--- pplsyu	2012-10-12 15:09:35.205546758 -0400
+++ pplsyu	2012-10-12 15:09:17.679294570 -0400
@@ -10,6 +10,9 @@
   --metalink-file=-
   --conf-path=/etc/ppl.conf
 )
+# find up-to-date mirror
+reflector -c "United States" -a 1 -f 3 --sort rate --save /etc/pacman.d/mirrorlist
+
 # sync databases
 pacman -Sy
 #pm2ml -ysd var/lib/pacman/sync | aria2c "${aria2_args[@]}"

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#13 2012-10-13 22:42:51

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: pm2ml: generate metalinks for pacman download

That's intentional. I have never used an on-the-fly generated mirrorlist to update the database in any of my update scripts (powerpill/bauerbill/pm2ml/parisync/etc). This will probably change once the official database are finally signed, but until then I leave the choice of which mirror to trust up to the user. Packages can be pulled in from random mirrors because they are checked against the database. Even before the advent of package signing I felt that the checksums provided sufficient reliability.

That said, the ppl* scripts are just examples for deriving your own scripts. I myself use parisync most of the time.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB