You are not logged in.

#1 2010-02-15 11:56:48

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,026

package distribution through a bit torrent network

make pacman a bittorrent client, and servers seeders.

would this be too complicated and not worth the trouble? im sure someone already thought of this.

if noone is seeding, servers would offload traffic from each other.

and if users choose to seed, well, we all know how fast movies download, right?

any downsides of this package distribution topology? after all, this was the reason bt got created in the first place.

im sure package signing would be a requirement..

Offline

#2 2010-02-15 12:06:22

Mr.Elendig
#archlinux@freenode channel op
From: The intertubes
Registered: 2004-11-07
Posts: 3,714

Re: package distribution through a bit torrent network

It would actually be quite easy. It wouldn't even have to be done in pacman itself, could use the xfter command and have it done as a community project.

One, and only one official tracker, which only tracks the latest packages. (It's possible to block old torrents quite easily)
Any mirror who feels like it would do http seeding. And yes, package signing would be a plus. but not strictly needed when you have a trusted tracker.


Evil #archlinux@freenode channel op and general support dude.
. files on github, Screenshots, Random pics and the rest

Offline

#3 2010-02-15 12:26:51

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,026

Re: package distribution through a bit torrent network

Mr.Elendig wrote:

It would actually be quite easy. It wouldn't even have to be done in pacman itself, could use the xfter command and have it done as a community project.

if you are suggesting i start this...i would, but im not actually up for the job. (for obvious reasons)...

as to blocking old packages, hmmm, im not sure this would be a requirement. since pacman always tries to download the last and latest. with bittorrent, these packages will be around for a longer period for users with breakage (always better).

Offline

#4 2010-02-15 13:48:23

urist
Member
Registered: 2009-02-22
Posts: 248

Re: package distribution through a bit torrent network

This would alienate a good chunk of users I assume, unless a few good mirrors do provide http access. I know my current university seems to block bittorrent.

Offline

#5 2010-02-15 14:10:37

rson451
Member
From: Annapolis, MD USA
Registered: 2007-04-15
Posts: 1,233
Website

Re: package distribution through a bit torrent network

urist wrote:

This would alienate a good chunk of users I assume, unless a few good mirrors do provide http access. I know my current university seems to block bittorrent.

Not if it was done via the XferCommand as Mr.Elendig suggested.  Then the user would have to specifically set that up.


archlinux - please read this and this — twice — then ask questions.
--
http://rsontech.net | http://github.com/rson

Offline

#6 2010-02-15 15:19:24

Mr.Elendig
#archlinux@freenode channel op
From: The intertubes
Registered: 2004-11-07
Posts: 3,714

Re: package distribution through a bit torrent network

Even if it was done it pacman itself, it should be an optional option

[options]
# Uncomment the following line if you want to use the bittorent backend for transfers:
# UseTorrent

Evil #archlinux@freenode channel op and general support dude.
. files on github, Screenshots, Random pics and the rest

Offline

#7 2011-05-29 06:35:01

xer0
Member
Registered: 2008-03-09
Posts: 9

Re: package distribution through a bit torrent network

This is a great idea.
Hell, just replace wget with a custom built torrent-get that fetches a torrent file and starts to download and your're done.
This would also solve problems with unsynced mirrors,
and one could download from all mirros at once if they're all seeding the torrent.

It's win win. The only down side is slow startup time, which is unnecessary for small files.

Offline

#8 2011-05-29 06:44:33

xer0
Member
Registered: 2008-03-09
Posts: 9

Re: package distribution through a bit torrent network

Also, even if all peers can't contribute much because of firewalls, slow upload etc., this solution still provides automatic load balancing of the mirrors as well as better utilization because one can download from all mirrors at once.

Maybe fetching each torrent file individually would be slow, but torrent files are small and one could download a set of them with rsync to a local cache.

Last edited by xer0 (2011-05-29 06:50:52)

Offline

#9 2011-06-01 22:32:04

ConnorBehan
Trusted User (TU)
From: Long Island NY
Registered: 2007-07-05
Posts: 1,356
Website

Re: package distribution through a bit torrent network

This has been proposed many times and as expected... no code was ever written. Are any of you truly willing to program this?


6EA3 F3F3 B908 2632 A9CB E931 D53A 0445 B47A 0DAB
Great things come in tar.xz packages.

Offline

#10 2011-06-02 12:19:46

franz1789
Member
Registered: 2008-04-22
Posts: 56

Re: package distribution through a bit torrent network

I read that an experimental feature of MLDonkey is the ability to download some chunks from a p2p and some from another, then merging the file.. applying it in pacman/external script would solve most of the problem, since it could try downloading files thru torrent, then if it's not seeded enough (or torrent is blocked) it could switch to http/ftp.. or it could download some file well seeded in torrent and others not in http.. or stuff like this..
ok sorry, this is crazy..

Offline

#11 2011-06-03 19:38:45

3])
Member
From: Netherlands
Registered: 2009-10-12
Posts: 215

Re: package distribution through a bit torrent network

Short solution:
I guess another 'temporary' way to do this right away without installing or coding is to...

a) have someone download the entire archlinux repo and update it periodically
b) upload the torrent file to start seeding the arch repo
c) have everyone else leech and then seed it as well
d) change the directory where pacman 'fetches' its repository to the directory where you are saving the whole archlinux repo

However, the issue with this is that any updates would mess the system up. Not to mention the large repo, perhaps people can choose what portions of the repo they want.


Long solution:

Doesn't sound too hard to implement though. Python does have a torrent based library, perhaps looking at Deluge torrent clients source code could help. Or it could be C++ for speed and optimization.

I. Give it the ability/privileges to delete and update files you 'requested' through bit torrent (perhaps it could be seen as an automatic 'pacman' package chooser).
II. Perhaps add that killer security feature that people have talked about?
III. Make sure no one can share corrupt-rootkit infested pacman repository files (related to II)
IV. partial package downloading... franz1789 has good comments on that

I could be regurgitating some facts stated on this thread though.

Last edited by 3]) (2011-06-03 19:39:08)


“There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.”-- C.A.R. Hoare

Offline

#12 2011-06-03 20:48:15

Maki
Member
From: Skopje, Macedonia
Registered: 2007-10-16
Posts: 345
Website

Re: package distribution through a bit torrent network

1. Mirror the arch repos, write scripts that would make an .torrent file of every new package and block the old packages from sharing.
2. Make a nice aur like web interface for the tracker, with emphasis on ease of use by scripts.
3. yaourt like pacman wrapper that would download every package from the tracker via torrent.
4. A daemon that would seed your cached packages.


If it ain't broke, broke it then fix it.

Offline

#13 2011-06-10 03:54:04

Ranguvar
Member
From: Our collective subconscious
Registered: 2008-08-12
Posts: 2,515
Website

Re: package distribution through a bit torrent network

This would indeed be very cool, but considering the number of generous high-speed universities and similar donating bandwidth, there's not a big practical impetus.
I doubt solid code will surface unless someone comes along who is extremely interested in this project and has the skills to make it happen, regardless of its actual benefit (not that it'd be useless).

Offline

#14 2011-06-10 05:15:45

tladuke
Member
Registered: 2009-07-23
Posts: 171

Re: package distribution through a bit torrent network

thanks to this thread I realized you can use aria2 for XferCommand.

Couldn't you use aria2 as the torrent client?

Offline

#15 2011-06-11 12:06:46

songandsilence
Member
From: Burlingame, KS, USA
Registered: 2010-12-01
Posts: 28

Re: package distribution through a bit torrent network

The solution is a bit simpler than that.

1: Include on the CD image transmission-cli
2: Create torrents ONLY of specfic package groups (gnome, kde, xfce, lxde, LAMP, etc) However, break these groups down into more specific things (gnome-core, gnome-desktop, gnome-libs, gnome-extra-core, gnome-extra-desktop, gnome-extra-libs, etc; kde-core, kde-libs, kde-plasma, kde-apps, etc) The smaller torrents are easier to maintain, and help fit the rolling release model better, even if the updates via torrent are a bit slower than the normal rolling release model.
3: Upload THESE torrents to mirrors, along with trackers. If I had a better box, I'd happily seed these.
4: Invoke a script to download these torrents through transmission, extract them with tar into /tmp/torrent_name, add this folder as a temporary repo, then use 'pacman -U /path/to/extracted_torrent/*.pkg.tar.gz" to install them all in one fell swoop.
5: This script then runs a quick -Syyu to catch whatever might not have made it into the new torrents just yet.

This method is best suited for a fresh install, since most users will want a fully functioning desktop.

The commands might go something like this:

wget http://torrents.archlinux.org/gnome/gno … 86.torrent  <--this is a fake link, don't click it.
transmission gnome-core-x86.torrent
tar -xzvf gnome-core-x86.tar.gz (plus whatever options put the extracted files in /tmp)
pacman -U /path/to/files/*.pkg.tar.gz
pacman -Syu
exit

Seems simple enough to implement, even though it's more of a workaround than a solution, it should be alright.

Last edited by songandsilence (2011-06-11 12:10:15)

Offline

#16 2013-10-05 20:32:14

ivan.kolmycheck
Member
Registered: 2013-03-30
Posts: 2

Re: package distribution through a bit torrent network

Actualy, it's a quite good idea. I think, alot of users won't keep the full mirror, but they do keep the packages in the cache for some time. They could be seeding them.

With packages signed, there's no difference how do they come - by torrent, http or even from CD/flash drive (in case of isolated machines, for example). smile

Offline

#17 2013-10-05 20:35:51

karol
Archivist
Registered: 2009-05-06
Posts: 25,423

Re: package distribution through a bit torrent network

Arch Linux has a rather extensive net of mirrors, so it doesn't need to rely on torrents, as has already been mentioned in this thread.

Last edited by karol (2013-10-05 20:36:33)

Online

#18 2013-12-15 16:41:37

jameh0
Member
Registered: 2013-04-06
Posts: 32

Re: package distribution through a bit torrent network

Something that hasn't been mentioned is that this has been implemented for debian

Also, a paper by the author: http://camrdale.debian.net/Resume/apt-p2p.pdf

It's coded in Python. Maybe someone could adapt this. I'd look into this in my *spare time*.


Life is very short, and there's no time for fussing and fighting, my friends.

Offline

#19 2013-12-15 16:45:37

karol
Archivist
Registered: 2009-05-06
Posts: 25,423

Re: package distribution through a bit torrent network

What is the use case for this? Who needs it?

Online

#20 2013-12-15 17:08:35

jameh0
Member
Registered: 2013-04-06
Posts: 32

Re: package distribution through a bit torrent network

Maybe it's an idealistic thing, but I know that most of my package downloads come through Waterloo University, and AFAIK other universities or (larger) companies do the same.

Having a distributed system would lower the barrier for individuals to provide upload to the Arch network. Personally, this is reason enough.

Performance is another question, but with a large amount of users, I think it could be very efficient.


Life is very short, and there's no time for fussing and fighting, my friends.

Offline

#21 2013-12-15 17:24:05

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,633
Website

Re: package distribution through a bit torrent network

karol wrote:

What is the use case for this? Who needs it?

A full p2p package network would allow anyone to distribute binary packages on par with the official packages. Users wishing to share their own binary repos would no longer need (access to) a dedicated server. Basically, everything on the AUR would be available in binary format somewhere, which would be very useful for packages with long compilation times and/or many build dependencies. It would open the doors for architecture-optimized binary package distribution through some sort of tagging system. With time all architectures would be represented. It would also make the entire package distribution system far more robust, as everything currently relies on a single server to distribute new packages to the mirrors.

It would also eliminated the arbitrariness of the current "Trusted Users" and even the devs, not all of whom are equally trusted. It would be easier for users to decide whom to trust on an individual basis, including others who are not officially associated with the distro. The current trust system would remain, but it would not be "imposed" to the same degree.

Instead of ordering repos, it would be possible to order packagers (with a matching dependency graph), which would make it easier to do things such as pull in all the Haskell packages from arch-haskell without having to give arch-haskell priority over everything in the community repo.



Heinrich Hertz wrote:

I do not think that the wireless waves I have discovered will have any practical application.

When you see no use for something, it may be that it has no use, but quite often it is you who simply fails to see the use.

Offline

#22 2013-12-15 22:46:58

Allan
Developer
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,379
Website

Re: package distribution through a bit torrent network

Xyne wrote:
karol wrote:

What is the use case for this? Who needs it?

A full p2p package network would allow anyone to distribute binary packages on par with the official packages. Users wishing to share their own binary repos would no longer need (access to) a dedicated server. Basically, everything on the AUR would be available in binary format somewhere, which would be very useful for packages with long compilation times and/or many build dependencies. It would open the doors for architecture-optimized binary package distribution through some sort of tagging system. With time all architectures would be represented. It would also make the entire package distribution system far more robust, as everything currently relies on a single server to distribute new packages to the mirrors.

There are advantages to not providing packages via bittorrent.  Primarily, I ofter release packages, close my laptop and go to work.  So the signle server at leasts guarantees seeds.  Many of the AUR packages would have ~0 seeds, making that not very useful.

Xyne wrote:

It would also eliminated the arbitrariness of the current "Trusted Users" and even the devs, not all of whom are equally trusted. It would be easier for users to decide whom to trust on an individual basis, including others who are not officially associated with the distro. The current trust system would remain, but it would not be "imposed" to the same degree.

I already disable some Trusted User's keys in my pacman keyring.

Xyne wrote:

Instead of ordering repos, it would be possible to order packagers (with a matching dependency graph), which would make it easier to do things such as pull in all the Haskell packages from arch-haskell without having to give arch-haskell priority over everything in the community repo.

Until I package something haskell as part of a rebuild and then someone at Arch Haskell decides to provide a glibc build.  Crappy example I know, but there is a point there!

If bittorrent were to be used, I recommend it using web seeds with our mirrors and only used to spread the load for large downloads.

Offline

#23 2013-12-16 01:01:54

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,633
Website

Re: package distribution through a bit torrent network

Allan wrote:

There are advantages to not providing packages via bittorrent.  Primarily, I ofter release packages, close my laptop and go to work.  So the signle server at leasts guarantees seeds.  Many of the AUR packages would have ~0 seeds, making that not very useful.

Torrents would not be mutually exclusive with current online repos and web seeding would be possible. Besides, someone who wants to share a built package this way would presumably seed it and/or make sure that it was available.


Allan wrote:

I already disable some Trusted User's keys in my pacman keyring.

I wonder which. Feel free to pm me some names. Have you signed those keys with your Arch Linux master key?

Allan wrote:

Until I package something haskell as part of a rebuild and then someone at Arch Haskell decides to provide a glibc build.  Crappy example I know, but there is a point there!

I don't get the example. Are you trying to highlight a problem with circular dependencies?

I don't doubt that there would be issues requiring careful consideration. I just wanted to show that there are some interesting possibilities for a p2p package distribution system.




Allan wrote:

If bittorrent were to be used, I recommend it using web seeds with our mirrors and only used to spread the load for large downloads.

This wouldn't be mutually exclusive with custom package distribution. It would simply be self-contained and "official". As for limiting it to large packages, I don't really see the point. Let it pull in everything all at once from whatever source can send the bits fast enough, in parallel.

Offline

#24 2014-01-04 18:26:20

thiagowfx
Member
Registered: 2013-07-09
Posts: 532

Re: package distribution through a bit torrent network

Allan wrote:

I already disable some Trusted User's keys in my pacman keyring.

I don't get this. Is it really necessary? This statement makes me worry about something I don't really know what or why. I mean, I've read it this way: some official packages should not be trusted. Please tell me I've misunderstood your statement.

Last edited by thiagowfx (2014-01-04 18:26:58)

Offline

#25 2014-01-10 00:25:44

ConnorBehan
Trusted User (TU)
From: Long Island NY
Registered: 2007-07-05
Posts: 1,356
Website

Re: package distribution through a bit torrent network

Unfortunately, I think that's the gist of it. Even though they were quickly fixed, I have definitely committed some derps in [community].

If a TU makes a mistake but works hard on improving, I think being untrusted on Allan's box is a better punishment than getting the boot altogether.


6EA3 F3F3 B908 2632 A9CB E931 D53A 0445 B47A 0DAB
Great things come in tar.xz packages.

Offline

Board footer

Powered by FluxBB