You are not logged in.

#1 2009-03-21 16:59:33

z.s.tar.gz
Member
From: Rural Florida
Registered: 2008-12-12
Posts: 298

Bittorrent & Pacman Idea

Here is my idea: Create a program (or even script) that would use bittorrent and pacman for fast downloads of big files (AKA kde or gnome)

1. Tracker with compressed groups of packages (don't download individual)
2. Once done downloading, extract
3. pacman -U everything for install

Don't know the practicality of this, but that's what ideas are for.


I need to find a way out so everyone can find their way out.
Resregietd Lunix Uesr: 485581

Offline

#2 2009-03-21 17:20:46

daf666
Member
Registered: 2007-04-08
Posts: 470
Website

Re: Bittorrent & Pacman Idea

Hmm.. interesting.., like everyone shares their /var/cache/pacman/pkg dir.
I would..
is this what you mean?

Offline

#3 2009-03-21 17:23:04

z.s.tar.gz
Member
From: Rural Florida
Registered: 2008-12-12
Posts: 298

Re: Bittorrent & Pacman Idea

I really don't know, I was thinking more along the lines of using pacman the same, but using bittorrent. Less load on servers means faster downloads, right?

like:

# bitman -S kde

Last edited by z.s.tar.gz (2009-03-21 17:24:10)


I need to find a way out so everyone can find their way out.
Resregietd Lunix Uesr: 485581

Offline

#4 2009-03-21 17:44:10

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Bittorrent & Pacman Idea

Just curious... have you seen powerpill yet? If you just want simultaneous/segmented downloads, that should be good enough.

I've thought of using torrents to share packages before too (especially for old packages no longer in the repos that live on in people's caches), but I never got further than a few suggestions. There might be a bittorrent protocol in pkgd's future if I ever get around to it (I have ideas for extending its topology floating around in my head).


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#5 2009-03-21 17:58:03

skottish
Forum Fellow
From: Here
Registered: 2006-06-16
Posts: 7,942

Re: Bittorrent & Pacman Idea

Xyne wrote:

I've thought of using torrents to share packages before too (especially for old packages no longer in the repos that live on in people's caches), but I never got further than a few suggestions. There might be a bittorrent protocol in pkgd's future if I ever get around to it (I have ideas for extending its topology floating around in my head).

Since powerpill using aria2, it seems like you've already done quite a bit of the work.

Offline

#6 2009-03-21 20:06:36

ammon
Member
Registered: 2008-12-11
Posts: 413

Re: Bittorrent & Pacman Idea

Great! I like it!
It would decentralize repository and speed things up. They are doing ti with git, so why not with pacman.
And +  everyone can put custom compiled packages (their own cflags, like core2 or such) and share it... So i can download firefox for my proc and share it to next peer.

Offline

#7 2009-03-21 20:58:30

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: Bittorrent & Pacman Idea

There is a catch with that, in some places p2p traffic is very throttled and I've seen cases of getting corrupted p2p data. Throttling can be done by the isp/school and data corruption will only go away with encryption.
But thats a nice idea, sharing older packages and/or custom packages.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#8 2009-03-21 21:15:46

markp1989
Member
Registered: 2008-10-05
Posts: 431

Re: Bittorrent & Pacman Idea

daf666 wrote:

Hmm.. interesting.., like everyone shares their /var/cache/pacman/pkg dir.
I would..
is this what you mean?

sounds like a prety good idea, like said it would saave server resources, i  would be woried about people altering package files thou. not sure if that could be a problem or if im just being paranoid


Desktop: E8400@4ghz - DFI Lanparty JR P45-T2RS - 4gb ddr2 800 - 30gb OCZ Vertex - Geforce 8800 GTS - 2*19" LCD
Server/Media Zotac GeForce 9300-ITX I-E - E5200 - 4gb Ram - 2* ecogreen F2 1.5tb - 1* wd green 500gb - PicoPSU 150xt - rtorrent - xbmc - ipazzport remote - 42" LCD

Offline

#9 2009-03-21 21:36:39

godane
Member
From: NH, USA
Registered: 2008-02-03
Posts: 241
Website

Re: Bittorrent & Pacman Idea

If it was to be done it would still have to uses the md5 from the mirrors in order for it to be made sure that the packages is same has in the repo.

This is just my theory if it was going to be done.


I'm working on a live cds based on Archlinux. http://godane.wordpress.com/

Offline

#10 2009-03-21 23:31:26

Ranguvar
Member
Registered: 2008-08-12
Posts: 2,544

Re: Bittorrent & Pacman Idea

BitTorrent does integrity checking on every single piece -- no extra checking needed besides the normal md5 pacman does.

Offline

#11 2009-03-21 23:52:26

sokuban
Member
Registered: 2006-11-11
Posts: 412

Re: Bittorrent & Pacman Idea

That sounds like a really nice idea. Sure, we don't really need it since there are so many mirrors and all and its not like there is too much a load on them (is there?), but it would be nice since then old packages etc could still be floating around.

But I don't know about custom packages. I'm not against the idea of allowing them to integrate into the system, but I would never use them myself. (Is paranoid of unofficial packages) So make sure there is enough seperation so people who only use official packages can do so.

EDIT: I don't understand the idea of the original poster. I just think that everyone sharing their /var/cache/pacman/pkg/ would be cool.

Last edited by sokuban (2009-03-22 21:47:07)

Offline

#12 2009-03-22 00:36:31

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Bittorrent & Pacman Idea


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#13 2009-03-22 00:38:50

skottish
Forum Fellow
From: Here
Registered: 2006-06-16
Posts: 7,942

Re: Bittorrent & Pacman Idea

Offline

#14 2009-03-22 01:26:19

godane
Member
From: NH, USA
Registered: 2008-02-03
Posts: 241
Website

Re: Bittorrent & Pacman Idea

Can't we just added in rsync support? This could allow us to save bandwidth by updating the old packages in are cashe folder.

More like copy older package with new name/version then add a .rsync at the end untill it fully updated with diff.

Could be better then xdelta since you have make it before hand for it to work.

PS My understanding is that rsync does it on the fly. Where xdelta, you have make the diff of the package on server then download and manually added it in.


I'm working on a live cds based on Archlinux. http://godane.wordpress.com/

Offline

#15 2009-03-22 21:17:07

z.s.tar.gz
Member
From: Rural Florida
Registered: 2008-12-12
Posts: 298

Re: Bittorrent & Pacman Idea

I don't know. It was just an idea.
Seems that it has more problems than solutions, so 'oh well'


I need to find a way out so everyone can find their way out.
Resregietd Lunix Uesr: 485581

Offline

#16 2009-03-22 22:02:12

haxit
Member
From: /home/haxit
Registered: 2008-03-04
Posts: 1,247
Website

Re: Bittorrent & Pacman Idea

This could be potentially dangerous.


Archi686 User | Old Screenshots | Old .Configs
Vi veri universum vivus vici.

Offline

#17 2009-03-23 00:26:36

z.s.tar.gz
Member
From: Rural Florida
Registered: 2008-12-12
Posts: 298

Re: Bittorrent & Pacman Idea

Yes it could be. But then again, what can't be dangerous?

The concept is good, but doesn't pose that big of an advantage over standard pacman, so at the moment it is somewhat pointless to implement bittorrent into package management.

Last edited by z.s.tar.gz (2009-03-23 00:29:14)


I need to find a way out so everyone can find their way out.
Resregietd Lunix Uesr: 485581

Offline

#18 2009-03-23 01:14:03

CheesyBeef
Member
Registered: 2008-06-04
Posts: 190

Re: Bittorrent & Pacman Idea

On the other hand, we could use Bittorrent to transfer distro ISO releases!  Has anyone thought of that yet?!?

Offline

#19 2009-03-23 01:18:16

skottish
Forum Fellow
From: Here
Registered: 2006-06-16
Posts: 7,942

Re: Bittorrent & Pacman Idea

CheesyBeef wrote:

On the other hand, we could use Bittorrent to transfer distro ISO releases!  Has anyone thought of that yet?!?

http://www.archlinux.org/download/

Offline

#20 2009-03-23 01:41:06

CheesyBeef
Member
Registered: 2008-06-04
Posts: 190

Re: Bittorrent & Pacman Idea

skottish wrote:
CheesyBeef wrote:

On the other hand, we could use Bittorrent to transfer distro ISO releases!  Has anyone thought of that yet?!?

http://www.archlinux.org/download/

no wai.

tongue

Offline

#21 2009-03-23 10:18:34

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: Bittorrent & Pacman Idea

Would it be that hard to implament? Ie, (thinking out loud here)

Xyne (Thank you for all your great work btw) has a pkgd daemon which distributes packages over LAN. If you have a central tracker (could be hosted on gerold) which creates a torrent when a package is uploaded to ftp.archlinux, it can be SHA/MD5 summed at the tracker and the torrent keeps its own integrity check so it would be (in therory) pretty hard to spoof the download. Back to Xyne's pkgd, it does what we need already, a central server to distribute files, and all that would need distributing would be the torrent files, then some backend to initiate the download via torrent to the package cache and then run pacman again.

I might have a go at dry coding something later

Offline

#22 2009-03-23 13:22:51

genisis300
Member
From: Uk
Registered: 2008-01-15
Posts: 284

Re: Bittorrent & Pacman Idea

i'd be interested in helping out with this even it's a proof of concept as i think to work out if this is even worth doing is to test it out.

It's a very intersting idea.


"is adult entertainment killing our children or is killing our children entertaining adults?" Marilyn Manson

Offline

#23 2009-03-24 05:52:23

Xilon
Member
Registered: 2007-01-01
Posts: 243

Re: Bittorrent & Pacman Idea

I'm not sure that BitTorrent would be the best solution. A P2P model would work, sure, but BitTorrent requires .torrent files. Maintaining these .torrent files would be tedious. Every single package would require one of these .torrent files, otherwise there's no way of knowing what to download or from who (well is some form of magnets for bittorrent but I haven't looked into it). You can't just share files in a repo-wide .torrent, and you can't just share your cache, because the torrent will quickly get invalidated and people won't be able to finish it (DHT might help).

A more traditional P2P system would work well though. Having some daemon running in the background, sharing files in the cache would work. There would still need to be some way of identifying specific packages and versions. I'm not sure that $pkgname-$version-$release-$arch.pkg.tar.gz would be sufficient, as corrupt downloads may occur.

Having a distributed network of caches would be great, but is it really necessary? There are plenty of mirrors for packages, and xdeltas are (at least partially) supported. Those two things already reduce network traffic by a significant amount.

Offline

#24 2009-03-24 07:44:25

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Bittorrent & Pacman Idea

skottish wrote:

There you go using the search function again...

guilty! cool


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#25 2009-03-24 12:17:13

genisis300
Member
From: Uk
Registered: 2008-01-15
Posts: 284

Re: Bittorrent & Pacman Idea

could the tracker not be made by a script as suggested everytime a new package is uploaded a tracker is made at the same time?


"is adult entertainment killing our children or is killing our children entertaining adults?" Marilyn Manson

Offline

Board footer

Powered by FluxBB