You are not logged in.

#51 2009-09-07 14:08:59

wantilles
Member
From: Athens - Greece
Registered: 2007-03-29
Posts: 327

Re: pkgd: a daemon for sharing pkgs over your LAN

Allan wrote:

Also, try doing a "pacman -Syu" on multiple systems with that setup.

As I pointed out earlier above, it is very easy to do a "pacman -Sy", once per architecture, only on the server, via ssh.

Then you can do a "pacman -Su" on each client, in turn, via ssh again.

Allan wrote:

Two pacman instances trying to download the same package to the cache at the same time really makes things interesting. I ran into that while using multiple chroots with the same pacman cache.

Then do not do it at the same time, but do it in turn.

Again, you will never have to download a package more than once.

Offline

#52 2009-09-07 14:53:08

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

this should save a lot of bw from the mirrors... nice package...

Offline

#53 2009-09-07 15:48:14

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

@wantilles
I think your method adds a level of complexity that isn't necessary. You think the same of my method. It's really a matter of opinion and it's up to each user to decide what's right for them. There's really no need for you to push your method further in this thread and I do not feel inclined to argue with you over what I consider to be misunderstandings.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#54 2009-09-07 16:26:13

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

in the manpage, its not clear how you set up master/slave servers...maybe it needs a bit more clarification...

under my setup, ive got one server running 247, and clients that connect to it..

server+clients have pkgdd running and in their settings, they point to the server.

is this all that is needed? or should i just setup pkgdd on the server, and the clients which dont talk to each other dont need the daemon? (only pacman.conf tweaked)? i thought that if the server would require the clients to serve pkgs to it, they would need the daemon running too. is this asumption correct?

EDIT: should the daemon script test for root privileges? when run without those, it fails and drops a screenload of errors without exiting gracefuly..

Last edited by eldragon (2009-09-07 18:16:55)

Offline

#55 2009-09-08 07:06:49

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I'll update the manpage. *edit* Done.

You need to set up a master pkgd on one server and then set up slave pkgds on all of the other servers and point them to the master pkgd. The master pkgd can only retrieve packages from itself and the slave pkgds that it knows about.

If you don't want to run the daemon as root then you will need to start it on a non-privileged port (>1024).

Last edited by Xyne (2009-09-08 07:34:49)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#56 2009-09-08 09:52:46

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

If you don't want to run the daemon as root then you will need to start it on a non-privileged port (>1024).

the piece that is unclear is the master-slave setting...i managed to do this myself. (aka, set a server an slave daemons...) but this master-slave naming is not mentioned anywere (what qualifies as master, what qualifies as slave, etcetera)...

im running the daemon on the default unprivileged port. the write access to the cache dir seems to be troublesome. and i dont want to leave it world-writable. there was some other control file that complained, i dont remember which.

Offline

#57 2009-09-08 10:37:33

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

As noted in my post, I've updated the man page.

I don't understand why pkgd needs write privileges on your system. Unless I've completely forgotten something in the way that I've coded it, it should only need to read from the cache. What makes you think it needs write privileges?

Anyway, update and read the new man page then let me know if it makes things clearer. Check the info page too as it might make it clearer what pkgd actually does.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#58 2009-09-08 22:32:56

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

As noted in my post, I've updated the man page.

I don't understand why pkgd needs write privileges on your system. Unless I've completely forgotten something in the way that I've coded it, it should only need to read from the cache. What makes you think it needs write privileges?

Anyway, update and read the new man page then let me know if it makes things clearer. Check the info page too as it might make it clearer what pkgd actually does.

thanks, just received the upgrade...the man page is a lot clearer now wink

one question: should there be an advice on laptops running pkgdd? im setting a policy to start/stop the daemon when connecting to my network only but this seems quite hackish to me.
maybe a desirable feature would be to support authentication. i know this is not needed within trusted lans...but laptops could be troublesome.

Offline

#59 2009-09-08 22:55:22

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

How would authentication work? The current set-up requires pkgd to act as a very limited http server that serves packages to pacman based on http requests (as well as a respond to a custom protocol to query packages on slave server). I do not see a simple way to provide authentication within that framework.

I'm open to ideas, but it seems that if you can configure your system to only run it when on your LAN then that should be the best solution. After all, there's no point in running it off the LAN as it would only check your own packages, which pacman does already.

If it's just an issue with the "404 Not Found" errors, then you can use powerpill (in [community]) to avoid them completely.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#60 2009-09-08 23:04:39

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

hmmm, you are partially right. having pkgd running is not an issue. but i found a security issue with having pacman querying a lan server within an uncontrolled environment. if there is no way to authenticate the server against a private key, it could be easily spoofed, and pacman would end up receiving phony updates (rootkit hazzard?). is it hard to add a public/private key pair verification protocol like ssh does?

EDIT: i just realized what im asking is actually out of the scope of your program sad no easy way without messing with pacman....or making pacman connect to localhost and actually make the connection with pkgd which might convolute things .... so dont mind me.. maybe this small security issue could go in the man page too wink

Last edited by eldragon (2009-09-08 23:11:27)

Offline

#61 2009-09-09 00:45:11

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

You could improve the security of that setup by removing the pkgd servers from pacman.conf and adding them to powerpill.conf. That would guarantee that pacman would never try to download a database from pkgd server so there would be no risk of accidentally pulling one from a spoofed server (powerpill never touches the database and pacman would not know about the pkgd servers). Powerpill would still query the pkgd servers when downloading packages but it would check them against the md5sums in the database that you pulled from the official mirror. Although that might not offer NSA security, it should be good enough to let you sleep at night. wink

After all, the chances that someone is going to run a pkgd server on some random network with malicious packages that have been engineered to generate the same md5sums as the official packages is fairly remote.

You could also look at using a local proxy and tunnelling the connection through ssh. I don't actually know how to set that up but I think it's possible. If you do do that, let me know as I would be interested to know how.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#62 2009-09-09 13:03:28

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

i dont know if what im proposing here is even possible or it would require a lot of code rewrite, but could it be possible to do the following?

get clients on the lan connect directly to its own pkgd daemon, and have this daemon do the talking with the master server? in this way authentication / encription, etcetera could be handled by the daemons transparently...

what do you think?

Offline

#63 2009-09-09 15:06:45

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've considered that before but then I would need to pass the pacman configuration file around so that each pkgd would know which repos it was supposed to talk to. It would also be problematic in cases where one would prefer a certain download manager that isn't available on the pkgd server.

The best way to do that would probably be to have a pkgd running on the requesting server so that the download, if necessary, can be passed back to it instead of passed off to another server. Aside from making the configuration file available, it would also avoid duplicating packages unnecessary. It would still require code though both to keep track of which which slave pkgd is running on the requesting computer and a way for each pkgd to access the host's pacman configuration file, including custom ones specified on the command line at run time, which pacman does not pass along with the request to the pkgd.

At best I could make something like that work with a custom wrapper but I can't currently make that work with pacman itself.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#64 2009-09-23 02:41:40

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

A couple of feature requests:

- When logging a "dropped request", could you log $client_ip? It'd be nice to know who's sending me bogus data without resorting to tcpdump.

- Could the logfile use unbuffered IO so I can read it in real time?

- Since this listens on an unprivileged port by default, could the initscript start it as user "nobody" instead of root? Maybe even have a "User" config option so it can drop privs on startup? (Actually, it looks like it would need its own user and group, since we can't su to "nobody" or "http".)

Last edited by ataraxia (2009-09-23 02:47:53)

Offline

#65 2009-09-23 04:48:35

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've added the client ip to the "dropped request" line.

You can run pkgd in the foreground and watch the output in real time that way. I'll consider adding an option to unbuffer log output, but that involves changing the common log module and adding options to pkgd to control it.  Running it in the foreground just seems simpler to me at the moment.

I'll look into creating a pkgd user and group. Feel free to bump the thread with the name of a package that I can use as a template for setting that up.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#66 2009-09-23 14:21:45

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

I thought maybe the shared nature of the Log module might make the unbuffered request a bit more trouble than it's worth.

The "boinc" package in community has an install script the creates a user and group for it, and an initscript that uses "su -c" to start it with that user.

Offline

#67 2009-09-23 16:20:24

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've added a pkgd user and group for running the daemon. Let me know if there are any problems.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#68 2009-09-23 20:06:54

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

I've added a pkgd user and group for running the daemon. Let me know if there are any problems.

The start method of the initscript frequently returns failure when pkgd actually started properly. I'm guessing it's a race between starting pkgd and looking for the PID - it might not be totally up yet when the PID request comes in?

Other than that, looks like it works great!

Offline

#69 2009-09-23 21:46:51

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

Sorry to say, but I've found a showstopper. There are tools that break if your first pacman mirror isn't a "real" or "full" mirror (i.e., they won't go on to the next "actual" mirror and try again). One of these is "pkgfile -u" from pkgtools, which I depend on. Even though my reading of the code shows that it shouldn't do that, it's what I see.

Adding to that is my nervousness in using pkgd over the internet at large rather than just a LAN. (This is what actually makes it useful, but it doesn't feel safe.)

By all means, I thank you for what you've done for me so far (on this app as well as your others). Hopefully others find my feature requests valuable so it wasn't a waste of your time.

Offline

#70 2009-09-23 23:31:57

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've added a loop to the daemon launcher to wait up to 5 seconds for pkgd to start so the race problem should be solved. Thanks for reporting it.

I've also move the pkgd user home directory as I realized that there was a possible security risk with the previous one.


I don't recommend using pkgd over the internet so I'm not going to argue for that. I don't think there is any real risk for exploitation considering how simple it is and I do keep an eye out for exploits when I code, but I just don't have the experience to be able  to guarantee that it is secure. You could always try to tunnel it through ssh or run it over a vpn. If you do, I would be interested in how it works out.


Anyway, even if you don't use it and no one ever even notices the new features, at least I learned a little when implementing them, so it wasn't a complete waste of time.



Out of curiosity, is "pkgfile -u" able to retrieve packages from pkgd itself? Does it work when regular mirrors lack the file?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#71 2009-09-24 00:16:53

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

Last night I tried fuzzing pkgd with some random crap, and some valid-looking requests with random noise in the URLs, and it did the right thing in each case. For too-long URLs and non-GET input, it dropped them. For input that could have been valid package requests, it tried to serve them. In no case did it crash, hang up, write data to the local machine, or serve data that it shouldn't. My security requirements are a bit high - the only thing I run that listens is sshd, which I run on a nonstandard port and accept only pubkey logins with. Compared to that, anything that serves potentially large files to anyone who knows what to ask for makes me feel naked lol

"pkgfile -u" downloads the *.files.tar.gz from pacman's first mirror and unpacks them to /var/cache/pkgtools/lists. It finds the mirror by parsing the output from "pacman --debug". It doesn't do any downloading of actual packages. It would break just the same if my first actual mirror didn't have the files.

Offline

#72 2009-10-14 17:08:02

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

on one of the updates, the user pkgd was created, and it was given a shell, this all nice and dandy, but it appears as the list of users allowed to login. something i dont like much wink.

i edited /etc/passwd and replaced the shell from /bin/bash to /bin/true or one of the equivalents. but when this is done, the service does not start.

i know its set with a ! and it cannot actually login, gnome 2.28 is quite stubborn and still lists it.


on the security side of things... i was wondering if it was feasable to make each computer in the network try to coonnect to its own pkgdd@localhost, and make all communications through them. this could allow login / encryption and what not.. similar like how unison works.

just a thought.

Offline

#73 2009-10-14 20:41:24

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,449
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

@eldragon
How do other packages create custom users for their daemons? I used "boinc" as a template when I implemented the user as requested above but didn't look into it beyond that as it worked. I'll try to figure this out soon but it's one of those things that might take longer than expected and I don't have the time right now. Post a solution if you find one and want me to implement it.

I don't know how unison works but I actually have thought about passing through a local pkgd before. A local pkgd would be capable of checking the host's pacman.conf and downloading packages on the fly transparently, thus avoiding the error. It would increase the complexity of the code though so I ultimately decided against it.

I don't know if it would really be worth it for security though. In nearly all cases the packages in the cache are not sensitive and pkgd is not accessible outside of the LAN. Although I would like to learn more about secure file-sharing etc, it also seems that building in extra security to enable deployment on an insecure network (read: a network where you're actually worried that someone might get a package in your cache) would be turning pkgd into a much more general p2p app. As the current functionality can be secured through a vpn or ssh tunnel, I don't think it's necessary to make it more secure right now.

Having said/written that, I do like the idea of a secure p2p package sharing app that would effectively create a megarepo from all the caches of users running the application. That would be great for getting older versions of packages, lessening server load, etc. Security would definitely be a serious consideration in that case.



*edit*
I should also mention that sensitive packages can be kept in a separate cache that pkgd doesn't use. You can specify multiple caches in pacman.conf and on the command line so you don't even have to worry about pkgd sharing a sensitive package.

Last edited by Xyne (2009-10-14 20:44:50)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#74 2009-10-14 21:15:44

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pkgd: a daemon for sharing pkgs over your LAN

You can "cheat" by having the shell set to /sbin/nologin or some equivalent thing, and still be able to run things with su under that account, if you use the "-p" flag (in addition to "-c") to su.

Offline

#75 2009-10-14 21:45:07

eldragon
Member
From: Buenos Aires
Registered: 2008-11-18
Posts: 1,028

Re: pkgd: a daemon for sharing pkgs over your LAN

ataraxia wrote:

You can "cheat" by having the shell set to /sbin/nologin or some equivalent thing, and still be able to run things with su under that account, if you use the "-p" flag (in addition to "-c") to su.

thats what i was going to check right now. how to get su to work with a user with no shell wink

will tweak and report back

Offline

Board footer

Powered by FluxBB