You are not logged in.

#1 2003-12-03 20:28:56

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Arch's server security

Well, the Gentoo Servers just got hacked. This follows Debians and the LK's servers getting broken into.

I was just wondering what type of security is in place on the Arch Servers? I think the smaller distros may become targets soon too, as we can be caught "with our pants down".

Yeah, so I think we should prepare, if we haven't already.

sad


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#2 2003-12-04 00:03:23

zen_guerrilla
Member
From: Greece
Registered: 2002-12-22
Posts: 259

Re: Arch's server security

Some info on the debian attack :
Andrew Morton discovered the 'do_brk' problem back in 25/09/2003 & a fix for it was first implemented in 2.6.0-test6, but didn't catch up with 2.4.22 release & no advisory was issued since it wasn't remotely exploitable.
The attackers of debian servers used old/unused, sniffed passwords to log in & then gained root perms via the 'do_brk' exploit, applied the SucKIT chkrootkit (which alters /sbin/init among others) & thus sniffed more passwords. Then used the exact same method to gain access to the other 3 servers.

The odd thing is that they (the attackers) didn't touch the pkg repositories, so IMHO they either hadn't enough time or were planning to use these hosts to make a DDoS to another site (e.g. M$ - conspiracy theory smile) & create bad press (from the same ignorant journalists that were warning everyone about the sco case smile) for linux...

Offline

#3 2003-12-10 23:47:41

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

Sorry to revive this thread, but I didn't recieve an answer exactly.

The gentoo servers were running an IDS, package signing, etc. Basically, has Arch taken these types of measures?

I secure my machine decently (as much as an average user can), but I am basically at the will of all the arch developers. I trust you guys, and I like to know if you guys do things to help verify that the packages on your servers are untampered.

I guess this question is mainly aimed towards Judd, as I believe he was the one to set up the server(s).


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#4 2003-12-11 17:46:26

kakabaratruskia
Member
From: Santiago, Chile
Registered: 2003-08-24
Posts: 596

Re: Arch's server security

But what does the md5sums check, the integrity of the arch package, or the integrity of the program, in the pkgbuild. Cause if it just checks the program integrity, someone could just put code like: "rm -r /" (well, maybe that one not), in the pkgbuild.


And where were all the sportsmen who always pulled you though?
They're all resting down in Cornwall
writing up their memoirs for a paper-back edition
of the Boy Scout Manual.

Offline

#5 2003-12-11 21:27:09

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

MD5sums are not good enough.

The MD5sums and the package are on the same server, so if it gets hacked, the person can simply replace the MD5sum.

What would be better is PGP signing, as thats a whole lot harder to crack.


Yes, I could have emailed judd, but this should be discussed publically. I guess the mailing list was a good place, sorry. tongue


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#6 2003-12-11 21:45:01

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

contrasutra wrote:

MD5sums are not good enough.

The MD5sums and the package are on the same server, so if it gets hacked, the person can simply replace the MD5sum.

What would be better is PGP signing, as thats a whole lot harder to crack.


Yes, I could have emailed judd, but this should be discussed publically. I guess the mailing list was a good place, sorry. tongue

Well then, you have the source.  I'll expect a patch... what, tomorrow?

Where would you store the PGP keys?  On the server?  Or the developer's machines?  I bet most of the developers don't have PGP keys or if they do they don't use them for very much.  It's another leap to change our package process.


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#7 2003-12-11 23:07:31

RdsArts
Member
Registered: 2003-10-17
Posts: 32

Re: Arch's server security

Fools! PGP keys are the opiate of the masses. No, there is only one way to secure the server.

First, all packages must be handled by bittorrent. Then, we send the updates to the package list. How?
Bittorrent.
Ofcourse, that will need updates. And how will be do that?
Bittorrent?
NO!
Sneakernet. A small team of highly trained and bonded group of elite software ninjas which will provide USB disk and floppy patches against the main tree at a rate of one per hour.

Of course, to ensure that no one can possibly buy off the ninjas, they must be ritualistically killed each hour and replaced with fresh software ninjas.

Ninji..

The plural form is unimportant!

Now, of course, we can't very well store the main directory from which we pull updates in something as "open" as Linux. No! We need to move this to OpenBSD. Then have the server unplugged, covered in sulfiric acid, and patrolled by a team of angry fire ants who are on a 12-on 12-off hour schedule.

We then need to bury this under at least 300 feet of concrete, and fill the keyboard port with some seal. The shinto have a great one for warding off spirits, but I have plans for that elsewhere....

Yes.... Elsewhere...

... what were we talking about again?

Offline

#8 2003-12-11 23:17:17

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

Do you really think a seal could fit in the keyboard port?  I think they might have too much blubber...  Or do you just mean parts of a seal?  Cause that would make sense...

But really... who can trust a computer that exists at all?  We should store all of our packages on parts of a computer... we can then disassemble it and scatter the pieces to various parts of the world.  Updates can come in the form of temporal waves being propagated from my giant head.

The ninja idea was good, but flawed.  What do we do with all the ninja's you've already ordered I wonder...


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#9 2003-12-12 01:18:28

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

I dont see this as a laughing matter.

As it stands now, someone could hack the arch server, post an update to some major package, and even within a few minutes, people could be infected.


PGP signing doesn't have to be part of Pacman.  The signatures are kept on dev machines or on a seperate server. This way, you can run periodic checks (scripted) that compare the PGP keys of the packages on the server, with that of the "trusted" signatures. Im not saying it would be checked every time you download a package.

Its really not that complicated. Most distributions do it.


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#10 2003-12-12 03:26:41

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

Judd may very well have taken great precautions and the server is as secure as it should be, but I dont know that.

Convince me? All you've done is mock me for asking. Yes, I should have emailed judd, but the thread is here, so unless you want to lock it, I wont spam people's inbox. Since you are so sure, enlighten us in how he has secured everything. I trust you guys, and I dont think it's too odd. In the 3 posts you made making light of the situation, you could have just as easily told me the answer to my question.


And if you really want, I can start signing my packages and uploading the signatures. The TUR's can now upload "extra" files, so it's completely possible.

See how easy that was? Yes, I suppose PGP could be cracked (and in some cases, it has), but it's just another layer of protection.

The point of signing is not to prevent hackers, but to provide a way to verify if the package on the server was actually made by the developer.

Now, apeiro could already be doing this, I really don't know.  :?


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#11 2003-12-12 03:58:04

RdsArts
Member
Registered: 2003-10-17
Posts: 32

Re: Arch's server security

Xentac wrote:

Do you really think a seal could fit in the keyboard port? I think they might have too much blubber... Or do you just mean parts of a seal? Cause that would make sense...

Na, you just need to get one that's USB-ready and has a PS2-to-USB converter.

Xentac wrote:

The ninja idea was good, but flawed. What do we do with all the ninja's you've already ordered I wonder...

I told you I had plans...

contrasutra wrote:

I dont see this as a laughing matter.

Everything is a laughing matter. I take a deep interest in security, but I like making jokes more because taking things too seriously leads to making rash decisions. It's much more fun to laugh at things even while your serious. smile

contrasutra wrote:

As it stands now, someone could hack the arch server, post an update to some major package, and even within a few minutes, people could be infected.

But if a computer is world-facing, then it is always open to being hacked. Nothing changes that. PGP keys can be forged, servers compromised for months without anyone knowing, ESR could get root on any SCO server. Bad things happen.

The Debian break in wasn't caught by PGP. It was caught by watching the kernel OOPs, and it happened because someone compromised a developer's password. Gentoo was attacked by a rsync hole. Kernel.org was hacked via a anon CVS. While PGP isn't a bad idea, it won't make anything magickally unhackable.

contrasutra wrote:

PGP signing doesn't have to be part of Pacman.  The signatures are kept on dev machines or on a seperate server. This way, you can run periodic checks (scripted) that compare the PGP keys of the packages on the server, with that of the "trusted" signatures. Im not saying it would be checked every time you download a package.

You still have to source it from somewhere. What if that gets hacked?

PGP is nice, but if it's not needed all it's adding is more complexity ( that will break, as that is the first rule of programming, and Humans Make Mistakes(TM) ). The big problem is that gives a false sense of security. Like the Iraq war and OpenBSD. No matter what you do, it's all a question of how vigilant the admins are, and I'm assuming the Arch admins are excellent.

Besides, if your going to upload a malicious package, your likely already in as root, which means your going to get that script and make it OK your package for you. And if not, then your going to be caught even if they don't because your likely a skript kidde1 who just found out nmap will let your script your "amazing hacks."

There are no silver bullets. Well, there are, but they're literally silver bullets, and as such are not really useful in a discussion about admining. Unless you have a alibi.

Offline

#12 2003-12-12 05:14:16

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

Wait a second... who's private key do we use?  Do all the developers share one?  What if one of their machines is compromised?  How do we securely update the signed packages on the verification machine?  How does that do more than md5sums do (it's still verifying that the package that's there is the package you wanted there)?  Does each developer make their own PGP key?  How do we know that we're not being man-in-the-middled when we get their key?

If the server stores the keys that adds a whole bunch of other issues, so let's just ignore that.

What if a package is updated?  Then the PGP key or md5sum has to be updated immediately, otherwise we'll think that something's up.  How do we do that quickly, automatically?  If it's automatic and initiated by the server then you have the same problems... if it's not then I can't see an easy way that's not error prone.  Can you suggest one?

I take security things seriously and it does bring up a good point.  But what do you expect when you say, "hey guys!  I think things are broken!  You need to fix them!"


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#13 2003-12-12 18:19:54

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

I DONT think things are broken. This was an inquiry thread. I was ASKING. PGP doesnt have to be the answer. No change might be needed.

Here's how I see it working:

Each dev has their own PGP key. All the other developers sign each other's keys with their keys, creating trust. This can actually be done in person maybe. Don't some of you work together?

Now, the public keys are uploaded to a keyserver, it can be run by Arch, or whatever you want.

Each package is now signed.

If the key server gets hacked, and the public keys are replaced, we'll know, because It won't have been signed by all the other developers.

Also, it's a lot easier to compare one key(or 3 or 4) on a server, then EVERY SINGLE md5sum for a package. Because of the way it works, the only way to make the packages look valid (once they've been hacked) is to hack and change the public keys.

Not to mention, that most users would already have the proper public keys on their computer, and would be unlikely to redownload a hacked signature in only a few hours/days. This is unlike a package, where people could be downloading it within a few minutes.


There's probably some flaw in this, but the way I see it, its not TOO complicated. It's a change, yes, but it would make things a little better.


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#14 2003-12-12 18:40:32

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

Only two of the Archlinux developers work together... and one of them isn't very active.  I'm the third that lives in the same city but that's it.

That means that all devs have to have keys.  We also need a place to store the signatures (that makes it easy to compare to the original files).

I don't understand what you're saying about comparing keys instead of comparing md5sums.  Even if we compared keys to find that one of them was changed, we'd still have to look at which packages were signed by that hacked key.  And there's only one md5sum for a package.

This does make it easier to find in the event of breakin but still doesn't stop people from downloading infected packages.


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#15 2003-12-12 21:01:51

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

Let me see if I can explain it:

With md5sums, for someone to properly fool pacman, they would have to:
1. Hack the server
2. Replace the package with the hacked one
3. Replace the md5sum (on the same server), with one for the hacked package

For someone to successfully fool a person with PGP, they would have to:
1. Hack the server
2. Change the package to the hacked version
3. Change the signature to a proper sig for the hacked file. This would require using a non-official signature, so they would also have to...
4. Hack the key server, change the key of EVERY developer, and resign the key with all the "new" dev keys.
5. Get  people to download the hacked keys and replace it with the old keys. This is the hardest part, as your pgp key wont change very often (if at all), so people won't do this on a whim.

The PGP method requires a lot more work, and people are more likely to notice it during the process.

As long as you make sure your copy of the devs' key is proper, then you can never get a hacked file.

Another thing Arch can do, which is probably simpler, is to make sure a package and its md5sum are on seperate servers. Its probability (i think), 2 servers are harder to hack than one.

Also, it takes about 5 minutes to generate a PGP key, and any good mail client can do gpg.


Well, it doesn't have to be "in person" for the key signing, but it can be via a private chatroom, or by mail if you really wanted to be secure. Basically, just not from a public keyring or something.


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#16 2003-12-12 21:29:07

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

I'm aware of the differences between md5sums and PGP key signing.

Picture this situation though.  We implement PGP package signing.  We have all of the packages signed and all the signatures up to date on every upgrade.

Someone hacks into the server and changes the package files.  Before someone with proper authentication is notified (the script runs every 2 hours, let's say), 5 people update their packages.  Those 5 people now have hosed systems.  PGP package signing did nothing for them because there was no check at the package manager level.  Unless you want an email for every updated package that contains the signature file.  At that point why have pacman do any downloading automatically anyway?

After that, we notice the packages were modified and fix them, but the damage is still done.

At that point we're putting more trust in the developer's machines.  It'd probably be easier for an attacker to hack into a developer's machine, make a malicious package, sign it with a proper key, and then send it up.  Then the package looks good and is still bad.  There's always going to be a point of failure.

My point is that what you're suggesting can be done with md5sums handled in a slightly different way.


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#17 2003-12-12 22:01:04

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

OK, a few things:

1. Same problem exists with md5sums.

2. Yes, a devs machine is "hackable", but you are forgetting two things:

a. How does the hacker know where the dev's machine is? Unless it's running a server, its not nearly as easy to find.

b. Even if the person does hack their machine, they have to hack the PGP private signature too. This is EXTREMELY hard, and is very unlikely.

3. You're right, this doesnt prevent downloading hacked packages, but it could be adapted to.

How about pacman uses the dev keys, like it uses md5sums now, but the keys have to have been imported manually. Now, the md5sum checking is automatic, so its easy to get a hacked sum. So you can verify packages against keys, but you cant automatically download a hacked key.

If you are forced to download/update the keys by hand, you are more likely to get accurate ones.  And like I said, someone could run a script every 2 hours to verify the accuracy of the keys.

Arch is not for beginners, so downloading a PGP key shouldn't cause problems.


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#18 2003-12-12 22:10:41

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

1.  Never said the problem didn't exist with md5sums, but it offers the same coverage as PGP keys.  Why make things more difficult than they have to be?

2. a. I'm on IRC, aren't I?  I send email, don't I?  Two easy ways to get IPs of developer's machines.

b. They don't have to hack the PGP signature.  Just hijack the gpg binary/process or even better, just set up a keygrabber.  Then you can capture the password anyway (or just tell gpg to sign these files as well as the other files that it's supposed to sign).  Digital signatures show intent of a computer, actual signatures show intent of a person.

3. Could be adapted to... isn't that why I asked you for a patch in the first place?

That means gpg has to be in base.  You need a way of getting keys in, potentially, without network or nice apps like web browsers or wget.  It creates all sorts of chicken and egg problems.

Anyway... we've done a lot of painting here (you can always paint a picture blacker).  I'm sure that I can always come up with a response to some situation that you give me.  We'll go on for quite some time if it's not stopped now.

To answer your initial question, the best form of package validation we have right now is md5sums.  If you think we need more, then you should figure out a workable solution that's not difficult to implement and offer us an implementation plan listing the things needed, required steps, security concerns addressed, and problems unaddressed.  Otherwise I'm sure we'll get something on our own sometime.


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

#19 2003-12-13 01:17:46

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

I forgot about the email, IM, irc thing tongue. That is a definite problem.


I think the best and easiest way to greatly increase Arch security is to keep the md5sums and packages on a different server, like I said before.

Is this a possibility?

Also, I couldn't patch anything, my BASH is sub-par.

And, like my original post asked, What type of programs do the arch servers run to help prevent/detect intrusion? Are the servers running an IDS? I know that helped the gentoo folks.

Lastly, do you devs compare md5sums of the server packages with the md5sums on your computer? Maybe you could set a cron job for this? Actually, im a hypocrite, I dont check the TUR md5sums. If you guys start to do that every once and a while, i'll be sure to check on the TUR as well.

Im done arguing. tongue


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#20 2004-03-23 22:46:29

kakabaratruskia
Member
From: Santiago, Chile
Registered: 2003-08-24
Posts: 596

Re: Arch's server security

I think the packages in /incoming are not pacman files. They are just tar.gz that contain the PKGBUILD and in some cases the package. So you may want to look at the PKGBUILD before installing anything from incoming.


And where were all the sportsmen who always pulled you though?
They're all resting down in Cornwall
writing up their memoirs for a paper-back edition
of the Boy Scout Manual.

Offline

#21 2004-03-24 03:04:09

tsykoduk
Member
From: A Chair
Registered: 2003-10-31
Posts: 48
Website

Re: Arch's server security

but what I'm reading here is "we can't be 100% secure so why bother"

What I read here was 'the tools that we have are secure. Adding more complexity would not add that much more security. Why make things harder?"

A perfect example of this. I had a client a few years ago that was chasing 9's - you know; how many 9's of up time?

He was at 99.99 uptime ( 3 days a year down). He really wanted that 5th 9 - to get to 8 hours a year down. So, I came in with a proposal - new rack, new servers w a 4 hour response time from the vendor (vendor would keep parts in the depot in town for it), farging huge UPS array. The servers would be clustered, so if any one failed, the backup would step in. Each of the clusters used a single RAID 50 HDD array.

The client loved the idea. Until I gave him the cost. It was quite extreme. Those extra 72 hours of up time were just not worth the head ache.

To come back around somewhat to the topic...  8)

We could design a pacman system/add on that would do wonders. However, the additional trouble that it would cause might make the system unworkable. What we have now works - it's easy, and you know that you got the package unbroken from the server.

Using arch in a corprate enviroment I would NOT point my boxen at the public pacman servers. I would have a development box sitting on my desk. I would test each package agenst my mix of apps before letting it near my production servers.

I would roll my own internal update server - and push the 'certified tested and clean' packages to it. I would then ONLY allow the production boxes to pull from that/those servers.

IMHO security is in the hands of us - the users/sysadmins. If my boxes at home are rooted and trashed, do I sue Arch? Do I sue my firewall vendor? Nope - I learn from the mistake and press on. Perhaps I download updates less frequently. Perhaps I only use abs and build vs the source. Perhaps I stuff a seal into the keyboard interface, say matra's over my computer and pray that the gaki do not steal it...

Who knows?

Offline

#22 2004-03-24 04:44:01

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: Arch's server security

I think my original point was that we DONT KNOW that the Arch servers are secured in the least bit. At this point, all the developers have told us is "they're secure". If they really cared, they would be very open about their server security policy like some other distributions (gentoo, debian).

That was my original point, but the argument is over. Basically, we have to trust the Arch developers.


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#23 2004-03-24 06:11:14

tsykoduk
Member
From: A Chair
Registered: 2003-10-31
Posts: 48
Website

Re: Arch's server security

I agree - I just was trying to rebut the perceved 'why try' post  8)

Offline

#24 2004-04-11 20:01:40

Xentac
Forum Fellow
From: Victoria, BC
Registered: 2003-01-17
Posts: 1,797
Website

Re: Arch's server security

Ok, I'm resurrecting this thread to have the last word.

I've talked to apeiro about package integrity checking and he does this (I've just started): each night he creates a new md5sum list for all the packages.

That way we know which packages could possibly have been modified.  Anyone can do this, if they have a local mirror.


I have discovered that all of mans unhappiness derives from only one source, not being able to sit quietly in a room
- Blaise Pascal

Offline

Board footer

Powered by FluxBB