My arch system got cracked!

print · 2010-03-28 13:25:12

. . . Because I'm an idiot. This is a first for me, that I'm aware of. I have been running Linux for years now and I have never had any security issues before. However, this morning some crackers coming from a computer in Romania brute-forced a test account I left unprotected during a fruitless attempt to install Lexmark drivers a week or two ago.

Luckily, they installed a program called pscan2 that absolutely grinds CPU. If not for that, I probably wouldn't have noticed it at all. I feel like a real newb after this and I have already kicked myself for the obvious (in hindsight) security flaws that were present in my system.

Here's a list of the dumb things I was doing that led to the successful attack:

* Running ssh on port 22
* Not using the OpenSSH

AllowUsers

directive. By the way, why is this not a default option in Arch's packaged sshd_config file? The stock sshd_config file should contain AllowUsers directive. If most people are like me then they only login from one or two non-privileged users. All others should be blacklisted by default.
* Having

sshd : ALL

in /etc/hosts.allow

As you can see this was a recipe for disaster. Luckily I caught them in the act, and I deleted the test user and killed their scanning process. I'm fairly confident that they were scanning other computers and not mine.

I have also run chkrootkit and it reports nothing. Does anyone have any other suggestions for what I should do to ensure the integrity of my system post-breach?

Finally, I can provide a link to a tarball of the files that they installed in /var/tmp if anyone is interested.

Last edited by print (2010-03-28 16:01:09)

gazj · 2010-03-28 13:29:45

Re-install is the only 100% full proof way. They could have left anything on your system anywhere! I would not risk anything less then a re-install

Last edited by gazj (2010-03-28 13:30:11)

wonder · 2010-03-28 13:31:22

wasn't me

print · 2010-03-28 13:33:17

@gazj If I'd discovered it later I'd be inclined to agree with you. However, I have two up-to-date systems here and I compared the md5sums ofthe /bin and /usr/bin binaries on the compromised machine with the uncompromised machine and they are all the same. Plus I caught them after 20 minutes, and they were just running some stupid port scanning crap out of /var/tmp. They wouldn't have been able to escalate privileges that quickly and the permissions on my user directories are all locked down.

@wonder I updated it to note that they hackers were coming from a computer in Romania, not that they were necessarily in Romania. But I wouldn't put it past you . . .

Last edited by print (2010-03-28 13:36:00)

Mahara · 2010-03-28 14:03:08

Finally, I can provide a link to a tarball of the files that they installed in /var/tmp if anyone is interested.

I'd like to see those files if you don't mind.

gazj · 2010-03-28 14:05:47

http://bbs.archlinux.org/viewtopic.php?pid=348015 [+]

Contains scripts for checking for files that are not within pacmans control. It may help

.:B:. · 2010-03-28 14:33:57

print wrote:

.The stock sshd_config file should contain AllowUsers directive. If most people are like me then they only login from one or two non-privileged users. All others should be blacklisted by default.

No it should not. I'm quite sure you picked Arch because of its philosophy, and now that you see the drawbacks of that philosophy, you want to see things adapted. But that's not how it works . If you are talking about the OpenSSH project itself, you can always file a bug report.

The end responsibility lies always with the user. You have to secure your setup yourself.

A few tips (of which you already discovered a few yourself):
- a non-standard port (security by obscurity works fairly well in this case);
- whitelist the allowed users;
- enforce the use of keys;
- monitor the connection attempts (and failed logins) and have the firewall or another application block them after N unsuccessful attempts.

The first and second will keep attackers out for a while; the third one will take ages to crack; and the fourth one will make it even harder. Applications you can use for this are e.g. fail2ban, but you can also set up iptables to handle this kind of thing. Both approaches have their pros and cons.

print · 2010-03-28 14:56:56

@.:B:. I'm using keys; I'm using denyhosts for #4.

As for AllowUsers: I've been using arch long enough to have completely assimilated The Arch Way. However, in this case I think AllowUsers is important enough (and a big enough security hole if you forget to add it in) that it would make sense to include it in the file, even if it's simply commented out. OTOH, if it's a lean sshd_config, the Kerberos and GSSAPI blocks, each of which are several lines apiece can probably be omitted as well .

Ramses de Norre · 2010-03-28 15:12:26

print wrote:

@.:B:. I'm using keys; I'm using denyhosts for #4.

Were you using that before they cracked your machine? If so, your keys got comprimised somehow and I would start thinking really hard how they managed to get access to your keys.

print · 2010-03-28 15:16:12

@Ramses de Norre: I'm using both keys and password authentication. They got in by using a dictionary attack against a test account that I carelessly left exposed.

Ramses de Norre · 2010-03-28 15:18:07

There is no gain in security if you use both keys and passwords.. You should disable passwords and use keys only.

toad · 2010-03-28 15:28:48

.:B:. wrote:

A few tips (of which you already discovered a few yourself):
- a non-standard port (security by obscurity works fairly well in this case);
- whitelist the allowedusers;
- enforce the use of keys;
- monitor the connection attempts (and failed logins) and have the firewall or another application block them after N unsuccessful attempts.

That would be a really cool wiki page called security for those of us not so well versed in those issues.

I am quite confident on changing to a non standard port which I believe is already explained in the openssh wiki but am not at all sure how to do the rest...

Xyne · 2010-03-28 15:31:38

print wrote:

My arch system got phracked!

fixed that for you

Those who do not learn from the overlord's mistakes are doomed to repeat them.

print · 2010-03-28 15:40:16

@Ramses de Norre: Correct, no gain in security -- only gain in convenience. For example, if you don't happen to have your key memorized

Ramses de Norre · 2010-03-28 16:15:35

print wrote:

@Ramses de Norre: Correct, no gain in security -- only gain in convenience. For example, if you don't happen to have your key memorized

Of course, but we were talking about enhancing security by using keys.

brebs · 2010-03-28 19:03:50

gazj wrote:

They could have left anything on your system anywhere!

They got a login into a "test" account. Why do you assume the baddies managed to get a root login? We have to assume that not all kernels have broken security.

Rorschach · 2010-03-28 19:04:19

@print: None of the things you listed in your first post was a misstake. I'm running ssh on port 22 for years on everymachine and that's just fine. Also that everyone can connect to your machine and all user accounts can do so is no problem. The problem is that you choose a password such short, that a bruteforce attack was successfull. Don't do this also just for test-accounts and you'll be fine

gazj · 2010-03-28 19:31:39

brebs wrote:

gazj wrote:
They could have left anything on your system anywhere!
They got a login into a "test" account. Why do you assume the baddies managed to get a root login? We have to assume that not all kernels have broken security.

Why assume that they didn't. Why take that chance? Although I'm sure /var/log/auth.log would give you a good indication

gazj · 2010-03-28 19:34:48

Rorschach wrote:

@print: None of the things you listed in your first post was a misstake. I'm running ssh on port 22 for years on everymachine and that's just fine. Also that everyone can connect to your machine and all user accounts can do so is no problem. The problem is that you choose a password such short, that a bruteforce attack was successfull. Don't do this also just for test-accounts and you'll be fine

I think your right, Make sure you have nothing that is in the dictionary. Just make up some sort of acronym which contains number and caps.

alAslld4m - archlinux A simple lightweight linux distribution for me

A complex password that is simple to remember. Sorry if I'm telling your grandmother how to suck eggs

Last edited by gazj (2010-03-28 19:35:26)

KimTjik · 2010-03-28 20:24:14

I still don't get one thing. You said: "I'm using both keys and password authentication". The likelihood that someone would crack a decent key seems slim, unless you're actually someone with state sensitive data. Thus I wonder what you mean by "both"; either you rely on password (user) or a key with password (you could choose to not be password protected, but why lower security?). It sounds to me like your configuration allows that if key doesn't match it challenge and goes for user password.

brebs · 2010-03-28 20:24:34

gazj wrote:

Why assume that they didn't. Why take that chance?

Because we are ALL assuming that our PCs haven't been hacked. We have to have a reasonable sense of security. Otherwise, you'd have me reinstalling my OS every day, twice on Sundays and immediately before I use online banking.

.:B:. · 2010-03-28 20:57:23

print wrote:

@.:B:. I'm using keys; I'm using denyhosts for #4.

You say 'use', but I said 'enforce' .

Toad: I can always give you a hand if you want to write up a page, but for the moment I'm not really anxious to write up something like that myself (mainly because of the work that would go in it). If you'd like to write it I'd be happy to give you pointers and relevant links though - it's always a lot fresher when you just set it up yourself (and I am by no means an authority on it).

Some pointers:

Whitelist allowed users
Should be exact matches with the usernames on the server), one line per user

AllowUsers JohnDoe
AllowUsers JaneDoe

Enforce the use of keys
To generate a set of keys and enforce the use of keys, check this link.

Block attackers after X failed attempts
Fail2ban is already wikified; so is Denyhosts, a similar application. If you want to use iptables, I think it can be done with the 'recent' module.

Only open port on request
A fifth security measure would be setting up port knocking, but every layer adds complexity (and I find port knocking adds a lot of complexity).

Manage and cache your keys
Finally, to manage your keys, you should install something like keychain, which will launch SSH and GPG agents and cache all your keys for your login session. If you use keys that have identical passphrases (note passphrases are preferred due to added complexity over regular passwords), it will only prompt you once for that passphrase/password, if you make sure it caches all those keys one behiind another. E.g. if you make keychain cache two keys with passphrase A, one key with passphrase B and three keys with passphrase A, it will ask you for passphrase A twice. If you cache five keys with passphrase A, then one key with passphrase B, you'll only be asked for passphrase A once. I have added an alias for this:

[stijn@hermes ~]$ alias keychain
alias keychain='keychain id_rsa-amalthea id_rsa-athena id_rsa-zeus id_rsa-mnemosyne id_rsa-bellerophon id_rsa-arethusa'

Now you might say 'Gosh, another command to run at startup that I can't run automatically', but you already have a net gain when you perform one login over SSH (one per passphrase at least ). So for the length of the session, this is an absolute win, and you can also run scripted transfers with scp e.g. when you're logged in, or mount SSHFS shares without needing user intervention to authenticate, etc. It's pure luxury - and laziness.

Another interesting link that details a lot of this stuff is Samhain's page on brute force SSH attacks.

Feel free to wikify

gazj · 2010-03-28 21:05:12

brebs wrote:

gazj wrote:
Why assume that they didn't. Why take that chance?
Because we are ALL assuming that our PCs haven't been hacked. We have to have a reasonable sense of security. Otherwise, you'd have me reinstalling my OS every day, twice on Sundays and immediately before I use online banking.

Point taken

print · 2010-03-29 00:02:35

@brebs I think you are on the right track. The unix security model is a natural sandbox, and it shows its strength here. That, and I did forensics on the files they installed and in my opinion they are total amateurs.

@Rorschach
You are totally correct, I shot myself in the foot here. However, I think that as .:B:. mentioned, security by obscurity is helpful in the case of these [expletive] script kiddies. I have noticed after switching that the port scans are cut down by about 99.9%.

@KimTjik:

I still don't get one thing. You said: "I'm using both keys and password authentication". The likelihood that someone would crack a decent key seems slim, unless you're actually someone with state sensitive data. Thus I wonder what you mean by "both"; either you rely on password (user) or a key with password (you could choose to not be password protected, but why lower security?). It sounds to me like your configuration allows that if key doesn't match it challenge and goes for user password.

I keep the state secrets on my XP machine;) What I meant is that I allow both password and key authentication. I believe that the way OpenSSH works is that if both are enabled then password authentication is the fallback, at least that's the way it has always worked for me. For example, if I attempt to log in and my keyfile has insufficient permissions, or if I supply the wrong key password a couple of times, the OpenSSH server will then prompt me for my user password. Does that help?

@.:B:. I highly recommend keychain as well.

If somebody wants to start a wiki page I will contribute what I have learned.

And here is a link to the malware for the curious: http://bit.ly/8YccUW

Last edited by print (2010-03-29 00:16:45)

KimTjik · 2010-03-29 10:54:21

I keep the state secrets on my XP machine;) What I meant is that I allow both password and key authentication. I believe that the way OpenSSH works is that if both are enabled then password authentication is the fallback, at least that's the way it has always worked for me. For example, if I attempt to log in and my keyfile has insufficient permissions, or if I supply the wrong key password a couple of times, the OpenSSH server will then prompt me for my user password. Does that help?

That confirms what I suspected. For higher security I prefer to not have password authentication as fallback and use a key with password. I set sshd_conf to only accept key and not to challenge if it doesn't match or is present.

Arch Linux

#1 2010-03-28 13:25:12

My arch system got cracked!

#2 2010-03-28 13:29:45

Re: My arch system got cracked!

#3 2010-03-28 13:31:22

Re: My arch system got cracked!

#4 2010-03-28 13:33:17

Re: My arch system got cracked!

#5 2010-03-28 14:03:08

Re: My arch system got cracked!

#6 2010-03-28 14:05:47

Re: My arch system got cracked!

#7 2010-03-28 14:33:57

Re: My arch system got cracked!

#8 2010-03-28 14:56:56

Re: My arch system got cracked!

#9 2010-03-28 15:12:26

Re: My arch system got cracked!

#10 2010-03-28 15:16:12

Re: My arch system got cracked!

#11 2010-03-28 15:18:07

Re: My arch system got cracked!

#12 2010-03-28 15:28:48

Re: My arch system got cracked!

#13 2010-03-28 15:31:38

Re: My arch system got cracked!

#14 2010-03-28 15:40:16

Re: My arch system got cracked!

#15 2010-03-28 16:15:35

Re: My arch system got cracked!

#16 2010-03-28 19:03:50

Re: My arch system got cracked!

#17 2010-03-28 19:04:19

Re: My arch system got cracked!

#18 2010-03-28 19:31:39

Re: My arch system got cracked!

#19 2010-03-28 19:34:48

Re: My arch system got cracked!

#20 2010-03-28 20:24:14

Re: My arch system got cracked!

#21 2010-03-28 20:24:34

Re: My arch system got cracked!

#22 2010-03-28 20:57:23

Re: My arch system got cracked!

#23 2010-03-28 21:05:12

Re: My arch system got cracked!

#24 2010-03-29 00:02:35

Re: My arch system got cracked!

#25 2010-03-29 10:54:21

Re: My arch system got cracked!

Board footer