Arch Linux

AquaSZS

Member

Registered: 2016-06-26

Posts: 23

[SOLVED] Defending against distributed brute-force SSH attempts

I have been receiving constant SSH login attempts lately, each from different IP addresses on different ports. Thankfully, I have set up RSA key login, so they have nearly zero chance of succeeding. However, the sheer volume of these attempts still concerns me and clogs system logs.

What can I do to mitigate this? Conventional suggestions like Fail2Ban won't work in this case, since the attacker never uses the same IP again.

Sample log (this goes on for days!):

Mar 14 00:01:43 march sshd[15526]: Connection closed by 121.229.59.100 port 49818 [preauth]
Mar 14 00:01:47 march sshd[15528]: Connection closed by 46.218.85.122 port 50536 [preauth]
Mar 14 00:12:34 march sshd[15534]: Connection closed by 158.174.171.23 port 53174 [preauth]
Mar 14 00:34:50 march sshd[15540]: Connection closed by 14.29.177.149 port 52466 [preauth]
Mar 14 00:38:52 march sshd[15542]: Connection closed by 187.217.199.20 port 49862 [preauth]
Mar 14 00:41:30 march sshd[15545]: Connection closed by 51.38.48.242 port 46436 [preauth]
Mar 14 00:42:22 march sshd[15548]: Connection closed by 202.107.238.14 port 33608 [preauth]
Mar 14 00:43:12 march sshd[15550]: Connection closed by 159.65.12.204 port 36232 [preauth]
Mar 14 00:50:39 march sshd[15553]: Connection closed by 201.192.152.202 port 60492 [preauth]
Mar 14 00:51:47 march sshd[15555]: Connection closed by 157.245.110.95 port 36942 [preauth]
...

Last edited by AquaSZS (2020-07-17 02:17:07)

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,334

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Yep.  I do use sshguard just to cut down on noise in the journal; and to provide a level of satisfaction as addresses are banned.

But, it is like farting in the wind.  Nothing beats locking down sshd by denying password logins.

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

kokoko3k

Member

Registered: 2008-11-14

Posts: 2,426

Re: [SOLVED] Defending against distributed brute-force SSH attempts

1) It is not "on", but "from" different ports.
...so changing the listening  port from time to time should help.

2) if it is feasible for your use cases, check port knocking too: https://wiki.archlinux.org/index.php/Port_knocking

Last edited by kokoko3k (2020-03-16 07:19:27)

---

Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

seth

Member

Registered: 2012-09-03

Posts: 60,969

Re: [SOLVED] Defending against distributed brute-force SSH attempts

In addition and if legit inbound is geographically limited: https://linoxide.com/linux-how-to/block … ip-addons/
While not a real security measure, moving away from ports 22 & 443 will keep the kids out.

You want to establish the IP/port blocks as far out as possible (eg. in your consumer router if this is a private system) to prevent DDoS effects.

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

kokoko3k

Member

Registered: 2008-11-14

Posts: 2,426

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Yep.  I do use sshguard just to cut down on noise in the journal

Can i ask you how you kept ssh logs out of systemd journal (if i understood properly!)?

---

Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

herOldMan

Member

Registered: 2013-10-11

Posts: 154

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Reject them at the gateway not the workstation.

I have pFSense firewall/gateway/routers deployed in several locations. They allow you to ban by subnet, country, and blocklists of bad actors.

AquaSZS

Member

Registered: 2016-06-26

Posts: 23

Re: [SOLVED] Defending against distributed brute-force SSH attempts

I tried setting up iptables following this wiki page, but I am still getting ssh attempts at random ports from random IP addresses. I thought this INPUT chain would reject all TCP connections other than the 2 ports I specified. Am I missing something here?

Here is my filter table. (Other tables omitted. Port numbers are arbitrary.)
Docker adds its own rules, but it shouldn't affect my INPUT chain.

# Generated by iptables-save v1.8.5 on Wed Jul 15 11:16:57 2020
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [83497:11921602]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -p tcp -m recent --set --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset
-A INPUT -p udp -m recent --set --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A TCP -p tcp -m recent --update --seconds 60 --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset
-A TCP -p tcp -m tcp --dport 1234 -j ACCEPT
-A TCP -p tcp -m tcp --dport 4321 -j ACCEPT
-A UDP -p udp -m recent --update --seconds 60 --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Wed Jul 15 11:16:57 2020

Last edited by AquaSZS (2020-07-15 02:38:15)

seth

Member

Registered: 2012-09-03

Posts: 60,969

Re: [SOLVED] Defending against distributed brute-force SSH attempts

sudo iptables -L -n -v --line-numbers

won't require you to parse the rules.

Is this still the same log pattern?

It is not "on", but "from" different ports.

Also: sshd isn't running inside/listening on the docker device, is it?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

AquaSZS

Member

Registered: 2016-06-26

Posts: 23

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Yes, the sshd log looks similar. Various unique IPs trying various ports.

Docker is only used occasionally to run some applications locally. No networking services like SSH run via Docker.

Here is `iptables -nvL --line-numbers`.

Chain INPUT (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     8644 5069K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
2       94  5640 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
3        3   120 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
4        1    48 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8 ctstate NEW
5     9884 1762K UDP        udp  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW
6       25  1220 TCP        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 ctstate NEW
7       12   504 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: SET name: TCP-PORTSCAN side: source mask: 255.255.255.255 reject-with tcp-reset
8     2895  536K REJECT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: SET name: UDP-PORTSCAN side: source mask: 255.255.255.255 reject-with icmp-port-unreachable
9     2232 71424 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-proto-unreachable

Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
2        0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
3        0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4        0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
5        0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
6        0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 9223 packets, 1273K bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
2        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
2        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain TCP (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        8   416 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: UPDATE seconds: 60 name: TCP-PORTSCAN side: source mask: 255.255.255.255 reject-with tcp-reset
2        5   300 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:1234
3        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4321

Chain UDP (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1     6989 1226K REJECT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: UPDATE seconds: 60 name: UDP-PORTSCAN side: source mask: 255.255.255.255 reject-with icmp-port-unreachable

I checked /proc/net/xt_recent and the ssh attempts do not seem to go through the TCP chain. Which of these is happening here?

1. SSH attempts are not handled by iptables as TCP packets.

2. My iptables rules are wrong for handling SSH attempts.

3. iptables is actually working. 'Connection closed by...' log messages are expected.

I will try setting up geographical iptables and see how that goes, but I am curious what is wrong with my understanding of iptables.

seth

Member

Registered: 2012-09-03

Posts: 60,969

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Mar 14 00:01:43 march sshd[15526]: Connection closed by 121.229.59.100 port 49818 [preauth]

49818 is the *remote* port, not your local one.

The message is from sshd, sshd will only listen on particular local ports and only respond to TCP inbound on those ports (by default 22)
The different PIDs are because ssh is socket activated (ie. sth. has to go in on eg. port 22 to trigger the ssh process)

The accepted inbound is

1     8644 5069K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED # any previously established connection
2       94  5640 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           # anything on the loopback device
4        1    48 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8 ctstate NEW # echo requests ("ping")
3        0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED # docker transfer
5        0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0            # docker transfer
6        0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0            # docker transfer
2        5   300 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:1234  # tcp inbound on 1234
3        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4321  # tcp inbound on 4321

Are 1234 & 4321 the real actual values and/or did you move sshd to those ports?
Also please take docker out of the equation, to ensure that sshd isn't (also) listening on the docker0 device.

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

drcouzelis

Member

From: Connecticut, USA

Registered: 2009-11-09

Posts: 4,092

Website

Re: [SOLVED] Defending against distributed brute-force SSH attempts

While not a real security measure, moving away from ports 22 & 443 will keep the kids out.

I understand that "obscurity is not security", but by having my SSH server on a non-standard port I have not had a single unknown attempt at access in 10 years, which may be the only other thing that is needed in this case to defend against distributed brute-force attempts.

seth

Member

Registered: 2012-09-03

Posts: 60,969

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Yeah, it is a *very* effective way to keep the kids out.
But you'll have to stress that it's not a security measure, because it's often sold as such - and it will do shit against a pro-attack.

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,334

Re: [SOLVED] Defending against distributed brute-force SSH attempts

I have updated my system since last I posted in this thread.  And, I have updated my router as well.   I no longer run sshguard, I still require keys (no password logins).  No root logins by ssh.   The router has some nice firewall features -- There are seven countries that are blocked at the router, and I estimate I see only a couple percent of what I had seen before.  With those seven countries blocked, the majority of the remaining attempts I see are from the Netherlands and the United States.

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,330

Website

Re: [SOLVED] Defending against distributed brute-force SSH attempts

... I have not had a single unknown attempt at access in 10 years

This may be in part due to using a nonstandard port, but this will also be hugely influenced by your server's web presence.  Is your server's domain/IP linked to by other sources, or in otherwords, how discoverable is it.

It wouldn't matter what port amazon.com switches to, I'm pretty sure they'll still get obscene numbers of attempts, while the server I put up in the basement could listen on port 22 for years and never get an outside ssh attempt.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

AquaSZS

Member

Registered: 2016-06-26

Posts: 23

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Mar 14 00:01:43 march sshd[15526]: Connection closed by 121.229.59.100 port 49818 [preauth]

49818 is the *remote* port, not your local one.

The message is from sshd, sshd will only listen on particular local ports and only respond to TCP inbound on those ports (by default 22)
The different PIDs are because ssh is socket activated (ie. sth. has to go in on eg. port 22 to trigger the ssh process)

Are 1234 & 4321 the real actual values and/or did you move sshd to those ports?
Also please take docker out of the equation, to ensure that sshd isn't (also) listening on the docker0 device.

OOOH. So that's what everyone was trying to tell me! Those large port numbers being client output ports makes a lot more sense.

I am using a different non-standard ports than 1234 and 4321. Those were just examples. Before I try GeoIP, I will try changing the listening ports and see how that affects the traffic.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,492

Re: [SOLVED] Defending against distributed brute-force SSH attempts

If you have IPv6 perhaps you could put sshd on a randomized address. If you have OpenVPN/Wireguard you could remove sshd from the public interface altogether.

A completely different approach would be to simply disable logging of fruitless login attempts. ;-)

Otherwise just live with it. I'm using a non-standard port myself - which adds a lot of inconvenience, I have aliases for ssh, scp, rsync, ... just so I don't have to specify it every time.

But even so it gets some login attempts from time to time. Nothing you can do about it.

At least for IPv4 it's also not directly related to webserver presence. Even if you sign up for a completely new machine new IP, you get login attempts long before you use it for anything. Those ssh bots are walking through the entire IP address space (the subset of it that is assigned to various ISPs and makes for a relevant target). They just make countless attempts all over the place so they find the one in a million successful one, like spammers sending just as many mails. It's just the kind of background noise you have to live with...

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,330

Website

Re: [SOLVED] Defending against distributed brute-force SSH attempts

I have aliases for ssh, scp ... just so I don't have to specify it every time.

Aliases?  Why haven't you simply set up your ssh config?

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

frostschutz

Member

Registered: 2013-11-15

Posts: 1,492

Re: [SOLVED] Defending against distributed brute-force SSH attempts

Why haven't you simply set up your ssh config?

Good point, guess I should do that :-)

drcouzelis

Member

From: Connecticut, USA

Registered: 2009-11-09

Posts: 4,092

Website

Re: [SOLVED] Defending against distributed brute-force SSH attempts

... I have not had a single unknown attempt at access in 10 years

This may be in part due to using a nonstandard port, but this will also be hugely influenced by your server's web presence.  Is your server's domain/IP linked to by other sources, or in otherwords, how discoverable is it.

Haha Yeah, my website has approximately zero discoverability. Pretty much just for some family and friends.

AquaSZS

Member

Registered: 2016-06-26

Posts: 23

Re: [SOLVED] Defending against distributed brute-force SSH attempts

After simply changing my ports, the attempts have stopped.

I was using non-standard ports before, but I guess it just got guessed somehow. With port scan protection rules set, it is unlikely that would happen again.