Wifi randomly drops over eduroam network

DanielPowerNL · 2017-10-15 18:39:32

My university uses Eduroam for its student wifi. Eduroam is a standard wifi network in use across many universities and educational institutions worldwide.

Network Specifications:
- Authentication: Protected EAP (PEAP)
- PEAP Version: My university doesn't specify, but I've tried Version 0 and Version 1, both work, but randomly drop.
- Inner Authentication: MSCHAPv2

Laptop Specifications:
- Lenovo Thinkpad T440s
- Network Controller: Intel Corporation Wireless 7260 (rev 83)
- Archlinux x86_64
- NetworkManager 1.8.4-1
- Network-Manager-Applet: 1.8.4-1

Logs
dmesg output: https://pastebin.com/raw/fBXfSYCs
- I'm not sure the dmesg output is relevant, since it doesn't output anything when the network drops. The "failed to remov ekey" and "Limiting TX power" messages appear upon reconnecting.
lspci output: https://pastebin.com/raw/D9iqfueY
journalctl -u NetworkManager output: https://pastebin.com/raw/kfytsqjq

Description of the issue:
Connecting to the network works fine. Speeds are normal. For all intents and purposes the wifi works perfect. Until it randomly drops. It could take 15 minutes to drop, or it could drop after just one minute. On rare ocassions I've had it last upwards of 30 minutes. But more often than not it drops within the first few minutes.
When the network drops, there is no indication. network-manager-applet still shows that I am connected. There are no notifications. I simply become unable to connect to any websites.
In order to restore connection, I have to manually disconnect and reconnect. The wifi will then continue to work normally again, until it drops.

Troubleshooting steps attempted so far
- Testing different distributions. The issue also occurs on Ubuntu, Debian, Antergos, Arch-Anywhere, as well as Archlinux.
- Tested a different laptop. This issue also occured on my old laptop, a Thinkpad Edge 15. I'm not sure what network card that laptop had, unfortunately, as the motherboard died and I no longer have it.
- Disabling IPv6
- Setting the options listed under https://wiki.archlinux.org/index.php/Wi … on#iwlwifi
- Reading this: https://wiki.archlinux.org/index.php/Wi … onnections which unfortunately doesn't seem to be related to my issue (maybe I'm wrong?)

Any assistance in this would be greatly appreciated.

Last edited by DanielPowerNL (2017-10-15 19:11:24)

GenkiSky · 2017-10-21 02:06:19

Interesting... The fact that it is reproducible across hardware and distribution kind of sounds like there's either an issue in the router (speaking of which, do you have the same issue at different locations on campus?) or an issue in your WiFi configuration. Chances are it's the latter of course. I do think it's worth sending an email to the IT department to ask, you can confirm that you're using the right authentication options/values. I would also try starting Wireshark/tcpdump and watching the traffic, and see if you see anything funny happening when the network connection dies. Also, are you able to ping the router when it stops working? And just to confirm, when your network connection stops, do you still have an IP address (that was assigned to you via DHCP presumably) that you can ping locally? It might also be interesting to see what happens when you ping that IP address from another device (e.g. your phone, or a school computer on the same network) through the connection dying. These are just some sanity checks.

lukasr · 2017-10-29 21:33:41

Hi,
I am having the same problem. I don't know exactly when the network drops but downloading a large file is not possible.
Do you guys have any solutions?

Actually i don't know but I think I am having this issue since a kernel upgrade but I didn't have time to try it out.

More over I'm having kind a same issue with my bluetooth speaker. They are disconnecting after some seconds.

seth · 2017-10-30 15:09:27

linux-firmware update inbetween? Try to move away the latest versions for your chip (you can also compare the present package to the formerly used one)

lukasr · 2017-10-31 16:21:29

Today it didn't happen, but if it does again I am going to try that. Thank you!

mcku · 2017-11-01 08:06:34

Is this happening on WPA2? I would expect instability in wifi everywhere nowadays. Just guessing.. Some vendors might be updating drivers for APs and STAs, because of the latest WPA2 attacks. AFAIK the key reinstation is part of normal operation and to work without it may degrade performance. Not to mention the bugs that may have been introduced by the fixes. (This is just a hypothesis and needs to be tested. )

DanielPowerNL · 2017-11-03 15:41:21

I apologize for my absense from this thread. I've still been experiencing the issue daily, but have been too busy with assignments and midterms to deal with it. I've been using USB tethering on my phone as a workaround.

luksar's issue does not seem to be related, as his occurred only after a software update.

@GenkiSky I do not think the issue is with the router, since Windows and OSX users do not experience the connection issues that I experience. Everybody complains that the network is slow sometimes, which is to be expected with a network that has as probably 10,000+ devices connected at a time. But nobody on Windows or OSX experiences drops that require manually reconnecting.

I unfortunatelly cannot test pinging anything, because my university's network doesn't allow pings for security reasons. I had someone on the Archlinux IRC explain why this might be a couple years ago, but I don't remember specfics. Trying to ping any ip address from the terminal results in 100% packet loss.

I will check out wireshark to see if I can determine anything from that, but I've never used it before. So I'm not sure I'll be able to make any conclusions from it.

Edit: You asked me to check if I still have an IP address when the network drops. Is there a way I can test that given that pings don't work whether I have a connection or not? What I can tell you is that networkmanager never shows me as disconnected. networkmanager-applet continues to show "Connected" and even updates the signal icon to the router (periodically moving between 2 and 3 bars) even when the connection has dropped.

Last edited by DanielPowerNL (2017-11-03 15:52:23)

seth · 2017-11-03 16:10:45

"ip addr" or "ip route" will show you your IPs
You can try to check the router with "nmap -Pn <router IP>"

DanielPowerNL · 2017-11-03 16:35:52

Thanks, I'll check this the next time the network goes down. I've been connected for about an hour now with Wireshark running and no drops. So I'm not sure if I've been extremely lucky, or if Wireshark's constant monitoring is preventing the network from timing out or something weird like that. It's hard to say at this point, but I'll continue testing. It's very rare for me to hold a connection this long on the network.

I can confirm that nap -Pn <router IP> does show that the host is up when the connection is working.

DanielPowerNL · 2017-11-03 18:12:10

The results are in. My connection dropped a few moments ago.

When watching Wireshark while the connection was up, I saw a constant stream of both Standard Query and Application Data. As well as some packets that looked like "993 -> 53376 [ACK] Seq=5399 Ack=10667...".

Once the connection went down, wireshark continued receiving "Standard Query" packets, but never received any more Application Data or other packets. Just continuous Standard Queries, and ocassionally a [TCP Retransmission].

nmap still found the host and said it was up. But instead of finding it in 0.05 seconds, it took 13 seconds each time I tried.

I still had an ip address according to ip route.

Once the connection drops like that, I'm forced to manually disconnect and reconnect with networkmanager. The connection will not recover on its own. (Or if it does, it takes a very long time, more than I've been willing to wait)

seth · 2017-11-03 18:54:54

For the record: your OP dmesg suggests a flawed attempt to block 11n (seems you added two parameters "11n" and "disable" instead of "11n_disable=1"
Fix that and also add "swcrypto=0" and "disable_11ac=true"
Do you have any bluetooth devices enabled?

DanielPowerNL · 2017-11-03 19:17:57

Thanks for pointing that out! It seems I left out the underscore. So I was hopeful fixing that would solve the issue. But I fixed it, and added the two options you suggested, rebooted, and still lost connection after a few minutes.

I do not have any bluetooth devices connected. I don't use bluetooth, and don't even have bluez or blueman installed.

I did find that I can ping my own ip address. Pinging any other ip address (including the router) fails, but pinging my own ip address works. I will test if I can do that when the connection drops the next time it happens.

Edit: I'll also add that the last time I lost connection, I decided to give it some time and see if it would recover on its own. I gave it 7 minutes before I gave up on it and manually reconnected.

Edit 2: I lost connection again, and I can ping myself even when the network stops working. I also found something very strange. It seems that my desktop client for Facebook "messengerfordesktop" continues to receive and send messages for a little while even after everything else stops working. So I'll be unable to load pages with Firefox, unable to query pacman, but Facebook Messenger continues receiving messages for a little while. Though after some time, Facebook Messenger stops working as well.

I also noticed that if I was on a trading page on the Cryptocurrency Exchange www.coinexchange.com, I'd continue seeing activity of the latest trades for a little while after the network stopped working for other things. Could this be because Messenger and CoinExchange are keeping up an active connection, and so they don't time out right away. It's very weird.

Last edited by DanielPowerNL (2017-11-03 19:31:55)

seth · 2017-11-03 20:00:12

can you still ping 8.8.8.8?

DanielPowerNL · 2017-11-03 20:01:50

No, I can't ping 8.8.8.8 even when I have a connection.

seth · 2017-11-03 20:25:00

The own IP is likely shortcut over the loopback device, what you see in those applications is probably processing of what's still on the TCP stack.
Might be a firmware bug, try to move away /usr/lib/firmware/iwlwifi-7260-17 (DO NOT DELETE IT! If this goes wrong, you might loose internet altogether!) and then downwards.
Reboot after each time (I guess reloading the module should work, but am not sure) and see whether you get a more reliable connection.

DanielPowerNL · 2017-11-03 20:56:42

I'll try that next time I get a chance to troubleshoot this. I have to stop for today. Hopefully I'll get a chance tomorrow or sometime this weekend.

Thank you so much for all your help with troubleshooting, I really appreciate it!

MartB · 2017-11-09 09:40:59

Same issue at a german university using eduroam here...
I tried the AC8260 and AC8265 from intel but it didnt change a thing.

My home wireless uses the same authentication protocols on a custom radius server and it works just fine.
Could be eduroams fault as it drops the network traffic but not the connection for me.

I tried the same basic troubleshooting as you did.

Any help is appreciated
Kind regards
MartB

Last edited by MartB (2017-11-09 09:41:20)

DanielPowerNL · 2017-11-09 12:37:16

I have no doubt there are issues with the eduroam network. But Windows and Mac users do not experience the same network drops that I and other Linux users have experienced. So surely something can be improved.

I'm focusing on my upcoming midterms, so I've just been using my phone with USB tethering, and haven't been able to focus on troubleshooting this for a while.

MartB · 2017-11-09 13:23:06

Yeah windows users dont seem to have that issue at all.
So it could be related to the wireless stack on (arch) linux.

Is your facility using cisco accesspoints by any chance ?
Im asking because i noticed plenty of errors with these no matter what kind of model is used.

If i capture packets and this starts happening i can see retransmits and RSTs on ssl pageloads and dns requests, but icmp pings still work fine.
So im not sure if its actually related to the stack or just to the overall quality of the eduroam network im on,

Good luck with your midterms!

Kind regards
MartB

seth · 2017-11-09 14:00:07

Did you try downgrading the firmware?

MartB · 2017-11-16 06:08:04

Yeah but im starting to doubt is a specific firmware issue.
I can still see some packets flow over the network but some are getting weird DUP-ACKs and RSTs from the servers.
So something might be corrupting packets on the network and or physical layer whenever this happens.
As to where it exactly starts to happen i dont know.

seth · 2017-11-16 08:16:19

Bufferbloat?

http://www.linux-magazin.de/ausgaben/20 … practices/ (german only!)
https://www.phoronix.com/scan.php?page= … ufferbloat
http://www.linuxjournal.com/content/que … work-stack

nstarclix · 2017-11-22 08:24:19

I have the very same problem, also only on eduraom in Germany. Yesterday I thought I found a fix, because I've been using systemd-networkd and systemd-resolved parallel to network-manager.
But disabling the services did not fix anything at all.
From all I know there seems to be a problem with DNSSEC. Right after I lose internet connection journalctl -f spams endlessly lines like:

Nov 14 12:00:15 ThinkPadX230 systemd-resolved[598]: Using degraded feature set (UDP+EDNS0+DO) for DNS server <dns-server-address>.
Nov 14 12:00:21 ThinkPadX230 systemd-resolved[598]: DNSSEC validation failed for question <dns-server-address> IN SOA: no-signature

After disabling systemd-networkd and systemd-resolved, the messages dissappeared from journalctl -f, but NetworkManager reports a time out and won't establish an internet connection again (wifi connection still persists!).

This happens after a random amount of time. It won't reconnect until I use the radio hardware switch on my ThinkPad X230. I am using the lts-kernel 4.9 with NetworkManager (1.10.0-1, Arch Linux).
Btw I have the same network-card as OP: Intel AC 7260. Anything else I could provide?

I don't know what to do anymore, this has been plagueing me for 2 months - every single day.

Right now I am testing a newer linux kernel "Linux ThinkPadX230 4.13.12-1-ARCH #1 SMP PREEMPT Wed Nov 8 11:54:06 CET 2017 x86_64 GNU/Linux", I will report back ...

Report: After 41 minutes the internet connection was lost again with kernel 4.13. Guess that does not matter as well.

Last edited by nstarclix (2017-11-22 08:46:11)

MartB · 2017-11-22 13:54:45

@Seth
It does not matter which qdisc i use fq_codel etc it doesnt work.
Im using cake on my home accesspoints with radius authentication and it works flawlessly.

@nstarclix
Im so sorry to hear that.
I forced my DNS to 9.9.9.9 and i will report back if it worked or not.
As a matter of fact this might also be an issue with intels wireless drivers given that everyone in this thread seems to have issues with intel cards.

Keep an eye on tcpdump when it happens, you will see that some traffic is still flowing.
So it might be physical corruption of the frames or just a broken dns server / client.

Im on 4.14 RC8 with a patch supplied by intel to run the latest firmware ucode 34.

Edit:
Not DNS related it just died on me again with.
Nov 22 14:59:27 martb NetworkManager[943]: <info> [1511359167.1298] connectivity: (wlp2s0) timed out

Kind regards
Martin

Last edited by MartB (2017-11-22 14:01:50)

nstarclix · 2017-11-23 23:57:03

I just realized that the /etc/modprobe.d/iwlwifi.conf seems to be having no effect at all. Has anyone had a similiar problem?
Also using "modprobe iwlwifi parameter" on terminal throw no errors, but seem to have no effect ...

options iwlwifi 11n_disable=1
options iwlwifi bt_coex_active=0
options iwlwifi swcrypto=1
options iwlwifi power_save=0
options iwlwifi 11n_disable=8

And nothing seems to work.

sudo mkinitcpio -p linux-lts works fine and I also have the modconf parameter in the mkinitcpio.conf

HOOKS="base udev autodetect keyboard keymap consolefont modconf block lvm2 filesystems fsck"

I never had these problems 2 years ago on my X220T ThinkPad with Ubuntu 14.04. Also my wifi seems to be really slowing down due to the iwlwifi.conf having no effect.

Also: eduroam problem still persists. I'm gonna try if I also have the same problem with a different linux distribution. I feel like throwing my ThinkPad against the wall. So frustrating.

Last edited by nstarclix (2017-11-24 00:22:51)

Arch Linux

#1 2017-10-15 18:39:32

Wifi randomly drops over eduroam network

#2 2017-10-21 02:06:19

Re: Wifi randomly drops over eduroam network

#3 2017-10-29 21:33:41

Re: Wifi randomly drops over eduroam network

#4 2017-10-30 15:09:27

Re: Wifi randomly drops over eduroam network

#5 2017-10-31 16:21:29

Re: Wifi randomly drops over eduroam network

#6 2017-11-01 08:06:34

Re: Wifi randomly drops over eduroam network

#7 2017-11-03 15:41:21

Re: Wifi randomly drops over eduroam network

#8 2017-11-03 16:10:45

Re: Wifi randomly drops over eduroam network

#9 2017-11-03 16:35:52

Re: Wifi randomly drops over eduroam network

#10 2017-11-03 18:12:10

Re: Wifi randomly drops over eduroam network

#11 2017-11-03 18:54:54

Re: Wifi randomly drops over eduroam network

#12 2017-11-03 19:17:57

Re: Wifi randomly drops over eduroam network

#13 2017-11-03 20:00:12

Re: Wifi randomly drops over eduroam network

#14 2017-11-03 20:01:50

Re: Wifi randomly drops over eduroam network

#15 2017-11-03 20:25:00

Re: Wifi randomly drops over eduroam network

#16 2017-11-03 20:56:42

Re: Wifi randomly drops over eduroam network

#17 2017-11-09 09:40:59

Re: Wifi randomly drops over eduroam network

#18 2017-11-09 12:37:16

Re: Wifi randomly drops over eduroam network

#19 2017-11-09 13:23:06

Re: Wifi randomly drops over eduroam network

#20 2017-11-09 14:00:07

Re: Wifi randomly drops over eduroam network

#21 2017-11-16 06:08:04

Re: Wifi randomly drops over eduroam network

#22 2017-11-16 08:16:19

Re: Wifi randomly drops over eduroam network

#23 2017-11-22 08:24:19

Re: Wifi randomly drops over eduroam network

#24 2017-11-22 13:54:45

Re: Wifi randomly drops over eduroam network

#25 2017-11-23 23:57:03

Re: Wifi randomly drops over eduroam network

Board footer