You are not logged in.

#1 2015-08-27 14:28:19

normalra
Member
Registered: 2015-08-20
Posts: 1

SSL error: decryption failed or bad record mac

Hello!

I'm having a hard time understanding the underlying cause of one particular issue which has been troubling me for a better part of the year. The symptoms of whatever is the problem occur spontaneously and irregularly with no apparent pattern and I would greatly appreciate any advice or insight you might have.


The Setup

Network is configured by the dhcpcd client on enp2s0 connecting to a 3G wireless
router with no custom configuration. Machine is running a DNS cacher -- dnsmasq. System has an up-to-date mirror and is freshly installed.
Hardware NIC lspci output:

02:00.0 Ethernet controller: Qualcomm Atheros AR8121/AR8113/AR8114 Gigabit or Fast Ethernet (rev b0)
	Subsystem: ASUSTeK Computer Inc. Device 14f5
	Flags: bus master, fast devsel, latency 0, IRQ 17
	Memory at fdfc0000 (64-bit, non-prefetchable) [size=256K]
	I/O ports at d800 [size=128]
	Capabilities: [40] Power Management version 2
	Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [58] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: ATL1E
	Kernel modules: atl1e

03:00.0 Network controller: Qualcomm Atheros AR9285 Wireless Network Adapter (PCI-Express) (rev 01)
	Subsystem: AzureWave AW-NE785 / AW-NE785H 802.11bgn Wireless Full or Half-size Mini PCIe Card
	Flags: bus master, fast devsel, latency 0, IRQ 18
	Memory at febf0000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [60] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [170] Power Budgeting <?>
	Kernel driver in use: ath9k
	Kernel modules: ath9k
The Symptoms

The best way to describe the issue would be to provide a verbatim transcription of what happens when attempting to download a file:

normal@universe ~ > wget -d --tries=1 https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz
Setting --tries (tries) to 1
DEBUG output created by Wget 1.16.3 on linux-gnu.

URI encoding = ‘UTF-8’
--2015-08-20 22:24:00--  https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz
Resolving www.python.org (www.python.org)... 185.31.19.223
Caching www.python.org => 185.31.19.223
Connecting to www.python.org (www.python.org)|185.31.19.223|:443... connected.
Created socket 3.
Releasing 0x0000000001791cf0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00000000017889a0
certificate:
  subject: CN=www.python.org,O=Python Software Foundation,L=Wolfeboro\\,,ST=NH,C=US,postalCode=03894-4801,street=16 Allen Rd,serialNumber=3359300,jurisdictionST=Delaware,jurisdictionC=US,businessCategory=Private Organization
  issuer:  CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
X509 certificate successfully verified and matches host www.python.org

---request begin---
GET /ftp/python/3.4.3/Python-3.4.3.tgz HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: www.python.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Server: nginx
Content-Type: application/octet-stream
Last-Modified: Wed, 25 Feb 2015 11:34:57 GMT
ETag: "54edb361-12a6153"
X-Clacks-Overhead: GNU Terry Pratchett
Content-Length: 19554643
Accept-Ranges: bytes
Date: Thu, 20 Aug 2015 20:24:01 GMT
Via: 1.1 varnish
Age: 43722
Connection: keep-alive
X-Served-By: cache-lhr6331-LHR
X-Cache: HIT
X-Cache-Hits: 1
Public-Key-Pins: max-age=600; includeSubDomains; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="5C8kvU039KouVrl52D0eZSGf4Onjo4Khs8tmyTlV3nU="; pin-sha256="5C8kvU039KouVrl52D0eZSGf4Onjo4Khs8tmyTlV3nU="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="TUDnr0MEoJ3of7+YliBMBVFB4/gJsv5zO7IxD9+YoWI="; pin-sha256="x4QzPSC810K5/cMjb05Qm4k3Bw5zBn4lTdO/nEW/Td4=";
Strict-Transport-Security: max-age=63072000; includeSubDomains

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 19554643 (19M) [application/octet-stream]
Saving to: ‘Python-3.4.3.tgz.1’

Python-3.4.3.tgz                 0%[                                                            ]  79.16K   226KB/s   in 0.3s   

Disabling further reuse of socket 3.
Closed 3/SSL 0x00000000017889a0
2015-08-20 22:24:02 (226 KB/s) - Read error at byte 81064/19554643 (error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac). Giving up.

Python's SSL module seems to exhibit the same issue:

normal@universe ~/code/cyanogenmod > repo init -u https://github.com/CyanogenMod/android.git -b cm-12.1
Get https://gerrit.googlesource.com/git-repo/clone.bundle
Traceback (most recent call last):
  File "/usr/bin/repo", line 771, in <module>
    main(sys.argv[1:])
  File "/usr/bin/repo", line 739, in main
    _Init(args)
  File "/usr/bin/repo", line 263, in _Init
    _Clone(url, dst, opt.quiet)
  File "/usr/bin/repo", line 523, in _Clone
    if _DownloadBundle(url, local, quiet):
  File "/usr/bin/repo", line 478, in _DownloadBundle
    buf = r.read(8192)
  File "/usr/lib/python2.7/socket.py", line 384, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/httplib.py", line 612, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 384, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1752)

At this point it's important to note that file transfers break exclusively over HTTPS and exclusively for larger files (>2M). Perhaps this issue is the same reason why transferring over the git:// protocol results in an unsuccessful clone.

The Experiments
  1. Perhaps we're dealing with unreliable internet service; another machine on the same network should exhibit the same symptoms
    Result: no issue on the other machine

  2. The previous machine is connected via wireless LAN, perhaps we should try the same
    Result: issue persists

  3. Let's use curl instead of wget for funsies
    Result: it is as strange as it seems, there are no issues! and the file is valid

  4. Using Android's USB tethering
    Result: no issues. this of course bypasses the network entirely (different ISP).

  5. Throttling bandwidth to ~50kbps using tc
    Result: this one really fascinates me; no issues

  6. Disabling AES-NI
    Result: pointless because the machine has no hardware-accelerated AES support but it has been suggested and it doesn't hurt to try; issue persists

The Conclusion?

Debugging this issue goes well beyond my abilities, but if it turns out to be something completely trivial I solemnly swear to pick up on solving logic and math puzzles. Hopefully that will teach me how to approach problems correctly and thoughtfully, without resorting to help "vampiricism".
Thank you!

Last edited by normalra (2015-08-27 15:09:06)

Offline

Board footer

Powered by FluxBB