Python sockets on loopback

sgr011 · 2018-10-20 10:23:19

The connection between the server and the client is working but when you try to send data through it brakes. I really doubt it is the code because it is working on 2 different windows machines. I've checked iptables and I have nothing blocked or forwarded. I'm using port 9001.
Here's the tcp dump for when it is running

listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
12:20:23.391545 IP localhost.34170 > localhost.etlservicemgr: Flags [SEW], seq 1651948385, win 43690, options [mss 65495,sackOK,TS val 1814916314 ecr 0,nop,wscale 7], length 0
12:20:23.391558 IP localhost.etlservicemgr > localhost.34170: Flags [S.E], seq 1480537830, ack 1651948386, win 43690, options [mss 65495,sackOK,TS val 1814916315 ecr 1814916314,nop,wscale 7], length 0
12:20:23.391569 IP localhost.34170 > localhost.etlservicemgr: Flags [.], ack 1, win 342, options [nop,nop,TS val 1814916315 ecr 1814916315], length 0
12:20:23.391770 IP localhost.etlservicemgr > localhost.34170: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 1814916315 ecr 1814916315], length 1
12:20:23.391781 IP localhost.34170 > localhost.etlservicemgr: Flags [.], ack 2, win 342, options [nop,nop,TS val 1814916315 ecr 1814916315], length 0
12:20:23.391802 IP localhost.etlservicemgr > localhost.34170: Flags [P.], seq 2:3, ack 1, win 342, options [nop,nop,TS val 1814916315 ecr 1814916315], length 1
12:20:23.391807 IP localhost.34170 > localhost.etlservicemgr: Flags [.], ack 3, win 342, options [nop,nop,TS val 1814916315 ecr 1814916315], length 0

Even stranger, a couple of times it does work but I haven't been able to reproduce this.

ewaller · 2018-10-20 15:02:37

Welcome to Arch Linux. I use sockets in Python all the time without issue. Could be that you are not using Python 3.7 on the Windows boxes and you are on Arch.

Those logs do not tell me anything. Can you share the code?

Edit: Are you going machine to machine, or are the client and host on the same machine? Are they on the same subnet? Are you communicating over the public Internet? Any firewalls?

Last edited by ewaller (2018-10-20 15:04:03)

sgr011 · 2018-10-20 15:32:36

class chunkSocket(object):
    def __init__(self,
            sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM),
            BUFFER_SIZE=1024):

        #make the scope of sock and BUFFER_SIZE class wide
        self.sock = sock
        self.BUFFER_SIZE = BUFFER_SIZE


    def send(self, msg, sendBytes=False):

        #if msg is not a bytes object convert the message into bytes
        #msg is assumed to be an integer so this is default
        if not sendBytes:
            #convert message to bytes
            msg = self.intToBytes(msg)

        #get length of message and send the binary data of the length
        msglen = len(msg)

        msglenNetworkBytes = self.intToBytes(msglen)

        self.sock.send(msglenNetworkBytes)

        while msglen > self.BUFFER_SIZE:

            #get the first Buffer_size number of bytes from message
            chunk = msg[:self.BUFFER_SIZE]
            #removes that chunk from the message
            msg = msg[self.BUFFER_SIZE:]

            #MSGLEN acts as bytes left to send
            #it is decreased by buffer size when a buffer size amount of bytes are sent
            msglen -= self.BUFFER_SIZE

            #send the chunk
            self.sock.send(chunk)

        #send the remainder of the bytes
        return self.sock.send(msg)    def recieve(self, recvBytes=False):

        #The first message sent will be the message length
        msglenNetworkBytes = self.sock.recv(self.BUFFER_SIZE)

        msglen = self.bytesToInt(msglenNetworkBytes)

        msgBytes = b''
        while msglen > self.BUFFER_SIZE:

            #recieves the chunk and adds it to msgBytes
            chunk = self.sock.recv(self.BUFFER_SIZE)
            msgBytes += chunk

            #msglen acts as bytes left to recieve
            msglen -= self.BUFFER_SIZE

        #recieves the remainder of the bytes
        lastChunk = self.sock.recv(self.BUFFER_SIZE)
        msgBytes += lastChunk

        #converts to integer if the kwarg is set to False (default)
        if not recvBytes:
            msg = self.bytesToInt(msgBytes)
            return msg
        else:
            return msgBytes

That's the code that actually does all of the sending and receiving

         self.serversocket = chunkSocket()
        self.serversocket.sock.bind(('127.0.0.1', PORT))
        self.serversocket.sock.listen(5)

That's the code that sets up the server

        
while True:
            clientsock, addr = self.serversocket.sock.accept()
            print(str(addr), 'connected')

            #makes an instance of the chunkSocket class with the client socket
            self.clientSocks[addr] = chunkSocket(sock=clientsock)
            self.usrNames[addr] = str(addr)

            #send e
            self.clientSocks[addr].send(self.serverRSA.e)
            #recv e
            self.clientPubKeys[addr] = self.clientSocks[addr].recieve()

            #send n
            self.clientSocks[addr].send(self.serverRSA.n)
            #recv n
            self.clientN[addr] = self.clientSocks[addr].recieve()

That's the code which accepts a new client, the send n and e stuff is for encryption, e is a small integer that gets converted into bytes and then sent through. By small I mean 128 - 256. That's where the program seems to fail as the client isn't receiving it.

         self.sock = chunkSocket()
        self.sock.sock.connect((addr, PORT))
    
        #recv e
        self.serverKey = self.sock.recieve()
        print('recv e')
        #send e
        self.sock.send(self.clientRSA.e)

and there's the client side code.

I was using Python 3.6 on Windows, I'm just sending the packets on the local loopback for testing so client and host are on the same machine. No firewalls just checked iptables and there's nothing. Thanks for the reply!

ewaller · 2018-10-20 15:36:57

I need to step away for a few hours, but I definitely want to look into this. I will get back to you.

sgr011 · 2018-10-20 17:52:33

Also just tested it over the internet between 2 Windows machines and it works (binding the server to socket.gethostname()) tried to do it with linux still no luck

ewaller · 2018-10-21 19:20:39

Okay, I've been playing with this all morning, and frankly, your code sucks

Okay, that is a bit harsh. But, what you provided is most definitely not complete and has errors that probably occurred during posting. Look at the definition of receive().

In any event, in the absence of anything clearer, this works:

The server:

#! /usr/bin/env python

import chunk

PORT=2718

serversocket = chunk.chunkSocket()
serversocket.sock.bind(('127.0.0.1', PORT))
serversocket.sock.listen(5)
serverRSA=b'FlurpityFlurp'


while True:
    clientSocks, clientPubKeys = {}, {}
    
    clientsock, addr = serversocket.sock.accept()
    print(str(addr), 'connected')

    #makes an instance of the chunkSocket class with the client socket
    clientSocks[addr] = chunk.chunkSocket(sock=clientsock)
    #usrNames[addr] = str(addr)
    #send e
    clientSocks[addr].send(serverRSA, sendBytes=True)
    print(clientSocks)
    #recv e
    clientPubKeys[addr] = clientSocks[addr].recieve(recvBytes=True)
    print(clientPubKeys)
    #send n
    clientSocks[addr].send(serverRSA, sendBytes=True)
    #recv n
    #clientN[addr] = clientSocks[addr].recieve(recvBytes=True)

The client

#! /usr/bin/env python

import chunk
PORT = 2718
addr = 'localhost'
clientRSA = b'fubarBletch'

sock = chunk.chunkSocket()
sock.sock.connect((addr, PORT))

# recv e
serverKey = sock.recieve(recvBytes=True)
print('recv e')
print(serverKey)
# send e
sock.send(clientRSA, sendBytes=True)

the class

import socket
import sys
import struct

class chunkSocket(object):
    def __init__(self,
            sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM),
            BUFFER_SIZE=1024):

        #make the scope of sock and BUFFER_SIZE class wide
        self.sock = sock
        self.BUFFER_SIZE = BUFFER_SIZE


    def send(self, msg, sendBytes=False):

        #if msg is not a bytes object convert the message into bytes
        #msg is assumed to be an integer so this is default
        if not sendBytes:
            #convert message to bytes
            msg = struct.pack('I', msg)
            print(msg)

        #get length of message and send the binary data of the length
        msglen = len(msg)

        msglenNetworkBytes = struct.pack('I', msglen)

        self.sock.send(msglenNetworkBytes)

        while msglen > self.BUFFER_SIZE:

            #get the first Buffer_size number of bytes from message
            chunk = msg[:self.BUFFER_SIZE]
            #removes that chunk from the message
            msg = msg[self.BUFFER_SIZE:]

            #MSGLEN acts as bytes left to send
            #it is decreased by buffer size when a buffer size amount of bytes are sent
            msglen -= self.BUFFER_SIZE

            #send the chunk
            self.sock.send(chunk, sendBytes=True)

        #send the remainder of the bytes
        return self.sock.send(msg)

    def recieve(self, recvBytes=False):

        #The first message sent will be the message length
        msglenNetworkBytes = self.sock.recv(4)
        print(msglenNetworkBytes)
        msglen = struct.unpack('I', msglenNetworkBytes)[0]

        msgBytes = b''
        while msglen > self.BUFFER_SIZE:

            #recieves the chunk and adds it to msgBytes
            chunk = self.sock.recv(self.BUFFER_SIZE)
            msgBytes += chunk

            #msglen acts as bytes left to recieve
            msglen -= self.BUFFER_SIZE

        #recieves the remainder of the bytes
        lastChunk = self.sock.recv(self.BUFFER_SIZE)
        msgBytes += lastChunk

        #converts to integer if the kwarg is set to False (default)
        if not recvBytes:
            print(msgBytes)
            msg = struct.unpack('I',msgBytes)[0]
            return msg
        else:
            return msgBytes

The things I found: The message length does not necessarily come in as a block of four bytes by itself when you specify a larger message size -- the next send gets tacked on to the end. I forced it to 4.
Also, your code, as written, does not send stuff as bytes -- unless I am not understanding how your data structures are represented. I could but guess.
I don't know what your int to bytes and int from bytes did -- you did not share them. I converted them to struct operations.

Also, your code is very trusting. It is very susceptible to malformed packets. You probably should add some sanity checks. When it comes to sockets, never expect the 'easy' path. Assume things are going to come of the rails -- and be delightfully surprised when you receive a properly formed packet.

Last edited by ewaller (2018-10-21 19:21:31)

sgr011 · 2018-10-21 20:18:18

Thank you a lot for this. But I was going to use struct but the encryption I'm using means that each message will be about 2kb in length (that's just one integer) so pack really doesn't work. I'll implement the rest of it to see if it works.

Arch Linux

#1 2018-10-20 10:23:19

Python sockets on loopback

#2 2018-10-20 15:02:37

Re: Python sockets on loopback

#3 2018-10-20 15:32:36

Re: Python sockets on loopback

#4 2018-10-20 15:36:57

Re: Python sockets on loopback

#5 2018-10-20 17:52:33

Re: Python sockets on loopback

#6 2018-10-21 19:20:39

Re: Python sockets on loopback

#7 2018-10-21 20:18:18

Re: Python sockets on loopback

Board footer