You are not logged in.

#1 2013-12-13 15:32:15

brettski
Member
Registered: 2013-06-21
Posts: 18
Website

Python Server/Client Application Development

Hey All!

I am writing a basic proof of concept server/client application for a project I am working on.

My goal is this:

1) End user launches client.
2) Client connects to server.
3) User selects one or more files to send to server.
4) Server saves a copy of each file.
5) Server terminates connection.
5) Server processes each file.
6) Upon completion of the processing server sets a "processing completed flag."
7) Client periodically checks with server for "processing completed flag."
8) Client securely reconnects

9) Client downloads processed files.

I have no background in programming servers/clients so this will be a learning experience for me and the workflow described above may change as I learn more about how this all works. Currently my hope is to first implement a system that will stay connected while the files are processed and once I have a better understanding as to how that works I can add more complexity to the system.

I will be writing this in Python 3.3 and the server will be hosted on a server which will (for now) be running Arch.

This thread will act as a sort of development journal but feel free to comment or post suggestions. I will for sure have questions as I go along too!

Offline

#2 2013-12-13 15:45:55

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,739

Re: Python Server/Client Application Development

Sounds fun.   If you want to run the server half under Apache, you might want to look at projects like Django.  The nice thing about running under Apache is that is much easier to make the system scale.  For example, you did not mention whether it was possible that there could be multiple clients trying to attach to a single server; or whether they might try it at the same time, or how many there might be.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#3 2013-12-13 16:59:35

brettski
Member
Registered: 2013-06-21
Posts: 18
Website

Re: Python Server/Client Application Development

This is the first step in a long term project. In the first few iterations there will only be a few clients but the hope is to scale it and in a few years it is possible that there will be hundreds of clients potentially connecting many times each day.

To provide a little more detail about my current setup:
Right now I have my own home server which I have set up to accept connections from the internet via ssh. I hope to adapt this machine to be the "testing server" while we are in this pre-pre-pre-alpha phase. Ultimately my immediate goal is to build a demo to convince a team that building our own server/client solution will be a viable and worth while pursuit.

Offline

#4 2013-12-16 15:38:35

brettski
Member
Registered: 2013-06-21
Posts: 18
Website

Re: Python Server/Client Application Development

So after doing some research I have decided against going with a standard ftp or scp option and I am going to be using Python's socket module.

The first answer to this StackOverflow Question was particularly helpful.
The author implements a basic server/client pair and also implements a netstring function.

Last night I was having some trouble with going from integers to bytes and back (sockets are only capable of sending/receiving bytecode). Turns out the best way (so far as I can tell) is to go integer -> string -> byte and then byte -> string -> integer.

Now that I have all that figured out, I am ready to start writing my own server and client.

Offline

#5 2013-12-16 19:50:29

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Python Server/Client Application Development

brettski wrote:

Turns out the best way (so far as I can tell) is to go integer -> string -> byte and then byte -> string -> integer..

Do you mean regular, non-byte strings? That cannot possibly be the best way to do it. It would be like using Russian as an intermediary language for translating Spanish to French.

Maybe the following will set you on the right path:
http://stackoverflow.com/questions/6187 … -in-python
http://stackoverflow.com/questions/4445 … int-python

Looking further, this is probably the right way to do it:

...
some_int = 57
bytes = struct.pack('i', bytes)
sock.send(bytes)
...

See http://docs.python.org/3/library/struct.html for details.


As for threaded servers, you may find some useful examples here.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#6 2013-12-16 22:20:24

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 649

Re: Python Server/Client Application Development

brettski wrote:

So after doing some research I have decided against going with a standard ftp or scp option and I am going to be using Python's socket module.

I wouldn't use either of those. Best and certainly easiest way is to just to use standard web browser with web server and file uploads and downloads.

Offline

#7 2013-12-16 22:52:36

cris9288
Member
Registered: 2013-01-07
Posts: 348

Re: Python Server/Client Application Development

bulletmark wrote:
brettski wrote:

So after doing some research I have decided against going with a standard ftp or scp option and I am going to be using Python's socket module.

I wouldn't use either of those. Best and certainly easiest way is to just to use standard web browser with web server and file uploads and downloads.

Could you comment on why you think a browser based implementation would be better? As someone that's not very experience in this kind of thing, my first inclination was to use sockets, but only because that's the only thing I know.

Offline

#8 2013-12-17 01:15:18

brettski
Member
Registered: 2013-06-21
Posts: 18
Website

Re: Python Server/Client Application Development

Xyne wrote:

Do you mean regular non-byte strings?

I was having a misunderstanding as to what was actually going on with how python differentiates between a string/bytestring(bytes)/byte. All of the examples I have looked at are from Python 2.x which seems not to care what is fed into a socket... which is not the case in Python 3.3.3. It looks like bytestrings are going to be the easiest for implementing a netstring function.

Do you happen to know if I would run into trouble opening a file in binary mode and appending it's contents to a bytestring before transmitting it (this is how I have seen it done in 2.x)?

bulletmark wrote:

I wouldn't use either of those. Best and certainly easiest way is to just to use standard web browser with web server and file uploads and downloads.

A browser based solution would definitely make my life a bit simpler smile. That said, in the future the client is going to do more than simply upload and download files.

Offline

#9 2013-12-17 01:57:55

bulletmark
Member
From: Brisbane, Australia
Registered: 2013-10-22
Posts: 649

Re: Python Server/Client Application Development

cris9288 wrote:

Could you comment on why you think a browser based implementation would be better? As someone that's not very experience in this kind of thing, my first inclination was to use sockets, but only because that's the only thing I know.

Sockets is about low level transport, browsers uses sockets after all, so comparing sockets to web technology is like comparing a tyre to a car.

In 2013, if you are developing an architecture whereby a central server services multiple client machines then you use a web app. No client machines need any special software installed (browser is the universal client). There are a gazillion python web frameworks to choose from. Django is the most popular. Development using one of these will be much easier and quicker than developing a traditional non-web application.

Last edited by bulletmark (2013-12-17 02:00:38)

Offline

#10 2013-12-17 07:41:12

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Python Server/Client Application Development

brettski wrote:

Do you happen to know if I would run into trouble opening a file in binary mode and appending it's contents to a bytestring before transmitting it (this is how I have seen it done in 2.x)?

You can do that, but why would you? Send your bytestring through the socket, then send the file contents through the socket. It makes absolutely no difference on the other end if they were concatenated on your end before being piped through the socket. Doing so will just incur overhead from combining the bytes (memory reallocation) with no benefit. The only possible reason to do that would be if you need to do some sort of processing that will only work on the combined sequence of bytes.  Otherwise you process them separately and then send them through.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB