Dual computer remote setup solutions

Hacksaurus_Babu · 2020-11-09 18:14:04

Hello

I am posting this so I can get some possible solutions on how to configure a remote setup between 2 computers remotely.

First I'll give some background to my current setup. Right now, I have a laptop that dual boots Windows and Linux, I do most of my work on this system for school, work, etc. My main driver is of course Linux, and I really just have Windows for games and to make some school programs work more smoothly. Now I am going to get a new desktop computer, which is going to be my new "gaming" setup which will ideally only have Windows installed. Thus making my laptop my "work" computer and my desktop my "gaming" computer.

The reason why I most likely won't install Linux on my desktop is really out of convenience, I rather just work on the same system with the same files without the worry of having to sync up files and configurations files (which I have a lot of, and I edit a lot). Also, I don't really need that much performance in the majority of cases, but there are some cases where I do need that processing power.

Sometimes, I do some machine learning which does require a lot of processing power which I would like to use my desktop computer. I use Jupyter for those types of projects, but it doesn't completely solve the issue is that I don't want to have to deal with syncing up files.

What are some potential solutions where I can still use my laptop, but still exploit the processing power of a more powerful Windows machine in a way that preferrable doesn't resort to file-sync?

natervance · 2020-11-09 19:12:19

From my reading of what you're trying to do, you basically want your Windows desktop to provide CPU and GPU cycles for computing tasks, but keep Windows installed. This sounds to me like cluster/grid computing. The restraint that the "cluster" (of the one desktop) must support Windows narrows options down quite a bit. My university uses HTCondor for this task, which has a daemon running on all lab computers (including Windows) all over campus, and when a computer "goes to sleep", it doesn't actually sleep, but rather starts processing jobs that are submitted to the HTCondor system. It's a spiffy setup, but in your case is probably much more trouble than it's worth (and it copies files around under the hood - unavoidable if you want to execute something in particular, though).

What I would probably do is write some sort of remoteExec script which takes a directory and a command as arguments, rsyncs the directory over to the Windows machine (e.g., using WSL), runs the code remotely, and copies the results back. I'd advise this kind of roll-your-own thing for the following reasons:

1. Any pre-built clustering solution is going to be more powerful and also more complex than what you need because it'll support large clusters, which adds unnecessary complexity.
2. Most of the effort in rolling your own solution is in network communication between machines and setting up a good execution environment. From experience, this is also the source of most of the effort in configuring clustering software, so using a pre-made solution doesn't actually make things easier for the cluster size = 1 case.

Another option: if the only software you need to execute remotely is python, then maybe you'll find all you need in RPyC. You will have to modify your code to take advantage of the remote resources, which I don't think is what you're going for, but maybe it'll spark some ideas.

Hacksaurus_Babu · 2020-11-09 19:24:22

I've heard of docker, which is a fast way to get Linux on a Windows machine, what do you suggest if I install docker on my Windows desktop machine. Thus having a "Linux install", without having to dual-boot.

What I'm primarily want to avoid is to sync all my programming files / all my configuration files since I update them a lot. Hence avoiding a full-blown OS. But I think I can handle a small containerized installation whose sole purpose is to just occasionally use the GPU/CPU of my desktop computer.

natervance · 2020-11-09 19:30:02

Having never personally attempted that route before, I'd just say go for it! Can't see any reason not to try.

Hacksaurus_Babu · 2020-11-09 19:34:01

In your previous post you stated "The restraint that the "cluster" (of the one desktop) must support Windows narrows options down quite a bit. " So if I did have Linux, (which docker I presume will let me). Do you know what other options there are?

Now that I am reading into docker more ti seems to be what I need, "distributed computing" was the keyword I needed

Slithery · 2020-11-09 19:37:10

Have you looked into WSL to run a Linux container on a Windows machine? It's probably a simpler solution than docker.

natervance · 2020-11-09 19:37:44

I've used slurm in the past, and I'm aware that torque is also quite popular. If your tasks lend themselves, GNU Parallel is a super easy option.

Hacksaurus_Babu · 2020-11-09 19:38:13

I have heard of that, but it would seem that docker was designed to be containerized and more flexible so I'll try that before I'll try WSL.

Slithery · 2020-11-09 19:53:18

Have you looked into WSL to run a Linux container on a Windows machine? It's probably a simpler solution than docker.

Arch Linux

#1 2020-11-09 18:14:04

Dual computer remote setup solutions

#2 2020-11-09 19:12:19

Re: Dual computer remote setup solutions

#3 2020-11-09 19:24:22

Re: Dual computer remote setup solutions

#4 2020-11-09 19:30:02

Re: Dual computer remote setup solutions

#5 2020-11-09 19:34:01

Re: Dual computer remote setup solutions

#6 2020-11-09 19:37:10

Re: Dual computer remote setup solutions

#7 2020-11-09 19:37:44

Re: Dual computer remote setup solutions

#8 2020-11-09 19:38:13

Re: Dual computer remote setup solutions

#9 2020-11-09 19:53:18

Re: Dual computer remote setup solutions

Board footer