You are not logged in.

#1 2019-04-26 22:20:08

teateawhy
Member
From: GER
Registered: 2012-03-05
Posts: 1,133
Website

Application checkpointing of interactive python scripts in batch mode

Hi,

This is a question related to virtual consoles/consoles, even if it might not seem like that in the beginning.
I'm currently running a python script repeatedly for scientific computations with different inputs. The backend software (gurobi.com) is essentially searching through a large search tree. Unfortunately the backend software only has the ability to save the best result, but the search tree is lost when the program exits. The search tree would have to be pointlessly recomputed again if i want to revisit the same problem later on. If the script is interactive (python -i), the python script does not exit, and the search tree stays in memory, and i can revisit the problem, run commands from python, and that works well in principle. The problem is, i can't keep all of the processes for all the datasets in memory, because that would eat up too much RAM.
To remedy that, i want to do application checkpointing.
I have installed CRIU https://aur.archlinux.org/packages/criu/, and with that i'm able to checkpoint and restore the interactive script, as long as it's running on a virtual console.
Now i need to run and checkpoint many datasets, so i'm looking for a method to run the interactive script on a virtual console, in a batch processing way, then checkpoint it after the computation has finished.
Can someone help me with that, or has perhaps, a better idea to achieve the original goal of checkpointing a python console? Thanks!

Offline

Board footer

Powered by FluxBB