You are not logged in.

#1 2013-08-06 21:36:14

ParanoidAndroid
Member
Registered: 2012-10-14
Posts: 114

parallel Backup script written in python

I'm writing a backup script in python, based on one I wrote in BASH earlier on. Both of these leverage rsync, but I decided to move to a python implementation because python is much more flexible than BASH. My goals with the new implementation are to back up the system, home folder, and my documents folder to a set of multiple primary, secondary, and tertiary disks, respectively. The method is as follows:

1. check for the existence of disks and create folders which will contain mountpoints for each category of disk.
2. decrypt and mount disks found under subfolders within those category folders, creating the mountpoints if they don't exist.
3. syncronize the aforementioned data to the mounted disks using rsync, doing all three classes of disk in parallel.
4. unmount and close disks

This is really my first serious python program, and I realize that it's a bit complicated. My code is rather sloppy, as well, perhaps understandably so given my novice status. My only other programming experience is with BASH scripts, but I digress.

Here is the code for the script (about 250 lines). It is written as a series of functions, and I'm uncertain as to whether functions or objects would work better. Additionally, I'm sure there's a python function provided by the os module analogous to the sync system call, but I've yet to find it in my python desk reference. The backup functions need work, and I'm still trying to figure out how to get them to loop through the mounted disks in each folder and copy to them. I require assistance in determining how to write the backup functions to do as outlined above, and how to run them in parallel. This is still a work in progress, mind.

#!/usr/bin/python

# Backup Script

#### preferences ####
# set your primary/secondary backup disks, encryption, and keyfile (if applicable) here.

# backup disks
# primary and secondary backups. As used here,
# primary backups refer to backups of the entire system made to an external drive
# secondary backups refer to backups made of individual folders, such as documents

# primary backup disks by UUID:
global PDISKS 
PDISKS = ("/dev/disk/by-uuid/16d64026-28bd-4e1f-a452-74e76bb4d47b","")

# secondary backups by UUID.
global SDISKS 
SDISKS = ()

# tertiary disks by UUID:
global TDISKS 
TDISKS = ("/dev/disk/by-uuid/39543e6e-cf50-4416-9669-e97a6abd2a37","")

# backup paths
# these are the paths of the folders you wish to back up to secondary
# and tertiary disks, respectively. Primary disks are set to back up the
# contents of the root filesystem (/*). NO TRAILING SLASHES.

global SBACKUP 
SBACKUP = "/home/bryant"

global TBACKUP 
TBACKUP = "/home/bryant/docs"

# use encryption:
use_encryption = True

# keyfile
# set the full path to your keyfile here
# this assumes a single keyfile for all backup disks
# set this to None if you don't have a single keyfile for all of your backups

keyfile = "/usr/local/bin/backup.keyfile"

#######

# import modules
import os, subprocess, sys

### preliminary functions ###
# these do the setup and post-copy work

def check_dirs():
    """checks that the folders which contain the mountpoints exist, creates them if they don't"""
    print("checking for mountpoints...")
    p = os.path.isdir("/mnt/pbackup")
    if p == True:
        print("primary mountpoint exists.")
    elif p == False:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/pbackup")

    s = os.path.isdir("/mnt/sbackup")
    if s == True:
        print("secondary mountpoint exists.")
    elif s == False:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/sbackup")

    t = os.path.isdir("/mnt/tbackup")
    if t == True:
        print("tertiary mountpoint exists.")
    elif t == False:
        print("mountpoint /mnt/tbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/tbackup")
    
def mount_disks():
    """mounts available backup disks in their respective subdirectories"""
    pfolder = 1
    sfolder = 1
    tfolder = 1
    
    pmapper = "pbackup" + str(pfolder)
    smapper = "sbackup" + str(sfolder)
    tmapper = "tbackup" + str(tfolder)

    for pdisk in PDISKS:
        e = os.path.islink(pdisk)
        if e == True:
            subprocess.call("sync",shell=True)
            kf=os.path.isfile(keyfile)
            if kf == True:
                print("keyfile found. Using keyfile to decrypt...")
                subprocess.call("sudo cryptsetup luksOpen " + pdisk + " " + pmapper + " --key-file " + keyfile,shell=True)

            if kf == False:
                print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                subprocess.call("sudo cryptsetup luksOpen " + pdisk + " " + pmapper,shell=True)

        f = os.path.isdir("/mnt/pbackup/pbak" + str(pfolder))
        if f == True:
            subprocess.call("mount " + "/dev/mapper/" + pmapper + " /mnt/pbak" + str(pfolder),shell=True)
            pfolder += 1
        elif f == False:
            os.mkdir("/mnt/pbackup/pbak" + str(pfolder))
            subprocess.call("mount " + "/dev/mapper/" + pmapper + " /mnt/pbak" + str(pfolder),shell=True)           
            pfolder += 1


    for sdisk in SDISKS:
        e = os.path.islink(sdisk)
        if e == True:
            subprocess.call("sync",shell=True)
            kf=os.path.isfile(keyfile)
            if kf == True:
                print("keyfile found. Using keyfile to decrypt...")
                subprocess.call("sudo cryptsetup luksOpen " + sdisk + " " + smapper + " --key-file " + keyfile,shell=True)

            if kf == False:
                print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                subprocess.call("sudo cryptsetup luksOpen " + sdisk + " " + smapper,shell=True)

        f = os.path.isdir("/mnt/sbackup/sbak" + str(sfolder))
        if f == True:
            subprocess.call("mount " + "/dev/mapper/" + smapper + " /mnt/sbackup/sbak" + str(sfolder),shell=True)
            sfolder += 1
        elif f == False:
            os.mkdir("/mnt/sbackup/sbak" + str(folder))
            subprocess.call("mount " + "/dev/mapper/" + smapper + " /mnt/sbackup/sbak" + str(sfolder),shell=True)           
            sfolder += 1


    for tdisk in TDISKS:
        e = os.path.islink(tdisk)
        if e == True:
            subprocess.call("sync",shell=True)
            kf=os.path.isfile(keyfile)
            if kf == True:
                print("keyfile found. Using keyfile to decrypt...")
                subprocess.call("sudo cryptsetup luksOpen " + tdisk + " " + tmapper + " --key-file " + keyfile,shell=True)

            if kf == False:
                print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                subprocess.call("sudo cryptsetup luksOpen " + tdisk + " " + tmapper,shell=True)

        f = os.path.isdir("/mnt/tbackup/tbak" + str(tfolder))
        if f == True:
            subprocess.call("mount " + "/dev/mapper/" + tmapper + " /mnt/pbak" + str(tfolder),shell=True)
            tfolder += 1
        elif f == False:
            os.mkdir("/mnt/tbackup/tbak" + str(tfolder))
            subprocess.call("mount " + "/dev/mapper/" + tmapper + " /mnt/tbak" + str(tfolder),shell=True)           
            tfolder += 1

def umount_disks():
    """unmounts and relocks disks"""
    subprocess.call("umount /mnt/pbackup*",shell=True)
    subprocess.call("umount /mnt/sbackup*",shell=True)
    subprocess.call("umount /mnt/tbackup*",shell=True)
    subprocess.call("cryptsetup luksClose /dev/mapper/pbackup*",shell=True)    
    subprocess.call("cryptsetup luksClose /dev/mapper/sbackup*",shell=True)
    subprocess.call("cryptsetup luksClose /dev/mapper/tbackup*",shell=True)

def check_disks():
    """checks to see how many disks exist, exits program if none are attached"""
    pdisknum = 0
    sdisknum = 0
    tdisknum = 0

    for pdisk in PDISKS:
        p = os.path.islink(pdisk)
        if p == True:
            pdisknum += 1
        elif p == False:
            print("disk " + pdisk  + " not detected.")
    
    for sdisk in SDISKS:
        s = os.path.islink(sdisk)
        if s == True:
            sdisknum += 1
        elif s == False:
            print("disk " + sdisk + " not detected.")
    
    for tdisk in TDISKS:
        t = os.path.islink(tdisk)
        if t == True:
            tdisknum += 1
        elif t == False:
            print("disk " + tdisk + " not detected.")

    total = pdisknum + sdisknum + tdisknum
    if total == 0:
        print("ERROR: no disks detected.")
        sys.exit()
    elif total > 0:
        print("found " + str(total) + " attached backup disks")
        print(str(pdisknum) + " Primary")
        print(str(sdisknum) + " secondary")
        print(str(tdisknum)  + " tertiary")
    return total, pdisknum, sdisknum, tdisknum

### backup functions ###
# these need serious work. Need to get them to loop through available mounted
# disks in their categories and then execute rsync

def pbackup(): 
    """calls rsync to backup the entire system to all pdisks"""

    dirs = os.listdir("/mnt/pbackup")
    for dir in dirs:
        m = os.path.ismount(dir)
        if m == True:
            subprocess.call("sync",shell=True)
            print("syncing disks with rsync...")
#            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} /* /mnt/pbackup/" + dir + "/ --exclude={/sys/*,/mnt/*,/proc/*,/dev/*,/lost+found,/media/*,/tmp/*,/home/*/.gvfs/*,/home/*/downloads/*,/opt/*,/run/*",shell=True)
            print("test1")
            subprocess.call("sync",shell=True)
            print("disk synced with disk " + pdisk + ".")
            print("sync with disk " + pdisk + " complete.")
        elif m == False:
            continue

def sbackup():
    """calls rsync to backup everything under SBACKUP folder to all sdisks"""

    dirs = os.listdir("/mnt/sbackup")
    for dir in dirs:
        m = os.path.ismount(dir)
        if m == True:
            subprocess.call("sync",shell=True)
#            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} SBACKUP/* /mnt/sbackup/" + dir + "/",shell=True)
            print("test2")
            subprocess.call("sync",shell=True)
            print("disk synced with disk " + pdisk + ".")
            print("sync with disk " + sdisk + " complete.")
        elif m == False:
            continue

def tbackup():
    """calls rsync to backup everything under TBACKUP folder to all tdisks"""

    dirs = os.listdir("/mnt/tbackup")
    for dir in dirs:
        m = os.path.ismount(dir)
        if m == True:
            subprocess.call("sync",shell=True)
#            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} TBACKUP/* /mnt/sbackup/" + dir + "/",shell=True)
            print("test3")
            subprocess.call("sync",shell=True)
            print("disk synced with disk " + pdisk + ".")
            print("sync with disk " + sdisk + " complete.")
        elif m == False:
            continue

#### main ####

# check for root access:
r=os.getuid()
if r != 0:
    print("ERROR: script not run as root.\n\tThis script MUST be run as root user.")
    sys.exit()

elif r == 0:
# program body
    check_dirs()
    check_disks()
    mount_disks()
#    pbackup()
#    sbackup()
    tbackup()
    umount_disks()

    print("backup process complete.")

Last edited by ParanoidAndroid (2013-08-07 20:01:07)

Offline

#2 2013-08-06 21:39:55

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 18,563
Website

Re: parallel Backup script written in python

On-topic, but better off in Programming & Scripting: moving there...


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#3 2013-08-07 21:22:35

berbae
Member
From: France
Registered: 2007-02-12
Posts: 1,100

Re: parallel Backup script written in python

Hello

I could advice that you take time to read and study tutorials about python programming; they should be easy to find on the web.
You could also search on the web other backup programs written in python; maybe you don't need to reinvent the wheel and there already exists a program which does what you want or that you can more easily adapt to your need.

With a little more studies you could have avoided this:

    p = os.path.isdir("/mnt/pbackup")
    if p == True:
        print("primary mountpoint exists.")
    elif p == False:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/pbackup")

which should be :

    if os.path.isdir("/mnt/pbackup"):
        print("primary mountpoint exists.")
    else:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/pbackup")

A boolean variable is either True or False, and it can directly be tested, you don't need to use a comparison operator like ==:

boolean_var = True
if boolean_var:
    ...

Best wishes for the success of your programming project.

Offline

#4 2013-08-08 00:03:44

ParanoidAndroid
Member
Registered: 2012-10-14
Posts: 114

Re: parallel Backup script written in python

I have indeed searched on line. Nothing that quite covers what I'm trying to do. I'm well acquainted with the Python language, but it's been forever since I actually used the language... I knew I would miss stuff. Thank you for pointing out the Boolean-evaluation bit.

Offline

#5 2013-08-08 00:04:46

ParanoidAndroid
Member
Registered: 2012-10-14
Posts: 114

Re: parallel Backup script written in python

I've run into a problem on line 149. I'm asking the program to list the directories under the top-level backup directories under /mnt, check to see if each one is a mountpoint, and if it is unmount it. It does this, but it appears to recurse into the directories under the directories I'm asking it to check. The output is:

checking for mountpoints...
primary mountpoint exists.
secondary mountpoint exists.
tertiary mountpoint exists.

disk /dev/disk/by-uuid/16d64026-28bd-4e1f-a452-74e76bb4d47b not detected.
found 1 attached backup disks
0 Primary
0 secondary
1 tertiary
keyfile found. Using keyfile to decrypt...
mounting tbackup1 at /mnt/tbak1
test3
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
not a mountpoint
Device /dev/mapper/pbackup* is not active.
Device /dev/mapper/sbackup* is not active.
backup process complete.

here is the code for the entire script. It's been much modified from the previously posted version, so I included all of the code versus the section in question for reference. As I said, the section that seems to be causing the issue is on line 149.

#!/usr/bin/python

# Backup Script

#### preferences ####
# set your primary/secondary backup disks, encryption, and keyfile (if applicable) here.

# backup disks
# primary and secondary backups. As used here,
# primary backups refer to backups of the entire system made to an external drive
# secondary backups refer to backups made of individual folders, such as documents

# primary backup disks by UUID:
global PDISKS 
PDISKS = ["/dev/disk/by-uuid/16d64026-28bd-4e1f-a452-74e76bb4d47b"]

# secondary backups by UUID.
global SDISKS 
SDISKS = []

# tertiary disks by UUID:
global TDISKS 
TDISKS = ["/dev/disk/by-uuid/39543e6e-cf50-4416-9669-e97a6abd2a37"]

# backup paths
# these are the paths of the folders you wish to back up to secondary
# and tertiary disks, respectively. Primary disks are set to back up the
# contents of the root filesystem (/*). NO TRAILING SLASHES.

global SBACKUP 
SBACKUP = "/home/bryant"

global TBACKUP 
TBACKUP = "/home/bryant/docs"

# use encryption:
use_encryption = True

# keyfile
# set the full path to your keyfile here
# this assumes a single keyfile for all backup disks
# set this to None if you don't have a single keyfile for all of your backups

keyfile = "/usr/local/bin/backup.keyfile"

#######

# import modules
import os, subprocess, sys

### preliminary functions ###
# these do the setup and post-copy work

def check_dirs():
    """checks that the folders which contain the mountpoints exist, creates them if they don't"""
    print("checking for mountpoints...")
    if os.path.isdir("/mnt/pbackup"):
        print("primary mountpoint exists.")
    else:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/pbackup")

    if os.path.isdir("/mnt/sbackup"):
        print("secondary mountpoint exists.")
    else:
        print("mountpoint /mnt/pbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/sbackup")

    if os.path.isdir("/mnt/tbackup"):
        print("tertiary mountpoint exists.")
    else:
        print("mountpoint /mnt/tbackup does not exist.\n\tcreating...")
        os.mkdir("/mnt/tbackup")
    
def mount_disks(wdisk):
    """mounts available backup disks in their respective subdirectories"""
    pfolder = 1
    sfolder = 1
    tfolder = 1
    
    pmapper = "pbackup"
    smapper = "sbackup"
    tmapper = "tbackup"

    if wdisk == "p":
        for pdisk in PDISKS:
            if os.path.islink(pdisk):
                subprocess.call("sync",shell=True)
                if os.path.isfile(keyfile):
                    print("keyfile found. Using keyfile to decrypt...")
                    subprocess.call("sudo cryptsetup luksOpen " + pdisk + " " + pmapper + str(pfolder) + " --key-file " + keyfile,shell=True)
    
                else:
                    print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                    subprocess.call("sudo cryptsetup luksOpen " + pdisk + " " + pmapper + str(pfolder),shell=True)

            if os.path.isdir("/mnt/pbackup/pbak" + str(pfolder)):
                print("mounting " + pmapper + str(pfolder) + " at /mnt/pbak" + str(pfolder))
                subprocess.call("mount " + "/dev/mapper/" + pmapper + str(pfolder) + " /mnt/pbackup/pbak" + str(pfolder),shell=True)
                pfolder += 1
            else:
                os.mkdir("/mnt/pbackup/pbak" + str(pfolder))
                subprocess.call("mount " + "/dev/mapper/" + pmapper + str(pfolder) + " /mnt/pbackup/pbak" + str(pfolder),shell=True)           
                pfolder += 1

    elif wdisk == "s":
        for sdisk in SDISKS:
            if os.path.islink(sdisk):
                subprocess.call("sync",shell=True)
                if os.path.isfile(keyfile): 
                    print("keyfile found. Using keyfile to decrypt...")
                    subprocess.call("sudo cryptsetup luksOpen " + sdisk + " " + smapper + str(sfolder) + " --key-file " + keyfile,shell=True)

                else:
                    print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                    subprocess.call("sudo cryptsetup luksOpen " + sdisk + " " + smapper + str(sfolder),shell=True)

            if os.path.isdir("/mnt/sbackup/sbak" + str(sfolder)):
                print("mounting " + smapper + str(sfolder) + " at /mnt/sbak" + str(sfolder))
                subprocess.call("mount " + "/dev/mapper/" + smapper + str(sfolder) + " /mnt/sbackup/sbak" + str(sfolder),shell=True)
                sfolder += 1
            else:
                os.mkdir("/mnt/sbackup/sbak" + str(folder))
                subprocess.call("mount " + "/dev/mapper/" + smapper + str(sfolder) + " /mnt/sbackup/sbak" + str(sfolder),shell=True)           
                sfolder += 1

    elif wdisk == "t":
        for tdisk in TDISKS:
            if os.path.islink(tdisk):
                subprocess.call("sync",shell=True)
                if os.path.isfile(keyfile):
                    print("keyfile found. Using keyfile to decrypt...")
                    subprocess.call("sudo cryptsetup luksOpen " + tdisk + " " + tmapper + str(tfolder) + " --key-file " + keyfile,shell=True)

                else:
                    print("keyfile not found or keyfile not set. \t\nAsking for passphrase...")
                    subprocess.call("sudo cryptsetup luksOpen " + tdisk + " " + tmapper + str(tfolder),shell=True)

            if os.path.isdir("/mnt/tbackup/tbak" + str(tfolder)):
                print("mounting " + tmapper + str(tfolder) + " at /mnt/tbak" + str(tfolder)) 
                subprocess.call("mount " + "/dev/mapper/" + tmapper + str(tfolder) + " /mnt/tbackup/tbak" + str(tfolder),shell=True)
                if os.path.islink(tdisk):
                    tfolder += 1
            else:
                os.mkdir("/mnt/tbackup/tbak" + str(tfolder))
                subprocess.call("mount " + "/dev/mapper/" + tmapper + " /mnt/tbackup/tbak" + str(tfolder),shell=True) 
                tfolder += 1

def umount_disks():
    """unmounts and relocks disks"""
    pdirs = os.listdir("/mnt/pbackup")
    sdirs = os.listdir("/mnt/sbackup")
    tdirs = os.listdir("/mnt/tbackup")

    for pdir in pdirs:
        if os.path.ismount("/mnt/pbackup/" + pdir):
            subprocess.call("umount /mnt/pbackup/" + pdir,shell=True)
        else:
            print("not a mountpoint")

    for sdir in sdirs:
        if os.path.ismount("/mnt/sbackup/" + sdir):
            subprocess.call("umount /mnt/sbackup/" + sdir,shell=True)
        else:
            print("not a mountpoint")

    for tdir in tdirs:
        if os.path.ismount("/mnt/tbackup/" + tdir):
            subprocess.call("umount /mnt/tbackup/" + tdir,shell=True)
        else:
            print("not a mountpoint")
    
    subprocess.call("cryptsetup luksClose /dev/mapper/pbackup*",shell=True)    
    subprocess.call("cryptsetup luksClose /dev/mapper/sbackup*",shell=True)
    subprocess.call("cryptsetup luksClose /dev/mapper/tbackup*",shell=True)

def check_disks():
    """checks to see how many disks exist, exits program if none are attached"""
    pdisknum = 0
    sdisknum = 0
    tdisknum = 0

    for pdisk in PDISKS:
        if os.path.islink(pdisk):
            pdisknum += 1
        else:
            print("\ndisk " + pdisk  + " not detected.")
    
    for sdisk in SDISKS:
        if os.path.islink(sdisk):
            sdisknum += 1
        else:
            print("\ndisk " + sdisk + " not detected.")
    
    for tdisk in TDISKS:
        if os.path.islink(tdisk):
            tdisknum += 1
        else:
            print("\ndisk " + tdisk + " not detected.")

    total = pdisknum + sdisknum + tdisknum

    if total == 0:
        print("\nERROR: no disks detected.")
        sys.exit()
    elif total > 0:
        print("found " + str(total) + " attached backup disks")
        print(str(pdisknum) + " Primary")
        print(str(sdisknum) + " secondary")
        print(str(tdisknum)  + " tertiary")
    return total, pdisknum, sdisknum, tdisknum

### backup functions ###
# these need serious work. Need to get them to loop through available mounted
# disks in their categories and then execute rsync

def pbackup(): 
    """calls rsync to backup the entire system to all pdisks"""

    dirs = os.listdir("/mnt/pbackup")
    for dir in dirs:
        if os.path.ismount("/mnt/pbackup/" + dir) == True:
            subprocess.call("sync",shell=True)
            print("syncing disks with rsync...")
            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} /* /mnt/pbackup/" + dir + "/ --exclude={/sys/*,/mnt/*,/proc/*,/dev/*,/lost+found,/media/*,/tmp/*,/home/*/.gvfs/*,/home/*/downloads/*,/opt/*,/run/*",shell=True)
            subprocess.call("sync",shell=True)
        else:
            continue

def sbackup():
    """calls rsync to backup everything under SBACKUP folder to all sdisks"""

    dirs = os.listdir("/mnt/sbackup")
    for dir in dirs:
        if os.path.ismount("/mnt/sbackup/" + dir):
            subprocess.call("sync",shell=True)
            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} " + SBACKUP + "/* /mnt/sbackup/" + dir + "/",shell=True)
            subprocess.call("sync",shell=True)
        else:
            continue

def tbackup():
    """calls rsync to backup everything under TBACKUP folder to all tdisks"""

    dirs = os.listdir("/mnt/tbackup")
    for dir in dirs:
        if os.path.ismount("/mnt/tbackup/" + dir):
            subprocess.call("sync",shell=True)
            subprocess.call("rsync --progress --human-readable --numeric-ids --inplace --verbose --archive --delete-after --hard-links --xattrs --delete --compress --skip-compress={*.jpg,*.bz2,*.gz,*.tar,*.tar.gz,*.ogg,*.mp3,*.tar.xz,*.avi} " + TBACKUP + "/* /mnt/sbackup/" + dir + "/",shell=True)
            subprocess.call("sync",shell=True)
        else:
            continue

#### main ####

# check for root access:
r=os.getuid()
if r != 0:
    print("ERROR: script not run as root.\n\tThis script MUST be run as root user.")
    sys.exit()

elif r == 0:
# program body
    check_dirs()
    
    d=check_disks()
    if d[1] > 0:
        mount_disks("p")
        pbackup()
        
    elif d[2] > 0:
        mount_disks("s")
        sbackup()

    elif d[3] > 0:
        mount_disks("t")
        tbackup()

    umount_disks()
    print("backup process complete.")

Last edited by ParanoidAndroid (2013-08-11 00:32:02)

Offline

#6 2013-08-26 22:18:46

windows_me
Member
From: England
Registered: 2013-08-14
Posts: 36
Website

Re: parallel Backup script written in python

Python does support parallelisation in the same way that IE6 supports web browsing, i.e. its generally a bit of a pain to do.  If you have the choice then you might be better off using something like Java, Mono or Ada that has built-in support for threading.  If you do stick to Python for it, look into libraries that run each parallel task as a separate process rather than threads, as it'll save you many headaches.


[10:04:21] Time for weekly full server backup.
[10:04:25] Redirecting it to "/dev/null" to make it go faster.
[10:04:53] Backup done! Amazing how fast modern technology is!

Offline

#7 2013-08-29 07:37:00

Darksoul71
Member
Registered: 2010-04-12
Posts: 319

Re: parallel Backup script written in python

IMO doing multi-process scripts is not exactly hard to do. You have queues and multi-process commands.

I have use pexpect for various multiprocess scripts myself:
https://pypi.python.org/pypi/pexpect/2.4

There are several good examples in the Python documentation as well:
http://docs.python.org/2/library/multiprocessing.html


My archlinux x86_64 host:
AMD E350 (2x1.6GHz) / 8GB DDR3 RAM / GeForce 9500GT (passive) / Arch running from 16GB USB Stick

Offline

#8 2013-08-29 21:32:03

Nisstyre56
Member
From: Canada
Registered: 2010-03-25
Posts: 82

Re: parallel Backup script written in python

windows_me wrote:

Python does support parallelisation in the same way that IE6 supports web browsing, i.e. its generally a bit of a pain to do.  If you have the choice then you might be better off using something like Java, Mono or Ada that has built-in support for threading.  If you do stick to Python for it, look into libraries that run each parallel task as a separate process rather than threads, as it'll save you many headaches.

Python supports concurrency, it just doesn't support parallelism. For something like this parallelism is unnecessary, so Python supports everything you'd need to do (although OS threads can still be unwieldy, so a lightweight threading library may be better suited).

For an explanation of the difference see: http://vimeo.com/49718712 (Rob Pike explaining the difference)

Last edited by Nisstyre56 (2013-08-29 21:33:01)


In Zen they say: If something is boring after two minutes, try it for four. If still boring, try it for eight, sixteen, thirty-two, and so on. Eventually one discovers that it's not boring at all but very interesting.
~ John Cage

Offline

Board footer

Powered by FluxBB