You are not logged in.

#1 2012-05-07 18:10:46

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

[SOLVED] Duplicity getting 'stuck' when attempting to restart a backup

Hey all,

After a backup attempt failed partway through (first-time full backup) with this error:

AsyncScheduler: scheduling task for asynchronous execution
No handlers could be found for logger "paramiko.transport"
AsyncScheduler: a previously scheduled task has failed; propagating the result immediately
AsyncScheduler: task execution done (success: False)
Backend error detail: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1391, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1384, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1354, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 500, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 399, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/site-packages/duplicity/asyncscheduler.py", line 151, in schedule_task
    return self.__run_asynchronously(fn, params)
  File "/usr/lib/python2.7/site-packages/duplicity/asyncscheduler.py", line 215, in __run_asynchronously
    with_lock(self.__cv, wait_for_and_register_launch)
  File "/usr/lib/python2.7/site-packages/duplicity/dup_threading.py", line 100, in with_lock
    return fn()
  File "/usr/lib/python2.7/site-packages/duplicity/asyncscheduler.py", line 207, in wait_for_and_register_launch
    check_pending_failure() # raise on fail
  File "/usr/lib/python2.7/site-packages/duplicity/asyncscheduler.py", line 191, in check_pending_failure
    self.__failed_waiter()
  File "/usr/lib/python2.7/site-packages/duplicity/dup_threading.py", line 201, in caller
    value = fn()
  File "/usr/lib/python2.7/site-packages/duplicity/asyncscheduler.py", line 183, in <lambda>
    (waiter, caller) = async_split(lambda: fn(*params))
  File "/usr/bin/duplicity", line 398, in <lambda>
    async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename, vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 296, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/site-packages/duplicity/backends/sshbackend.py", line 191, in put
    raise BackendException("sftp put of %s (as %s) failed: %s" % (source_path.name,remote_filename,e))
BackendException: sftp put of /tmp/duplicity-9_jssu-tempdir/mktemp-uqQRIn-42 (as duplicity-full.20120507T060530Z.vol41.difftar
.gpg) failed: Server connection dropped: 
BackendException: sftp put of /tmp/duplicity-9_jssu-tempdir/mktemp-uqQRIn-42 (as duplicity-full.20120507T060530Z.vol41.difftar
.gpg) failed: Server connection dropped: 

the process keeps getting stuck when I try to restart it. There's almost no CPU use and no useful error messages. Here's the output (the version and parameters are included) with -v debug:

Using archive dir: /root/.cache/duplicity/1498ef4c9557164d69523e5aec4c78ce                                           [83/4524]
Using backup name: 1498ef4c9557164d69523e5aec4c78ce
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.giobackend Failed: No module named gio
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.cloudfilesbackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Main action: inc
================================================================================
duplicity 0.6.18 (February 29, 2012)
Args: /usr/bin/duplicity --use-agent -vdebug --encrypt-key YYYYYYYY --sign-key XXXXXXX --full-if-older-than 60D --exclude /mn
t/backup/bfd-training-data --exclude /mnt/backup/fsarchiver --exclude /mnt/backup/media --exclude /mnt/backup/usapparel --asyn
chronous-upload /mnt/backup scp://user@destinatino.com//backups/homeserver
Linux homeserver 3.0.29-1-lts #1 SMP PREEMPT Mon Apr 23 09:41:11 CEST 2012 x86_64 AMD Sempron(tm) Processor 3400+
/usr/bin/python2 2.7.3 (default, Apr 24 2012, 00:00:54) 
[GCC 4.7.0 20120414 (prerelease)]
================================================================================
Using temporary directory /tmp/duplicity-OjWCPd-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-OjWCPd-tempdir/mkstemp-GZ2ieI-1
Temp has 490532864 available, backup will use approx 60293120.
Local and Remote metadata are synchronized, no sync needed.
41 files exist on backend
2 files exist in cache

Extracting backup chains from list of files: ['duplicity-full.20120507T060530Z.manifest.part', 'duplicity-full-signat[51/4524]
0507T060530Z.sigtar.part', 'duplicity-full.20120507T060530Z.vol1.difftar.gpg', 'duplicity-full.20120507T060530Z.vol2.difftar.g
pg', 'duplicity-full.20120507T060530Z.vol3.difftar.gpg', 'duplicity-full.20120507T060530Z.vol4.difftar.gpg', 'duplicity-full.2
0120507T060530Z.vol5.difftar.gpg', 'duplicity-full.20120507T060530Z.vol6.difftar.gpg', 'duplicity-full.20120507T060530Z.vol7.d
ifftar.gpg', 'duplicity-full.20120507T060530Z.vol8.difftar.gpg', 'duplicity-full.20120507T060530Z.vol9.difftar.gpg', 'duplicit
y-full.20120507T060530Z.vol10.difftar.gpg', 'duplicity-full.20120507T060530Z.vol11.difftar.gpg', 'duplicity-full.20120507T0605
30Z.vol12.difftar.gpg', 'duplicity-full.20120507T060530Z.vol13.difftar.gpg', 'duplicity-full.20120507T060530Z.vol14.difftar.gp
g', 'duplicity-full.20120507T060530Z.vol15.difftar.gpg', 'duplicity-full.20120507T060530Z.vol16.difftar.gpg', 'duplicity-full.
20120507T060530Z.vol17.difftar.gpg', 'duplicity-full.20120507T060530Z.vol18.difftar.gpg', 'duplicity-full.20120507T060530Z.vol
19.difftar.gpg', 'duplicity-full.20120507T060530Z.vol20.difftar.gpg', 'duplicity-full.20120507T060530Z.vol21.difftar.gpg', 'du
plicity-full.20120507T060530Z.vol22.difftar.gpg', 'duplicity-full.20120507T060530Z.vol23.difftar.gpg', 'duplicity-full.2012050
7T060530Z.vol24.difftar.gpg', 'duplicity-full.20120507T060530Z.vol25.difftar.gpg', 'duplicity-full.20120507T060530Z.vol26.diff
tar.gpg', 'duplicity-full.20120507T060530Z.vol27.difftar.gpg', 'duplicity-full.20120507T060530Z.vol28.difftar.gpg', 'duplicity
-full.20120507T060530Z.vol29.difftar.gpg', 'duplicity-full.20120507T060530Z.vol30.difftar.gpg', 'duplicity-full.20120507T06053
0Z.vol31.difftar.gpg', 'duplicity-full.20120507T060530Z.vol32.difftar.gpg', 'duplicity-full.20120507T060530Z.vol33.difftar.gpg
', 'duplicity-full.20120507T060530Z.vol34.difftar.gpg', 'duplicity-full.20120507T060530Z.vol35.difftar.gpg', 'duplicity-full.2
0120507T060530Z.vol36.difftar.gpg', 'duplicity-full.20120507T060530Z.vol37.difftar.gpg', 'duplicity-full.20120507T060530Z.vol3
8.difftar.gpg', 'duplicity-full.20120507T060530Z.vol39.difftar.gpg', 'duplicity-full.20120507T060530Z.vol40.difftar.gpg', 'dup
licity-full.20120507T060530Z.vol41.difftar.gpg']
File duplicity-full.20120507T060530Z.manifest.part is not part of a known set; creating new set
File duplicity-full-signatures.20120507T060530Z.sigtar.part is not part of a known set; creating new set
Ignoring file (rejected by backup set) 'duplicity-full-signatures.20120507T060530Z.sigtar.part'
File duplicity-full.20120507T060530Z.vol1.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol2.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol3.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol4.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol5.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol6.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol7.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol8.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol9.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol10.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol11.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol12.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol13.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol14.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol15.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol16.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol17.difftar.gpg is part of known set

File duplicity-full.20120507T060530Z.vol18.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol19.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol20.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol21.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol22.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol23.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol24.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol25.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol26.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol27.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol28.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol29.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol30.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol31.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol32.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol33.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol34.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol35.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol36.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol37.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol38.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol39.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol40.difftar.gpg is part of known set
File duplicity-full.20120507T060530Z.vol41.difftar.gpg is part of known set
Found backup chain [Sun May  6 23:05:30 2012]-[Sun May  6 23:05:30 2012]
Last full backup left a partial set, restarting.
Last full backup date: Sun May  6 23:05:30 2012
Collection Status
-----------------
Connecting with backend: SftpBackend
Archive dir: /root/.cache/duplicity/1498ef4c9557164d69523e5aec4c78ce

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------

Chain start time: Sun May  6 23:05:30 2012
Chain end time: Sun May  6 23:05:30 2012
Number of contained backup sets: 1
Total number of contained volumes: 41
 Type of backup set:                            Time:      Num volumes:
                Full         Sun May  6 23:05:30 2012                41
-------------------------
No orphaned or incomplete backup sets found.
RESTART: Volumes 41 to 42 failed to upload before termination.
         Restarting backup at volume 41.
Registering (mktemp) temporary file /tmp/duplicity-OjWCPd-tempdir/mktemp-Fx_YTQ-2
^CRemoving still remembered temporary file /tmp/duplicity-OjWCPd-tempdir/mktemp-Fx_YTQ-2
Removing still remembered temporary file /tmp/duplicity-OjWCPd-tempdir/mkstemp-GZ2ieI-1
INT intercepted...exiting.

Does anyone have some experience with this? It's a fairly good sized backup (well, for me at least) so I'd really hate to have to start over every time one fails!

Thanks,
Scott

Last edited by firecat53 (2012-05-08 18:23:19)

Offline

#2 2012-05-08 18:21:40

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: [SOLVED] Duplicity getting 'stuck' when attempting to restart a backup

Ok, solved-ish. There were actually 3 issues:
1. There's a problem in the 0.6.18 new python paraminko ssh internals that was preventing it from uploading more than about 1GB at a time, possibly because it's reusing the same ssh connection which some webhosts will kick off after a certain time or amount.
2. The --num-retries parameter wasn't being read by the ssh backend
3. There's a bug in some encryption validation code that had to be removed before a 'resume backup' operation would work correctly (it won't read the GPG encryption key correctly).

So I reverted to 0.6.17 and removed a previously applied patch from the code found here.

So far it seems to be working. The dev who answered me on Launchpad said that there's a memory leak with 0.6.17 so watch out when doing any backups over ~100GB or so.

Scott

Offline

Board footer

Powered by FluxBB