You are not logged in.

#51 2010-08-08 21:15:36

Nihathrael
Member
From: Freising, Germany
Registered: 2007-10-21
Posts: 82
Website

Re: pycp: cp and mv with a progressbar

Nice tool. Just tested it, works nicely. Great work!


Unknown Horizons - Open source real-time strategy game with the comfy Anno 1602 feeling!

Offline

#52 2010-08-14 14:35:05

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Thanks smile

Just a quick question, for those who care.

I stumbled upon an  excellent fileutils patch :
http://www.beatex.org/web/advancedcopy.html

A screenshot:

1469 files copied so far...                          130.2 MiB / 298.7 MiB
[|||||||||||||||||||||||||||||------------------------------------] 43.6 %
Copying at  23.4 MiB/s (about 0h 0m 31s remaining)
BigFolder/automation/delegates/class-list.jpeg         1.5 MiB /   2.0 MiB
[|||||||||||||||||||||||||||||||||||||||||||||||------------------] 75.0 %

So I figured out pycp should do something similar.

(unfortunately, python-progressbar does not play well with several lines of progressbars)

So I ended up with something that looks like this:

Copying from old/ to new/
 33% - [#############                          ] - (1 files / 3)
 50% - [#######       ] - File #2  -  24.97 B/s | ETA:  00:00:02

What do you think?


The code to produce this is here:

import time
import sys
from progressbar import ProgressBar
from progressbar import ProgressBarWidget
from progressbar import Percentage
from progressbar import FileTransferSpeed
from progressbar import ETA
from progressbar import Bar


def transfer_file(i):
    small_pbar = ProgressBar(
      widgets = [
        Percentage(),
        " - ",
       Bar(marker='#', left='[', right=']'),
        " - ",
        "File #%i " % (i+1),
        " - ",
        FileTransferSpeed(),
        " | ",
        ETA()])
    small_pbar = small_pbar.start()
    for i in range(0, 3):
        time.sleep(1)
        small_pbar.update(50*i)


class SoFarWidget(ProgressBarWidget):
    def update(self, pbar):
        return "(%i files / %i)" % \
            (pbar.currval, pbar.maxval)

print "Copying from old/ to new/"
total_pbar = ProgressBar(
    widgets = [
      Percentage(),
      " - ",
      Bar(marker="#", left="[", right="]"),
      " - ",
      SoFarWidget()],
    maxval=3)

total_pbar = total_pbar.start()
for i in range(0, 3):
    total_pbar.update(i)
    print
    transfer_file(i)
    if i == 2:
        sys.stderr.write("\033[2A")
        total_pbar.finish()
        break
    sys.stderr.write("\033[2A")
    sys.stderr.write("\033[K\n")
    sys.stderr.write("\033[K\n")
    sys.stderr.write("\033[2A")

print "done"

Note the use of ANSI escape characters, which won't work fine on windows ....

Last edited by Yannick_LM (2010-08-14 14:48:02)

Offline

#53 2010-10-30 17:29:23

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Hi, for those still interested:

Version 6.0 is out !

ChangeLog:

* pycp no longer depends on progressbar
* add pycp -g option to display a global pbar on
  several lines
* fix problems with latest python update

Screenshot:

pycp -g ~/test/tbbt/ /tmp
 29% -  45.11 M/s - [#########                        ] - 2 on 4 ETA  : 00:00:11
 19% /home/yannick/test/tbbt/t2.avi [#####                     ] ETA  : 00:00:01

Offline

#54 2011-02-13 17:35:53

Falstaff
Member
Registered: 2008-11-03
Posts: 79

Re: pycp: cp and mv with a progressbar

Neat little package.

Offline

#55 2011-02-14 00:14:58

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Thanks a lot smile

Offline

#56 2011-10-30 04:30:02

tamikana
Member
Registered: 2010-04-06
Posts: 17

Re: pycp: cp and mv with a progressbar

symbolic link support would be really nice big_smile

Offline

#57 2011-10-30 15:49:36

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Hum. It's trickier than it sounds because pycp gets really confused when copying files which size is 0.
(as empty file or symlinks)

I'm still trying to figure out a solution ....

Offline

#58 2011-10-30 16:09:22

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Feels a little hackish, but I've pushed a fix.

Please upgrade pycp-git and tell me if it's okay.

Offline

#59 2011-11-13 16:14:25

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Version v6.1 is out, with a not that bad support of symlinks, and a little surprise for you archers made by aldrik
(thanks, aldrick)

Offline

#60 2011-11-14 14:47:49

clod89
Member
Registered: 2011-11-14
Posts: 18

Re: pycp: cp and mv with a progressbar

Maybe it's me missing something but i can't copy directories right, most of the times if i do pycp dir/ documents/ it will copy all the files in dir to documents without creating a dir/ directory in the documents folder, other times instead it works fine

Offline

#61 2011-11-14 17:17:16

Zom
Member
From: Sweden
Registered: 2007-10-27
Posts: 406

Re: pycp: cp and mv with a progressbar

Try 'pycp dir documents/' instead, in case it uses the same syntax as rsync.

From the manpage of rsync:

       A  trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination.  You can
       think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but
       in both cases the attributes of the containing directory are transferred to the containing directory on the destination.

Offline

#62 2011-11-14 18:42:45

clod89
Member
Registered: 2011-11-14
Posts: 18

Re: pycp: cp and mv with a progressbar

Zom wrote:

Try 'pycp dir documents/' instead, in case it uses the same syntax as rsync.

From the manpage of rsync:

       A  trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination.  You can
       think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", but
       in both cases the attributes of the containing directory are transferred to the containing directory on the destination.

Yes that was it :-) , still it's quite uncomfortable when using autocompletion as it always adds / at the end of the directory

Offline

#63 2011-11-14 22:17:46

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Hum. This is not correct.

The behavior is the same as cp.

When you run
up

pycp source dest

the behavior changes depending on whether or not 'dest' exists.

If 'dest' exist, you will end with 'source' contents in 'dest/source', and if 'dest' does not exist, the contents of 'source' will be copied
in a new directory named 'dest'

Adding a '/' won't change that.

Or maybe there is something else I'm missing ...

Offline

#64 2011-11-14 23:03:32

clod89
Member
Registered: 2011-11-14
Posts: 18

Re: pycp: cp and mv with a progressbar

What happens is that when a directory is copied with this syntax "dir/" supposing that dir/ contains test1 test2 "pycp dir/ documents/"  will result in documents/test1 documents/test2 if used wihtout "/"  the command "pycp dir documents/" will result in documents/dir/test1 documents/dir/test2, i hope i made it more clear, is it intended or is it a bug?

Last edited by clod89 (2011-11-14 23:13:33)

Offline

#65 2011-11-14 23:18:21

SS4
Member
From: !Rochford, Essex
Registered: 2010-12-05
Posts: 699

Re: pycp: cp and mv with a progressbar

I'm having a problem with recursively copying directories.

[23:12:58] $ mkdir ~/testing/pycp-dir
$  pycp -sg /mnt/Documents/NR/ ~/testing/pycp-dir/

This has essentially done

cp /mnt/Documents/NR/* ~/testing/pycp-dir/

. In other words, it's dumped the contents of the SRC file into DEST instead of copying the folder over first and then it's contents


Edit: I can confirm the poster above: omitting the final "/" from the source dir it copies as expected

[23:13:48] $  pycp -sg /mnt/Documents/NR ~/testing/pycp-dir/ 

Also works with a dir containing no subfolders

Last edited by SS4 (2011-11-14 23:21:10)


Rauchen verboten

Offline

#66 2011-11-16 18:50:53

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Hum.

Looks like we really have a bug here.

I don't lilke the behavior depending on wether or not you have a final '/' or whether or not you have subfolders.

And the goal (as the name implies) for pycp is to have exactly the same behavior as cp.

Still not sure where the bug really is.

I'll have a look maybe next week (sorry, quite busy here)

If you want this bug to be fixed quicker, you can always try and write a (failing) automatic test.

It's not that hard (well, if you no a little bit of Python, of course) and it would really help me fix it.


EDIT:
just to be sure: you're using pycp version 6.1, right?

Last edited by Yannick_LM (2011-11-16 18:52:41)

Offline

#67 2011-11-18 14:36:05

clod89
Member
Registered: 2011-11-14
Posts: 18

Re: pycp: cp and mv with a progressbar

Yannick_LM wrote:

EDIT:
just to be sure: you're using pycp version 6.1, right?

The latest from the git

Offline

#68 2011-11-20 16:59:38

shadowdancer
Member
Registered: 2011-11-16
Posts: 15

Re: pycp: cp and mv with a progressbar

This is cool man!

Anybody alias cp to pycp and mv to pymv already?

Still unsure to do that smile

Offline

#69 2011-11-20 17:52:12

SS4
Member
From: !Rochford, Essex
Registered: 2010-12-05
Posts: 699

Re: pycp: cp and mv with a progressbar

shadowdancer wrote:

This is cool man!

Anybody alias cp to pycp and mv to pymv already?

Still unsure to do that smile

I have smile although It'd be even better with support for the -u flag. From man cp:

 -u, --update
              copy only when the SOURCE file is newer than the destination file or when the destination file is missing

Last edited by SS4 (2011-11-20 17:52:30)


Rauchen verboten

Offline

#70 2011-11-21 21:22:45

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Huhu...

I'll try to fix the bugs before implementing new features wink

But that's a neat idea.

Offline

#71 2012-05-04 12:27:26

Natanji
Member
Registered: 2009-09-22
Posts: 125

Re: pycp: cp and mv with a progressbar

Hey there,
first of all, thanks for the great tool! I would love to see, however, if you could improve the pymv tool. This would allow us to just alias mv to pymv and have nice progress bars at all times.

Right now, the main reason for this failing is that mv can just rename files if they reside on the same partition/mount point. mv'ing a 1 GB file to a subdirectory, for instance, only takes a fraction of a second. With pymv it takes forever.

Do you see any way for pymv *only* using pycp with delete_afterwards if the mv is not a rename, but actually has to copy over data to a different partition? That would be amazing.

Last edited by Natanji (2012-05-04 12:27:51)

Offline

#72 2012-05-04 19:15:05

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Well, my reasoning for this was: if it's going to take a fraction of a second, why would you need a progressbar in the first place ?

But I guess it won't be very hard to implement. I'll have a look wink

Offline

#73 2012-12-17 02:00:55

jmartl109
Member
Registered: 2011-03-17
Posts: 24

Re: pycp: cp and mv with a progressbar

This is very nice!

I have a feature request: any chance for it to handle errors that occur during the copying process, instead of just stopping? For example, suppose I want to copy a bunch of files to a usb drive formatted FAT, but a few of the filenames have windows-illegal characters in them. In this scenario I'd like all the non-problematic files copied without my intervention, and then a list of error messages at the end saying which files failed to copy.

Offline

#74 2012-12-17 19:37:00

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

This is very nice!

Thanks, I appreciate it.

Any chance for it to handle errors that occur during the copying process, instead of just stopping?

I think what you need is already implemented.

Try with --ignore-errors ;-)

Offline

#75 2012-12-20 04:03:50

jmartl109
Member
Registered: 2011-03-17
Posts: 24

Re: pycp: cp and mv with a progressbar

Try with --ignore-errors ;-)

Whoa, I somehow totally missed that. Thanks.

Offline

Board footer

Powered by FluxBB