You are not logged in.

#1 2009-02-13 22:04:21

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

pycp: cp and mv with a progressbar

Hi there.

So, I had once to copy a bunch of big files from one partition to another.
Just made of few `cp', but I was really feeling like there should have been a progress-bar somewhere ...

I thought that it would be quite easy to write a program that would fork cp as an other process, monitor the size of the source, and the size of the destination, and use this to print a nice progress-bar.

This is how pycp was born, sources were put on gitorious, and a package was made on AUR.

So, feel free to download it and send me any remarks you could have.

Please consider this as an exercise : my own purpose was to learn a bit more about python, AUR, and git wink
(BTW, setup.py is very nice)

And (who knows ?) ,this script could be useful to you after all tongue

EDIT:
Mandatory screenshot

$~ pycp -g ~/test/tbbt/ /tmp
 29% -  45.11 M/s - [#########                        ] - 2 on 4 ETA  : 00:00:11
 19% /home/yannick/test/tbbt/t2.avi [#####                     ] ETA  : 00:00:01

Note that sources are now hosted on github:
http://github.com/yannicklm/pycp  (gitorious project has been deleted)

Package is available on AUR:
https://aur.archlinux.org/packages/pycp

Last edited by Yannick_LM (2015-09-11 16:46:48)

Offline

#2 2009-02-14 00:14:59

evr
Arch Linux f@h Team Member
Registered: 2009-01-23
Posts: 554

Re: pycp: cp and mv with a progressbar

it's funny because i had actually thought about writing something like this in the past.  I like your version, it seems to work well.  The one thing I might suggest is that you should be able to list a directory as the destination and it will keep the same filename, like a regular cp command.  Right now this just throws me an error. Ex:

pycp /media/share/Linux/archlinux-2008.06-core-i686.iso /tmp/

Error: '/media/share/Linux/archlinux-2008.06-core-i686.iso' and '/media/share/Linux/archlinux-2008.06-core-i686.iso' are the same file

anyway, just a suggestion! thanks for putting it up!

Offline

#3 2009-02-14 06:22:47

joephantom
Member
From: Latinoamérica
Registered: 2008-01-09
Posts: 94
Website

Re: pycp: cp and mv with a progressbar

¡Cool!


By striving to do the impossible, man has always achieved what is possible. Those who have cautiously done no more than they believed possible have never taken a single step forward - Mikhail Bakunin

Offline

#4 2009-02-14 09:57:08

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Yup, right now it does not work at all with dirs...

(and I just realised, thanks to your example, that it's not that  obvious. Anyway, I'll try to fix this.)

Offline

#5 2009-02-14 10:13:01

Wintervenom
Member
From: /home/wintervenom
Registered: 2008-08-20
Posts: 1,011
Website

Re: pycp: cp and mv with a progressbar

Could it eventually be a drop-in replacement for cp, so that one can alias cp='pycp' and it will use the same parameters, but will give a progress?


My [GitHub] and my [artwork].
Data is guaranteed to stay crunchy in milk.
Keep your _✍ hand strong.

Offline

#6 2009-02-14 10:34:07

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Wintervenom wrote:

Could it eventually be a drop-in replacement for cp, so that one can alias cp='pycp' and it will use the same parameters, but will give a progress?

Yup, this is the idea. The keywork is *eventually* wink


Anyway, I just uploaded a new version, fixing a few bugs.

Offline

#7 2009-02-14 16:18:38

evr
Arch Linux f@h Team Member
Registered: 2009-01-23
Posts: 554

Re: pycp: cp and mv with a progressbar

well that was fast, thanks! smile btw, i'm really liking this program. I've always thought cp should optionally have a progressbar

Offline

#8 2009-02-14 23:52:22

Stefan Husmann
Member
From: Germany
Registered: 2007-08-07
Posts: 1,000

Re: pycp: cp and mv with a progressbar

Not to destroy your enthusiams,  but there is a tool in AUR with a similar task: vcp. It works quite well.

Online

#9 2009-02-15 00:00:04

Square
Member
Registered: 2008-06-11
Posts: 429

Re: pycp: cp and mv with a progressbar

Also, there is another program that does this called bar. It's in [community].


 

Offline

#10 2009-02-15 00:38:38

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Not to destroy your enthusiams,  but there is a tool in AUR with a similar task: vcp. It works quite well.

Hu, hu wink

I promiss I made some research before submitting.
(and was expecting something like that to happen)

Thanks for putting that out, guys.

(And vcp is indeed quite nice)


Well, anyway re-inventing the wheel is part of the process isn't it?

Last edited by Yannick_LM (2009-02-15 00:41:29)

Offline

#11 2009-02-15 15:03:06

scj
Member
From: Sweden
Registered: 2007-09-23
Posts: 158

Re: pycp: cp and mv with a progressbar

Also, there is a more generalized program for this kind of thing: pv.

For instance, a naive file copy command with progress bar would be:

pcp(){
cat "$1" | pv -s $(ls -s --block-size 1 | cut -d ' ' -f 1) > "$2"
}

Of course, it doesn't deal with globbing and that sort of stuff.

Offline

#12 2009-02-15 21:32:00

seenxu
Member
Registered: 2008-08-25
Posts: 95

Re: pycp: cp and mv with a progressbar

Good job, very nice and useful script! more straightforward than pv, vcp or bar etc.

If it also has a speed indicator and the elapsed time/total time consumed in copying files, it would be great. smile

Offline

#13 2009-02-18 21:31:23

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Version 1.3.1 comes with a file transfer speed and a ETA.

Enjoy!

Last edited by Yannick_LM (2009-02-18 21:31:32)

Offline

#14 2009-02-18 21:56:52

seenxu
Member
Registered: 2008-08-25
Posts: 95

Re: pycp: cp and mv with a progressbar

Yannick_LM wrote:

Version 1.3.1 comes with a file transfer speed and a ETA.

Enjoy!

woo, thx. big_smile

Offline

#15 2009-02-21 18:46:58

seenxu
Member
Registered: 2008-08-25
Posts: 95

Re: pycp: cp and mv with a progressbar

Yannick_LM wrote:

Version 1.3.1 comes with a file transfer speed and a ETA.

Enjoy!

is it also possible to add a mv feature. big_smile

Offline

#16 2009-02-22 04:50:26

Ghost1227
Forum Fellow
From: Omaha, NE, USA
Registered: 2008-04-21
Posts: 1,418
Website

Re: pycp: cp and mv with a progressbar

can you make it work with multiple files?

pycp file1 file2 file3 /path/to/destdir

Offline

#17 2009-02-22 21:36:59

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Ghost1227 wrote:

can you make it work with multiple files?

Done in version 2.1, which is also able to copy directories.

is it also possible to add a mv feature.

That's planned wink

Offline

#18 2009-02-24 10:14:14

robmaloy
Member
From: Germany
Registered: 2008-05-14
Posts: 263

Re: pycp: cp and mv with a progressbar

some changes in pycp.py to:
support some options
implemented -o|--overwrite option
doesnt overwrite anymore when doing "pycp SOURCE SOURCE DESTINATION_DIRECTORY"
skips existing files instead of canceling whole operation

if knew how to use git, i'd upload it there big_smile


EDIT: maybe, to behave more like the real "cp", files should be overwritten by default, unless the -u option is specified, but thatsyour decision


btw: nice little app cool

64c64
<     def __init__(self, source, destination):
---
>     def __init__(self, source, destination, cpopts):
68a69
>     self.cpopts        = cpopts
134c135
<         opts, args = getopt.getopt(sys.argv[1:], "hv", ["help", "version"])
---
>         opts, args = getopt.getopt(sys.argv[1:], "hvo", ["help", "version", "overwrite"])
139a141,142
>     cpopts = []
> 
146c149
<         elif opt == "--version":
---
>         elif opt in ("-v", "--version"):
148a152,154
>     elif opt in ("-o", "--overwrite"):
>         cpopts.append("overwrite")
> 
174c180
<             recursive_copy(source, destination)
---
>             recursive_copy(source, destination, cpopts)
179c185
< def _prepare_copy(source, destination):
---
> def _prepare_copy(source, destination, cpopts):
189a196
>     skip = False
199a207,215
> 
>         source_dir, source_file = path.split(source)
>         destination_file = path.join(destination, source_file)
>         if ((path.exists(destination_file)) and (not 'overwrite' in cpopts)):
>                     #print "Error: file '" + destination_file + "' already exists"
>                     #exit(1)
>                     print "file '" + destination_file + "' already exists"
>             skip = True
> 
203,206c219,225
<             else:
<                 # refusing to override an exiting file
<                 print "Error: file '" + destination + "' already exists"
<                 exit(1)
---
>         
>         elif not 'overwrite' in cpopts:
>                 # refusing to override an exiting file if the -o option is not specified
>                 #print "Error: file '" + destination + "' already exists"
>                 #exit(1)
>                 print "file '" + destination_file + "' already exists"
>         skip = True
212c231,232
<             exit(1)
---
>             #exit(1)
>         skip = True
229c249
<     return new_destination
---
>     return new_destination, skip
233c253
< def recursive_copy(source, destination):
---
> def recursive_copy(source, destination, cpopts):
237c257
<     new_destination = _prepare_copy(source, destination)
---
>     new_destination, skip = _prepare_copy(source, destination, cpopts)
239,245c259,266
<     if path.isdir(source):
<         for file_name in listdir(source):
<             recursive_copy(path.join(source,      file_name),
<                            path.join(new_destination, file_name))
<     else:
<         copy_manager = CopyManager(source, new_destination)
<         copy_manager.copy()
---
>     if (not skip):
>         if path.isdir(source):
>             for file_name in listdir(source):
>                 recursive_copy(path.join(source,      file_name),
>                                path.join(new_destination, file_name), cpopts)
>         else:
>             copy_manager = CopyManager(source, new_destination, cpopts)
>             copy_manager.copy()

Last edited by robmaloy (2009-02-24 10:24:08)


☃ Snowman ☃

Offline

#19 2009-02-24 22:29:45

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Wow, already a patch!
Thanks, I was not expecting so much interest.

robmaloy wrote:

EDIT: maybe, to behave more like the real "cp", files should be overwritten by default, unless the -u option is specified, but that's your decision

-u is for --ubuntu I presume ? (1)
</troll>

I'm a huge fan of
" alias cp='cp -i' "
so I'm in favor of keeping the skipping behavior wink

Having a configuration file just for this looks a bit overkil, but may be fun to implement, so who knows ?

Anyway, version 2.2 comes with robmaloy patch.

(1) EDIT:
I'm also learning about debian (or ubuntu) packaging.
In case someone really bored would like to have a look, it's in a PPA:
https://launchpad.net/~yannick-lm/+archive/ppa

Last edited by Yannick_LM (2009-02-24 22:35:38)

Offline

#20 2009-02-25 16:07:46

evr
Arch Linux f@h Team Member
Registered: 2009-01-23
Posts: 554

Re: pycp: cp and mv with a progressbar

just so you know, i'm getting a md5 checksum error when upgrading to the latest version (2.2-1)

Offline

#21 2009-02-25 21:16:24

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Sorry, my mistake.

Fixed in a new package release

pycp-2.2-2

Offline

#22 2009-02-26 02:11:27

evr
Arch Linux f@h Team Member
Registered: 2009-01-23
Posts: 554

Re: pycp: cp and mv with a progressbar

no worries! keep up the good work!

Offline

#23 2009-03-01 11:50:24

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

Version 3.0 is out!

new: man pages, improved setup.py
add a pymv script.

Right now, it just calls pycp with an argument delete_afterwards

Enjoy!

Offline

#24 2009-03-01 13:30:06

seenxu
Member
Registered: 2008-08-25
Posts: 95

Re: pycp: cp and mv with a progressbar

Yannick_LM wrote:

Version 3.0 is out!


add a pymv script.

it just calls pycp with an argument delete_afterwards

cool! big_smile


but I don't see any changes in git, haven't committed the code?

Last edited by seenxu (2009-03-01 13:31:38)

Offline

#25 2009-03-01 16:02:56

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: pycp: cp and mv with a progressbar

seenxu wrote:

but I don't see any changes in git, haven't committed the code?

Just did wink

Offline

Board footer

Powered by FluxBB