You are not logged in.
I need to backup some files from a remote server and store them as tar.bz2.
What I could do now is
rsync -avz --progress SRC DEST
tar -cvjf DEST.tar.bz2 DEST
rm -rf DESTHow could I do the download and compression with one step? For the cases where I won't have enough local space to accommodate two copies of the remote resources.
Offline
Assuming you do this from your local machine, you could ssh into the remote machine along with a command to create a tar archive, but output to stdout, then pipe that into a file. Example:
ssh user@host "tar -cvjf - SRC" > DEST.tar.bz2The big difference here is that the data is compressed on the server, I am assuming the server has tar in its path. Another, perhaps better option, would be to perform the command like this, in case you don't care about preserving full path.
ssh user@host "cd SRC && tar -cvjf - *" > DEST.tar.bz2Offline
OK, I forgot to mention that the storage space on the server is much more limited than the local one. It is impossible to create the tar ball on the server due to shortage of disk space.
In fact, my specific case is that the remote server is running out of space, so I want to backup some resources to a local machine and then clean up the remote one.
Offline
That tarball is not written to disk, it is written to stdout. This will not take up any disk space on the server.
Offline
Yes, you are right about the pipe!
Here are two more questions:
Firstly, since the connection to the remote server is not always stable, I would really love to have the possibility of "resume download". When I tried with rsync, the transfer was actually interrupted and I had to restart the transfer several times. With rsync it was no problem at all. But with
ssh user@host "tar -cvjf - SRC" > DEST.tar.bz2 , it will start the compression and transfer from scratch every time?
Secondly, how could I reverse the whole process so that it would work "out-of-the-box" if I re-upload the backup files. I know I could achieve it with rsync, since I can ensure that all the owners, groups and permissions are preserved and sync the folders in the way I like. Can tar and ssh transfer also do that?
Thank you!
Offline
I honestly don't know a way to make it work like rsync, which is possibly why rsync exists in the first place. This method will simply send the data directly over the network, but since it goes to a pipe I don't think there is any way to "resume" the operation if connection is lost. The way I see it, you can either use your original method if the connection is too unstable. Otherwise you could try to work with only a few files at a time to limit the amount of data sent over the network.
One way you could reverse the process would be:
cat ARCHIVE | ssh user@host "tar -xvjf -"This will extract into the home directory of the user. If you want to extract with / as base directory, I think you can do it like this.
cat ARCHIVE | ssh user@host "tar -xvjf - -C /"I have not tested this properly. There is one issue here. You need to be root on the server in order to extract the archive with ownership of files and permissions preserved.
The originally created archive will preserve permissions and ownership properly, you can verify that yourself. If you extract a tar archive as root (or with sudo) it will extract files with original ownership. I honestly don't know right now how you could run tar on the server as root. Putting sudo into the ssh command will not work without the '-t' option, but that will not let you pipe the archive contents. There may be some options to get this working, but I am not sure how.
There are probably someone who has a much better idea than me here. But with my idea, you actually need to ssh into the server as root, and allowing root access over ssh is generally not advisable. Personally, I would just suggest unpacking the archives locally with sudo so ownership is intact, then just using rsync to the server.
Offline