Stream output from terminal to another computer over the network... ?

kdar · 2011-12-21 21:26:25

How can I stream output (for example something from terminal) from one computer to a file in another computer through the network?

I am thinking of something like > used in regular settings:
$ cat testfile.txt > /dev/lp0

Awebb · 2011-12-21 21:40:55

sshfs? nfs?

Ashren · 2011-12-21 21:46:58

netcat

kdar · 2011-12-22 04:58:05

I was able to do it just with just ssh.

I didn't have sshfs or nfs on the system and netcat seemed a bit problematic that you had to open port on the receiving side (at least from my first understanding, never dealt with netcat before).

This what I did:

gzip -c9 file.txt | ssh user@remote-server "cat > /home/file.txt.bz2"

Also had to get ssh key generated to not enter password everytime.

Last edited by kdar (2011-12-22 04:59:24)

fsckd · 2011-12-22 05:48:24

kdar wrote:

netcat seemed a bit problematic that you had to open port on the receiving side (at least from my first understanding

Eh? You've misunderstood. You'll need something listening on the other side no matter what protocol you use. netcat is a little more flexible as it is a little more raw. Use netcat -l to receive data.

Example,
In one terminal run netcat -l -p 9001 > outfile
In another terminal run echo y helo thar | netcat -c 127.0.0.1 9001
Then open outfile and see the message.

hopla · 2011-12-22 10:50:32

kdar wrote:

This what I did:

gzip -c9 file.txt | ssh user@remote-server "cat > /home/file.txt.bz2"

That is the best way to do it kdar, it's certainly how I solve those kinds of problems. There are also endless variations on this, for example, say you also want to see the output that goes to the remote server on *your* terminal (this is not very efficient however ):

gzip -c9 file.txt | ssh user@remote-server "tee /home/file.txt.bz2"

The pipe functionality of SSH is very cool

Blµb · 2011-12-27 09:49:35

Just keep in mind that the above 'tee' version sends the output back over the network, and you can avoid this by, for example, tee-ing to stderr - locally

gzip -c9 file.txt | tee /dev/fd/2 | ssh user@remote "cat > /home/file.txt.gz"

I was actually looking for something like that for a while, until I remembered /dev/fd/* - I should add that to the ooh nice thread xD

Last edited by Blµb (2011-12-27 09:50:26)

hopla · 2011-12-27 21:04:13

Ooh, that is nice! Thx

kdar · 2012-01-06 19:56:33

What do you do in reverse?

If you have a gzip file on a server and you want to sent it over network and then decompress it.... Would this command be most efficient?

ssh remote-user@server-adr "cat /input.pdf.gz" | gunzip -fc > /home/local-user/input.pdf

I am a bit not sure if it is a best way to do it, since it seems to give surprisingly larger time to complete (than I initially thought it would).

Last edited by kdar (2012-01-06 20:27:01)

hopla · 2012-01-06 20:09:47

Hmm I don't know. You are doing 2 things at once here, which is smart, but makes it hard to say which one is slowing the operation down. Is is the network transfer of the gzipped file? Or is it the gunzipping and writing of the file to disk?

Try to do just the transfer with scp or

time ssh -p 6166 remote-user@server-adr "cat /input.pdf.gz" | cat > /home/local-user/input.pdf

and then gunzip. Time both operations separately.

kdar · 2012-01-06 20:40:35

It might be network transfer, but when I look at the compression case, when I compress first and then send the output over network... the final time change seem to be minimal.

For example... if I compress a file and send output to the server over the network, it increases time by about 15% or by 0.4 seconds.

On the other hand, when I do the opposite: sending compressed file back and then decompressing it, it increases total time by 96% or adds about 1 second.

I am comparing both cases to compression and decompression without network, just to local filesystem.

Last edited by kdar (2012-01-06 20:41:36)

hopla · 2012-01-06 21:11:49

Off the top of my head:

How do the compressed and uncompressed files compare? What's the compression ratio? How big of a file are we talking about here? (megabytes, gigabytes?) Is something else using the network at the time? Is compression enabled in SSH? (which might slow down the transfer, when SSH tries to compress and already compressed data stream) Are you taking the time of compression into account when you say "it increases time by 15%/96%" ? Or just the transfer of the uncompressed file vs the compressed file?

kdar · 2012-01-06 21:18:24

File is 3.69MB. Compression ratio is around 2.3. And nothing else was using network.

kdar · 2012-01-06 21:23:02

File is 3.69MB. Compression ratio is around 2.3. And nothing else was using network.

I didn't know ssh has enabled compression option, I will have to check this.

Yes. I am comparing to a case when you do similar operations only on one single computer. For example, compressing and saving output file to a different filename or decompressing and saving output to a different filename. Calculated time includes both compressing and saving output or decompressing and saving output.

When I do it over network, I am trying to:
(1 compression) compress a file and pipe it over network by ssh.
(2 decompression) send compressed file with ssh and pipe, decompress and save file.

But maybe (1) compression case has less time impact just because it doesn't actually spends anytime on saving the output (which is done on the server side).

*oops, sorry for double post.

Last edited by kdar (2012-01-06 21:32:40)

hopla · 2012-01-06 21:40:27

kdar wrote:

Calculated time includes both compressing and sending or both receiving and decompressing/saving.

Ok, in this case you have to take in account the compression speed vs network speed. Gzip has a max compression speed of about 20MB/s with level 1 (least compression), default is level 6 and that's around 10MB/s. When you got a 1Gigabit network connection you can get 125MB/s, 100Mbit gives 12,5MBs.

So... it could be that your uncompressed file will have already transferred in the time it takes to compress+transfer.

Also note, if you are doing the latter, you can do that in one go, without having to store an intermediate file:

For sending to server:
gzip file | ssh user@server "gunzip > file"

For receiving from server:
ssh user@server "gzip file" | gunzip > file

As I said, you can also do the compression with SSH, using the -C flag. But in both cases, on a fast network connection, you best use the level 1 compression setting (see man ssh on how to set gzip compression level when using the -C option). Or a faster compression algorithm, like lzjb or lzop (but gzip on level 1 is pretty fast already).

Arch Linux

#1 2011-12-21 21:26:25

Stream output from terminal to another computer over the network... ?

#2 2011-12-21 21:40:55

Re: Stream output from terminal to another computer over the network... ?

#3 2011-12-21 21:46:58

Re: Stream output from terminal to another computer over the network... ?

#4 2011-12-22 04:58:05

Re: Stream output from terminal to another computer over the network... ?

#5 2011-12-22 05:48:24

Re: Stream output from terminal to another computer over the network... ?

#6 2011-12-22 10:50:32

Re: Stream output from terminal to another computer over the network... ?

#7 2011-12-27 09:49:35

Re: Stream output from terminal to another computer over the network... ?

#8 2011-12-27 21:04:13

Re: Stream output from terminal to another computer over the network... ?

#9 2012-01-06 19:56:33

Re: Stream output from terminal to another computer over the network... ?

#10 2012-01-06 20:09:47

Re: Stream output from terminal to another computer over the network... ?

#11 2012-01-06 20:40:35

Re: Stream output from terminal to another computer over the network... ?

#12 2012-01-06 21:11:49

Re: Stream output from terminal to another computer over the network... ?

#13 2012-01-06 21:18:24

Re: Stream output from terminal to another computer over the network... ?

#14 2012-01-06 21:23:02

Re: Stream output from terminal to another computer over the network... ?

#15 2012-01-06 21:40:27

Re: Stream output from terminal to another computer over the network... ?

Board footer