[SOLVEDish] bash: wget doesn't get full webpage when piped

inkdinky · 2012-01-25 23:27:23

Method 1

wget ${1} -O ${Webpage} 2>&1 | grep -q "200 OK"

Method 2

wget ${1} -O ${Webpage} & 2>&1 | grep -q "200 OK"

Method 1 works on an PIII @750MHz running Ubuntu 10.10
Method 1 doesn't work on a PIV @2.4 GHz or a Celeron @1GHz running Arch

By "doesn't work" I get about 3/4 of a page of text of the html webpage I'm going after.
Have I somehow screwed up the way the pipes work or are my assumptions about pipes wrong? Or is it the processor speeds?
As far as the pipes go I thought the output of the first command was piped to the second command, yet here it seems that somehow the piping is cutting the previous command short????

Thanks in advance.

Last edited by inkdinky (2012-01-26 19:00:41)

karol · 2012-01-26 00:01:22

What about Method 2? Does it work?

inkdinky · 2012-01-26 00:23:10

Yeah. Method 2 works.
This just makes me wonder if there is an over-arching problem with timing and or things happening too fast when programming in bash. I say this bc as I rework the script I'm having problems with the ftp using here documents and commands. Not sure what the problem is there just yet but it isn't a problem on the Ubuntu box yet it is on the Arch Celeron.

karol · 2012-01-26 00:31:39

Are you using the bash on Ubuntu? IIRC the default shell is dash.

inkdinky · 2012-01-26 00:43:52

Well I am calling it with bash
bash script.sh
and the shebang is followed by /bin/bash in the script. I'm calling the script via
bash script.sh
on all machines.

inkdinky · 2012-01-26 00:45:49

echo $SHELL
returned
/bin/bash
on all machines.

alphaniner · 2012-01-26 00:47:01

I don't think Method 2 is doing what you expect. The '&' end should the wget command, invalidating the redirection; effectively like running:

$ wget ${1} -O ${Webpage}
$ 2>&1 | grep -q "200 OK"

I'm in Windows now, but '2>&1' is a valid command in cygwin. Surprising to me anyway.

I think the correct syntax should be

wget ${1} -O ${Webpage} 2>&1 & | grep -q "200 OK"

Unless I'm completely missing the point.

Last edited by alphaniner (2012-01-26 00:49:42)

inkdinky · 2012-01-26 01:53:44

@alphaniner Ok I'll buy that it isn't doing what I'm expecting, but your proposed solution doesn't work.

wget [url]http://www.yahoo.com[/url] -O /tmp/Yahoo 2>&1  & | grep -q "200 OK"; less /tmp/Yahoo
bash: syntax error near unexpected token `|'

So.....
The point still remains that Method 1 fails on the higher speed machines for some reason.

wget [url]http://www.yahoo.com[/url] -O /tmp/Yahoo 2>&1 | grep -q "200 OK"; less /tmp/Yahoo

results in a partial webpage

wget [url]http://www.yahoo.com[/url] -O /tmp/Yahoo & 2>&1  | grep -q "200 OK"; less /tmp/Yahoo

results in the full webpage.

alphaniner · 2012-01-26 02:04:58

Yeah, my mistake. When you put the & on the end you're effectively doing

$ wget ...
$ | grep ...

which is obviously not going to work.

But I stand by the point that in method 2 you're piping the output of 2>&1 (which is precisely nothing) rather than any output of wget, so the fact that succeeds is pretty much irrelevant.

Unfortunately my cygwin install doesn't include wget, so I can't do any testing with the command.

Maybe if you explain exactly what you're trying to accomplish. It seems like you want some kind of notification of wget's successful completion, and there are a lot of ways to accomplish that.

Last edited by alphaniner (2012-01-26 02:24:39)

fschiff · 2012-01-26 02:07:24

I'd have to look it up but doesn't 2>&1
redirect stderr to stdout?

Is that what you want?

alphaniner · 2012-01-26 02:21:27

Nevermind.

Last edited by alphaniner (2012-01-26 02:22:28)

Peasantoid · 2012-01-26 02:32:59

The -q option to grep causes it to exit immediately when a match is found (see grep(1)). When the reading process of a pipe exits, the writing process receives SIGPIPE, which in most cases terminates it, though it can be caught. Hence, the moment grep reads "200 OK", wget is signaled. There may or may not be enough time for it to finish writing to ${Webpage}.

Method 2 does not do what you want at all. It's equivalent to writing this:

$ wget ${1} -O ${Webpage} &
$ 2>&1 | grep -q "200 OK"

The first line runs wget in the background. The second line pipes the output of the command 2>&1, which outputs nothing, to grep.

Edit for further information:

If you want to test for successful retrieval of ${1}, you can check the exit status of wget. If you want to check specifically for the string "200 OK", you can use grep in a way that does not cause it to prematurely close the pipe. Also, note that the curly braces around your variable names are superfluous.

Last edited by Peasantoid (2012-01-26 02:43:29)

inkdinky · 2012-01-26 02:53:51

I appreciate the help guys/gals.
I guess I really just want to make sure that wget exited successfully and wasn't redirected. So I do want the "200 ok" and not some other exit code.
From Peasantoid's explanation I think that the old PIII is slow enough that wget finishes before grep is done.
So is there a preferred method to capturing wget's exit status? man wget didn't seem to be much help.

Peasantoid · 2012-01-26 03:05:25

inkdinky wrote:

I guess I really just want to make sure that wget exited successfully and wasn't redirected. So I do want the "200 ok" and not some other exit code.

Do you mean a redirect as in HTTP 3xx or receiving the Location header from the server? In that case, wget will print "200 OK" no matter what happens, so long as it eventually ends up at a URI that yields HTTP 200. If you grep for "200 OK", you will also need to check that only one request occurred. You might try grep -v "$something" against the output, where $something is a string that only occurs in the event of a redirect.

inkdinky · 2012-01-26 03:09:12

I'm not really expecting a redirect but is there a better way to get the status code from wget? As it stands my way isn't foolproof.

Peasantoid · 2012-01-26 03:29:04

Just to clarify, do you mean "status code" as in the exit status of wget or the HTTP status? If you want the HTTP status, you need to retrieve the wget output in a way that avoids the SIGPIPE problem. For example, you could read the output into a string and grep against that, or you could use the pipe and suppress grep's output without using -q. If it's the exit status, you can use $?, &&, ||, if, and so on.

alphaniner · 2012-01-26 04:00:04

Maybe you could use process substitution instead of a pipe:

grep "200 OK" < <(wget ... 2>&1)

As I understand the concept, it's sort of like redirecting the output of wget to a file then grepping the file, without actually making a file*. It may have the same limitation that Peasantoid mentioned, but I don't think so.

* The first < is actually input redirection. But when grep isn't told to search a file, it searches stdin. And input redirection sends the contents of a file to stdin. So

grep "yadda" file

and

grep "yadda" < file

do pretty much the same thing I think. The result is the same anyway.

Last edited by alphaniner (2012-01-26 04:55:52)

inkdinky · 2012-01-26 18:53:57

Thanks. I'll look into the methods offered.
I really appreciate the help!

Arch Linux

#1 2012-01-25 23:27:23

[SOLVEDish] bash: wget doesn't get full webpage when piped

#2 2012-01-26 00:01:22

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#3 2012-01-26 00:23:10

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#4 2012-01-26 00:31:39

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#5 2012-01-26 00:43:52

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#6 2012-01-26 00:45:49

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#7 2012-01-26 00:47:01

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#8 2012-01-26 01:53:44

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#9 2012-01-26 02:04:58

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#10 2012-01-26 02:07:24

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#11 2012-01-26 02:21:27

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#12 2012-01-26 02:32:59

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#13 2012-01-26 02:53:51

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#14 2012-01-26 03:05:25

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#15 2012-01-26 03:09:12

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#16 2012-01-26 03:29:04

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#17 2012-01-26 04:00:04

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

#18 2012-01-26 18:53:57

Re: [SOLVEDish] bash: wget doesn't get full webpage when piped

Board footer