[Solved] Peculiar behaviour with grep.

IrvineHimself · 2017-05-02 05:47:19

In the code snippet below, I have two greps, the first grep checks that a record exists and, if not, creates it. The second grep retrieves a record for modification. The peculiarity is that the only way it doesn't break standard out is if the first grep uses the ‘-q’ option and the second grep uses either ‘none’ or the ‘-a’ option.

if !(grep -q ":$RecordNme:" "$TextFile") ; then         # If RecordNme record doesn't exists?
    RecordField=":$RecordNme:0:0:0:0:0"                 # create field
    RecordCount=$(expr $RecordCount + "1")              # Increment 'net' RecordCount
    echo "$RecordField" >> "$TextFile"                  # append to "TextFile"
fi
###
### Not sure why, but (grep -q ...) above and (grep -a ...) below is the only way that doesnt break stndout?
###
RecordFieldOld=$(grep -a ":$RecordNme:" "$TextFile")         # get old RecordNme record
RecordFC="$(echo $RecordFieldOld | cut -d':' -f3)"              # Isolate the 2nd field
RecordFH="$(echo $RecordFieldOld | cut -d':' -f4)"              # Isolate the 3rd field
RecordFM="$(echo $RecordFieldOld | cut -d':' -f5)"              # Isolate the 4th field 
RecordFL="$(echo $RecordFieldOld | cut -d':' -f6)"              # Isolate the 5th field
RecordFU="$(echo $RecordFieldOld | cut -d':' -f7)"              # Isolate the 6th field 
###
### Note: The first field is empty in order to fully encapsulate 'RecordName' in colons, otherwise I have
### a problem with false positives from records with similar names. For example  'Doe' would break if 
### either/both 'John-Doe' and 'Jane-Doe' were also records.
###
case $RecordChange in    ### increment relevant count 
    #
    ### update record
    #
esac
RecordFieldNew=":$RecordNme:$RecordFC:$RecordFH:$RecordFM:$RecordFL:$RecordFU"
sed -i "s/$RecordFieldOld/$RecordFieldNew/g" "$TextFile"                                                       # Update record in TextFile
#
### echo the colour highlighted names of the affected records to stndout in three columns
#

For example, changing the first grep from the ‘-q’ to the ‘-a’ option, (or none,) works for the first dozen records, before starting to echo the full record along with the formatted record name.

Changing the second grep to ‘-q’ doesn’t work at all, reporting the error:

 sed: -e expression #1, char 0: no previous regular expression

In a way, the error message for the second grep with ‘-q’ kind of makes sense, because, according to “grep –help” the -q option assumes a binary file. But, this doesn’t explain why anything other than the ‘-q’ option in the first grep should break standard out. Especially when you consider it is supposed to be just a plain text file.

Very confused
Irvine

Last edited by IrvineHimself (2017-05-02 09:32:09)

aiBo · 2017-05-02 06:53:14

man grep wrote:

-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit immediately with zero status if any
match is found, even if an error was detected. Also see the -s or --no-messages option.

So in your second grep you don't want to use -q, as you want to catch standard output in the variable. Your sed error is caused by the variable RecordFieldOld being empty. Before using that variable you should check if it is empty or not and handle the case that it is empty anyways.

The switch for binary content with grep is -a.

For your first grep -q is fine, as you just care for the return code of the command but not for the particular lines. Your construct is still bogus, as you use extended globbing syntax "!()" (for details see bash manpage section EXPANSION, Pathname Expansion, Pattern matching). So, either put a space between the ! and ( or remove the parenthesis as they are not needed at all. A space after the ! is needed, tho.

jasonwryan · 2017-05-02 06:56:46

...and all that field isolation (and the second grep), could be more elegantly handled and with one pass using Awk.

Xyne · 2017-05-02 06:58:31

When you run the first grep after "if", it is run as usual and its exit status is checked to determine if the result should be considered true or false. Because grep normally prints results to stdout, you need the -q flag to suppress the output. This changes nothing when the record is absent from the file because the result of grep is empty. Once the record has been added, grep will print the line with the record, which is why it seems to work in the beginning before the records are added.

In the second case, you are capturing the output from grep using subshell invocation with $(...). If you add -q, it captures an empty string in RecordFieldOld and all of the subsequent fields are also empty. When you get to the replacement with sed, you are telling sed to replace "", which it can't do.

Note that the first grep is also invoked in a subshell with (...). Parenthesis are not needed for checking commands in Bash and should be avoided because of this. I don't known that is related to the problem though. The need for -a on the second invocation in order to detect that the file is text is very strange if the first invocation works. Did you omit steps between the greps in the example that you posted?

Here's a slightly modified version of what you posted to demonstrate ways to avoid redundant subshell invocations by using built-in integer arithmetic and arrays:

if !grep -q ":$RecordNme:" "$TextFile" ; then           # If RecordNme record doesn't exists?
    RecordField=":$RecordNme:0:0:0:0:0"                 # create field
    RecordCount=$((RecordCount + 1))                    # Increment 'net' RecordCount
    echo "$RecordField" >> "$TextFile"                  # append to "TextFile"
fi
###
### Not sure why, but (grep -q ...) above and (grep -a ...) below is the only way that doesnt break stndout?
###
RecordFieldOld=$(grep ":$RecordNme:" "$TextFile")         # get old RecordNme record
IFS=: RecordFields=(${RecordFieldOld#:})
RecordFC="${RecordFields[1]}"              # Isolate the 2nd field
RecordFH="${RecordFields[2]}"              # Isolate the 3rd field
RecordFM="${RecordFields[3]}"              # Isolate the 4th field
RecordFL="${RecordFields[4]}"              # Isolate the 5th field
RecordFU="${RecordFields[5]}"              # Isolate the 6th field
###
### Note: The first field is empty in order to fully encapsulate 'RecordName' in colons, otherwise I have
### a problem with false positives from records with similar names. For example  'Doe' would break if
### either/both 'John-Doe' and 'Jane-Doe' were also records.
###
case "$RecordChange" in    ### increment relevant count
    #
    ### update record
    #
esac
RecordFieldNew=":$RecordNme:$RecordFC:$RecordFH:$RecordFM:$RecordFL:$RecordFU"
sed -i "s/$RecordFieldOld/$RecordFieldNew/g" "$TextFile"                                                       # Update record in TextFile
#
### echo the colour highlighted names of the affected records to stndout in three columns
#

Note the increment of RecordCount, and the use of the input field separator (IFS) along with a string to convert the record fields to array elements. ${RecordFieldOld#:} strips off the first ":" to avoid an empty field.

IrvineHimself · 2017-05-02 09:31:29

Thanks, all of you, very clear and concise.

As you probably gathered, I have only just started making a serious effort to master bash syntax. Believing there was only one problem, I spent most of last night figuring out why it was breaking with 'Doe'/'Jane-Doe'/'John-Doe'. Once I figured out it was a problem with the field, (rather than grep,) and had a version that worked, I was tempted to pass the grep problem off for another day. However, I am now really glad I asked. I have learned a lot from your answers.

Xyne wrote:

..... Did you omit steps between the greps in the example that you posted? ......

No, the only code that is missing is what was inside the case statement, and that just incremented one of the fields of the record. There was some other assignments completely unrelated to the problem, (for example highlighting colour codes and temporary counters,) I just omitted them for clarity.

Thanks again

A lot less confused

Irvine

Arch Linux

#1 2017-05-02 05:47:19

[Solved] Peculiar behaviour with grep.

#2 2017-05-02 06:53:14

Re: [Solved] Peculiar behaviour with grep.

#3 2017-05-02 06:56:46

Re: [Solved] Peculiar behaviour with grep.

#4 2017-05-02 06:58:31

Re: [Solved] Peculiar behaviour with grep.

#5 2017-05-02 09:31:29

Re: [Solved] Peculiar behaviour with grep.

Board footer