How to use sed on a literal string containing regex meta characters?

ninian · 2013-11-15 15:23:08

I want to delete the lines from a file which match an arbitrary literal string.
The string is contained in a variable, and may itself contain various regex meta characters such as [,],^,$,|, etc
So, my problem is that sed wants to interpret these meta characters itself. For example:

x='abc[xyz$'
sed -i "/$x/d" file

produces a sed error about the unmatched [, and no doubt it wants to interpret the $ too.
(Of course, I could try and change the string $x so that every possible regex meta character was escaped with a \, but that seems immensely cumbersome. Also, can't see a way of using single or double quotes cleverly.)

Sorry if I am being thick about this, but any ideas or alternative methods would be most welcome!

frostschutz · 2013-11-15 15:45:55

With sed, you have to escape meta characters indivudually. That's just how it works. Of course you could do the escaping using sed as well, by replacing all meta characters (including \ and /) with \character.

If you want to match strings literally, use a different tool. Or do it in Bash.

while read line
do
    [ "$line" != "$x" ] && echo "$line"
done < file

In Perl/PCRE instead of escaping indivudually there is \Q...\E but sed does not understand that, and \Q\E does not solve all problems either, if there is a \E in the $x you have to escape it with \E\\E\Q or something like it.

Last edited by frostschutz (2013-11-15 15:46:42)

ninian · 2013-11-15 15:52:38

frostschutz wrote:

With sed, you have to escape meta characters indivudually. That's just how it works. Of course you could do the escaping using sed as well, by replacing all meta characters (including \ and /) with \character.
If you want to match strings literally, use a different tool. Or do it in Bash.
while read line
do
    [ "$line" != "$x" ] && echo "$line"
done < file
In Perl/PCRE instead of escaping indivudually there is \Q...\E but sed does not understand that, and \Q\E does not solve all problems either, if there is a \E in the $x you have to escape it with \E\\E\Q or something like it.

Thank you for your swift reply. Looks like sed can't be much of a friend in these circumstances!
Probably will have to go with the Bash file handling method.
Oh well ...

moetunes · 2013-11-15 18:57:23

Using quotes differently

sed -i '/"$x"/d' file

stops sed from erroring out.

frostschutz · 2013-11-15 19:04:07

moetunes wrote:

Using quotes differently
sed -i '/"$x"/d' file
stops sed from erroring out.

Certainly. It also stops sed from doing what the OP wanted. In particular, it probably does nothing.

zorro · 2013-11-16 01:29:29

You could automatically escape the meta characters

x='abc[xyz$'
echo $x | sed 's/\([[$]\)/\\\1/g'

generates: abc\[xyz\$

This can then be processed by sed

sed "/$(echo $x | sed 's/\([[$]\)/\\\1/g')/d"

Trilby · 2013-11-16 01:39:40

Would grep be an option? It would do exactly what was requested:

grep -Fv "$x" file

ninian · 2013-11-16 11:47:56

zorro wrote:

You could automatically escape the meta characters
x='abc[xyz$'
echo $x | sed 's/$[[$]$/\\\1/g'

generates: abc\[xyz\$
This can then be processed by sed
sed "/$(echo $x | sed 's/$[[$]$/\\\1/g')/d"

Thank you zorro, yes I can see that will work, if I extend it to escape all 12 (?) regex metachars.
Also, to cope with embedded spaces and match whole lines in the file, I've tweaked your expression thus:

sed "/^$(echo "$x" | sed 's/\([[$]\)/\\\1/g')$/d"

ninian · 2013-11-16 11:59:29

Trilby wrote:

Would grep be an option? It would do exactly what was requested:
grep -Fv "$x" file

Of course, of course, I forgot all about grep's -F option! sed is not sacred ...
I just need to add the -x option to match whole lines and rewrite the file after deleting the lines:

grep -Fvx "$x" file >file.$$; mv file.$$ file

Thank you Trilby!

Arch Linux

#1 2013-11-15 15:23:08

How to use sed on a literal string containing regex meta characters?

#2 2013-11-15 15:45:55

Re: How to use sed on a literal string containing regex meta characters?

#3 2013-11-15 15:52:38

Re: How to use sed on a literal string containing regex meta characters?

#4 2013-11-15 18:57:23

Re: How to use sed on a literal string containing regex meta characters?

#5 2013-11-15 19:04:07

Re: How to use sed on a literal string containing regex meta characters?

#6 2013-11-16 01:29:29

Re: How to use sed on a literal string containing regex meta characters?

#7 2013-11-16 01:39:40

Re: How to use sed on a literal string containing regex meta characters?

#8 2013-11-16 11:47:56

Re: How to use sed on a literal string containing regex meta characters?

#9 2013-11-16 11:59:29

Re: How to use sed on a literal string containing regex meta characters?

Board footer