[Solved] Bash and RegEx

IrvineHimself · 2019-10-29 14:36:29

Recently, I have been trying to gain a better understanding of regular expressions, ~~Deleted Text~~.

During the course of my experiments, I was considering trying to replace, (in all my shells,) $(command-pipe) with `command-pipe`. Note: I am not really interested in the actual regular expression, I believe something like sed -ir 's/(\$$)(.*)($)/`\1`/g' * will work.

What I am interested in is the limitations of using regular expressions in a bash shell?

For example: To test out a few ideas, I was using grep and noticed the following peculiarity:

#!/bin/bash

cd ${HOME}/Work/UsrLib/TestDir

# Echo lines containg '$('
grep -rP '\$\(' * | \
    while read Line ; do
        FileName=`echo "${Line}" | cut -d ':' -f1`
        OldString=`echo "${Line}" | cut -d ':' -f2-`
        CommandPosix=`echo "${OldString}" | grep -oP '\$\(\K.*?(?=\))'`
        CommandBash=$(echo "${OldString}" | grep -oP '\$\(\K.*?(?=\))')
        echo "${Line}"
        echo "${OldString}"
        echo "CommandPosix = ${CommandPosix}"
        echo "CommandBash = ${CommandBash}"
        echo "${OldString}" | grep -oP '\$\(\K.*?(?=\))'
        echo
    done

A sample of the output from the above is: (Note: I edited this slightly to clean out surplus Tabs/Spaces)

....

Shared/TerminalTricks:MyName=$(logname)
MyName=$(logname)
CommandPosix = 
CommandBash = logname
logname

Shared/TerminalTricks:Result=$(echo -e "$1" | sed -e 's/^[[:space:]]*//g')
Result=$(echo -e "$1" | sed -e 's/^[[:space:]]*//g')
CommandPosix = 
CommandBash = echo -e "$1" | sed -e 's/^[[:space:]]*//g'
echo -e "$1" | sed -e 's/^[[:space:]]*//g'

Shared/TerminalTricks:Result=$(echo -e "$1" | sed -e 's/[[:space:]]*$//g')
Result=$(echo -e "$1" | sed -e 's/[[:space:]]*$//g')
CommandPosix = 
CommandBash = echo -e "$1" | sed -e 's/[[:space:]]*$//g'
echo -e "$1" | sed -e 's/[[:space:]]*$//g'

Shared/TerminalTricks: Result=$(echo "${Result}" | grep -o . | LC_COLLATE=C sort | tr -d "n") ;;
Result=$(echo "${Result}" | grep -o . | LC_COLLATE=C sort | tr -d "n") ;;
CommandPosix = 
CommandBash = echo "${Result}" | grep -o . | LC_COLLATE=C sort | tr -d "n"
echo "${Result}" | grep -o . | LC_COLLATE=C sort | tr -d "n"

Shared/TerminalTricks: Result=$(echo "${Result}" | tr '[:upper:][:lower:]' '[:lower:][:upper:]')
Result=$(echo "${Result}" | tr '[:upper:][:lower:]' '[:lower:][:upper:]')
CommandPosix = 
CommandBash = echo "${Result}" | tr '[:upper:][:lower:]' '[:lower:][:upper:]'
echo "${Result}" | tr '[:upper:][:lower:]' '[:lower:][:upper:]'

....

As you can see, the version `command-pipe` breaks completely if regular expressions are involved. Looking further into this, I read on Stack Exchange, that neither Bash nor Posix are particularly adept at handling regular expressions, though the latest versions of Bash offer some functionality in this regard.

Trying to keep things simple, I won't go into detail, (unless asked,) but the issue of using regular expressions in a shell, ~~Deleted Text~~ has come up in other things I am working on. So,

~~Deleted Text~~.
~~Deleted Text~~.
More broadly, what are the limitations of using regular expressions in a Bourne shell?
Are there shells more suited to using regular expressions?

Irvine

Edited to remove erroneous references to posix compliance, see below

Last edited by IrvineHimself (2019-10-29 16:57:27)

goeb · 2019-10-29 14:58:46

IrvineHimself wrote:

During the course of my experiments, I was considering trying to replace, (in all my shells,) the Bashism $(command-pipe) with the posix version of `command-pipe`.

Why do you think $(command) is a Bashism and not POSIX?

See Command Substitution.

Scimmia · 2019-10-29 15:00:23

This whole thing would seem to be moot, as $(command) is POSIX, not a bashism.

IrvineHimself · 2019-10-29 15:21:14

I apologise, I was absolutely positive that I had read that back ticks were preferred to $() because the latter was not posix compliant.

However, it still leaves the more general issue of regular expressions in bash:

What are the limitations of using regular expressions in a Bourne shell?
Are there shells more suited to using regular expressions?

I will edit the title and original post accordingly
Irvine

V1del · 2019-10-29 15:36:45

sed and grep have nothing to do with your shell, all your examples use sed and grep, so you probably have to more precisely reformulate your question.

From what I'm gathering what you are actually asking is by what mechanism can I best escape a regex so that it gets passed to the interpreter unaltered, in which case use $(command) so that you can use single quotes for escaping purposes, which is both posixly and bashismly correct.

Backquotes can have quite a few side effects here:

pubs.opengroup wrote:

Within the backquoted style of command substitution, <backslash> shall retain its literal meaning, except when followed by: '$', '`', or <backslash>. The search for the matching backquote shall be satisfied by the first unquoted non-escaped backquote; during this search, if a non-escaped backquote is encountered within a shell comment, a here-document, an embedded command substitution of the $(command) form, or a quoted string, undefined results occur. A single-quoted or double-quoted string that begins, but does not end, within the "`...`" sequence produces undefined results.

Last edited by V1del (2019-10-29 16:02:08)

Trilby · 2019-10-29 16:27:07

The above is spot on. There aren't shells "more suited" to using regular expressions, especially as the shell may not use them at all ... I'm not 100% but I can't think of an example off the top of my head where the shell itself uses regular expressions (most shells have globbing, which may be superficially related, but is quite different).

Also note that there is not a universally aggreed upon definition of what the syntax or rules are for "regular expressions". There are different styles among different tools. Even with grep (from coreutils) one can use flags for basic, extended, or perl regular expressions. There are also other flavors among other tools.

But it does seem your issues in this thread really are just about quoting/escaping as noted above.

goeb · 2019-10-29 16:53:50

Trilby wrote:

I'm not 100% but I can't think of an example off the top of my head where the shell itself uses regular expressions

Bash has the the =~ operator for its test ([[ $something =~ $regex ]]), according to the man page it supports POSIX extended regular expression syntax.

IrvineHimself · 2019-10-29 16:56:16

Trilby wrote:

... But it does seem your issues in this thread really are just about quoting/escaping as noted above.

That and combining it all inside a pipe

I was looking at another example, and realised that what appeared to be the regular expression scrambling the order of execution was a result of the pipe.

Thank you for your input and patience. For hobbyists like me, regular expressions, particularly in the context of a shell, are very much akin to "Alice heading down the rabbit hole": They are completely opaque, with even the simplest expressions appearing as little more than an, eye-rolling, random jumble of characters.

Your comments have helped quite a bit, and given me quite a lot of ideas for further reading. As such, I will mark this as solved

Irvine

IrvineHimself · 2019-10-29 17:04:03

goeb wrote:

.... Bash has the the =~ operator for its test ([[ $something =~ $regex ]]), according to the man page it supports POSIX extended regular expression syntax.

It was the discovery of this operator that originally made me decide to gain a better understanding of regex, and ultimately lead me to start this thread.

Irvine

Arch Linux

#1 2019-10-29 14:36:29

[Solved] Bash and RegEx

#2 2019-10-29 14:58:46

Re: [Solved] Bash and RegEx

#3 2019-10-29 15:00:23

Re: [Solved] Bash and RegEx

#4 2019-10-29 15:21:14

Re: [Solved] Bash and RegEx

#5 2019-10-29 15:36:45

Re: [Solved] Bash and RegEx

#6 2019-10-29 16:27:07

Re: [Solved] Bash and RegEx

#7 2019-10-29 16:53:50

Re: [Solved] Bash and RegEx

#8 2019-10-29 16:56:16

Re: [Solved] Bash and RegEx

#9 2019-10-29 17:04:03

Re: [Solved] Bash and RegEx

Board footer