You are not logged in.

#1 2020-10-11 08:56:45

jonno2002
Member
Registered: 2016-11-21
Posts: 129

some questions about bash scripting

ive been teaching myself how to write bash scripts and been getting most things with trial and error and searching for answers but theres a few things ive been failing at lately while trying to use variables to keep from repeating the same words/lines, ive combined the problems into this little test script in the hopes that someone can correct me.

#!/bin/bash

var=foo
var2='sed 's/foo/bar/g''
var3='awk '{ print $1; }''
string="echo $var"
string2="var4=$var"

#this works
$string|$var2

#this doesnt
$string|$var3

#neither does this
$string2
echo $var4

exit 0

and when run i get this:

./test: line 5: print: command not found
./test: line 5: }: command not found
bar
./test: line 17: var4=foo: command not found

ive tried using (") instead of (') and vice versa which fixed the first problem with the "sed" variable but cannot get the other 2 scenarios to work

Offline

#2 2020-10-11 09:49:39

solskog
Member
Registered: 2020-09-05
Posts: 247

Re: some questions about bash scripting

eval --help

#!/bin/bash

var=foo
var2='sed 's/foo/bar/g''
var3='awk "{ print $1; }"'
string="echo $var"
string2="eval var4=$var"

#this works
$string|$var2allway

#this doesnt
$string|eval $var3

#neither does this
$string2
echo $var4

exit 0

But, I would be very careful with 'eval' taking unknown arguments e.g: user input.

Last edited by solskog (2020-10-11 11:16:56)

Offline

#3 2020-10-11 10:16:54

Awebb
Member
Registered: 2010-05-06
Posts: 5,594

Re: some questions about bash scripting

Bash does ot like the way you try to put commands in variables. Look up alias or eval (as suggested).

Offline

#4 2020-10-11 10:26:13

jonno2002
Member
Registered: 2016-11-21
Posts: 129

Re: some questions about bash scripting

thanks that clears that up, so its better practice to just repeat commands which takes up more lines/words/size/space in a script rather than put them into variables to save space/size of script.

Offline

#5 2020-10-11 10:38:23

ayekat
Member
Registered: 2011-01-17
Posts: 1,398
Website

Re: some questions about bash scripting

var3='awk '{ print $1; }''

Nesting quotes doesn't work like that. That line is doing the following things:

  • Invoking print $1, with the variable var='awk {' set.

  • Invoking }

Hence the error messages you're seeing.

From what I've gathered, your goal here is to store a shell command (namely the invocation of awk) to a variable.
I'll try to break this down step by step (because it's sunday and I haven't got better things to do).

(the tl;dr of this is: Use functions!)

---

First, we're trying to run some awk code:

{ print $1; }

We can do this by invoking awk and passing the awk code as an argument.
However, because the code contains spaces, we have to wrap the code in quotes (otherwise the shell would split it up into multiple arguments):

awk '{ print $1; }'
awk "{ print $1; }"     # ... uhm

There is an issue with the dollar sign ($), though: it is interpreted by the shell itself, and will cause that part of the string to be replaced before it's passed to awk.
With single quotes, this is prevented: between single quotes, nothing is treated specially by the shell (except the closing single quote).
But with double quotes, many things are interpreted specially (including that dollar sign). To prevent that, we have to escape it. So the correct options would be either of these:

awk '{ print $1; }'
awk "{ print \$1; }"     # now it's fine

Alright, now we want to assign it to a shell variable. A naive approach would be one of the following:

var3=awk '{ print $1; }'
var3=awk "{ print \$1; }"

But that won't work, because a variable assignment must follow the structure key=value, and value must not contain any spaces.
This is because, in shell, it is possible to do something like this:

foo=bar somecommand

This would invoke somecommand and pass it an additional environment variable (foo=bar).
With the above approach, it would try to invoke {print $1; } with the environment variable var3=awk set, which is clearly not what we want.

To prevent spaces from screwing things up, we have to wrap the string in quotes again.
But here's the issue: string quoting only happens from one quote to the next. What if we want to put quotes in a quoted string?
This is what is currently happening in your code:

     quoted             "quoted" (actually it's just a zero-length string)
      ====              \/
var3='awk '{ print $1; }''
           =============
             unquoted

To wrap quotes inside quotes, these are the valid variants:

# Wrapping single quotes:
foo="some text with 'quotes' in it"

# Wrapping double quotes:
foo='some text with "quotes" in it'
foo="some text with \"quotes\" in it"

(you cannot put single quotes in single-quoted strings, because you cannot escape things inside single quotes)

Now that we've got our tools together, we can properly quote the variable assignment, with one of these:

var3='awk "{ print \$1; }"'
var3="awk '{ print \$1; }'"
var3="awk \"{ print \\\$1; }\"'"    # double escaping!

(as you can see, things get a bit confusing once you start nesting string)

But here, we're blocked.
What you're trying to do there is to then run the content of var3 as shell code.
But that will fail, because the shell will resolve a variable only once, and then run it like that.

In particular, let's take the following line:

$string | $var3

$string will be transformed to echo foo (i.e. invoking echo with foo as argument), which is fine.
$var will be transformed to awk "{ print \$1; }", which looks fine, but what is actually happening is that awk is invoked with four arguments:

  • "{

  • print

  • \$1;

  • }"

This will lead to the following awk error:

"{
^ unterminated string

To work around this, you would need to somehow tell the shell to split the string in a very specific way (namely into two parts: awk and { print $1; }).
Unless some shell wizard in here can come up with a dark solution to work around this (without resorting to `eval`), I'm going to claim that you cannot achieve what you're trying to do the way you do just with variables.
Instead, consider declaring a function:

function3() {
  awk '{ print $1; }'
}

echo foo | function3

This is both cleaner and more readable.

Closing this:

jonno2002 wrote:

thanks that clears that up, so its better practice to just repeat commands which takes up more lines/words/size/space in a script rather than put them into variables to save space/size of script.

No. Use functions.

---

--edit: Also, "storing" variable assignments into variables and running them like that is not supported. It will simply try to run var4=foo as a command.

Last edited by ayekat (2020-10-11 10:55:39)


{,META,RE}PKGBUILDSpacman-hacks (includes makemetapkg and remakepkg) │ dotfiles

Offline

#6 2020-10-11 11:06:07

jonno2002
Member
Registered: 2016-11-21
Posts: 129

Re: some questions about bash scripting

wow thanks for that very informative write up, just tried some functions and they work as expected

Offline

#7 2020-10-11 12:53:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 24,447
Website

Re: some questions about bash scripting

And note that the only reason the (un)quotes were not a problem in sed is that no quotes are actually necessary around the sed script  that you used: s/foo/bar/g is a single token and does not contain any $ or { or other characters that the shell would try to interpret or expand before basing that whole "word" to sed as an argument.  In otherwords, here's how the shell interprets those two lines - despite your likely goal of having nested quotes within quotes, the shell interprets the red parts as quoted and the blue parts as unquoted (and I've put underscores to show spaces with a color:

var2=sed_s/foo/bar/g
var3=awk_{ print $1; }

When the shell sees the unquoted { print $1; } it does a few different things with it as noted in previous posts, but when the shell sees s/foo/bar/g unquoted, it just leaves it as it is.  So var2 could also be defined approriately as any of the following (though I'd only recommend the first two):

var2="sed s/foo/bar/g"
var2='sed s/foo/bar/g'
var2=sed\ s/foo/bar/g
var2=sed" "s/foo/bar/g
var2=sed' 's/foo/bar/g

While the comments on eval are worth considering, I don't think they are directly relevant to any of the hurdles you are facing here.  Using any of the above definitions of var2 would make the following command do what you'd expect:

echo foo | $var2

There is no need to add an 'eval' to this.

Of course, for general purposes, the advice to use functions rather than code in variables is spot on (there are cases when putting code in a variable makes sense, but not for anything in this thread).

Last edited by Trilby (2020-10-11 12:54:24)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#8 2020-10-12 02:17:59

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,706

Re: some questions about bash scripting

ayekat wrote:

To work around this, you would need to somehow tell the shell to split the string in a very specific way (namely into two parts: awk and { print $1; }).
Unless some shell wizard in here can come up with a dark solution to work around this (without resorting to `eval`), I'm going to claim that you cannot achieve what you're trying to do the way you do just with variables.
Instead, consider declaring a function:

If you only need to run one command with arguments, you could do use bash arrays.

var3=('awk' '{print $1}')

echo "foo bar" | "${var3[@]}"

But, this depends on bash for arrays and only permits you to correctly quote and pass arguments. You cannot evaluate shell syntax inside of an array, obviously... no different from string variables except you can choose how the array splits itself into words.

I strongly advise using functions simply due to their being the *correct* way to do things. One would use functions in other programming languages like C/C++ or python, so why avoid them in shell?


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#9 2020-10-12 02:28:38

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 24,447
Website

Re: some questions about bash scripting

Well you can do this is POSIX sh too if you really only do need one command with arguments:

set -- awk '{ print $1; }'
echo foo bar | "$@"

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#10 2020-10-12 04:51:42

Awebb
Member
Registered: 2010-05-06
Posts: 5,594

Re: some questions about bash scripting

OP might be confusing the whole thing with this kind of assignment

x = someclass_with_shit_name
x.somemethod

or

import someclass
x = someclass.has.some.nesting.going
x.on

But since bash is as Object oriented as a solipsist, there is no such a construct and either aliases or functions are the way to go.

EDIT: Okay, use functions. Aliases won't work in scripts unless you turn them on with shopt.

Last edited by Awebb (2020-10-12 05:28:03)

Offline

#11 2020-10-12 06:24:36

jonno2002
Member
Registered: 2016-11-21
Posts: 129

Re: some questions about bash scripting

thanks for all the input, so much to learn, since my last post ive learned functions, for loops, and the highly confusing printf which tbh i just trial-and-errored till i got it to do what i wanted.

Offline

#12 2020-10-12 06:32:10

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,706

Re: some questions about bash scripting

Trilby wrote:

Well you can do this is POSIX sh too if you really only do need one command with arguments:

set -- awk '{ print $1; }'
echo foo bar | "$@"

My "one command with arguments" is "don't store pipelines of multiple commands in the same variable".

Your "one command with arguments" is "once per script, you can store a POSIX sh array in argv and access it via $@".

tongue

Yes, indeed, you get one array. But it's not always enough...


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#13 2020-10-12 07:55:02

ayekat
Member
Registered: 2011-01-17
Posts: 1,398
Website

Re: some questions about bash scripting

Oh, arrays. I forgot this was about bash. Yeah, that might be a way to solve the splitting issue smile

About the parameter list, I've found one array to be surprisingly sufficient for most purposes.
$@ is individual for each function, so as long as you can structure your code such that you never need more than one array per function (not "once per script"), it should be fine.

… of course you'll quickly find that occasionally, this leads to some weird structuring in your code. I'd call it… "parameter list oriented programming".

Last edited by ayekat (2020-10-12 07:55:46)


{,META,RE}PKGBUILDSpacman-hacks (includes makemetapkg and remakepkg) │ dotfiles

Offline

#14 2020-10-12 12:53:01

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 24,447
Website

Re: some questions about bash scripting

ayekat wrote:

I'd call it… "parameter list oriented programming".

PLOP.  Nice.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#15 2020-10-12 15:32:35

zpg443
Member
Registered: 2016-12-03
Posts: 122

Re: some questions about bash scripting

ayekat wrote:

I'd call it… "parameter list oriented programming".

Not sure if that beats special high intensity training.

Offline

Board footer

Powered by FluxBB