[SOLVED] VIM trying to match curl brackets in double quotes with file

magicloud · 2019-04-19 12:28:06

This issue is only observed in Archlinux.

I have this code `url="${url/{USER\}/$username}"` in my sh script. Apparently, ${ should match with the last }, and {, \} around USER should not be check since they means nothing special.

But actually, VIM thinks the { before USER is coupling with the last }, thus all syntax highlight after that are totally messed.

Same script file looks good in vscode, Ubuntu VIM, Alpinelinux VIM. Copying Alpinelinux VIM (only the binary) to Archlinux environment gets same problem. So I think the issue is in VIM scripts. But after comparing /usr/share/vim/vim81 between Archlinux and Alpinelinux, and removed all different files in Archlinux, the problem is still there.

What should I do next? Thanks.

PS: All testing were done in container, so it is not the problem in my .vimrc.

Last edited by magicloud (2019-04-21 09:22:38)

WorMzy · 2019-04-19 12:53:07

Apparently, ${ should match with the last }, and {, \} around USER should not be check since they means nothing special.

That's not how I parse it. You have escaped the first close-curlybracket, so it is treated as a text character. Escape the corresponding open-curlybracket if you want that to be treated as text. Alternatively, escape the backslash if that should be text.

Note: I'm not at a PC right now, so I can't say whether the current line is valid bash. I would guess not, and if it works as you want it to, I'd say this is a happy coincidence.

magicloud · 2019-04-19 14:41:21

WorMzy wrote:

Apparently, ${ should match with the last }, and {, \} around USER should not be check since they means nothing special.
That's not how I parse it. You have escaped the first close-curlybracket, so it is treated as a text character. Escape the corresponding open-curlybracket if you want that to be treated as text. Alternatively, escape the backslash if that should be text.
Note: I'm not at a PC right now, so I can't say whether the current line is valid bash. I would guess not, and if it works as you want it to, I'd say this is a happy coincidence.

I cannot agree. Escaping sure is to make things "be treated as text". But is the "corresponding open-curlybracket" means anything here to shell? If it is not, why should I escape it?
And this line works with ash/bash(4, 5)/zsh. I do not have other shells to verify. Also, as mentioned, VIM in Ubuntu. So I'd say, there is something wrong here.

seth · 2019-04-19 15:41:13

That expansion works through heuristics.
If you remove the double quotes, it'll fail immediately (because of imbalanced braces)

Now, what does that behavior actually mean?

url="${url/{USER\}/$username}"

The first brace is read as variable (expansion) opener, the second one is read as string the third one is a string as well, so the escaping backslash should be a string either - do you really long to replace literally "{USER\}"?
If not, why do you add the backslash escape?
Right, to hint to the system that this isn't the closure for the variable.
However it's not handles literally, but as an escape, you'd now have to go for "\\\}" in order to match "{USER\}"

So, how does the system establish how to handle the backslash…?

Try to move the quotes inwards or quote the entire token as well as "'{USER\}'" - what happens?
What happens if you remove the escape?

I'm not saying the vim highlighter should not handle this like popular interactive shells, but don't fool yourself: the expansion at hand is ambigious at best and certainly not good style.

On topic: I'd check which highlighter is actually loaded on each system (there's at least sh.vim and zsh.vim) and compare (diff) those among the systems.

magicloud · 2019-04-19 16:03:54

seth wrote:

That expansion works through heuristics.
If you remove the double quotes, it'll fail immediately (because of imbalanced braces)
Now, what does that behavior actually mean?
url="${url/{USER\}/$username}"
The first brace is read as variable (expansion) opener, the second one is read as string the third one is a string as well, so the escaping backslash should be a string either - do you really long to replace literally "{USER\}"?
If not, why do you add the backslash escape?
Right, to hint to the system that this isn't the closure for the variable.
However it's not handles literally, but as an escape, you'd now have to go for "\\\}" in order to match "{USER\}"
So, how does the system establish how to handle the backslash…?
Try to move the quotes inwards or quote the entire token as well as "'{USER\}'" - what happens?
What happens if you remove the escape?
I'm not saying the vim highlighter should not handle this like popular interactive shells, but don't fool yourself: the expansion at hand is ambigious at best and certainly not good style.

On topic: I'd check which highlighter is actually loaded on each system (there's at least sh.vim and zsh.vim) and compare (diff) those among the systems.

Actually, it may not be like what you thought. The "\}" is a must here, if I want to replace "{cd}" (not "{cd\}") to "x", at least in bash5.

[shida@m11r3 glibc-2.28]$ a='ab{cd}ef'
[shida@m11r3 glibc-2.28]$ echo "${a/{cd}/x}"
ab}ef/x}
[shida@m11r3 glibc-2.28]$ echo "${a/{cd\}/x}"
abxef

I have tried checking if anything different on syntax/sh.vim. No. I tried to replace the whole /usr/share/vim/vim81 folder. No luck.

I tried switch the binaries (vim only) between systems. Only occurs on Archlinux. But I have no idea where else could the difference be caused....

seth · 2019-04-19 16:12:19

No, it's not.
You're using some sort of quote-escape hybrid, which, again is at least ambigious (to avoid stronger terms)

echo "${a/'{cd}'/x}"
echo "${a/"{cd}"/x}" # even this should work
echo ${a/\{cd\}/x}
echo "${a/\{cd\}/x}" # this will still work
echo ${a/{cd\}/x} # this will fail

The vim script seems to branch the different shells behind /bin/sh, so compare

ls -l /bin/sh

But again: your script is bogus. (there I said it)

PS: please don't full-quote posts.

magicloud · 2019-04-19 16:26:00

Sorry for the quoting.

All your ways work in bash5. Even the "this will fail" one. But that failed in zsh.

And I am not quite follow the "quote-escape hybrid". I cannot see the reason of the second one since it looks like shattering the var expanding. I can see how escaping both works. But I am with the shell, I cannot see why the opener should be escaped, since it means nothing but a char in bash syntax here.

The /bin/sh resolves the problem. Archlinux is pointing to bash. If links to, like busybox, VIM works well.

Last edited by magicloud (2019-04-19 16:27:40)

Trilby · 2019-04-19 17:38:38

Vim works fine for me even with your original (as odd as it is) as long as the shebang in the script is /bin/sh.

If the shebang is /bin/bash, then I get the symptoms you observe.

But why worry - you typed something odd that should be fixed and the parser for syntax highlighting couldn't get it right. No surprise. I'm actually a bit surprised that it does work as you intended with a /bin/sh shebang.

seth · 2019-04-19 19:23:06

I cannot see why the opener should be escaped

You seem to fail to understand that you expect the interpreter to handle the opening and closing brace differently - the opening one as a string but the closing one as a syntax element which you therefore need to escape.
Since you figured the failure in zsh, doesn't it give you pause that at least there the behavior depends on whether the entire expansion is put in quotes? (Which are not at all related to the inner brace group)

magicloud · 2019-04-20 08:00:30

Trilby wrote:

Vim works fine for me even with your original (as odd as it is) as long as the shebang in the script is /bin/sh.
If the shebang is /bin/bash, then I get the symptoms you observe.
But why worry - you typed something odd that should be fixed and the parser for syntax highlighting couldn't get it right. No surprise. I'm actually a bit surprised that it does work as you intended with a /bin/sh shebang.

The shebang is /bin/sh. But in my Archlinux, it was link to bash. I have no idea why it was like this.

magicloud · 2019-04-20 08:05:22

seth wrote:

Since you figured the failure in zsh, doesn't it give you pause that at least there the behavior depends on whether the entire expansion is put in quotes? (Which are not at all related to the inner brace group)

Of course. That is why my code style is "always in double quote".

And, actually, I am confused more. You said I expect "the opening one as a string but the closing one as a syntax element". While, I expect both as string, BUT shell takes the closing one as a syntax element. Thus it has to be escaped.

seth · 2019-04-20 11:37:01

You said I expect "the opening one as a string but the closing one as a syntax element".

No, I did not. Try reading it again:

seth wrote:

you expect the interpreter to handle the opening and closing brace differently - the opening one as a string but the closing one as a syntax element

The verb attached to the second part is "handle", not "expect".

You *want* both as string but for that you *expect* bash to treat them differently (or maybe rather ran into it by trial and erro) because you feed them differently.

Essentially you're relying on a missing feature ("{}" handling inside variable expansion) - if bash could do (or in a later version learns!) sth. like

${url/U{SZ}ER/$UID}

your approach falls apart.

If you want something that is a syntax element to be treated as string there're exactly two reliable ways:
1. escape it \{USER\}
2. put it in single quotes: '{USER}'

*Everything* else relies on unspecified behavior and luck.

Edit:

Of course. That is why my code style is "always in double quote".

That's complete nonsense. You put stuff into double quotes because the resulting token may contain eg. blanks - not to control the shell behavior.

Last edited by seth (2019-04-20 11:38:15)

magicloud · 2019-04-21 05:16:47

Sorry, I see your point now. I cannot say I already agreed, since this brings me some other questions on shell behaviors that I need to figure out.

But still, major problem is solved. Thanks.

seth · 2019-04-21 05:58:55

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Arch Linux

#1 2019-04-19 12:28:06

[SOLVED] VIM trying to match curl brackets in double quotes with file

#2 2019-04-19 12:53:07

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#3 2019-04-19 14:41:21

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#4 2019-04-19 15:41:13

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#5 2019-04-19 16:03:54

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#6 2019-04-19 16:12:19

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#7 2019-04-19 16:26:00

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#8 2019-04-19 17:38:38

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#9 2019-04-19 19:23:06

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#10 2019-04-20 08:00:30

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#11 2019-04-20 08:05:22

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#12 2019-04-20 11:37:01

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#13 2019-04-21 05:16:47

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

#14 2019-04-21 05:58:55

Re: [SOLVED] VIM trying to match curl brackets in double quotes with file

Board footer