Which characters are not excluded in double quotes?

darkbeanies · 2011-08-24 20:22:10

Hello, I was wondering if anyone knows which characters are not protected when executing stuff in double quotes? Like, if I do:

mv "some file with spaces.txt" "another file with spaces".txt

it moves the file.

But, as I found out today, it does not protect against filenames containing the accursed backquote (`) character.

And, my script ruined about 4,000 files...good job I'm a backup fanatic!

So, besides the backquote (a pox upon files containing this character), are there any other loopholes?

bernarcher · 2011-08-24 20:28:41

Probably the "EXPANSION" chapter in "man bash" will provide an answer.

darkbeanies · 2011-08-24 20:59:44

And the answer is...

$
`
\

and ! when history expansion is enabled.

And some other stuff too...

Last edited by darkbeanies (2011-08-24 21:03:24)

juster · 2011-08-25 14:39:30

If you are typing commands in then use single quotes, which don't expand anything. Double quotes expand everything, excluding glob (*?) and tilde (~) expansion. That's off the top of my head. You must be doing something crazy like using eval in a shell script instead of just writing mv "$SRC" "$DEST", which doesn't have this problem. Or maybe you are using find? I find it easier to loop over find's results in a for loop like, for file in $(find . -name bla) ; do ..., instead of using find's -exec. Then there is always the perl-rename package.

falconindy · 2011-08-25 15:18:53

juster wrote:

If you are typing commands in then use single quotes, which don't expand anything. Double quotes expand everything, excluding glob (*?) and tilde (~) expansion. That's off the top of my head.

No, double quotes will prevent glob expansion from occurring. Command substitution and parameter expansion, however, will still occur.

I find it easier to loop over find's results in a for loop like, for file in $(find . -name bla) ; do ..., instead of using find's -exec. Then there is always the perl-rename package.

This breaks when there's whitespace in the results. If you don't like using -exec, at least use a proper 'while read' loop.

juster · 2011-08-25 17:17:15

falconindy wrote:

juster wrote:
If you are typing commands in then use single quotes, which don't expand anything. Double quotes expand everything, excluding glob (*?) and tilde (~) expansion. That's off the top of my head.
No, double quotes will prevent glob expansion from occurring. Command substitution and parameter expansion, however, will still occur.

Um... that's what I said. Brace expansion (foo{bar,baz}) also does not occur in double-quotes. I avoid spaces in filenames but I probably should use while read anyways, thanks.

falconindy · 2011-08-25 17:25:27

juster wrote:

Um... that's what I said.

Why yes, yes it is. Oops.

darkbeanies · 2011-08-26 07:55:59

Hmm. Actually, I probably am doing something crazy...

See, I was doing (it's finished now, thank goodness) a script to extract and replace meta data in pdf documents. (Specifically, I needed to replace the ampersands and commas with semicolons). I would try and avoid spaces, quotes within filenames, etc. but this just looks offensive to my eyes ("the_genuine_history_of_the_life_of_dick_turpin.pdf" just makes me wanna puke. As does "aesops fables" without an apostrophe instead of "aesop's fables")

So, I thought wrapping stuff in double quotes would eliminate such problems. Am I right in thinking that the suggestion of single quotes would NOT protect against single quotes within filenames?

The problem only occured with four files containing the backquote. However, it made all my metadata "off" by like, four files by the time it got to the end, so all the author/tag info was incorrect after an error.

My script actually generated some more scripts which produced problems like

pdftk "A Dictionary of `Archaeology.pdf" dump_data output > "A Dictionary of `Archaeology".txt

(notice the backquote)

I really am barely able to script. I thought posting my full script for criticism might lead to some useful suggestions - I am sure it will make you feel ill...

#!/bin/bash

echo "HOLD TIGHT! HERE WE GO!!!"

#make a new directory named after current with a "2" on the end
SHORTDIR=$(basename `pwd`)
SHORTDIR2=$(echo $SHORTDIR | sed 's/$/2/')
mkdir $SHORTDIR2

echo "MAKING A BIG LIST, WOOHOO!!!"

#Make a list of all the files
ls -1 *.pdf > names
cat names | sed 's/^/"/' | sed 's/$/"/' > quotednames
ls -1 *.pdf | sed 's/\(.*\)\..*/\1/' > titles
cat titles | sed 's/^/"/' | sed 's/$/"/' > quotedtitles

echo "EXTRACTING METADATA FROM PDFS!!!"

#Extract the metadata with pdftk
cat quotednames | sed 's/^/pdftk /' | sed 's/$/ dump_data output > /'  > 1
cat quotedtitles | sed 's/$/.txt/' > 2
awk 'NR==FNR{a[FNR]=$0;next} {print a[FNR],$0}' 1 2 > 3
chmod +x 3
./3

echo "EXTERMINATING ALL METADATA, PRAY TO ALL THE GODS YOU CAN THINK OF!!!"

#Destroy all metadata...
cat quotednames | sed 's/^/exiftool -all= /' > death2meta
chmod +x death2meta
./death2meta

echo "ADDING NEW TITLES FROM FILENAMES!!!"
#Now rewrite the info catalog, by adding title from filename
awk 'NR==FNR{a[FNR]=$0;next} {print a[FNR],$0}' quotedtitles quotednames > 4
sed 's/^/exiftool -Title=/' 4 > 5
chmod +x 5
./5

echo "FIXING  METADATA FROM DUMP, SIT TIGHT!!!"
cat 2 | sed 's/^/sed -i /' | sed 's/$/ -e /' | sed "s/$/'/" | sed 's/$/s\/\, \/;\\ \/g/' | sed "s/$/'/" | sed 's/$/ -e /' | sed "s/$/'/" | sed 's/$/s\/\ \&amp; \/;\\ \/g/' | sed "s/$/'/" > 12
chmod +x 12
./12

echo "NEARLY DONE, PUTTING FIXED METADATA BACK!!!"
#put the metadata back
cat quotednames | sed 's/^/pdftk /' | sed 's/$/ update_info /' > 13
ls *.txt | sed 's/^/"/' | sed 's/$/"/' > 14
awk 'NR==FNR{a[FNR]=$0;next} {print a[FNR],$0}' 13 14 > 15
cat 15 | sed 's/$/ output /' > 16
cat 16 | sed "s/$/ $SHORTDIR2/" > 17
cat 17 | sed 's/$/\//' > 18
awk 'NR==FNR{a[FNR]=$0;next} {print a[FNR]$0}' 18 quotednames > 19
chmod +x 19
./19

Last edited by darkbeanies (2011-08-26 07:59:11)

falconindy · 2011-08-26 12:30:23

Quoting isn't literal, it's semantic. When you type something like:

mkdir "foo bar"

The quotes aren't passed literally to the shell as part of the argument, they're just a hint to the shell that the words "foo" and "bar" aren't separate arguments.

I'd love to point you here but it's currently down, so this will have to suffice.

Last edited by falconindy (2011-08-26 12:30:42)

darkbeanies · 2011-08-26 21:00:33

falconindy wrote:

The quotes aren't passed literally to the shell as part of the argument, they're just a hint to the shell that the words "foo" and "bar" aren't separate arguments.

I think I knew that...I meant that

mkdir "foo'bar"

will work,

mkdir 'foo"bar'

Will work, but

mkdir "foo"bar"

or

mkdir 'foo'bar'

Will not work

Am I right?

Thanks for the links!

bernarcher · 2011-08-26 22:04:35

In double quoted strings you can escape the double quote, thus

$ mkdir "foo\"bar"
$ ls -d foo*
foo"bar

This does not work with single quotes in single quoted strings, however.

falconindy · 2011-08-26 22:07:46

Right. How would the shell know what you mean? Escaping quotes within quotes becomes necessary... bernarcher covered most of it. If you need to escape a single quote within single quotes, i suggest using bash's $'' syntax, e.g.

$ echo $'foo\'bar'

s

Arch Linux

#1 2011-08-24 20:22:10

Which characters are not excluded in double quotes?

#2 2011-08-24 20:28:41

Re: Which characters are not excluded in double quotes?

#3 2011-08-24 20:59:44

Re: Which characters are not excluded in double quotes?

#4 2011-08-25 14:39:30

Re: Which characters are not excluded in double quotes?

#5 2011-08-25 15:18:53

Re: Which characters are not excluded in double quotes?

#6 2011-08-25 17:17:15

Re: Which characters are not excluded in double quotes?

#7 2011-08-25 17:25:27

Re: Which characters are not excluded in double quotes?

#8 2011-08-26 07:55:59

Re: Which characters are not excluded in double quotes?

#9 2011-08-26 12:30:23

Re: Which characters are not excluded in double quotes?

#10 2011-08-26 21:00:33

Re: Which characters are not excluded in double quotes?

#11 2011-08-26 22:04:35

Re: Which characters are not excluded in double quotes?

#12 2011-08-26 22:07:46

Re: Which characters are not excluded in double quotes?

Board footer