[RESOLVED] Trying to iterate over a list of files/directories

publicus · 2014-08-17 20:03:31

Hello,

I'm trying to solve a problem. Basically, I'm trying to replace "-" in my git repo with "_" in the names of the files/directories. What I'm having a hard time with is in the for-loop, I'd like get at a single file, one at a time, which I'm failing at.

I have found this source on iterating over files:
http://mywiki.wooledge.org/ParsingLs

However, since inserting [[ -e $file ]] | continue, I'm still not getting the results that I want. And this is where I'm stuck.

Ignore the usage function, I didn't post it since it's not really pertinent.

if [ -z $1 ] || [ -z $2 ] || [ -z $3 ]
then
  echo "You did not pass in the command line arguments correctly.  Please re-read how to use this script."
  echo ""

  usage

  exit 0
else
  var1=$2
  var2=$3

  if [ "${#var1}" -gt 1 ] || [ "${#var2}" -gt 1 ] 
  then
    echo "You passed in strings -- for 2nd and 3rd command line variables -- that were of length greater than 1.  Please re-read how to use this script."
    echo ""

    usage

    exit 0
  else
    foundFiles=$(find . -name "*$2*")

    echo "Found files: $foundFiles"
    echo ""

    for file in $foundFiles
    do  
      [[ -e $file ]] || continue

      echo "Old file: $file"

      newName=${file/$var1/$var2}
      echo "New file: $newName"
      #mv $file $newName
    done
  fi  
fi

Last edited by publicus (2014-08-23 12:53:35)

Trilby · 2014-08-17 20:56:03

First, to loop through items of an array it shoud be `for file in "${foundFiles[@]}"`. But you really shouldn't bother doing that either - just use find's exec's switch to mv the file.

publicus · 2014-08-18 00:24:51

This is what I have so far:

# check if something was passed in.  Usually, this means the directory that we
#   would like to manipulate.
if [ -z $1 ] || [ -z $2 ] || [ -z $3 ]
then
  echo "You did not pass in the command line arguments correctly.  Please re-read how to use this script."
  echo ""

  usage

  exit 0
else
  var1=$2
  var2=$3

  if [ "${#var1}" -gt 1 ] || [ "${#var2}" -gt 1 ] 
  then
    echo "You passed in strings -- for 2nd and 3rd command line variables -- that were of length greater than 1.  Please re-read how to use this script."
    echo ""

    usage

    exit 0
  else
    # TODO: When this script is complete, get rid of the below text.  Also,
    #   get rid of all other test outputs that are not needed.
    echo "Success!"
    echo ""

    # go through our files that we care about and rename them to something new.
    for file in `find $1 -name "*$2*"`
    do  
      # check if the file exists and then proceed, otherwise, skip over it.
      [[ -e $file ]] || continue

      echo "Old file: $file"

      newName=${file/$var1/$var2}
      echo "New file: $newName"
      mv $file $newName
    done
  fi  
fi

This... sort of works. It will go through only the first depth of the directory that I pass into it, but ignores the subsequent directories. When I run the script the second time, the subsequent directories/files are renamed. What's really freaky is that when I run find by itself, it gives me a nice depth-first search of all the files that match the criteria.

Why?

Trilby wrote:

First, to loop through items of an array it shoud be `for file in "${foundFiles[@]}"`. But you really shouldn't bother doing that either - just use find's exec's switch to mv the file.

Gotcha. The only problem is that I don't know how to change every char in there from, say, "-" to "_" (or back), in that one line. Although, I could write a small script that will do just that assuming that I pass the path to it and some more info.

Trilby · 2014-08-18 00:29:46

The above could work. The problem that you describe is not actually because it only does one level of depth, but rather that you're only telling it to replace the *first* instance of var1 with var2. Make the following change to change all occurrences.

- newName=${file/$var1/$var2}
+ newName =${file//$var1/$var2}

Also, you don't need to check every parameter in the first conditional, just check whether 3 were passed: "[[ $# -eq 3 ]]". Also, not the double brackets, if this is in bash you should use the [[ builtin. If you need full POSIX compliance though you can stick with the [ binary.

EDIT: I see you use '[' in most places, but '[[' in one. Pick one. If depending on bash is acceptable, [[ will be more efficient.

EDIT2: the exec syntax is shorter, but maybe a bit ugly:

find $1 -name '*$2*' -exec sh -c "file='{}'; mv \$file \${file//$2/$3}" \;

Last edited by Trilby (2014-08-18 00:40:00)

publicus · 2014-08-18 00:39:30

Trilby wrote:

The above could work. The problem that you describe is not actually because it only does one level of depth, but rather that you're only telling it to replace the *first* instance of var1 with var2. Make the following change to change all occurrences.
- newName=${file/$var1/$var2}
+ newName =${file//$var1/$var2}

I did that, still have the same problem.

UPDATE:

My latest code:

if [ -z $1 ] || [ -z $2 ] || [ -z $3 ]
then
  echo "You did not pass in the command line arguments correctly.  Please re-read how to use this script."
  echo ""

  usage

  exit 0
else
  var1=$2
  var2=$3

  if [ "${#var1}" -gt 1 ] || [ "${#var2}" -gt 1 ] 
  then
    echo "You passed in strings -- for 2nd and 3rd command line variables -- that were of length greater than 1.  Please re-read how to use this script."
    echo ""

    usage

    exit 0
  else
    for file in `find $1 -name "*$2*"`
    do  
      [ -e $file ] || continue

      echo "Old file: $file"

      newName=${file//$var1/$var2}
      echo "New file: $newName"
      mv $file $newName
    done
  fi  
fi

Thanks for your help trillby, when I get the 'meat' of the functionality working, I'll re-read what you've said and incorporate those changes.

Last edited by publicus (2014-08-18 00:41:10)

Xyne · 2014-08-18 01:46:22

publicus wrote:

for file in `find $1 -name "*$2*"`

This can result in unintended word expansion. If you can't use find's -exec argument, you can do this instead:

find "$1" -depth -name "*$2*" | while read path
do
  #...
done

Note the use of -depth to ensure that directory contents are processed first. What is likely happening in your current script is that it is renaming directories before recursing into them, which changes the path and thus prevents it from finding the contents.

publicus · 2014-08-20 00:52:02

This is what I have now:

# check if something was passed in.  Usually, this means the directory that we
#   would like to manipulate.
if [ -z $1 ] || [ -z $2 ] || [ -z $3 ]
then
  echo "You did not pass in the command line arguments correctly.  Please re-read how to use this script."
  echo ""

  usage

  exit 0
else
  var1=$2
  var2=$3

  if [ "${#var1}" -gt 1 ] || [ "${#var2}" -gt 1 ] 
  then
    echo "You passed in strings -- for 2nd and 3rd command line variables -- that were of length greater than 1.  Please re-read how to use this script."
    echo ""

    usage

    exit 0
  else
    # TODO: When this script is complete, get rid of the below text.  Also,
    #   get rid of all other test outputs that are not needed.
    echo "Success!"
    echo ""

    # go through our files that we care about and rename them to something new.
    for file in `find $1 -depth -name "*$2*"`
    do  
      # check if the file exists and then proceed, otherwise, skip over it.
      [ -e $file ] || continue

      echo "Old file: $file"

      newName=${file/%$var2/$var1}
      echo "New file: $newName"
      mv $file $newName
    done
  fi  
fi

For some reason, the string replace is not working. The newName has the same value as the file... why?

publicus · 2014-08-20 00:53:09

Xyne wrote:

publicus wrote:
for file in `find $1 -name "*$2*"`
This can result in unintended word expansion. If you can't use find's -exec argument, you can do this instead:
find "$1" -depth -name "*$2*" | while read path
do
  #...
done
Note the use of -depth to ensure that directory contents are processed first. What is likely happening in your current script is that it is renaming directories before recursing into them, which changes the path and thus prevents it from finding the contents.

That is exactly what is happening .

Koopa · 2014-08-20 04:52:27

This is an interesting problem to me. About seven years ago when I first installed linux it's a problem I wanted to solve, though slightly differently, I wanted to make my filenames all lowercase and replace all spaces with '_'.

I bother telling you how long ago it was because I realize the code I'm about to show definitely it's the prettiest, however it's worked for me and I haven't taken the time to improve it (one thing I'd like to do make it more efficient, as right now it tries to rename everything even if it doesn't contain uppercase characters or spaces, but seeing as I don't use it that often and it works in it's current state I don't care that much.) I've adapted it to replace '-' with '_' rather than making things lowercase and removing spaces.

#!/bin/bash
# Possible solution to publicus' problem on the arch forums.
# https://bbs.archlinux.org/viewtopic.php?id=185832

echo "Enter a number corresponding to depth of directories you wish to go down."
echo "Note: 0 is just the current directory."
echo -n "Enter an integer: "
read depthNum

for (( i = $depthNum ; i >= 0 ; i-- ))
do
    echo "Level - $i"
    while read N;
    do
        echo "$N" > /tmp/replace_in
        directory=`dirname "$N"`
        file=`basename "$N" | tr '-' '_'`
        echo "$directory/$file" > /tmp/replace_out
        input=`cat /tmp/replace_in`
        output=`cat /tmp/replace_out`
        echo "Renaming $input to $output"
        mv "$input" "$output" 2> /dev/null
    done < <(find ./* -maxdepth $i -mindepth $i)
done

This might not solve the problem as you had wanted, what it does is find all files at each depth level specified, splits up the directory names and basenames, replaces the characters in the basename, writes all these to a file which is then called in mv. Like I said it's definitely not the most efficient thing, and I do some things I'm not proud of, like redirecting stderr to /dev/null to avoid spitting out a bunch of redundant rename errors (mv: ‘./file_with_dash’ and ‘./file_with_dash’ are the same file), rather than actually trying to not rename files that don't need it.

I hope it helps in some way, even if it's just as an example of how not to write an efficient bash script.

-Koopa

Spider.007 · 2014-08-20 07:48:54

@Koopa next time, do it like this, have a look at http://www.tldp.org/LDP/abs/html/refcards.html#AEN22828 for an explanation of all this magic:

#!/bin/bash

echo "Enter a number corresponding to depth of directories you wish to go down."
echo "Note: 0 is just the current directory."
echo -n "Enter an integer: "
read depthNum

for (( i = $depthNum ; i >= 0 ; i-- ))
do
    echo "Level - $i"
    while read path;
    do
        file=${path##*/}
        file=${file/-/_}

        mv -v "$path" "${path%/*}/$file" 2>/dev/null
    done < <(find ./* -maxdepth $i -mindepth $i)
done

Koopa · 2014-08-20 13:28:00

@Spider.007 That's really nice, thanks for that. Putting my new knowledge to use I'd make this one slight change which will replace all '-'s with '_'s, rather than just the first.

 - file=${file/-/_}
 + file=${file//-/_}

-Koopa

publicus · 2014-08-23 03:18:50

On question, this is what I tried in my command line:

% stringZ=abcDEFGHIjkl
% echo $stringZ
abcDEFGHIjkl
% newVal=${stringZ/%jkl/444}
% echo $newVal
abcDEFGHI444
% stringZ=abcDEFGHIabc888
% echo $stringZ
abcDEFGHIabc888
% newVal=${stringZ/%abc/444}
% echo $newVal              
abcDEFGHIabc888
% newVal=${stringZ/%abc/444}
% ${stringZ/%abc/444} 
zsh: command not found: abcDEFGHIabc888
% ${"stringZ"/%abc/444}
zsh: bad substitution
% ${"stringZ"/%"abc"/444}
zsh: bad substitution
% ${"stringZ"/%"abc"/"444"}
zsh: bad substitution
% ${stringZ/%abc/444}    
zsh: command not found: abcDEFGHIabc888
% ${$stringZ/%abc/444}
zsh: bad substitution

Why can't I substitute the abc from the back of the string?

rockin turtle · 2014-08-23 03:48:09

newVal=${stringZ//abc/444}

publicus · 2014-08-23 03:57:28

rockin turtle wrote:

newVal=${stringZ//abc/444}

That just replaces all of the values in the string. What I'd like to do is replace the ones from the rear.

publicus · 2014-08-23 04:02:15

Spider.007 wrote:

@Koopa next time, do it like this, have a look at http://www.tldp.org/LDP/abs/html/refcards.html#AEN22828 for an explanation of all this magic:

#!/bin/bash

echo "Enter a number corresponding to depth of directories you wish to go down."
echo "Note: 0 is just the current directory."
echo -n "Enter an integer: "
read depthNum

for (( i = $depthNum ; i >= 0 ; i-- ))
do
    echo "Level - $i"
    while read path;
    do
        file=${path##*/}
        file=${file/-/_}

        mv -v "$path" "${path%/*}/$file" 2>/dev/null
    done < <(find ./* -maxdepth $i -mindepth $i)
done

I don't have advanced knowledge of the depth of my tree structure, that's why I'm not sure that this will work too well.

rockin turtle · 2014-08-23 04:45:12

Well, if you want to replace just the last 'abc', you could try this:

newVal="${stringZ%abc*}444${stringZ##*abc}"

although, this seems a little fragile to me for what you are trying to do (renaming files).

Spider.007 · 2014-08-23 07:02:25

@publicus; probably because you're using zsh; not bash. In bash; replacements starting from the end of the string works fine:

$ stringZ=abcDEFGHIjkl
$ echo ${stringZ/%jkl/xyz}
abcDEFGHIxyz

Last edited by Spider.007 (2014-08-23 07:03:27)

jasonwryan · 2014-08-23 07:04:47

FWIW: that works in zsh...
as does

echo ${stringZ/jkl#/xxx}

Trilby · 2014-08-23 11:50:31

publicus wrote:

% stringZ=abcDEFGHIabc888
% echo $stringZ
abcDEFGHIabc888
% newVal=${stringZ/%abc/444}
% echo $newVal              
abcDEFGHIabc888
Why can't I substitute the abc from the back of the string?

Because you are not substituting at the back of the string. The back of the string is 888 not abc. That replacement command would only replace abc if it was the very last thing in the string. Here's what you'd need:

newVal=${stringZ/%abc${stringZ##*abc}/444${stringZ##*abc}}

This wil replace (in stringZ) "abc" plus whatever is after the last "abc" with "444" plus whatever was after the last "abc". "888" is what was left after the last "abc", so this means "abc888" will be replaced with "444888".

publicus · 2014-08-23 12:16:52

Trilby wrote:

publicus wrote:
% stringZ=abcDEFGHIabc888
% echo $stringZ
abcDEFGHIabc888
% newVal=${stringZ/%abc/444}
% echo $newVal              
abcDEFGHIabc888
Why can't I substitute the abc from the back of the string?
Because you are not substituting at the back of the string. The back of the string is 888 not abc. That replacement command would only replace abc if it was the very last thing in the string. Here's what you'd need:
newVal=${stringZ/%abc${stringZ##*abc}/444${stringZ##*abc}}
This wil replace (in stringZ) "abc" plus whatever is after the last "abc" with "444" plus whatever was after the last "abc". "888" is what was left after the last "abc", so this means "abc888" will be replaced with "444888".

Yes! That is bloody brilliant (took me a second to figure out what was going on.)

I'll try to run it on the "prod" data and let you know.

Last edited by publicus (2014-08-23 12:17:15)

Xyne · 2014-08-23 12:17:44

I'm about to call 1-800-SOS-BASH because this is borderline abuse.

publicus · 2014-08-23 12:19:18

Xyne wrote:

I'm about to call 1-800-SOS-BASH because this is borderline abuse.

No pain, no gain .

publicus · 2014-08-23 12:53:15

Ok, partial success was achieved. If you had a file like this: foo1/foo2/some-file-over-here.txt only the last '-' would be changed. I said screw it and just ran the script multiple times until there were no other files were detected.

Methinks that using C/Python would make it easier to control for these oddities, since Bash scripting isn't the best tool 100% of the time.

Trilby · 2014-08-23 13:10:50

Ah ... you asked for a way to only change the last incidence of a substring in a string, then when it works you complain that it only changes the last incidence of a subtring in a string??

If you want to change all - to _ use ${string//-/_} as has already been well covered in this thread.

publicus · 2014-08-23 13:14:12

Trilby wrote:

Ah ... you asked for a way to only change the last incidence of a substring in a string, then when it works you complain that it only changes the last incidence of a subtring in a string??
If you want to change all - to _ use ${string//-/_} as has already been well covered in this thread.

Yes, it has and it has had some odd results that were not desired.

And I'm not complaining, merely reporting back of the results of the script. And my criticism is that of Bash... so... I'm not sure what the problem is...

Arch Linux

#1 2014-08-17 20:03:31

[RESOLVED] Trying to iterate over a list of files/directories

#2 2014-08-17 20:56:03

Re: [RESOLVED] Trying to iterate over a list of files/directories

#3 2014-08-18 00:24:51

Re: [RESOLVED] Trying to iterate over a list of files/directories

#4 2014-08-18 00:29:46

Re: [RESOLVED] Trying to iterate over a list of files/directories

#5 2014-08-18 00:39:30

Re: [RESOLVED] Trying to iterate over a list of files/directories

#6 2014-08-18 01:46:22

Re: [RESOLVED] Trying to iterate over a list of files/directories

#7 2014-08-20 00:52:02

Re: [RESOLVED] Trying to iterate over a list of files/directories

#8 2014-08-20 00:53:09

Re: [RESOLVED] Trying to iterate over a list of files/directories

#9 2014-08-20 04:52:27

Re: [RESOLVED] Trying to iterate over a list of files/directories

#10 2014-08-20 07:48:54

Re: [RESOLVED] Trying to iterate over a list of files/directories

#11 2014-08-20 13:28:00

Re: [RESOLVED] Trying to iterate over a list of files/directories

#12 2014-08-23 03:18:50

Re: [RESOLVED] Trying to iterate over a list of files/directories

#13 2014-08-23 03:48:09

Re: [RESOLVED] Trying to iterate over a list of files/directories

#14 2014-08-23 03:57:28

Re: [RESOLVED] Trying to iterate over a list of files/directories

#15 2014-08-23 04:02:15

Re: [RESOLVED] Trying to iterate over a list of files/directories

#16 2014-08-23 04:45:12

Re: [RESOLVED] Trying to iterate over a list of files/directories

#17 2014-08-23 07:02:25

Re: [RESOLVED] Trying to iterate over a list of files/directories

#18 2014-08-23 07:04:47

Re: [RESOLVED] Trying to iterate over a list of files/directories

#19 2014-08-23 11:50:31

Re: [RESOLVED] Trying to iterate over a list of files/directories

#20 2014-08-23 12:16:52

Re: [RESOLVED] Trying to iterate over a list of files/directories

#21 2014-08-23 12:17:44

Re: [RESOLVED] Trying to iterate over a list of files/directories

#22 2014-08-23 12:19:18

Re: [RESOLVED] Trying to iterate over a list of files/directories

#23 2014-08-23 12:53:15

Re: [RESOLVED] Trying to iterate over a list of files/directories

#24 2014-08-23 13:10:50

Re: [RESOLVED] Trying to iterate over a list of files/directories

#25 2014-08-23 13:14:12

Re: [RESOLVED] Trying to iterate over a list of files/directories

Board footer