You are not logged in.

#1 2012-11-10 13:41:22

grufo
Member
Registered: 2012-08-09
Posts: 100

[Bash] Massively replace text in all files of a directory

Hi everybody,
I wrote this small recursive function in order to massively replace some strings contained in all files of a directory (and all subdirectories). Any suggestions?

replaceText() {

	# set the temporary location
	local tFile="/tmp/out.tmp.$$"

	# call variables
	local script="$2"
	local opts="${@:3}"

	browse() {
		for iFile in "$1"/*; do
			if [ -d "$iFile" ];then
				# enter subdirectory...
				browse "$iFile"
			elif [ -f $iFile -a -r $iFile ]; then
				echo "$iFile"
				sed $opts "s/$(echo $script)/g" "$iFile" > $tFile && mv $tFile "$iFile"
			else
				echo "Skip $iFile"
			fi
		done
	}

	browse $1
}

Syntax:

replaceText [path] [script] [sed options (optional)]

For example (it will replace "hello" with "hi" in all files):

replaceText /home/user/mydir hello/hi

Note: It is case-sensitive.

Bye,
grufo

Last edited by grufo (2012-11-10 15:05:43)

Offline

#2 2012-11-10 14:12:37

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: [Bash] Massively replace text in all files of a directory

Use find instead? Your function might claim to be recursive, but it doesn't process anything past the first match, and will eventually blow the stack (segfaulting bash) if you pass a directory

find /some/dir -type f -name 'matching-these.*' -exec sed -i 's/foo/bar/' {} +

Offline

#3 2012-11-10 15:00:24

grufo
Member
Registered: 2012-08-09
Posts: 100

Re: [Bash] Massively replace text in all files of a directory

falconindy wrote:

Use find instead?
[...]

find /some/dir -type f -name 'matching-these.*' -exec sed -i 's/foo/bar/' {} +

Is it recursive?

falconindy wrote:

Your function might claim to be recursive, but it doesn't process anything past the first match, and will eventually blow the stack (segfaulting bash) if you pass a directory

Sorry, there was a banal error... Now it's fixed and it works fine (I hope)! ;-)

Offline

#4 2012-11-10 15:19:07

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: [Bash] Massively replace text in all files of a directory

Yes, find is recursive and extremely good at its job.

http://mywiki.wooledge.org/UsingFind

Your lack of quoting is dangerous, as is your code injection in sed. I'm not sure why you're echoing a var inside a command substitution inside a sed expression, but it's going to be subject to word splitting, all forms of expansion, and may very well break the sed expression entirely, leading to bad things. A contrived example, but passing something like 'foo//;d;s/bar/' should effectively delete the contents of every file the function touches.

I'll also point out that declaring a function within a function doesn't provide any amount of scoping -- 'browse' will be declared in the user's namespace after running this function for the first time.

Last edited by falconindy (2012-11-10 15:19:59)

Offline

#5 2012-11-10 16:03:34

grufo
Member
Registered: 2012-08-09
Posts: 100

Re: [Bash] Massively replace text in all files of a directory

falconindy wrote:

Yes, find is recursive and extremely good at its job.

http://mywiki.wooledge.org/UsingFind

Well smile

falconindy wrote:

Your lack of quoting is dangerous, as is your code injection in sed. I'm not sure why you're echoing a var inside a command substitution inside a sed expression, but it's going to be subject to word splitting, all forms of expansion, and may very well break the sed expression entirely, leading to bad things. A contrived example, but passing something like 'foo//;d;s/bar/' should effectively delete the contents of every file the function touches.

So, if you consider it dangerous, you can adopt the whole "sed syntax" and confirm before continue...:

replaceText() {

	# set the temporary location
	local tFile="/tmp/out.tmp.$$"

	# call variables
	local sedArgs="${@:2}"

	browse() {
		for iFile in "$1"/*; do
			if [ -d "$iFile" ];then
				# enter subdirectory...
				browse "$iFile"
			elif [ -f $iFile -a -r $iFile ]; then
				echo "$iFile"
				sed $sedArgs "$iFile" > $tFile && mv $tFile "$iFile"
			else
				echo "Skip $iFile"
			fi
		done
	}

	while true; do
	    read -p "Do you want to apply \"sed $sedArgs\" to all files contained in the directory $1? [y/n] " yn
	    case $yn in
		[Yy]* ) browse $1; break;;
		* ) exit;;
	    esac
	done

}

Syntax:

replaceText [parent directory] [sed arguments]

Example:

replaceText /your/path -r 's/OldText/NewText/g'

or, if you want to work directly with the current directory...

replaceText() {

	# set the temporary location
	local tFile="/tmp/out.tmp.$$"

	# call variables
	local sedArgs="$@"

	browse() {
		for iFile in "$1"/*; do
			if [ -d "$iFile" ];then
				# enter subdirectory...
				browse "$iFile"
			elif [ -f $iFile -a -r $iFile ]; then
				echo "$iFile"
				sed $sedArgs "$iFile" > $tFile && mv $tFile "$iFile"
			else
				echo "Skip $iFile"
			fi
		done
	}

	while true; do
	    read -p "Do you want to apply \"sed $sedArgs\" to all files contained in the directory $PWD? [y/n] " yn
	    case $yn in
		[Yy]* ) browse $PWD; break;;
		* ) exit;;
	    esac
	done

}

Syntax:

replaceText [sed arguments]

Example:

replaceText -r 's/OldText/NewText/g'

What about?

falconindy wrote:

I'll also point out that declaring a function within a function doesn't provide any amount of scoping -- 'browse' will be declared in the user's namespace after running this function for the first time.

See:

function1() {
	function2() {
		echo "Ciao"
	}
	function2
}

function2 # error
function1 # works

Offline

#6 2012-11-10 16:10:38

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: [Bash] Massively replace text in all files of a directory

function1() {
	function2() {
		echo "Ciao"
	}
	function2
}

function1 # works
function2 # works

This is what falconindy said.  AFTER the outer function has been called, the inner function will be defined - not before.

Anyhow, this looks like it was a good way to experiment and learn some bash, but given that find does do a better job much more simply, there isn't a lot of purpose in perfecting your script.

We've all reinvented the wheel before, it's fun, it's a chance to learn.  But don't expect people to start buying your square wheels! wink


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#7 2012-11-10 16:47:47

grufo
Member
Registered: 2012-08-09
Posts: 100

Re: [Bash] Massively replace text in all files of a directory

Trilby wrote:

This is what falconindy said.  AFTER the outer function has been called, the inner function will be defined - not before.

Well, thank you! smile However we have the "unset" command...

function1() {
	function2() {
		echo "Ciao"
	}
	function2
	unset function2
}

function1 # works
function2 # error

...so we can change the two scripts in these ways:

#1

replaceText() {

	# set the temporary location
	local tFile="/tmp/out.tmp.$$"

	# call variables
	local sedArgs="${@:2}"

	browse() {
		for iFile in "$1"/*; do
			if [ -d "$iFile" ];then
				# enter subdirectory...
				browse "$iFile"
			elif [ -f $iFile -a -r $iFile ]; then
				echo "$iFile"
				sed $sedArgs "$iFile" > $tFile && mv $tFile "$iFile"
			else
				echo "Skip $iFile"
			fi
		done
	}

	read -p "Do you want to apply \"sed $sedArgs\" to all files contained in the directory $1? [y/n] " yn
	case $yn in
		[Yy]* ) browse $1;
	esac

	unset browse;
}

#2

replaceText() {

	# set the temporary location
	local tFile="/tmp/out.tmp.$$"

	# call variables
	local sedArgs="$@"

	browse() {
		for iFile in "$1"/*; do
			if [ -d "$iFile" ];then
				# enter subdirectory...
				browse "$iFile"
			elif [ -f $iFile -a -r $iFile ]; then
				echo "$iFile"
				sed $sedArgs "$iFile" > $tFile && mv $tFile "$iFile"
			else
				echo "Skip $iFile"
			fi
		done
	}

	read -p "Do you want to apply \"sed $sedArgs\" to all files contained in the directory $PWD? [y/n] " yn
	case $yn in
		[Yy]* ) browse $PWD;
	esac

	unset browse

}

smile

Trilby wrote:

Anyhow, this looks like it was a good way to experiment and learn some bash, but given that find does do a better job much more simply, there isn't a lot of purpose in perfecting your script.

We've all reinvented the wheel before, it's fun, it's a chance to learn.  But don't expect people to start buying your square wheels! wink

Ahahah ok smile I was only looking for a simple method to massively replace some strings for a job of mines. I don't want to sell wheels wink

Offline

Board footer

Powered by FluxBB