sed to substitute out "bad" words in an array

graysky · 2010-10-24 09:57:58

I need to have sed read in a file (single column of words) and remove any that are on a blacklist as defined in an array. I'm kinda there but I don't know how to do process the entire array and THEN write out the resulting "censored" file.

Example input list (let's call the file 'foo'):

corn
romaine
arugula
keyboard
carrot
yogurt
peas

Here is the bash script that only works when there is exactly one word in the ignore array:

#!/bin/bash
ignore=(keyboard yogurt)
for i in ${ignore}; do
 sed -e "s/$i//g" -e '/^$/d' foo>bar
done

The flaw in my design here I think is that I keep overwriting 'bar' rather than doing all the processing, then writing out 'bar'. Suggestions are welcomed - I'm always glad to learn and thanks in advance.

Last edited by graysky (2010-10-24 10:03:37)

Allan · 2010-10-24 10:17:36

Are you sure this is not homework?

graysky · 2010-10-24 10:23:03

@Allan - Homework... I haven't done homework in more years than I care to recall! I want to implement this feature in my AUR package for modprobed_db. The script within keeps track of kernel modules probed so that users can build a kernel with 'make localmodconfig' and only get the modules they need. The blacklist array will contain modules that get built by packages like virtualbox and nvidia, for example: (nvidia vboxdrv vboxnetflt vboxnetadp) that we don't want probed for the compilation. In the real case, the input file will be /var/log/modprobe.long that my script in that package generates - I only presented the problem in simple "salad" terms to make it easier for the community to see what I want to do

Last edited by graysky (2010-10-24 10:29:47)

Allan · 2010-10-24 10:35:24

OK then... had to check

allan@mugen ~ 
> cat foo.txt 
one
two
three

allan@mugen ~ 
> sed '/^two$/d' foo.txt 
one
three

portix · 2010-10-24 10:38:56

If you use an array it has to be

for i in ${ignore[*]}; do ...

to iterate over all elements.

graysky · 2010-10-24 10:44:27

@allan - I can do that part... I just can't parlay the code into working with my array.

@portix - I tried your suggestion but got the same result.

#!/bin/bash
ignore=(keyboard yogurt)
for i in ${ignore[*]}; do
 sed -e "s/$i//g" -e '/^$/d' foo>bar
done

So 'bar' still contains the blacklisted word 'keyboard':

corn
romaine
arugula
keyboard
carrot
peas

Last edited by graysky (2010-10-24 10:45:45)

olvar · 2010-10-24 11:07:12

after your sed command you may want to move the new file to the old:

for i in.... ; then
   sed .... foo > bar
   mv bar foo
fi

so you are actually filtering the stuff out

Last edited by olvar (2010-10-24 11:08:30)

falconindy · 2010-10-24 14:06:49

build a filter and call sed once. backup optional.

modfilter=""
for ign in "${ignore[@]}"; do
  modfilter+='/\<'"$ign"'\>/d;'
done

sed -i.orig "$modfilter" inputlist

barto · 2010-10-24 23:39:20

I would do the filtering with grep -v, using extended regexp.

#!/bin/bash
ignore=(keyboard yogurt)
grep -Ev "`echo ${ignore[*]} | sed 's/ /|/g'`" foo > bar

falconindy · 2010-10-24 23:52:54

Word boundaries are important here. You probably don't want to ignore snd-sgalaxy, sg, videobuf-dma-sg, and ipmi_msghandler if you only put 'sg' in the ignore array.

graysky · 2010-10-25 06:56:30

@barto - thanks for the code! I updated my package accordingly.

Last edited by graysky (2010-10-25 06:56:49)

barto · 2010-10-25 23:17:44

@falconindy - Thanks for your notice, you are right.
Adding the option -x to grep to make it match whole lines should solve the issue.

Arch Linux

#1 2010-10-24 09:57:58

sed to substitute out "bad" words in an array

#2 2010-10-24 10:17:36

Re: sed to substitute out "bad" words in an array

#3 2010-10-24 10:23:03

Re: sed to substitute out "bad" words in an array

#4 2010-10-24 10:35:24

Re: sed to substitute out "bad" words in an array

#5 2010-10-24 10:38:56

Re: sed to substitute out "bad" words in an array

#6 2010-10-24 10:44:27

Re: sed to substitute out "bad" words in an array

#7 2010-10-24 11:07:12

Re: sed to substitute out "bad" words in an array

#8 2010-10-24 14:06:49

Re: sed to substitute out "bad" words in an array

#9 2010-10-24 23:39:20

Re: sed to substitute out "bad" words in an array

#10 2010-10-24 23:52:54

Re: sed to substitute out "bad" words in an array

#11 2010-10-25 06:56:30

Re: sed to substitute out "bad" words in an array

#12 2010-10-25 23:17:44

Re: sed to substitute out "bad" words in an array

Board footer