You are not logged in.

#1 2013-10-01 23:09:16

andrekp
Member
Registered: 2012-06-07
Posts: 112

[Solved]Sed pattern matching help

Ok I have a file like:

foo
cloo
zoo

And I want to have a script where: If it finds a line "foo" in a file, I want it to change the file to be:

foo bar
cloo
zoo

However, if the file already has "bar" appended to the line (as per the second example above), I want it to ignore the line.

A sed line:

sed -i '/foo/ s/$/ bar/' ~/myfile  

Will do the trick as far as appending " bar" to lines that say "foo"
but it will also append " bar" to lines that already say "foo bar"

What I need is code that will look for lines that say "foo" and append the " bar", but ignore instances where the " bar" has already been appended.

In English: Find lines that start with "foo" but do not include " bar" and append bar to the end of the line.

I am not married to using sed, but it needs to be something I can toss into a bash script and run.  This seems simple, but hours later...

Thanks

Last edited by andrekp (2013-10-03 19:31:53)

Offline

#2 2013-10-01 23:13:13

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: [Solved]Sed pattern matching help

sed -i '/bar/! s/foo.*$/& bar/' ~/myfile

Last edited by cfr (2013-10-01 23:14:14)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#3 2013-10-01 23:14:02

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: [Solved]Sed pattern matching help

How about

sed -i '/foo$/ s/$/ bar/' ~/myfile

Offline

#4 2013-10-02 07:42:13

Roken
Member
From: South Wales, UK
Registered: 2012-01-16
Posts: 1,251

Re: [Solved]Sed pattern matching help

or

 grep -v bar | sed -i 's/foo/foo bar/g'

Ryzen 5900X 12 core/24 thread - RTX 3090 FE 24 Gb, Asus Prime B450 Plus, 32Gb Corsair DDR4, Cooler Master N300 chassis, 5 HD (1 NvME PCI, 4SSD) + 1 x optical.
Linux user #545703

Offline

#5 2013-10-02 11:53:46

andrekp
Member
Registered: 2012-06-07
Posts: 112

Re: [Solved]Sed pattern matching help

Thanks guys!

I just picked the first one and it worked great for me.  I'll try the others as well. 

Sed (and I guess regex) is something I have yet to really understand.  Any resource you might particularly recommend to learn it?

on:

sed -i '/bar/! s/foo.*$/& bar/' ~/myfile

What does the "/foo.*$/& " sequence do?  (the characters after foo)

Thanks again,
Andre

Offline

#6 2013-10-02 12:02:22

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: [Solved]Sed pattern matching help

awk '/^foo/ && ! /bar$/ {sub(/$/," bar");} // {print;}' file

Last edited by Trilby (2013-10-02 12:04:12)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Online

#7 2013-10-02 17:18:37

sitquietly
Member
From: On the Wolf River
Registered: 2010-07-12
Posts: 219

Re: [Solved]Sed pattern matching help

andrekp wrote:

.....Sed (and I guess regex) is something I have yet to really understand.  Any resource you might particularly recommend to learn it?

You could get a copy of the classic and very helpful book Mastering Regular Expressions.  The older version is fine for understanding sed pattern matching and used copies are available for the cost of shipping. ($4 total).  It's one of the few books I still keep on hand (actually my old copy got left in the National Wilderness Area in my cabin when I abandoned it and I had to buy a new copy last year).  I read it, I need it, but I do not enjoy it. big_smile  Regular expressions are tedious and very helpful.  [The alternative is to use Icon and "string scanning" but that's another topic...I do love Icon and it is NOT tedious.].

Offline

#8 2013-10-02 21:17:05

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: [Solved]Sed pattern matching help

Roken wrote:

or

 grep -v bar | sed -i 's/foo/foo bar/g'

This won't work. What are you looking for bar in? Assuming you insert the filename there, you'll then be running sed only on the result of that (so the -i will do nothing). That is, you'll lose all lines which include bar in the original file.


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#9 2013-10-02 21:22:11

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: [Solved]Sed pattern matching help

andrekp wrote:

on:

sed -i '/bar/! s/foo.*$/& bar/' ~/myfile

What does the "/foo.*$/& " sequence do?  (the characters after foo)

So . means "any occurrence of any character". * means "zero or more occurrences of the previous character". So .* means "any string of characters of zero or greater length". $ indicates end of line. So this will match foo followed by anything and will substitute for foo and whatever follows to the end of the line. The & just prints out whatever the search string matched so you don't lose it.

e.g. This would match

foo
arm foo bow
afoocooloo=(\!@7

And the result would repeat the string in each case and append bar:

foo bar
arm foo bow bar
afoocooloo=(\!@7 bar

EDIT: O'Reilly's Classic Shell Scripting includes chapters on sed and awk. But awk scares me yikes.

EDIT: Correct "one" as "zero" as pointed out below.

Last edited by cfr (2013-10-02 22:09:36)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#10 2013-10-02 21:48:32

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: [Solved]Sed pattern matching help

cfr wrote:

... * means "one or more occurrences of the previous character". So .* means "any string of characters of zero or greater length"...

Just a pedantic quibble - as your second quoted sentence implies, * matches zero or more instances.

The O'Reilly book "Sed & Awk" covers the subject very well, but also these online manuals are worth looking at:

https://www.gnu.org/software/sed/manual/
http://www.delorie.com/gnu/docs/sed/sed_toc.html

https://www.gnu.org/software/gawk/manual/
http://www.delorie.com/gnu/docs/gawk/gawk_toc.html

Obviously they are GNU versions of the commands.

One final tip, if you are testing, then you can omit the -i in sed and the results will go to standard output. That way you don't accidentally trash your data.
Obviously working on copies also avoids that, but different people prefer different methods. Also I think GNU sed is the only version that has -i, so it's not portable (for whatever that's worth in each case).


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#11 2013-10-02 22:10:39

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: [Solved]Sed pattern matching help

skanky wrote:

Just a pedantic quibble - as your second quoted sentence implies, * matches zero or more instances.

Thanks. Not sure why I wrote that - I've edited my explanation to avoid confusion.


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#12 2013-10-03 19:22:50

andrekp
Member
Registered: 2012-06-07
Posts: 112

Re: [Solved]Sed pattern matching help

Thanks for ALL of your help, everyone.  Works great.

Just to show what I was REALLY doing, I run openbox with obmenu-generator and I was trying to have a special menu entry for certain programs that I use a lot so that I could find them easily and immediately.  This entailed creating a special category for the program in the respective /usr/share/applications/<program>.desktop file on the "Categories=" line.

A typical line might read:

Categories=GNOME;GTK;Office;Viewer;Graphics;2DGraphics;VectorGraphics;

And I wanted to add my special category, "Favorites" at the end.  Leaving out lots of detail, this would allow me to have a special menu item for certain programs.  At first I just added it manually like:

Categories=GNOME;GTK;Office;Viewer;Graphics;2DGraphics;VectorGraphics;Favorites

but I found that the <program>.desktop file would get replaced on updates and overwrite my changes every so often.  So I figured I'd write a script to do the updating and just run that when I saw the need.

And additional wrinkle was that some programs had the line by default as:

Categories=GNOME;GTK;Office;Viewer;Graphics;2DGraphics;VectorGraphics;

and others as:

Categories=GNOME;GTK;Office;Viewer;Graphics;2DGraphics;VectorGraphics

Note the final ; or lack thereof, so I had to account for that as well.  (Actually, I could ignore it and just add ;Favorites to the end, since the system ignored ";;" as being an error - but I wanted to do it cleanly.)

My final code became:

sudo sed -i 's/\(Categories.*\);$/\1/;/Favorites/! s/Categories.*$/&;Favorites/' /usr/share/applications/evince.desktop

which first strips off the final ; if found on the Categories line, then adds ;Favorites to the end of that line.  I just created a line in a bash script for each program I wanted to form the favorites menu, as above.  Works well.  May or may not be the best way, but...

Anyway, maybe this will give an idea helpful to someone else.   

(Personally, I think it would be nice if the .desktop files, which are really just configuration files, should warn you when they overwrite, but that's probably a losing argument.)

Thanks again,
Andre

Offline

#13 2013-10-03 19:38:54

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: [Solved]Sed pattern matching help

*headdesk*

Just put your prefered .desktop file in ~/.local/share/applications.  These override the system defaults and don't get overwritten with updates.

Another X-Y problem deflated.

(edit: typo)

Also, the following would have worked too:

sed -i '/^Categories/ s/;*$/;YourThingy;/'

Last edited by Trilby (2013-10-03 19:55:49)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Online

#14 2013-10-03 20:35:51

andrekp
Member
Registered: 2012-06-07
Posts: 112

Re: [Solved]Sed pattern matching help

Trilby wrote:

*headdesk*

Just put your prefered .desktop file in ~/.local/share/applications.  These override the system defaults and don't get overwritten with updates.

Another X-Y problem deflated.

(edit: typo)

Also, the following would have worked too:

sed -i '/^Categories/ s/;*$/;YourThingy;/'

I thought of that, but I like to avoid having duplicate configuration files that serve no real purpose except to scratch a minimal itch.  Besides, it's always possible that the original might be important one day...  Maybe, maybe not.  It was just as easy for me to just add a call to my new Favorites script to my usually-run system upgrade script and leave it all on autopilot.  Does what I need it to do and I never have to think about it again.

Thanks

Offline

#15 2013-10-03 20:39:41

2ManyDogs
Forum Moderator
Registered: 2012-01-15
Posts: 4,645

Re: [Solved]Sed pattern matching help

The point of having user-specific configs in a user-owned directory is specifically for this kind of "minimal itch." Of course it's your system, so you can do whatever you want. In the future though it might be helpful if you explain a little more about what you're trying to do and why, so people don't feel like they've wasted their time.

Trilby, I learned a few things about sed and awk from this thread, so it wasn't useless.

Last edited by 2ManyDogs (2013-10-03 20:46:10)


How to post. A sincere effort to use modest and proper language and grammar is a sign of respect toward the community.

Offline

#16 2013-10-03 23:46:21

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: [Solved]Sed pattern matching help

Trilby wrote:

*headdesk*

Just put your prefered .desktop file in ~/.local/share/applications.  These override the system defaults and don't get overwritten with updates.

Another X-Y problem deflated.

(edit: typo)

Also, the following would have worked too:

sed -i '/^Categories/ s/;*$/;YourThingy;/'

This.

Also: you run system updates on autopilot? That is an insanely bad idea. Not only is tweaking the system .desktop files this way not the right solution, it is the wrong solution to a problem which needs not to be solved in the first place.

As was said above:

2ManyDogs wrote:

Of course it's your system, so you can do whatever you want. In the future though it might be helpful if you explain a little more about what you're trying to do and why, so people don't feel like they've wasted their time.

EDIT: Note that if you are really worried about missing updates to .desktop files, you should have your script copy them to your home directory on update, overwriting your previous version, and then run your sed script on that version. (Don't do this as root, obviously.)

Last edited by cfr (2013-10-03 23:50:28)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#17 2013-10-05 16:36:23

andrekp
Member
Registered: 2012-06-07
Posts: 112

Re: [Solved]Sed pattern matching help

Assuming cfr's post was aimed at ME.  I don't do updates on autopilot in the literal sense.  I have a script that I run with various things like updating the mirror list, doing pacman -Syu, cleaning the cache, etc., is separate steps.  I watch it as it runs and it allows user intervention between each step.  It is NOT autopilot in any literal way, it is just automating command entry.

Interesting idea on copying the desktop files to my directory, etc.  I may consider that, but I'm not sure there's really a good reason not to just update the originals in place.  I mean I can think of a number of paranoid reasons, but likely problems?  I don't know.  I'm just not a fan of having multiple config files all over the place that will need to be remembered, occasionally checked, maintained, etc.  Especially for something that really is of minimal import.

Thanks all.

Offline

#18 2013-10-05 16:39:29

andrekp
Member
Registered: 2012-06-07
Posts: 112

Re: [Solved]Sed pattern matching help

2ManyDogs wrote:

The point of having user-specific configs in a user-owned directory is specifically for this kind of "minimal itch." Of course it's your system, so you can do whatever you want. In the future though it might be helpful if you explain a little more about what you're trying to do and why, so people don't feel like they've wasted their time.

Trilby, I learned a few things about sed and awk from this thread, so it wasn't useless.

Really now...  I seriously don't think I've given ANY reason for anybody to think they've "wasted their time."  My question was specific to what I needed to do and it was properly answered.  I think simplifying my question so that I could ask it precisely was a lot better than some highly detailed question with lots of extra details.

Offline

#19 2013-10-09 17:18:08

Roken
Member
From: South Wales, UK
Registered: 2012-01-16
Posts: 1,251

Re: [Solved]Sed pattern matching help

cfr wrote:
Roken wrote:

or

 grep -v bar | sed -i 's/foo/foo bar/g'

This won't work. What are you looking for bar in? Assuming you insert the filename there, you'll then be running sed only on the result of that (so the -i will do nothing). That is, you'll lose all lines which include bar in the original file.

And to the follow up replies: Just goes to show I should test before posting. FWIW, I always sed to STDOUT first before I commit changes for exactly this reason.


Ryzen 5900X 12 core/24 thread - RTX 3090 FE 24 Gb, Asus Prime B450 Plus, 32Gb Corsair DDR4, Cooler Master N300 chassis, 5 HD (1 NvME PCI, 4SSD) + 1 x optical.
Linux user #545703

Offline

Board footer

Powered by FluxBB