Arch Linux

oliver

Member

Registered: 2007-12-12

Posts: 448

[solved] grepping a file - can this be turned into a one-liner?

Using bash

This is probably really dumb and easy but I'm banging my head against a wall

I have a list of servers called 'allservers' in the following format

LON001
LON002
NYC001
NYC002
MUN001
MUN002
PAR001
PAR002

I have a file called 'includefile' in the following format

#
# servers to process
#

# all london servers
^LON[0-9]{3}

# all paris servers
^PAR[0-9]{3}

I want to grep out all the servers in the include file, but we're talking 100s (not just LON/PAR), so I really need to keep it as a separate file

The following command matches everything (all 8 entries in the example)

grep -E -f includefile allservers

If I remove the blank lines from the exclude file, it works and I only see the LON and PAR servers as expected

Right now I'm getting around this by removing the blank lines from the file with sed as follows

sed  '/^$/d' includefile > includefile.tmp
grep -E -f includefile.tmp allservers

The question is, can I do this in one go (and ideally expand it so I'm not wasting cycles grepping the comments too)?

Something like

grep -E -f (sed '/^$/d' includefile) allservers

Last edited by oliver (2013-05-08 20:44:23)

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [solved] grepping a file - can this be turned into a one-liner?

Not sure I got it right:

$ grep -E "LON|PAR" allservers
LON001
LON002
PAR001
PAR002

You can write the output to a file, pipe it to less etc.

Last edited by karol (2013-05-08 20:29:01)

oliver

Member

Registered: 2007-12-12

Posts: 448

Re: [solved] grepping a file - can this be turned into a one-liner?

Those examples were simplified.  I really need to use the external file because currently there are 61 entries in there and it's created by another group (I just grab it, copy it to my laptop and execute my script against it)

It just annoys me that I sed out the blanks and hashes before running the grep - hence the question.

Raynman

Member

Registered: 2011-10-22

Posts: 1,539

Re: [solved] grepping a file - can this be turned into a one-liner?

You can do:

grep -E -f <(sed '/^\(#.*\)\?$/d' includefile) allservers

and grep seems to follow the convention that a single dash (as a filename argument) means to read from stdin, so an alternative is:

sed '/^\(#.*\)\?$/d' includefile | grep -E -f - allservers

The sed expression now also removes lines starting with #.

Edit: if the patterns always match the complete server name, I would go with skanky's solution.

Last edited by Raynman (2013-05-08 20:41:58)

skanky

Member

From: WAIS

Registered: 2009-10-23

Posts: 1,847

Re: [solved] grepping a file - can this be turned into a one-liner?

Try

grep -E -o -f includefile allservers

It could be done with awk as well, but wouldn't be a one liner.

Last edited by skanky (2013-05-08 20:37:57)

---

"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,347

Re: [solved] grepping a file - can this be turned into a one-liner?

You may want to look into awk

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

oliver

Member

Registered: 2007-12-12

Posts: 448

Re: [solved] grepping a file - can this be turned into a one-liner?

you guys rule!

Both of the following worked on my test cases

grep -E -f <(sed '/^\(#.*\)\?$/d' includefile) allservers

grep -E -o -f includefile allservers

I don't control the include file (obviously I can modify it once it's local but I'd rather not) so all I know for sure is that it will have POSIX regex. 

You're curing my ADD - it bothered me to have that over multiple lines :-)

alphaniner

Member

From: Ancapistan

Registered: 2010-07-12

Posts: 2,810

Re: [solved] grepping a file - can this be turned into a one-liner?

Both of the following worked on my test cases

grep -E -f <(sed '/^\(#.*\)\?$/d' includefile) allservers

Here's the same using awk instead of sed:

grep -E -f <(awk '!/^(#.*)?$/' includefile) allservers

Alternately, if the lines from includefile you want to 'keep' will always begin with '^', you can do something like:

grep -E -f <(awk '/^\^/' includefile) allservers

---

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

thisoldman

Member

From: Pittsburgh

Registered: 2009-04-25

Posts: 1,172

Re: [solved] grepping a file - can this be turned into a one-liner?

This works too:

$ grep -E -f <(grep -Ev '^$|^#' includefile) allservers

Procyon

Member

Registered: 2008-05-07

Posts: 1,819

Re: [solved] grepping a file - can this be turned into a one-liner?

@alphaniner:
If you're gonna use awk, it can do it by itself:

awk 'BEGIN{ while ( getline ERE < "includefile" ) { if ( ERE !~ /^$|^#/ ) { REG=REG ERE "|" } } REG=substr(REG, 0, length(REG) -1) } $0 ~ REG' allserver

oliver

Member

Registered: 2007-12-12

Posts: 448

Re: [solved] grepping a file - can this be turned into a one-liner?

thanks again everyone for the great suggestions

alphaniner

Member

From: Ancapistan

Registered: 2010-07-12

Posts: 2,810

Re: [solved] grepping a file - can this be turned into a one-liner?

@Procyon

I figured as much, but my awk-fu is not so powerful.

---

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

skanky

Member

From: WAIS

Registered: 2009-10-23

Posts: 1,847

Re: [solved] grepping a file - can this be turned into a one-liner?

@alphaniner:
If you're gonna use awk, it can do it by itself:

awk 'BEGIN{ while ( getline ERE < "includefile" ) { if ( ERE !~ /^$|^#/ ) { REG=REG ERE "|" } } REG=substr(REG, 0, length(REG) -1) } $0 ~ REG' allserver

Nice one. I don't like too complicated awk scripts on one line, but that's a little simpler than I was envisaging, with most of the work in BEGIN.

---

"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle