Arch Linux

fflarex

Member

Registered: 2007-09-15

Posts: 466

[SOLVED] sh: strange behavior of uniq

So I'm currently writing some scripts to make life easier on myself, and I ran into some odd behavior when I tried using uniq in a pipeline. This isn't the exact code I'm using, but it shows the problem:

# Outputs "y" to stdout immediately
yes | uniq

# Doesn't write anything to file
yes | uniq > file

I need my script to only print when something has changed, but I can't avoid polling. I thought that using uniq would be an easy solution, and it worked when I tried it on the terminal, but not in my script (for the reason above). Anyone know why this is, or how I can work around it?

Last edited by fflarex (2010-03-20 22:25:37)

Procyon

Member

Registered: 2008-05-07

Posts: 1,819

Re: [SOLVED] sh: strange behavior of uniq

script -f can do this, but it also writes things like "Script started on ..."

Try script -f -c 'yes | uniq' file.txt

fflarex

Member

Registered: 2007-09-15

Posts: 466

Re: [SOLVED] sh: strange behavior of uniq

Not really a good solution, since I would have to use sed or something to filter that out. It might even be easier to reimplement uniq in sed than to filter that out. Thanks though.

EDIT: Actually, this might work after all. Here's what I'm using:

script -qfc 'yes | uniq' /dev/null > file

It's really, really ugly, but it'll work for now.

EDIT 2: Nevermind, that doesn't work either. It only outputs the first line; not any subsequent changes. Any other ideas?

Last edited by fflarex (2010-03-20 00:35:55)

falconindy

Developer

From: New York, USA

Registered: 2009-10-22

Posts: 4,111

Website

Re: [SOLVED] sh: strange behavior of uniq

uniq isn't going to work because it only detects duplicate fields when they're adjacent. You'll need to write a script using something like tcsh to determine if you've seen the output before and suppress it.

---

blog | ±github

fflarex

Member

Registered: 2007-09-15

Posts: 466

Re: [SOLVED] sh: strange behavior of uniq

uniq isn't going to work because it only detects duplicate fields when they're adjacent. You'll need to write a script using something like tcsh to determine if you've seen the output before and suppress it.

That's actually not the problem at all. That's the behavior I want. The problem is that it behaves differently depending on whether it's output goes to the terminal or a file.

Procyon

Member

Registered: 2008-05-07

Posts: 1,819

Re: [SOLVED] sh: strange behavior of uniq

I think this is caused by ulimit -p, but I'm not sure.

It's not uniq's fault, I only know of one program that can actually go through, sed -u:
yes | sed -u -n 1p > file.txt

You can find uniq written in sed here: http://www.gnu.org/software/sed/manual/ … .html#uniq

It needs a little editing:

#! /bin/sed -unf
1p

:b
N
/^\(.*\)\n\1$/ {
# The two lines are identical. Kill first line.
D
bb
}

# The lines are different. Print the last.
#D
s/^[^\n]*\n//
P
bb

At least, I think this is still correct...

fflarex

Member

Registered: 2007-09-15

Posts: 466

Re: [SOLVED] sh: strange behavior of uniq

Thanks, it works perfectly!