You are not logged in.

#1 2017-03-22 22:05:49

dahy
Member
Registered: 2017-03-05
Posts: 5

underscore in sort order

When sorting with the `sort` coreutil, why is underscore (`_`) placed between A-Z letters rather than before or after?

$ echo "$(echo {A..Z}) _ " | sed 's/ /=true\n/g' | sort

A=true
B=true
C=true
D=true
...
R=true
S=true
_=true
T=true
U=true
V=true
W=true
X=true
Y=true
Z=true
$ 
$ printenv | sort
...
USER=random_sort
_=/usr/bin/printenv
VISUAL=vim
...

Last edited by dahy (2017-03-22 22:09:10)

Offline

#2 2017-03-22 22:12:57

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: underscore in sort order

It isn't 'placed' anywhere, it's ignored (along with the = and the /), for more info read this...

https://www.gnu.org/software/coreutils/ … order_0021

Last edited by Slithery (2017-03-22 22:14:17)


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#3 2017-03-22 22:14:09

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 20,722

Re: underscore in sort order

I don't have the answer, but try echo "$(echo {A..Z}) _ " | sed 's/ /=true\n/g' | LC_ALL=c sort
O_o


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way

Offline

#4 2017-03-22 22:26:59

dahy
Member
Registered: 2017-03-05
Posts: 5

Re: underscore in sort order

It is most often due to having LANG set to ‘en_US.UTF-8’ set in your user environment. At that point sort appears broken because case is folded and punctuation is ignored because ‘en_US.UTF-8’ specifies this behavior.

I have `LC_ALL` set to `en_US.UTF-8`.

$ printenv | ag ^LC_ | sort
LC_ADDRESS=en_US.UTF-8
LC_ALL=en_US.UTF-8
LC_COLLATE=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_IDENTIFICATION=en_US.UTF-8
LC_MEASUREMENT=en_US.UTF-8
LC_MESSAGES=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
LC_NAME=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_PAPER=en_US.UTF-8
LC_TELEPHONE=en_US.UTF-8
LC_TIME=en_US.UTF-8
$ 

Using `vim` instead will allow me to keep those settings.

printenv | vim -c 'sort u' -

Offline

#5 2017-03-23 05:26:58

HiImTye
Member
From: Halifax, NS, Canada
Registered: 2012-05-09
Posts: 1,072

Re: underscore in sort order

this confuses me, as slithery points out, the characters are ignored, but then why are ignored characters placed squarely between S and T? eg.

[ tye@t: ~ ]$ echo "$(echo {A..Z}) _ / = " | sed 's/ /=true\n/g' | sort
A=true
...
S=true
==true
_=true
/=true
T=true
...
Z=true

this is the same behaviour as in the OP, so surely there is some sorting going on here, correct?

Last edited by HiImTye (2017-03-23 05:27:39)

Offline

#6 2017-03-23 07:48:26

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 78,892

Re: underscore in sort order

The relevant flag is LC_COLLATE and (at least in latin glyph using regions), you'll have a *much* simpler life if you set that to "C" in /etc/locale.conf

Why sort acts apparently "random" when sorting in UTF-8, I don't know either. Might be a bug, might just be utf-8 hackishnes *shrug*

Offline

#7 2017-03-23 08:08:40

ayekat
Member
Registered: 2011-01-17
Posts: 1,634

Re: underscore in sort order

HiImTye wrote:

but then why are ignored characters placed squarely between S and T?

I think that is due to the `true` that follows. If you put `false`, it will be placed between E and F.

I found another peculiarity in the sorting, though: Depending on the locale, lowercase letters come before or after uppercase letters. For instance, en_CA has AaBbCc... (whereas en_US, en_GB, de_CH, de_DE have aAbBcC...).
But depending on how the letters are passed to `sort`, en_CA may sort it like the other locales - compare the output of those two:

for i in {A..Z} {a..z}; do echo $i; done | LANG=en_CA.utf8 sort   # AaBbCc...
echo {A..Z} {a..z} | sed 's/ /\n/g' | lang=en_CA.utf8 sort        # aAbBcC...

I have absolutely no idea why this is so. When piping both into `xxd` instead of `sort`, the output is exactly the same.

I'm sorry, typo - it should've been `LANG=`, not `lang=`. Everything's fine.

Last edited by ayekat (2017-03-23 09:13:27)


pkgs │ hacks │ cfg │ blag

Offline

#8 2017-03-23 08:51:45

HiImTye
Member
From: Halifax, NS, Canada
Registered: 2012-05-09
Posts: 1,072

Re: underscore in sort order

ayekat wrote:
HiImTye wrote:

but then why are ignored characters placed squarely between S and T?

I think that is due to the `true` that follows. If you put `false`, it will be placed between E and F.

that's exacty right, changing the t to an f in true produces the behaviour you pointed out. faulty assumptions wink

Offline

#9 2017-03-23 10:55:37

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,546
Website

Re: underscore in sort order

It sounds like the problem has been identified and a good solution proposed.  I couldn't replicate the issue for two reasons: 1) I already had LC_COLLATE="C" - doesn't everybody?? and 2) I couldn't bear to use a nested echo and subshell for no reason tongue

echo {A..Z} "_ " | sed 's/ /=true\n/g' | sort

You could also do without the sed if you can live with one blank line:

echo -e "\n"{A..Z}"=true" "\n_=true" | sort

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#10 2017-03-23 17:46:44

HiImTye
Member
From: Halifax, NS, Canada
Registered: 2012-05-09
Posts: 1,072

Re: underscore in sort order

Trilby wrote:

You could also do without the sed if you can live with one blank line:

the sed version also produces one blank line wink

Offline

Board footer

Powered by FluxBB