suppressing hidden double quotes in returned array values

throbscottle · 2023-06-17 23:08:55

Ok title looks weird. It's what it looks like to me though.

So, I decided to try and supply the sonar screensaver with friendly names. Nmap gets me hostnames, arp-scan gets me whatever the manufacturer gave the device. It doesn't matter what names are actually used. I've been trying to combine the two.
So, I've got as far as creating an array with the address/hostname pairs generated by nmap, and looping through the elements. The results will be glued together and piped into a file with lines of address with one or both names, or no name as some will inevitably be.

What is happening though, if I pass an array element to arp-scan, it sees it's input in double quotes, which it then can't interpret. With echo, it outputs the elements with spaces between - I guess the spaces are indicative of the hidden double quotes. So either I'm putting something in that shouldn't be there, like extra newlines, or I know less about arrays than I thought I did!

Can anyone help correct what's going on with my array?

# generate list of friendly names for use with sonar screensaver

# show addresses only for UP IPV4 interfaces separated by a space, ignoring loopback
ADDRESS=`ip -4 -brief address show |grep -v lo|awk '{print $3 " "}'`

if [[ $ADDRESS == "192.168.1."* || $ADDRESS == "192.168.0."* ]]
        then ADDRESS="192.168.0.0/23"
fi # pick up if we are home or visiting somewhere

# make an array separated by the pipe symbol
IFS="|"
addlist=(`nmap -sn $ADDRESS|grep -v "Nmap done"|grep scan |awk 'NR==1 {print;next};{print $6 " " $5"|"}'|sed -E 's/\(//;s/\)//'|grep -vi "Nmap" |sed  -E "s/.Home//;s/.local//;s/^[ \w ]//"`

for add in ${addlist[@]} ;do
#        echo ${add}    
       arp-scan -x ${add}
done

Last edited by throbscottle (2023-06-17 23:10:44)

Trilby · 2023-06-17 23:28:34

I can't make heads or tails of that spaghetti code pipeline. Why are you passing nmap output through a dozen different text processing tools? Please provide the raw output of nmap and a description of what you want as a result.

throbscottle · 2023-06-18 18:23:54

It strips everything out so you just get the IP address followed by the hostname, if there is one. So it's like a very simple hosts file. Yes it's ugly and I'm sure I could do it better but time is short and my skills are few.

Anyway, I made some progress. Changing

$addlist[@]

to

"$addlist[*]"

got rid of the extra newlines, and not setting the IFS variable lets arp-scan work correctly, but then it's splitting the address/name pairs on the spaces, so that's no good. I suppose I could substitute the space for something then substitute back again (actually splitting the element into 2 extra variables might be the way to go here, considering the desired result!), but I'd prefer to learn how to handle the array elements properly.

I'll break down that pipeline.

I can be on either of 2 IP ranges at home, so that's what the gubbins with the $ADDRESS variable is doing, making sure both are captured, unless I'm on public wifi or something.
I'm on 192.168.1.57 at the moment - so I'll just use 192.168.1.0/24

nmap -sn 192.168.1.0/24

this gives a lot of lines like:

Nmap scan report for Linksys11555.Home (192.168.1.1)
Host is up (0.0043s latency).
Nmap scan report for Bedroom.Home (192.168.1.10)
Host is up (0.011s latency).
Nmap scan report for Meross_Smart_Plug.Home (192.168.1.11)
Host is up (0.056s latency).
Nmap scan report for 192.168.1.14
Host is up (0.021s latency).
...
Nmap done: 256 IP addresses (16 hosts up) scanned in 2.48 seconds

Pipe it through grep a couple times:

|grep  -v "Nmap done"|grep scan

gets rid of the last line, containing "Nmap done", which also contains the string "scan", and allows through only the other lines containing "scan", so I get output that looks like this:

Nmap scan report for Linksys11555.Home (192.168.1.1)
Nmap scan report for Bedroom.Home (192.168.1.10)
Nmap scan report for Meross_Smart_Plug.Home (192.168.1.11)
Nmap scan report for 192.168.1.14
Nmap scan report for NPI2FA7EE.Home (192.168.1.16)
Nmap scan report for SonosZP.Home (192.168.1.17)
Nmap scan report for 192.168.1.22
Nmap scan report for 192.168.1.24
Nmap scan report for 192.168.1.25

(Actually, now I think about it, just grepping "report" should do both jobs.)
So next, I need to get rid of "Nmap scan report for ", the brackets, .Home (or .local if I'm on .0 instead of .1) and put the address first. So (now modified, I changed the NR==1 to NR==0 ):

|awk NR==0 '{print;next};{print $6 " " $5"|"}'

using awk to just print the 6th and then 5th elements does a big lump of that, it also adds in the "|" for the IFS variable, important when handling spaces!

(192.168.1.1) Linksys11555.Home
(192.168.1.10) Bedroom.Home
(192.168.1.11) Meross_Smart_Plug.Home
 192.168.1.12
 192.168.1.14
(192.168.1.16) NPI2FA7EE.Home
(192.168.1.17) SonosZP.Home
 192.168.1.18
 192.168.1.22
 192.168.1.24

Obviously it's no good having those brackets there, so

|sed -E 's/\(//;s/\)//'

gets rid of them. However it still leaves an annoying the space at the start, so

|sed  -E "s/.Home//;s/.local//;s/^[ \w ]//"

gets rid of them, and also those unwanted "domain" names. The bit that says:

|grep -vi "Nmap"

is now redundant so I've deleted that.

So now, with echo in the for loop in place of arp-scan, I get output that looks like this:

192.168.1.1 Linksys11555 
192.168.1.10 Bedroom 
192.168.1.11 Meross_Smart_Plug 
192.168.1.14 
192.168.1.16 NPI2FA7EE 
192.168.1.17 
192.168.1.18 
192.168.1.22

, which looks like it should be perfect, right? If I copy any of these lines into

arp-scan -x

, I get

$ arp-scan -x 192.168.1.11 Meross_Smart_Plug
192.168.1.11    48:e1:e9:01:03:5d       Chengdu Meross Technology Co., Ltd.
192.168.1.11    48:e1:e9:01:03:5d       Chengdu Meross Technology Co., Ltd.

or

$ arp-scan -x 192.168.1.17
192.168.1.17    48:a6:b8:b6:13:71       Sonos, Inc.

Which referring to the list above, neatly highlights the whole purpose of the exercise.
But if I change echo back to arp-scan in the for loop, I get:

WARNING: get_host_address failed for "192.168.1.1 Linksys11555": Name or service not known - target ignored
WARNING: get_host_address failed for "
192.168.1.10 Bedroom": Name or service not known - target ignored
WARNING: get_host_address failed for "
192.168.1.11 Meross_Smart_Plug": Name or service not known - target ignored
WARNING: get_host_address failed for "
192.168.1.12": Name or service not known - target ignored

Where on earth is the

"
..."

construct coming from????
I can see that when echo is used, it doesn't show the quote marks. How do I pass the element to arp-scan in a way that doesn't include the quotes?

throbscottle · 2023-06-18 18:31:45

Slight progress.

Setting IFS==""

gets me a just one set of quotes around the whole listing - so arp-scan just rejects the first and last records.

Trilby · 2023-06-18 18:41:17

My point was that pipeline is so complex there is no telling what could be happening - and the problem is most certainly there*. As for the quotes, those are added to the error output by arp-scan itself. Start here:

nmap -sn $ADDRESS | sed -n 's/.* \([^ .]\+\)[.A-Za-z]* (\([0-9.]\+\))/\2 \1/p' | \
   while read ip name; do
      arp-scan -x $ip
   done

But why all the work to get the hosname if that isn't even used?

nmap -sn $ADDRESS | grep -Eo '[0-9]{3}\.[0-9]{3}\.[0-9.]*' | while read ip; do
   arp-scan -x $ip
done

*specifically, you set IFS=| so that newlines were no longer a field seperator, so when you fed content including newlines into the for loop, the newlines were preserved in the loop variable and passed as an argument to arp-scan. Arp-scan takes IP addresses, and IP addresses can't include newlines: so it reported an error with the "address" including a newline: this erroneous input data was printed in quotes by arp-scan.

Last edited by Trilby (2023-06-18 18:46:58)

throbscottle · 2023-06-18 19:12:02

And /this/ is why I ask people who know what they are talking about!

It's going to take me a while to unpack your sed and grep usages... Thanks for the education there!

The point of the exercise is at the top of the first post - it's to provide friendly names of any kind for the sonar screensaver to display, and combine the outputs of both nmap and arp-scan, since they produce different results.

So nmap produces the biggest list of devices, but only a small number of them have names. arp-scan produces a shorter list, but they all have names, and importantly, most of the names belong to different devices than the host names given in the nmap list.

So the final output would have on each line, IP address, then either hostname or arp name, or both (and a few will have none, but still be included).

Then the script would be run by cron twice a day or something (not sure if the screensaver can actually run a script - would be ideal if it can!).

Yes, I am completely mad...

Trilby · 2023-06-18 19:29:23

Ok. Wouldn't you want / need to use an arp-scan then only if nmap didn't provide a name? For example:

#!/bin/sh

nmap -sn $ADDRESS | \
	sed -n '/ for /{s/.*for //;s/\([^ \.]*\)[^ ]* (\([^)]*\))/\2 \1/;p};' | \
	while read ip name; do
		[ -n "$name" ] && echo $ip $name && continue
		arp-scan -x $ip # ... format this output?
	done

EDIT: having not used arp-scan before I realize the above is a bad approach. Don't run arp-scan for each address, just run it once with the -lx flags then `join' with the nmap output.

EDIT 2: having now read the arp-scan man page (which you should really do too), there's no reason for any of this to reach your goal, just use the following (in place of the entire script):

arp-scan -lxgdF '${ip}\t${name}\t${vendor}' | \
   awk -F'\t' '$1 == $2 { $2 = $3; }; { print $2; }'

That's it. Except you may want a "sub()" function in the last block if you want to get rid of ".Home" or whatever lan extension there may be on the hostnames.

Also, given that arp scans aren't always perfect, rather than running this as-is twice a day, you could run it several times with a short delay between each run, then "sort -u" the results for a more comprehensive list. E.g.:

for i in 1 2 3; do
   arp-scan -lxgdF '${ip}\t${name}\t${vendor}' | \
      awk -F'\t' '$1 == $2 { $2 = $3; }; { print $2; }'
   sleep 5
done | sort -u

Last edited by Trilby (2023-06-18 20:04:43)

throbscottle · 2023-06-20 22:31:14

Wow - that's just what I wanted - though the IP has gone missing in the final version - I'm sure I can fix it! Shame it doesn't pick up my house's other subnet though - maybe it's back to nmap for that.

Thank you so much.

So, I had another look at the arp-scan man page. I'd looked through the --help, obviously not very thoroughly or I'd have spotted the -d option if nothing else. I've learnt a few things here - once again, thanks

Trilby · 2023-06-20 22:55:13

throbscottle wrote:

Wow - that's just what I wanted - though the IP has gone missing in the final version

It hasn't "gone missing" it's just not displayed. If you want that too, change the "print $2" to "print $1, $2".

throbscottle · 2023-06-28 21:21:02

Ah, got it. Thanks again!

2ManyDogs · 2023-06-28 22:10:00

Please remember to mark your thread [SOLVED] (edit the title of your first post).

Arch Linux

#1 2023-06-17 23:08:55

suppressing hidden double quotes in returned array values

#2 2023-06-17 23:28:34

Re: suppressing hidden double quotes in returned array values

#3 2023-06-18 18:23:54

Re: suppressing hidden double quotes in returned array values

#4 2023-06-18 18:31:45

Re: suppressing hidden double quotes in returned array values

#5 2023-06-18 18:41:17

Re: suppressing hidden double quotes in returned array values

#6 2023-06-18 19:12:02

Re: suppressing hidden double quotes in returned array values

#7 2023-06-18 19:29:23

Re: suppressing hidden double quotes in returned array values

#8 2023-06-20 22:31:14

Re: suppressing hidden double quotes in returned array values

#9 2023-06-20 22:55:13

Re: suppressing hidden double quotes in returned array values

#10 2023-06-28 21:21:02

Re: suppressing hidden double quotes in returned array values

#11 2023-06-28 22:10:00

Re: suppressing hidden double quotes in returned array values

Board footer