You are not logged in.

#1 2009-05-08 12:31:32

einheitlix
Member
Registered: 2009-03-31
Posts: 16

[solved] The weirdest thing with ssh

Hi,

since quite recently I've been getting the weirdest error with ssh:

malte@bombadil $ ssh quarterback
ssh: Could not resolve hostname quarterback: Success

So now what's this about? "Could not resolve hostname... success" ? wink If ssh considers this to be a success, what's a failure then? big_smile

Anyways, what could be causing this? There is no problem with DNS or the resolving of hostnames in general here, except with ssh. For instance,

malte@bombadil $ ping quarterback
PING quarterback (134.96.240.71) 56(84) bytes of data.
64 bytes from quarterback (134.96.240.71): icmp_seq=1 ttl=64 time=0.187 ms
64 bytes from quarterback (134.96.240.71): icmp_seq=2 ttl=64 time=0.318 ms
64 bytes from quarterback (134.96.240.71): icmp_seq=3 ttl=64 time=0.221 ms
^C
--- quarterback ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.187/0.242/0.318/0.055 ms

malte@bombadil $ host quarterback
quarterback.cs.uni-sb.de has address 134.96.240.71
quarterback.cs.uni-sb.de mail is handled by 50 mail.uni-sb.de.
quarterback.cs.uni-sb.de mail is handled by 40 mail.cs.uni-sb.de.

nmap, ftp etc. also all work as expected, It's only ssh that's complaining.

Plus, quarterback is in my /etc/hosts anyways...

malte@bombadil $ grep quarterback /etc/hosts
134.96.240.71   quarterback.cs.uni-sb.de        quarterback

So, this really is a problem with ssh, not hostname resolution on a system-wide scale.

ssh'ing to a full host name also doesn't work:

malte@bombadil $ ssh quarterback.cs.uni-sb.de
ssh: Could not resolve hostname quarterback.cs.uni-sb.de: Success

However, ssh'ing to an IP works as excpected. So it seems to be a problem with ssh not being able to resolve hostnames correctly.

quarterback was only used as an example here, it is a machine in our LAN; however, I have the same problem with all hostnames and ssh, whatever machine I try to ssh to, whether I specify an alias or a complete hostname. It always works using the actual IP.

This problem occurs also on the only other machine running Arch here at work, and on no other machine, hence I assume it's really an Arch-specific problem with ssh.

In summary:
(1) the problem is ssh itself, not hostname resolution on the system
(2) it is Arch specific
(3) it has only been occurring for a short time, it used to work

My Arch system is up-to-date. openssh is version 5.2p1-1. However, even downgrading to openssh 5.1p1-2 does not help.

Has anybody else seen this? Any idea why this is happening?

Thanks,

Malte

Last edited by einheitlix (2009-05-15 13:51:17)

Offline

#2 2009-05-08 12:49:52

rson451
Member
From: Annapolis, MD USA
Registered: 2007-04-15
Posts: 1,233
Website

Re: [solved] The weirdest thing with ssh

Out of curiosity, have you been messing with /etc/ssh_config?  There is an option CheckHostIP that does is used to check for DNS spoofing, maybe that's giving you problems?


archlinux - please read this and this — twice — then ask questions.
--
http://rsontech.net | http://github.com/rson

Offline

#3 2009-05-08 13:36:03

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Well, no, I don't think that's it. I didn't mess with ssh_config.

root@bombadil $ grep CheckHostIP /etc/ssh/ssh_config 
#   CheckHostIP yes

It's commented out and has ever been using the default value (which is yes, as it seems).

However I tracked down the problem further. telnet also exhibits the same behaviour as ssh, so it must be some common dependency that has been updated and causes ssh and telnet to fail...

root@bombadil $ telnet quarterback
quarterback/telnet: lookup failure: Success

I have found out that it's actually glibc that's causing the problem. I updated it from 2.9-4 to 2.9.7. Here's why I'm sure...

openssh and telnet both depend on the package tcp_wrappers:

root@bombadil $ pacman -Qi openssh inetutils | grep ^Depends
Depends On     : openssl>=0.9.8k  zlib  pam  tcp_wrappers  heimdal>=1.2.1  
Depends On     : readline  ncurses  pam  tcp_wrappers

tcp_wrappers in turn depends on glibc:

root@bombadil $ pacman -Qi tcp_wrappers | grep ^Depends
Depends On     : bash  glibc

glibc is version 2.9-7:

root@bombadil $ pacman -Qi glibc | grep ^Version
Version        : 2.9-7

Downgrading glibc to version 2.9-4 makes everything work again:
(I also have to downgrade binutils since it depends on glibc-2.9-7)

root@bombadil $ pacman -U glibc-2.9-4-x86_64.pkg.tar.gz binutils-2.19.1-1-x86_64.pkg.tar.gz 
loading package data...
checking dependencies...
(2/2) checking for file conflicts                   [-----------------------------------------------------------------------] 100%
warning: /etc/nscd.conf saved as /etc/nscd.conf.pacsave
(1/2) upgrading glibc                               [-----------------------------------------------------------------------] 100%
Generating locales...
  de_DE.UTF-8... done
  de_DE.ISO-8859-1... done
  de_DE.ISO-8859-15@euro... done
  en_US.UTF-8... done
  en_US.ISO-8859-1... done
Generation complete.
(2/2) upgrading binutils                            [-----------------------------------------------------------------------] 100%

Now ssh works as usual, without problems concering hostname resolution.

So is this a bug in glibc? Is anybody else experiencing this with glibc-2.9-7?

Cheers,

Malte

Offline

#4 2009-05-08 13:40:20

rson451
Member
From: Annapolis, MD USA
Registered: 2007-04-15
Posts: 1,233
Website

Re: [solved] The weirdest thing with ssh

I've got the following and am having no issues.

[randy@voodoo ~] $ pacman -Qi glibc tcp_wrappers openssh | grep Ver
Version        : 2.9-7
Version        : 7.6-8
Version        : 5.2p1-1

Last edited by rson451 (2009-05-08 13:43:03)


archlinux - please read this and this — twice — then ask questions.
--
http://rsontech.net | http://github.com/rson

Offline

#5 2009-05-08 14:32:10

hutch
Member
From: Lusiana - Italy
Registered: 2008-10-16
Posts: 13

Re: [solved] The weirdest thing with ssh

rson451 wrote:

I've got the following and am having no issues.

[randy@voodoo ~] $ pacman -Qi glibc tcp_wrappers openssh | grep Ver
Version        : 2.9-7
Version        : 7.6-8
Version        : 5.2p1-1

I have those packages too and also for me ssh works fine

Offline

#6 2009-05-11 09:04:52

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Ok, then there is yet something else that is causing this, and using an older glibc version just works around it somehow...

But under this configuration I do get the ssh hostname lookup failure:

root@bombadil $ pacman -Qi glibc tcp_wrappers openssh | grep Ver
Version        : 2.9-7
Version        : 7.6-8
Version        : 5.2p1-1

And downgrading glibc does solve the ssh problem; under the following configuration (I only executed pacman -U glibc-2.9-4-x86_64.pkg.tar.gz binutils-2.19.1-1-x86_64.pkg.tar.gz between the above and the below code), I do not get the ssh problem:

root@bombadil $ pacman -Qi glibc tcp_wrappers openssh | grep Ver
Version        : 2.9-4
Version        : 7.6-8
Version        : 5.2p1-1

So what else could be causing this? Any ideas how I could track down the problem further? My workaround isn't really satisfying, since every pacman -Syu brings back the problem and I have to downgrade glibc again; this is no long-term solution... sad

Offline

#7 2009-05-11 09:17:03

the_isz
Member
Registered: 2009-04-14
Posts: 280

Re: [solved] The weirdest thing with ssh

Hi Malte,

are you by any chance using ipv6? I think I recently read about some troubles concerning ipv6 handling in glibc. You could start investigating by prohibiting the ipv6 module to load in /etc/rc.conf if you
haven't done so already. It's just a guess, but maybe this is it...

Offline

#8 2009-05-11 10:08:23

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Hi Timo,

thank you for taking the time to answer my post. Well, I followed your advice.

malte@bombadil $ grep ipv6 /etc/rc.conf
MODULES=(!ipv6)

...after reboot...

malte@bombadil $ lsmod | grep -e ^Mod -e ipv6
Module                  Size  Used by

As you can see ipv6 is not loaded. But the ssh problem occurs still...
(by the way, the ipv6 module gets loaded anyway after some time, e.g. when I start Opera; however, I did try ssh while the module wasn't loaded yet, and I did verifiy that it still wasn't loaded after the execution of ssh)

I now filed a bug report, since this is clearly a bug of some kind, albeit I do not know its full details:
http://bugs.archlinux.org/task/14678

Ideas still welcome wink

Last edited by einheitlix (2009-05-11 10:13:57)

Offline

#9 2009-05-11 10:18:17

the_isz
Member
Registered: 2009-04-14
Posts: 280

Re: [solved] The weirdest thing with ssh

Good luck with your bug, but I think posting the bug in an glibc bugtracking system might take you closer to a solution as this clearly seems to be a regression bug.

Btw: Do the other (non-Arch) machines use the same glibc version? Because otherwise your deduction, that this is an Arch bug, might be a little... flawful wink

Offline

#10 2009-05-11 11:09:42

bangkok_manouel
Member
From: indicates a starting point
Registered: 2005-02-07
Posts: 1,556

Re: [solved] The weirdest thing with ssh

looks very similar to http://bugs.archlinux.org/task/12215 which has been fixed already. I've checked and the patches are still in the current PKGBUILD, would be interesting to check if they apply well. If it happens to be the same problem, Jan gave me a nice workaround, quoting him:

JGC wrote:

I "fixed" this problem by installing pdns-recursor, binding it to 127.0.0.1 and pointing resolv.conf to this.

Offline

#11 2009-05-11 11:48:47

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [solved] The weirdest thing with ssh

I'm doing glibc-2.10 builds now so hopefully that improves things.  I really do not understand this and not much (read nothing of importance...) changed between glibc-2.9-4 and glibc-2.9-7...

Offline

#12 2009-05-11 13:00:26

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Hi people!

Ok:

I think posting the bug in an glibc bugtracking system might take you closer to a solution as this clearly seems to be a regression bug.

As I said before I do not think it is a glibc-only bug, since other people using glibc 2.9-7 do not experience this problem.
Hence it must be a problem somewhere in the toolchain, and downgrading glibc somehow works around the problem, but it must not be the case that it is entirely glibc's fault.
Thus I feel that the Arch bug tracking system is a good place.
But you might of course also be right and posting this to the glibc bug tracker would be more appropriate. It is difficult to know when you don't really know where the bug comes from wink

Do the other (non-Arch) machines use the same glibc version? Because otherwise your deduction, that this is an Arch bug, might be a little... flawful wink

A quite valid point indeed, old friend; and I should admit that I'm not quite sure about this.

The other machines are all either Ubuntu or FreeBSD, and there isn't any problem concerning name resolution in ssh with those machines.
Now FreeBSD is not even glibc-based, so these machines will not really help me to find out whether this is a glibc problem only (yeah, I know, there is Debian GNU/kFreeBSD which is glibc-based; but by FreeBSD I mean just plain standard FreeBSD wink)
As for the Ubuntu machines, well, they are Ubuntu Jaunty (9.04) and use libc6 2.9-4 (it's called libc6 in Ubuntu, but this is just glibc). So truly after all I wrote it doesn't seem surprising ssh is working there. However I did find libc6 2.9.9 packages (http://packages.ubuntu.com/jaunty/libc6), and after installing those it still works. So either this is a bug which wasn't there in 2.9-4, came in 2.9-7 and was fixed again in 2.9-9, or there is something else that is causing this. Unfortunately there are no libc6 2.9-7 packages for Ubuntu Jaunty (at least I did not find any), so I cannot check this.

looks very similar to http://bugs.archlinux.org/task/12215 which has been fixed already.

I don't think so. The bug in question caused wget to not resolve hostnames properly: but wget works fine for me both with glibc 2.9-4 and glibc 2.9-7. But yes, it might be something similar.

I've checked and the patches are still in the current PKGBUILD, would be interesting to check if they apply well. If it happens to be the same problem, Jan gave me a nice workaround

I will check to see if the workaround works for me, but I don't have time now. I will report back as soon as I get to do that.

I'm doing glibc-2.10 builds now so hopefully that improves things.

I'm looking forward to see if this helps smile

Last edited by einheitlix (2009-05-11 13:02:04)

Offline

#13 2009-05-14 20:25:47

fluchtpunkt
Member
Registered: 2009-05-14
Posts: 1

Re: [solved] The weirdest thing with ssh

I had exactly the same problem.

I have two computers, one with working dns resolving one without.
So I figured out that '/etc/nsswitch.conf' differs between these computers. Absolutely no idea why; maybe I missed some change some time ago.
The working one has

hosts: files dns

and the non working:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

removing the mdns stuff from /etc/nsswitch.conf worked for me.


HTH

Last edited by fluchtpunkt (2009-05-14 20:34:45)

Offline

#14 2009-05-15 12:07:09

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Hi!

Just to tell you that the problem is solved, using the help of Allan who did glibc 2.10 builds and the useful advice of Gerardo at the bug tracker smile Many thanks!

If anyone is interested, the problem occurred because we're using YP/NIS. My nsswitch.conf was configured in such a way that it tries to resolve hostnames though NIS first:

root@bombadil $ grep hosts /etc/nsswitch.conf
hosts: nis files dns

Changing this line to

root@bombadil $ grep hosts /etc/nsswitch.conf
hosts: files dns

...resolves the problem. Hence it had something to do with the combination of glibc 2.9-7 and NIS (or at least, our NIS server)...

Somehow the NIS server must have sent back a packet such that glibc was confused and thought the hostname was correctly resolved (which it probably was), but then it got lost somewhere in the toolchain. Or something like that. wink

Anyways I also tried bangkok_manouel's suggestion and tried to use pdns-recursor. However it did not help, which is not that surprising since I think that pdns did not even get involved: as you see from my nsswitch.conf, first hostname resolution is attempted through NIS, then through files (which is /etc/hosts I guess), then through dns (which is /etc/resolv.conf). It never even worked when I tried to ssh to a host name which had an entry in /etc/hosts, let alone a host name that would have been resolved using a nameserver in /etc/resolv.conf (and pdns would have got invoked only at that point).

Anyway, even without changing the line in /etc/nsswitch.conf as described above, everything works fine again with Allan's glibc 2.10.1 build from [testing]. So I just have wait a little until it appears in [core], and until then I can just use my modified /etc/nsswitch.conf smile

Cheers,

Malte

PS. Ah, I wrote this before I read fluchtpunkt's post. This is interesting, since it appears that this is not only a problem with our NIS server, but rather it looks like a general problem with the combination of glibc 2.9-7, and hostname resolution through various means like mdns4 or nis. Though admittedly only 2 occurrences of the problem (fluchtpunkt's and mine) have no statistical significance, so if anyone else is using nis or mdns it would be interesting to know whether it works for them or not. OTOH, since the problem disappears with glibc 2.10, this is no matter of big importance smile

Last edited by einheitlix (2009-05-15 12:22:09)

Offline

#15 2009-05-15 12:56:10

the_isz
Member
Registered: 2009-04-14
Posts: 280

Re: [solved] The weirdest thing with ssh

Nice to hear that your problem is solved.

einheitlix wrote:

[...] PS. Ah, I wrote this before I read fluchtpunkt's post.  [...]

Maybe it's actually time to follow my advice and set up email notification for forums wink

Offline

#16 2009-05-15 12:59:01

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Why, it wouldn't have made much of a difference, since I had already arrived at the very same solution one or two days ago with the help of the bug tracker smile
-> http://bugs.archlinux.org/task/14678

Though in principle I would agree with you that email notification is a nice feature inded wink

Last edited by einheitlix (2009-05-15 13:00:28)

Offline

#17 2009-05-15 13:02:21

the_isz
Member
Registered: 2009-04-14
Posts: 280

Re: [solved] The weirdest thing with ssh

In that case, you would have saved fluchtpunkt the effort of writing an answer to a solved thread wink

Btw: You should mark the thread as solved by prepending the string [solved] to the thread title.

Offline

#18 2009-05-15 13:48:12

einheitlix
Member
Registered: 2009-03-31
Posts: 16

Re: [solved] The weirdest thing with ssh

Ah... so what is your point? That email notification would be great because I would have saved fluchtpunkt some time, or what do you mean?

In fact I found his post very interesting, since I had already given thought to looking into our NIS server's configuration to find out if the error somehow originates there. Now it appears that the problem can occur as well under very different circumstances, e.g. not even using YP/NIS, but using things like mdns4 too. So at the very least he saved me the hassle of uselessly ploughing through our entire NIS server"s configuration. Thanks a lot for that! cool

Cheers,

Malte

Last edited by einheitlix (2009-05-15 13:49:06)

Offline

Board footer

Powered by FluxBB