You are not logged in.

#1 2013-11-14 18:12:42

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

RPC-gssd kernel null pointer dereference

 NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
BUG: unable to handle kernel NULL pointer dereference at 0000000000000c68
IP: [<ffffffffa0a3900c>] put_pipe_version+0x1c/0x80 [auth_rpcgss]
PGD 7b2fa067 PUD 764f1067 PMD 0 
Oops: 0000 [#1] PREEMPT SMP 
Modules linked in: rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss oid_registry nfs_acl xfs crc32c libcrc32c arc4 b43 bcma mac80211 cfg80211 joydev iTCO_wdt iTCO_vendor_support coretemp hp_wmi sparse_keymap ssb rfkill mmc_core pcmcia microcode pcmcia_core pcspkr psmouse serio_raw i2c_i801 lpc_ich 8139too snd_hda_codec_conexant snd_hda_intel 8139cp snd_hda_codec mii snd_hwdep snd_pcm snd_page_alloc snd_timer snd wmi thermal shpchp evdev battery ac processor soundcore nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_generic usbhid hid sd_mod sr_mod cdrom ata_generic pata_acpi ahci libahci ata_piix libata ehci_pci scsi_mod uhci_hcd ehci_hcd usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
CPU: 0 PID: 822 Comm: rpc.gssd Not tainted 3.12.0-1-ARCH #1
Hardware name: 
task: ffff88007a60c580 ti: ffff880076646000 task.ti: ffff880076646000
RIP: 0010:[<ffffffffa0a3900c>]  [<ffffffffa0a3900c>] put_pipe_version+0x1c/0x80 [auth_rpcgss]
RSP: 0000:ffff880076647e38  EFLAGS: 00010202
RAX: ffff88007a60c580 RBX: 0000000000000001 RCX: 00000000c0000100
RDX: ffff880076647fd8 RSI: ffff88007a60c580 RDI: 0000000000000000
RBP: ffff880076647e48 R08: ffff880076646000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000010 R14: ffff880053defa00 R15: ffff880053dc0d20
FS:  00007f1aa8c4d740(0000) GS:ffff88007ea00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000c68 CR3: 000000007650c000 CR4: 00000000000007f0
Stack:
ffff880053defa00 ffff880059cfe5c8 ffff880076647e60 ffffffffa0a3909d
ffff8800766865c0 ffff880076647ec0 ffffffffa0a39fe8 0000000000000000
0000000000000065 ffff880059cfe5c8 ffff8800766865c4 0000000100317f41
Call Trace:
[<ffffffffa0a3909d>] gss_release_msg+0x2d/0x80 [auth_rpcgss]
[<ffffffffa0a39fe8>] gss_pipe_downcall+0x248/0x540 [auth_rpcgss]
[<ffffffffa048a866>] rpc_pipe_write+0x56/0x70 [sunrpc]
[<ffffffff811a3a8d>] vfs_write+0xbd/0x1e0
[<ffffffff811a44e9>] SyS_write+0x49/0xa0
[<ffffffff814fabad>] system_call_fastpath+0x1a/0x1f
Code: 31 c0 e8 26 07 ab e0 31 c0 eb b4 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 41 54 49 89 fc 53 8b 1d ab 42 a6 ff e8 c6 8b 64 e0 85 db <49> 8b 84 24 68 0c 00 00 74 4b 3b 18 77 47 83 eb 01 48 63 db 48 
RIP  [<ffffffffa0a3900c>] put_pipe_version+0x1c/0x80 [auth_rpcgss]
RSP <ffff880076647e38>
CR2: 0000000000000c68
---[ end trace be1777d1c894f03f ]---
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=18002 jiffies, g=299650, c=299649, q=7598)
INFO: Stall ended before state dump start
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=72007 jiffies, g=299650, c=299649, q=25163)
INFO: Stall ended before state dump start
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=126013 jiffies, g=299650, c=299649, q=42479)
INFO: Stall ended before state dump start
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=180018 jiffies, g=299650, c=299649, q=59833)
INFO: Stall ended before state dump start
INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=234023 jiffies, g=299650, c=299649, q=79646)
INFO: Stall ended before state dump start

It fails to mount the nfs share, but was able to PXE boot from this same nfs server.  Will check to see if PXE client is using rpc-gssd (wasn't).  Also rpc-gssd as a systemd service is failing to restart.

Last edited by nomorewindows (2014-01-17 22:20:02)


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#2 2013-11-14 19:30:32

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Disabling rpc-gssd for now and seems to mount nfs shares without as much difficulty as previously had when started using rpc-nssd.

Last edited by nomorewindows (2013-11-14 19:47:06)


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#3 2013-11-14 20:03:54

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Trying this on another machine gives the following from systemctl status rpc-gssd:

rpc-gssd.service - RPC GSS-API client-side daemon
   Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; enabled)
   Active: active (running) since Thu 2013-11-14 13:15:19 CST; 49min ago
  Process: 217 ExecStart=/usr/bin/rpc.gssd $GSSD_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 221 (rpc.gssd)
   CGroup: /system.slice/rpc-gssd.service

systemd[1]: Started RPC GSS-API client-side daemon.
rpc.gssd[221]: ERROR: Cannot determine realm for numeric host address while getting realm(s) for host 'xxx.xxx.xxx.xxx'
rpc.gssd[221]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host xxx.xxx.xxx.xxx
rpc.gssd[221]: ERROR: No credentials found for connection to server xxx.xxx.xxx.xxx

Looking at the contents of /etc/krb5.conf is:

[libdefaults]
        default_realm = ATHENA.MIT.EDU

[realms]
# use "kdc = ..." if realm admins haven't put SRV records into DNS
        ATHENA.MIT.EDU = {
                admin_server = KERBEROS.MIT.EDU
                default_domain = MIT.EDU
                v4_instance_convert = {
                        mit = mit.edu
                        lithium = lithium.lcs.mit.edu
                }
        }
        ANDREW.CMU.EDU = {
                admin_server = vice28.fs.andrew.cmu.edu
        }

[domain_realm]
        .mit.edu = ATHENA.MIT.EDU
        mit.edu = ATHENA.MIT.EDU
        .media.mit.edu = MEDIA-LAB.MIT.EDU
        media.mit.edu = MEDIA-LAB.MIT.EDU
        .ucsc.edu = CATS.UCSC.EDU

[logging]
#       kdc = CONSOLE

Last edited by nomorewindows (2013-11-14 20:11:40)


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#4 2013-11-25 13:42:28

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Nfs-utils updated, restarted rpc-gssd and now the error is:

NFS: nfs4_discover_server_trunking unhandled error -512.  Exiting with error EIO

systemctl status rpc-gssd shows:

ERROR: can't open /var/lib/nfs/rpc_pipefs/nfs/clnta/info: No such file or directory
ERROR: failed to read service info

I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#5 2013-11-27 13:47:42

erm67
Member
From: Europe
Registered: 2007-08-01
Posts: 123

Re: RPC-gssd kernel null pointer dereference

mount
none on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)

Check if rpc_pipefs is mounted, nfs worked fine before even if I didn't had it in /etc/fstab but after a recent update I found that nfs was not working animore, and rpc_pipefs was not mounted, after adding

none            /var/lib/nfs/rpc_pipefs rpc_pipefs      default 0       0

in my fstab everything was back to normality.

Offline

#6 2013-11-28 03:16:10

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Running mount shows that:

rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)

Just updated the linux package when I did this, but may have already been there from previous kernel.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#7 2013-11-28 07:57:36

erm67
Member
From: Europe
Registered: 2007-08-01
Posts: 123

Re: RPC-gssd kernel null pointer dereference

Yeah I am still getting the kernel OOPS in gssd even with the rpc_pipefs mounted, but strangely not every time I mount the NFS share. I disabled the rpc-gssd service and wait some time at every mount for the moment.
After the kernel OOPS it is better to restart the computer to a clean state.
It is probably a bug in the kernel I think.

Last edited by erm67 (2013-11-28 08:02:17)

Offline

#8 2013-11-29 20:06:10

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

I wonder if it would work to disable the gssd options in /etc/conf.d/nfs-server.conf?
I don't really have any need for kerberos on my network, but others might be necessary.

Last edited by nomorewindows (2013-12-01 17:51:54)


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#9 2013-12-02 09:35:49

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: RPC-gssd kernel null pointer dereference

I'm following this thread too since I have similar problems. I had a kernel panic at boot with kernel linux-3.12.1-1-i686 and rpc-gssd.service. And since I just run this service on my client to shorten boot time with 15 secs and do not use kerberos at all, I'd really like to get rid of it. Google has not given me any clues to this, only that the rpc-gssd service occasionally produces OOPS on different plattforms.

So how can NOT using rpc-gssd create 15 secs longer boot time in a tiny NFS network, completely without kerberos used?
IS there a way to configure the client and the server to be more explicit about NOT using kerberos and by that eliminating the need for rpc-gssd service on the client?

Offline

#10 2013-12-02 17:34:46

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Does not look like setting NEED_GSSD to no disabled the kerberos.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#11 2013-12-11 17:02:18

OlafLostViking
Member
From: Lost
Registered: 2013-01-30
Posts: 58

Re: RPC-gssd kernel null pointer dereference

There's a flyspray bug discussing this problem.

Offline

#12 2013-12-15 23:30:08

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

One person stated that by blacklisting the rpcsec_gss_krb5 the undesired operation went away and I tried it and was able to have the same outcome.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#13 2013-12-25 23:29:02

chetwisniewski
Member
Registered: 2012-03-04
Posts: 30

Re: RPC-gssd kernel null pointer dereference

My network requires Kerberos, but I can't figure out how to tell what is the actual problem... When I run systemctl status rpc-gssd I get:

rpc-gssd.service - RPC GSS-API Client Daemon
   Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; enabled)
   Active: failed (Result: exit-code) since Wed 2013-12-25 13:50:19 PST; 24s ago
     Docs: man:rpc.gssd(8)
  Process: 1832 ExecStart=/usr/bin/rpc.gssd $GSSD_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 1833 (code=exited, status=1/FAILURE)

Dec 25 13:50:19 peter.zuzax.local systemd[1]: Starting RPC GSS-API Client Daemon...
Dec 25 13:50:19 peter.zuzax.local systemd[1]: Started RPC GSS-API Client Daemon.
Dec 25 13:50:19 peter.zuzax.local systemd[1]: rpc-gssd.service: main process exited, code=exited, status=1/FAILURE
Dec 25 13:50:19 peter.zuzax.local systemd[1]: Unit rpc-gssd.service entered failed state.

Confusing, failures mixed with success and no description as to why failures happened. Nothing in journalctl or dmesg either.

Offline

#14 2013-12-25 23:33:50

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

You are actually using Kerberos and it still doesn't work?


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#15 2014-01-03 20:52:47

rcincx
Member
From: NZ
Registered: 2013-03-22
Posts: 6

Re: RPC-gssd kernel null pointer dereference

erm67 wrote:

after adding

none            /var/lib/nfs/rpc_pipefs rpc_pipefs      default 0       0

in my fstab everything was back to normality.

This worked for me - rpc_pipefs was being mounted anyhow but actually specifying it in the fstab cleared this error up. Thanks!

Offline

#16 2014-01-04 09:58:35

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: RPC-gssd kernel null pointer dereference

I'm sorry to say that I went with samba instead. Samba was running on the server anyway, for some windows machines in the network. It mounts faster and seems to have less hazzles. It has slower transfer speeds though.

Offline

#17 2014-01-04 14:25:55

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

I have tried transferring files over the network and none have really reached the theoretical limit of 20MBps at 100Mbps Full Duplex.  Guess what I found?  Using sftp transferring files found on a FAT32 partition would consistently do that.  Wasn't expecting the ancient filesystem to be able to do that.  Figured the monstrous filesystems such as NTFS were supposed to be faster.  I can maybe get half that out of an NTFS and none of the other filesystems in Linux seem to go that fast either.  I'm not familiar with the statistics on *BSD, with apparently better tuned/performance filesystems, as I don't use them very often.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#18 2014-01-12 16:47:29

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: RPC-gssd kernel null pointer dereference

This appears to have been fixed in 3.12.7-2!  It took a set of six patches, and I still get the "RPC: AUTH_GSS upcall failed. Please check user daemon is running."  But my journal is not being spammed with crap and despite the "RPC: AUTH_GSS upcall failed" message, there is now no delay at all.


Edit: BTW, for those of you who might be looking for this fix, this fix is still in [testing] at the moment.

Last edited by WonderWoofy (2014-01-12 16:48:15)

Offline

#19 2014-01-14 21:59:56

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Commented out previous blacklist and seems to have worked.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#20 2014-01-17 22:19:45

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: RPC-gssd kernel null pointer dereference

Forgot to enable rpc-gssd service and when I do it starts to act up again.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

Board footer

Powered by FluxBB