Linux System hanged, unable to access it thru network or directly.

yungbloke1111 · 2011-01-22 08:06:47

Hi,

We have IBM X-Series Servers. Oracle Enterprise Linux 5.3 is installed over it..
Sytem is running for last 1 yr, suddenly it ihanged as if it is disconnected , we tried to access it but failed , as display is not coming on monitor like blank screen.

I turn it off and restarted, it worked fine.
But /var/log/messages or dmesg shows the following...

Jan 16 04:02:03 ahcdb syslogd 1.4.1: restart.
Jan 16 04:02:03 ahcdb nmbd[11744]: Got SIGHUP dumping debug info.
Jan 16 04:02:03 ahcdb nmbd[11744]: [2011/01/16 04:02:03, 0] nmbd/nmbd_workgroupdb.c:dump_workgroups(282)
Jan 16 04:02:03 ahcdb nmbd[11744]: dump_workgroups()
Jan 16 04:02:03 ahcdb nmbd[11744]: dump workgroup on subnet 172.16.0.56: netmask=255.255.255.192:
Jan 16 04:02:03 ahcdb nmbd[11744]: MYGROUP(1) current master browser = AHCTST
Jan 16 04:02:03 ahcdb nmbd[11744]: AHCDB 40819a03 (Samba Server Version 3.0.33-3.7.el5)
Jan 16 04:02:03 ahcdb nmbd[11744]: AHCTST 40849a03 (Samba Server Version 3.0.33-3.7.el5)
Jan 18 10:09:56 ahcdb syslogd 1.4.1: restart.
Jan 18 10:09:56 ahcdb kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 18 10:09:56 ahcdb kernel: Linux version 2.6.18-128.el5 (mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 08:45:05 EST 2009
Jan 18 10:09:56 ahcdb kernel: Command line: ro root=LABEL=/ rhgb quiet
Jan 18 10:09:56 ahcdb kernel: BIOS-provided physical RAM map:
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 0000000000100000 - 00000000bff4b380 (usable)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 00000000bff4b380 - 00000000bff57b40 (ACPI data)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 00000000bff57b40 - 00000000c0000000 (reserved)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 00000000d0000000 - 00000000e0000000 (reserved)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Jan 18 10:09:56 ahcdb kernel: BIOS-e820: 0000000100000000 - 0000000840000000 (usable)
Jan 18 10:09:56 ahcdb kernel: DMI 2.4 present.
Jan 18 10:09:56 ahcdb rpc.statd[4965]: Version 1.0.9 Starting
Jan 18 10:09:57 ahcdb rpc.statd[4965]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 12 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 32 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 16 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 60 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 13 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 33 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 17 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 61 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 14 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 34 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 18 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 62 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 15 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 35 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 19 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: PXM 0 -> APIC 63 -> Node 0
Jan 18 10:09:57 ahcdb kernel: SRAT: Node 0 PXM 0 0-c0000000
Jan 18 10:09:57 ahcdb kernel: SRAT: Node 0 PXM 0 0-840000000
Jan 18 10:09:57 ahcdb kernel: Bootmem setup node 0 0000000000000000-0000000840000000
Jan 18 10:09:57 ahcdb kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Jan 18 10:09:57 ahcdb kernel: disabling kdump
Jan 18 10:09:57 ahcdb kernel: ACPI: PM-Timer IO Port: 0x588
Jan 18 10:09:57 ahcdb hcid[5065]: Bluetooth HCI daemon
Jan 18 10:09:57 ahcdb sdpd[5069]: Bluetooth SDP daemon
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x0c] enabled)
Jan 18 10:09:57 ahcdb hcid[5065]: Register path:/org/bluez fallback:1
Jan 18 10:09:57 ahcdb kernel: Processor #12 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x20] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #32 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #16 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x3c] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #60 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x0d] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #13 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x21] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #33 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x11] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #17 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x3d] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #61 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x0e] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #14 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #34 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #18 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x3e] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #62 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0f] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #15 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x23] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #35 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x13] enabled)
Jan 18 10:09:57 ahcdb pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
Jan 18 10:09:57 ahcdb kernel: Processor #19 6:15 APIC version 20
Jan 18 10:09:57 ahcdb kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x3f] enabled)
Jan 18 10:09:57 ahcdb kernel: Processor #63 6:15 APIC version 20
Jan 18 10:09:58 ahcdb hidd[5206]: Bluetooth HID daemon
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x04] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x05] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x06] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x07] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb automount[5237]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x08] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x09] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0a] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0b] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0c] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0d] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0e] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: LAPIC_NMI (acpi_id[0x0f] dfl dfl lint[0x1])
Jan 18 10:09:58 ahcdb kernel: ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
Jan 18 10:09:58 ahcdb kernel: IOAPIC[0]: apic_id 15, version 32, address 0xfec00000, GSI 0-23
Jan 18 10:09:58 ahcdb kernel: ACPI: IOAPIC (id[0x10] address[0xfecff000] gsi_base[24])
Jan 18 10:09:58 ahcdb hpiod: 1.6.7 accepting connections at 2208...
Jan 18 10:09:58 ahcdb kernel: IOAPIC[1]: apic_id 16, version 17, address 0xfecff000, GSI 24-26
Jan 18 10:09:58 ahcdb kernel: ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[27])
Jan 18 10:09:58 ahcdb kernel: IOAPIC[2]: apic_id 14, version 17, address 0xfec01000, GSI 27-62
Jan 18 10:09:58 ahcdb kernel: ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[63])
Jan 18 10:09:58 ahcdb kernel: IOAPIC[3]: apic_id 13, version 17, address 0xfec02000, GSI 63-98
Jan 18 10:09:58 ahcdb kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 18 10:09:58 ahcdb kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 8 global_irq 8 high edge)
Jan 18 10:09:58 ahcdb kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
Jan 18 10:09:58 ahcdb kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Jan 18 10:09:58 ahcdb kernel: Setting APIC routing to clustered
Jan 18 10:09:58 ahcdb kernel: ACPI: HPET id: 0x10142201 base: 0xfde84000
Jan 18 10:09:58 ahcdb kernel: Using ACPI (MADT) for SMP configuration information
Jan 18 10:09:58 ahcdb kernel: Nosave address range: 000000000009b000 - 000000000009c000
Jan 18 10:09:58 ahcdb kernel: Nosave address range: 000000000009c000 - 00000000000a0000
Jan 18 10:09:58 ahcdb kernel: Nosave address range: 00000000000a0000 - 00000000000e0000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000000e0000 - 0000000000100000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000bff4b000 - 00000000bff4c000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000bff4c000 - 00000000bff57000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000bff57000 - 00000000bff58000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000bff58000 - 00000000c0000000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000c0000000 - 00000000d0000000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000d0000000 - 00000000e0000000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
Jan 18 10:09:59 ahcdb kernel: Nosave address range: 00000000fec00000 - 0000000100000000
Jan 18 10:09:59 ahcdb kernel: Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
Jan 18 10:09:59 ahcdb kernel: SMP: Allowing 16 CPUs, 0 hotplug CPUs
Jan 18 10:09:59 ahcdb kernel: Built 1 zonelists. Total pages: 8268757
Jan 18 10:09:59 ahcdb kernel: Kernel command line: ro root=LABEL=/ rhgb quiet
Jan 18 10:09:59 ahcdb kernel: Initializing CPU#0

and 
root@ahcdb ~]# lspci
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)
01:00.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 01)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 01)
03:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
04:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1078 PCI-Express Fusion-MPT SAS (rev 04)
05:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
0b:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
11:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
17:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
18:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
1d:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
1e:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
23:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)
24:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
24:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
29:00.0 PCI bridge: IBM CalIOC2 PCI-E Root Port (rev 01)

[root@ahcdb ~]# lspci -n
00:1c.0 0604: 8086:27d0 (rev 01)
00:1d.0 0c03: 8086:27c8 (rev 01)
00:1d.1 0c03: 8086:27c9 (rev 01)
00:1d.2 0c03: 8086:27ca (rev 01)
00:1d.3 0c03: 8086:27cb (rev 01)
00:1d.7 0c03: 8086:27cc (rev 01)
00:1e.0 0604: 8086:244e (rev e1)
00:1f.0 0601: 8086:27b8 (rev 01)
00:1f.1 0101: 8086:27df (rev 01)
01:00.0 0300: 1002:515e (rev 02)
02:00.0 0200: 14e4:1639 (rev 01)
02:00.1 0200: 14e4:1639 (rev 01)
03:00.0 0604: 1014:0308 (rev 01)
04:00.0 0100: 1000:0062 (rev 04)
05:00.0 0604: 1014:0308 (rev 01)
0b:00.0 0604: 1014:0308 (rev 01)
11:00.0 0604: 1014:0308 (rev 01)
17:00.0 0604: 1014:0308 (rev 01)
18:00.0 0c04: 1077:2432 (rev 03)
1d:00.0 0604: 1014:0308 (rev 01)
1e:00.0 0c04: 1077:2432 (rev 03)
23:00.0 0604: 1014:0308 (rev 01)
24:00.0 0200: 8086:105e (rev 06)
24:00.1 0200: 8086:105e (rev 06)
29:00.0 0604: 1014:0308 (rev 01)

--MOD EDIT : Added code tags to make it more readable, even though the thread is closed. -- Inxsible
Please help me out to resolve this issue

Thanks
Alig

Last edited by Inxsible (2011-01-22 17:04:53)

Allan · 2011-01-22 11:22:22

Hang on.... this is the Arch Linux forums..... Don't you pay Oracle for this support?

tomk · 2011-01-22 11:24:56

Maybe he'd like to pay us instead.

Closed.

Arch Linux

#1 2011-01-22 08:06:47

Linux System hanged, unable to access it thru network or directly.

#2 2011-01-22 11:22:22

Re: Linux System hanged, unable to access it thru network or directly.

#3 2011-01-22 11:24:56

Re: Linux System hanged, unable to access it thru network or directly.

Board footer