You are not logged in.

#1 2016-04-28 03:49:18

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

System hosed after upgrade

After a pacman -Syu my machine has gone into "read-only" mode, I've managed to boot into a recovery image, and chroot back into the environment - but not sure what to investigate next.

When I boot directly into the machine I get the following:

[ OK ] Started Login Service.
[ OK ] Stopped MariaDB database server.
[FAILED] Failed to start MariaDB database server.
See 'systemctl status mysqld.service' for details.
[ OK ] Stopped MariaDB database server.
[FAILED] Failed to start MariaDB database server.
See 'systemctl status mysqld.service' for details.
[ OK ] Stopped MariaDB database server.
[FAILED] Failed to start MariaDB database server.
See 'systemctl status mysqld.service' for details.
[FAILED] Failed to start /etc/rc.local Compatibility.
See 'systemctl status rc-local.service' for details.
[ OK ] Stopped MariaDB database server.
[FAILED] Failed to start MariaDB database server.
See 'systemctl status mysqld.service' for details.
random: nonblocking pool is initialized

Arch Linux 4.5.0-x86_64-linode65 (hvc0)

li382-54 login: perl[2715]: segfault at 5 ip 00007f3e0d2dac0e sp 00007fff4684ca50 error 4 in Socket6.s
o[7f3e0d2d7000+5000]
audit: type=1701 audit(1461759317.961:2): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=2715 comm="pe
rl" exe="/usr/bin/perl" sig=11
audit: type=1325 audit(1461759318.021:3): table=filter family=2 entries=0
audit: type=1325 audit(1461759318.021:3): table=mangle family=2 entries=0
audit: type=1325 audit(1461759318.021:3): table=nat family=2 entries=0
audit: type=1325 audit(1461759318.021:3): table=raw family=2 entries=0
audit: type=1325 audit(1461759318.021:3): table=security family=2 entries=0
audit: type=1325 audit(1461759318.021:3): table=filter family=10 entries=0
audit: type=1325 audit(1461759318.021:3): table=mangle family=10 entries=0
audit: type=1325 audit(1461759318.021:3): table=raw family=10 entries=0

Not sure what to do, I've chrooted into the system - but obviously because it's in "read only" mode when it boots it doesn't save any of the logs for later viewing, I looked at the log files while the machine was upgrading before the reboot and nothing much seems to be wrong - though I may not be looking in the wrong places.

UPDATE:

Looking at the errors I noticed that the mysql service seems to have been problematic, so I unlinked the systemd file in /etc and rebooted - less errors, but still read only mode:

[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Started Periodic Command Scheduler.
         Starting Login Service...
         Starting /etc/rc.local Compatibility...
         Starting dhcpcd on eth0...
         Starting Permit User Sessions...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Login Service.
         Starting Verify integrity of password and group files...
         Starting Rotate log files...
         Starting Update man-db cache...
         Starting Update locate database...
[  OK  ] Started Getty on tty1.
[  OK  ] Started Serial Getty on hvc0.
[  OK  ] Reached target Login Prompts.
[FAILED] Failed to start Update locate database.
See 'systemctl status updatedb.service' for details.
[FAILED] Failed to start Rotate log files.
See 'systemctl status logrotate.service' for details.
[  OK  ] Started Verify integrity of password and group files.
[  OK  ] Started Update man-db cache.
random: nonblocking pool is initialized
[FAILED] Failed to start /etc/rc.local Compatibility.
See 'systemctl status rc-local.service' for details.
perl[2711]: segfault at 5 ip 00007fdb0c4e9c0e sp 00007fff6d9f4ea0 error 4 in Socket6.so[7fdb0c4e6000+5000]
audit: type=1701 audit(1461815588.693:2): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=2711 comm="perl" exe="/usr/bin/perl" sig=11
audit: type=1325 audit(1461815588.717:3): table=filter family=2 entries=0
audit: type=1325 audit(1461815588.717:3): table=mangle family=2 entries=0
audit: type=1325 audit(1461815588.717:3): table=nat family=2 entries=0
audit: type=1325 audit(1461815588.717:3): table=raw family=2 entries=0
audit: type=1325 audit(1461815588.717:3): table=security family=2 entries=0
audit: type=1325 audit(1461815588.717:3): table=filter family=10 entries=0
audit: type=1325 audit(1461815588.717:3): table=mangle family=10 entries=0
audit: type=1325 audit(1461815588.717:3): table=raw family=10 entries=0

Arch Linux 4.4.4-x86_64-linode64 (hvc0)

li382-54 login: audit_printk_skb: 12 callbacks suppressed
audit: type=1006 audit(1461815605.453:4): pid=2824 uid=0 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=1 res=1
audit: type=1006 audit(1461815605.483:5): pid=2826 uid=0 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=2 res=1

Arch Linux 4.4.4-x86_64-linode64 (hvc0)

li382-54 login:

Last edited by beatbreaker (2016-04-28 03:56:11)

Offline

#2 2016-04-28 04:07:05

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

In the system I've also managed to inspect what are the failing services - I'll need to chroot again but I can turn off services and try to eliminate the problem from there - the rc-local.service I think looks a little suspect.

~ # ❯❯❯ systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● logrotate.service            loaded failed failed Rotate log files
● longview.service             loaded failed failed Longview Agent
● nginx.service                loaded failed failed A high performance web serve
● postgresql.service           loaded failed failed PostgreSQL database server
● rc-local.service             loaded failed failed /etc/rc.local Compatibility
● systemd-remount-fs.service   loaded failed failed Remount Root and Kernel File
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and
● systemd-update-utmp.service  loaded failed failed Update UTMP about System Boo
● updatedb.service             loaded failed failed Update locate database

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

9 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Offline

#3 2016-04-28 04:10:17

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: System hosed after upgrade

What is a rc-local.service? That isn't Arch...


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#4 2016-04-28 04:25:13

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

It is Arch, it was my own way to work around the fact that systemd moved away from having an rc.local file to execute on boot.

Now that I know how to write a systemd service file I should probably replace it - Anyway I'm going to remove that service and reboot.

As you can see, it's nothing too crazy anyway

❯❯❯ cat /etc/rc.local
#!/bin/bash
#
# /etc/rc.local: Local multi-user startup script.

redis-server /etc/redis.conf
/usr/bin/forever start --sourceDir=/srv/http/paste.andreacchio.net server.js
#/usr/bin/npm start /srv/http/paste.andreacchio.net/ &

## Disabling pastboard due to vlunerability
#npm start /srv/http/pasteboard.andreacchio.net/ &

❯❯❯ cat /etc/systemd/system/multi-user.target.wants/rc-local.service
[Unit]
Description=/etc/rc.local Compatibility

[Service]
Type=forking
ExecStart=/etc/rc.local
TimeoutSec=0
#StandardInput=tty
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Anyway to help narrow it down I disabled a bunch of those services - but still no dice:

[  OK  ] Reached target Swap.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Started Daily locate database update.
[  OK  ] Reached target Basic System.
         Starting Login Service...
         Starting Permit User Sessions...
         Starting dhcpcd on eth0...
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Started Periodic Command Scheduler.
[  OK  ] Started Daily verification of password and group files.
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily man-db cache update.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Reached target Timers.
[  OK  ] Started Permit User Sessions.
         Starting Verify integrity of password and group files...
         Starting Rotate log files...
         Starting Update locate database...
         Starting Update man-db cache...
[  OK  ] Started Serial Getty on hvc0.
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[FAILED] Failed to start Update locate database.
See 'systemctl status updatedb.service' for details.
[FAILED] Failed to start Rotate log files.
See 'systemctl status logrotate.service' for details.
[  OK  ] Started Verify integrity of password and group files.
[  OK  ] Started Update man-db cache.
random: nonblocking pool is initialized

Arch Linux 4.4.4-x86_64-linode64 (hvc0)

li382-54 login: perl[2699]: segfault at 5 ip 00007f1e6172ec0e sp 00007ffdd3252720 error 4 in Socket6.so[7f1e6172b000+5000]
audit: type=1701 audit(1461818075.891:2): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=2699 comm="perl" exe="/usr/bin/perl" sig=11
audit: type=1325 audit(1461818075.915:3): table=filter family=2 entries=0
audit: type=1325 audit(1461818075.915:3): table=mangle family=2 entries=0
audit: type=1325 audit(1461818075.915:3): table=nat family=2 entries=0
audit: type=1325 audit(1461818075.915:3): table=raw family=2 entries=0
audit: type=1325 audit(1461818075.915:3): table=security family=2 entries=0
audit: type=1325 audit(1461818075.915:3): table=filter family=10 entries=0
audit: type=1325 audit(1461818075.915:3): table=mangle family=10 entries=0
audit: type=1325 audit(1461818075.915:3): table=raw family=10 entries=0

Last edited by beatbreaker (2016-04-28 04:42:35)

Offline

#5 2016-04-28 05:49:50

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: System hosed after upgrade

This is one of those Linode images? I recall seeing other threads about borked upgrades: might be worth searching through those...


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#6 2016-04-28 06:13:52

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

Yeah It's a Linode. I've searched around but can't find anything similar to what I've got - even Perl seems to crap itself, any ideas behind that, or anywhere else I can investigate?

~/ uname -a
Linux li382-54.members.linode.com 4.4.4-x86_64-linode64 #1 SMP Tue Mar 8 11:21:53 EST 2016 x86_64 GNU/Linux
~/ sudo systemctl --failed
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
● logrotate.service              loaded failed failed Rotate log files
● longview.service               loaded failed failed Longview Agent
● systemd-remount-fs.service     loaded failed failed Remount Root and Kernel File Systems
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories
● systemd-update-utmp.service    loaded failed failed Update UTMP about System Boot/Shutdown
● updatedb.service               loaded failed failed Update locate database

Offline

#7 2016-04-28 06:34:20

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

Humm... I think I'm getting close. Here I can see that /dev/xvda is ext3 filesystem type

❯❯❯ mount
/dev/xvda on / type ext3 (ro,relatime,data=ordered)

But, then I saw this using dmesg:

[11617385.534757] EXT4-fs (xvda): mounting ext3 file system using the ext4 subsystem
[11617385.541015] EXT4-fs (xvda): barriers disabled
[11617385.542599] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
[11617385.542634] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
[11617385.548836] devtmpfs: mounted

Not sure this has anything to do with it, but it seems that EXT4 is trying to mount the ext3 file system, then it reverts to read only mode for some reason

Offline

#8 2016-04-28 06:51:43

x33a
Forum Fellow
Registered: 2009-08-15
Posts: 4,587

Re: System hosed after upgrade

beatbreaker wrote:

Not sure this has anything to do with it, but it seems that EXT4 is trying to mount the ext3 file system, then it reverts to read only mode for some reason

That is normal. See the details here: https://bugs.archlinux.org/task/25341

Offline

#9 2016-04-28 06:57:04

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

x33a wrote:

That is normal. See the details here: https://bugs.archlinux.org/task/25341

Yeah I found that with a little more Googling, ok, so I'm still not sure where the problem is coming from - here's my dmesg

root@li382-54 ~ # ❯❯❯ dmesg
[    0.000000] x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WC  WP  UC  UC
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.4-x86_64-linode64 (maker@build) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Tue Mar 8 11:21:53 EST 2016
[    0.000000] Command line: root=/dev/xvda xencons=tty console=tty1 console=hvc0 nosep nodevfs ramdisk_size=32768 ip_conntrack.hashsize=8192 nf_conntrack.hashsize=8192 ro  devtmpfs.mount=1
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Released 0 page(s)
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000007fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] Hypervisor detected: Xen
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x80000 max_arch_pfn = 0x400000000
[    0.000000] MTRR: Disabled
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 24576
[    0.000000] BRK [0x02365000, 0x02365fff] PGTABLE
[    0.000000] BRK [0x02366000, 0x02366fff] PGTABLE
[    0.000000] BRK [0x02367000, 0x02367fff] PGTABLE
[    0.000000] BRK [0x02368000, 0x02368fff] PGTABLE
[    0.000000] BRK [0x02369000, 0x02369fff] PGTABLE
[    0.000000] BRK [0x0236a000, 0x0236afff] PGTABLE
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x7fc17000-0x7fc1afff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x000000007fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000007fffffff]
[    0.000000] On node 0 totalpages: 524191
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3999 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8128 pages used for memmap
[    0.000000]   DMA32 zone: 520192 pages, LIFO batch:31
[    0.000000] p2m virtual area at ffffc90000000000, size is 400000
[    0.000000] Remapped 0 page(s)
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] e820: [mem 0x80000000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.4.4-pre (preserve-AD)
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6370452778343963 ns
[    0.000000] setup_percpu: NR_CPUS:48 nr_cpumask_bits:48 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s94424 r8192 d32552 u1048576
[    0.000000] pcpu-alloc: s94424 r8192 d32552 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1
[    0.000000] xen: PV spinlocks enabled
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 515978
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/xvda xencons=tty console=tty1 console=hvc0 nosep nodevfs ramdisk_size=32768 ip_conntrack.hashsize=8192 nf_conntrack.hashsize=8192 ro  devtmpfs.mount=1
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory: 2035332K/2096764K available (9892K kernel code, 1240K rwdata, 4108K rodata, 1524K init, 660K bss, 61432K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=48 to nr_cpu_ids=2.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
[    0.000000] Using NULL legacy PIC
[    0.000000] NR_IRQS:4352 nr_irqs:48 0
[    0.000000] xen:events: Using FIFO-based ABI
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2800.110 MHz processor
[11625826.372713] Calibrating delay loop (skipped), value calculated using timer frequency.. 5602.56 BogoMIPS (lpj=9333700)
[11625826.372725] pid_max: default: 32768 minimum: 301
[11625826.373005] Security Framework initialized
[11625826.373378] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[11625826.373964] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[11625826.374187] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[11625826.374200] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[11625826.374492] Initializing cgroup subsys io
[11625826.374502] Initializing cgroup subsys memory
[11625826.374514] Initializing cgroup subsys devices
[11625826.374521] Initializing cgroup subsys freezer
[11625826.374527] Initializing cgroup subsys net_cls
[11625826.374533] Initializing cgroup subsys perf_event
[11625826.374541] Initializing cgroup subsys net_prio
[11625826.374547] Initializing cgroup subsys pids
[11625826.374555] Initializing cgroup subsys debug
[11625826.374609] CPU: Physical Processor ID: 1
[11625826.374613] CPU: Processor Core ID: 11
[11625826.375367] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8
[11625826.375374] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0, 1GB 4
[11625826.545081] ftrace: allocating 40360 entries in 158 pages
[11625826.576796] cpu 0 spinlock event irq 1
[11625826.576816] Could not initialize VPMU for cpu 0, error -38
[11625826.576880] Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
[11625826.577200] SMP alternatives: switching to SMP code
[11625826.636426] installing Xen timer for CPU 1
[11625826.636464] cpu 1 spinlock event irq 8
[11625826.637628] x86: Booted up 1 node, 2 CPUs
[11625826.637954] devtmpfs: initialized
[11625826.638689] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6370867519511994 ns
[11625826.638837] xor: automatically using best checksumming function:
[11625826.670721]    avx       : 24620.400 MB/sec
[11625826.670969] NET: Registered protocol family 16
[11625826.670987] xen:grant_table: Grant tables using version 1 layout
[11625826.671000] Grant table initialized
[11625826.673486] PCI: setting up Xen PCI frontend stub
[11625826.673494] PCI: pci_cache_line_size set to 64 bytes
[11625826.744417] raid6: sse2x1   gen()  6298 MB/s
[11625826.801296] raid6: sse2x1   xor()  5580 MB/s
[11625826.858148] raid6: sse2x2   gen()  8241 MB/s
[11625826.915005] raid6: sse2x2   xor()  6312 MB/s
[11625826.972016] raid6: sse2x4   gen()  7917 MB/s
[11625827.028919] raid6: sse2x4   xor()  6738 MB/s
[11625827.028932] raid6: using algorithm sse2x2 gen() 8241 MB/s
[11625827.028936] raid6: .... xor() 6312 MB/s, rmw enabled
[11625827.028941] raid6: using ssse3x2 recovery algorithm
[11625827.029011] ACPI: Interpreter disabled.
[11625827.029039] xen:balloon: Initialising balloon driver
[11625827.039695] xen_balloon: Initialising balloon driver
[11625827.040029] vgaarb: loaded
[11625827.040167] SCSI subsystem initialized
[11625827.040270] libata version 3.00 loaded.
[11625827.040340] pps_core: LinuxPPS API ver. 1 registered
[11625827.040348] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[11625827.040365] PTP clock support registered
[11625827.040456] dmi: Firmware registration failed.
[11625827.040477] PCI: System does not support PCI
[11625827.040483] PCI: System does not support PCI
[11625827.040900] clocksource: Switched to clocksource xen
[11625827.050403] FS-Cache: Loaded
[11625827.050557] CacheFiles: Loaded
[11625827.050613] pnp: PnP ACPI: disabled
[11625827.055185] NET: Registered protocol family 2
[11625827.055431] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[11625827.055520] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[11625827.055571] TCP: Hash tables configured (established 16384 bind 16384)
[11625827.055622] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[11625827.055647] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[11625827.055715] NET: Registered protocol family 1
[11625827.055972] RPC: Registered named UNIX socket transport module.
[11625827.055987] RPC: Registered udp transport module.
[11625827.055993] RPC: Registered tcp transport module.
[11625827.055999] RPC: Registered tcp NFSv4.1 backchannel transport module.
[11625827.056027] PCI: CLS 0 bytes, default 64
[11625827.056342] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer
[11625827.056358] hw unit of domain pp0-core 2^-16 Joules
[11625827.056365] hw unit of domain package 2^-16 Joules
[11625827.056371] hw unit of domain dram 2^-16 Joules
[11625827.056855] Scanning for low memory corruption every 60 seconds
[11625827.066295] AVX2 or AES-NI instructions are not detected.
[11625827.066312] AVX2 instructions are not detected.
[11625827.066622] futex hash table entries: 512 (order: 3, 32768 bytes)
[11625827.066662] audit: initializing netlink subsys (disabled)
[11625827.066687] audit: type=2000 audit(1461826509.734:1): initialized
[11625827.066860] Initialise system trusted keyring
[11625827.069878] VFS: Disk quotas dquot_6.6.0
[11625827.069921] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[11625827.070273] DLM installed
[11625827.070849] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[11625827.071235] FS-Cache: Netfs 'nfs' registered for caching
[11625827.071408] NFS: Registering the id_resolver key type
[11625827.071426] Key type id_resolver registered
[11625827.071432] Key type id_legacy registered
[11625827.071445] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[11625827.071462] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[11625827.071856] FS-Cache: Netfs 'cifs' registered for caching
[11625827.071961] Key type cifs.spnego registered
[11625827.071973] Key type cifs.idmap registered
[11625827.071985] romfs: ROMFS MTD (C) 2007 Red Hat, Inc.
[11625827.072142] fuse init (API version 7.23)
[11625827.072565] JFS: nTxBlock = 8192, nTxLock = 65536
[11625827.075313] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[11625827.076616] gfs2: GFS2 installed
[11625827.076845] FS-Cache: Netfs 'ceph' registered for caching
[11625827.076860] ceph: loaded (mds proto 32)
[11625827.086572] NET: Registered protocol family 38
[11625827.086602] Key type asymmetric registered
[11625827.086611] Asymmetric key parser 'x509' registered
[11625827.086620] Key type pkcs7_test registered
[11625827.086718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[11625827.086731] io scheduler noop registered
[11625827.086740] io scheduler deadline registered
[11625827.086757] io scheduler cfq registered (default)
[11625827.087202] xen:xen_evtchn: Event-channel device installed
[11625827.123221] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[11625827.128979] brd: module loaded
[11625827.131398] loop: module loaded
[11625827.132719] nbd: registered device at major 43
[11625827.141972] drbd: initialized. Version: 8.4.5 (api:1/proto:86-101)
[11625827.141987] drbd: built-in
[11625827.141993] drbd: registered as block device major 147
[11625827.142126] rbd: loaded
[11625827.142511] tun: Universal TUN/TAP device driver, 1.6
[11625827.142518] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[11625827.142637] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[11625827.142645] e1000: Copyright (c) 1999-2006 Intel Corporation.
[11625827.142667] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[11625827.142673] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[11625827.142700] PPP generic driver version 2.4.2
[11625827.142771] PPP BSD Compression module registered
[11625827.142781] PPP Deflate Compression module registered
[11625827.142807] PPP MPPE Compression module registered
[11625827.142818] NET: Registered protocol family 24
[11625827.142834] PPTP driver version 0.8.5
[11625827.143040] xen_netfront: Initialising Xen virtual ethernet driver
[11625827.388197] i8042: PNP: No PS/2 controller found. Probing ports directly.
[11625828.022648] blkfront: xvda: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[11625828.033827] blkfront: xvdb: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[11625828.057604] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x285ca91565f, max_idle_ns: 440795346597 ns
[11625828.412684] i8042: No controller found
[11625828.412877] md: linear personality registered for level -1
[11625828.412885] md: raid0 personality registered for level 0
[11625828.412891] md: raid1 personality registered for level 1
[11625828.412898] md: raid10 personality registered for level 10
[11625828.413018] md: raid6 personality registered for level 6
[11625828.413026] md: raid5 personality registered for level 5
[11625828.413032] md: raid4 personality registered for level 4
[11625828.413042] md: multipath personality registered for level -4
[11625828.413217] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[11625828.413337] device-mapper: raid: Loading target version 1.7.0
[11625828.413359] dmi-sysfs: dmi entry is absent.
[11625828.413406] GACT probability NOT on
[11625828.413418] Mirror/redirect action on
[11625828.413646] netem: version 1.3
[11625828.413684] u32 classifier
[11625828.413690]     Performance counters on
[11625828.413694]     Actions configured
[11625828.413744] Netfilter messages via NETLINK v0.30.
[11625828.413767] nfnl_acct: registering with nfnetlink.
[11625828.413792] nf_conntrack version 0.5.0 (8192 buckets, 65536 max)
[11625828.413914] ctnetlink v0.93: registering with nfnetlink.
[11625828.414014] nf_tables: (c) 2007-2009 Patrick McHardy <kaber@trash.net>
[11625828.414034] nf_tables_compat: (c) 2012 Pablo Neira Ayuso <pablo@netfilter.org>
[11625828.414301] xt_time: kernel timezone is -0000
[11625828.414315] ip_set: protocol 6
[11625828.414395] ipip: IPv4 over IPv4 tunneling driver
[11625828.414553] gre: GRE over IPv4 demultiplexor driver
[11625828.414563] ip_gre: GRE over IPv4 tunneling driver
[11625828.414836] IPv4 over IPsec tunneling driver
[11625828.415729] ip_tables: (C) 2000-2006 Netfilter Core Team
[11625828.415980] ipt_CLUSTERIP: ClusterIP Version 0.8 loaded successfully
[11625828.416011] Initializing XFRM netlink socket
[11625828.416236] NET: Registered protocol family 10
[11625828.416813] ip6_tables: (C) 2000-2006 Netfilter Core Team
[11625828.418236] sit: IPv6 over IPv4 tunneling driver
[11625828.418590] ip6_gre: GRE over IPv6 tunneling driver
[11625828.418767] NET: Registered protocol family 17
[11625828.418788] NET: Registered protocol family 15
[11625828.418809] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[11625828.419551] Bridge firewalling registered
[11625828.419569] Ebtables v2.0 registered
[11625828.419859] 8021q: 802.1Q VLAN Support v1.8
[11625828.419882] sctp: Hash tables configured (established 256 bind 256)
[11625828.419989] Key type dns_resolver registered
[11625828.420009] Key type ceph registered
[11625828.420087] libceph: loaded (mon/osd proto 15/24)
[11625828.420098] mce: Unable to init device /dev/mcelog (rc: -5)
[11625828.420266] AVX version of gcm_enc/dec engaged.
[11625828.420272] AES CTR mode by8 optimization enabled
[11625828.422325] registered taskstats version 1
[11625828.422340] Loading compiled-in X.509 certificates
[11625828.423214] Btrfs loaded
[11625828.424450] Key type encrypted registered
[11625828.424634] console [netcon0] enabled
[11625828.424641] netconsole: network logging started
[11625828.424657] hctosys: unable to open rtc device (rtc0)
[11625828.424846] md: Waiting for all devices to be available before autodetect
[11625828.424851] md: If you don't use raid, use raid=noautodetect
[11625828.425109] md: Autodetecting RAID arrays.
[11625828.425114] md: Scanned 0 and added 0 devices.
[11625828.425118] md: autorun ...
[11625828.425121] md: ... autorun DONE.
[11625828.427122] EXT4-fs (xvda): mounting ext3 file system using the ext4 subsystem
[11625828.429866] EXT4-fs (xvda): barriers disabled
[11625828.433699] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
[11625828.433722] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
[11625828.437792] devtmpfs: mounted
[11625828.438969] Freeing unused kernel memory: 1524K (ffffffff82138000 - ffffffff822b5000)
[11625828.438981] Write protecting the kernel read-only data: 16384k
[11625828.443568] Freeing unused kernel memory: 336K (ffff8800019ac000 - ffff880001a00000)
[11625828.444652] Freeing unused kernel memory: 2036K (ffff880001e03000 - ffff880002000000)
[11625828.444664] ------------[ cut here ]------------
[11625828.444673] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0xf2/0x5cd()
[11625828.444681] x86/mm: Found insecure W+X mapping at address ffff880000000000/0xffff880000000000
[11625828.444688] Modules linked in:
[11625828.444695] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.4.4-x86_64-linode64 #1
[11625828.444700]  0000000000000000 0000000000000009 ffffffff81596777 00000000000000e1
[11625828.444707]  0000000000000200 ffff88007cffbdd8 ffffffff810e0dbf 0000000000000201
[11625828.444714]  ffffffff8104f43e ffffffff819a439c 0000000000000000 ffff88007cffbec8
[11625828.444721] Call Trace:
[11625828.444729]  [<ffffffff81596777>] ? dump_stack+0x63/0x84
[11625828.444736]  [<ffffffff810e0dbf>] ? warn_slowpath_common+0x9d/0xb5
[11625828.444741]  [<ffffffff8104f43e>] ? note_page+0xf2/0x5cd
[11625828.444749]  [<ffffffff819a439c>] ? _raw_spin_unlock_irqrestore+0x16/0x17
[11625828.444754]  [<ffffffff810e0e82>] ? warn_slowpath_fmt+0x4a/0x4f
[11625828.444759]  [<ffffffff8104f43e>] ? note_page+0xf2/0x5cd
[11625828.444766]  [<ffffffff81006239>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[11625828.444771]  [<ffffffff8104fbc6>] ? ptdump_walk_pgd_level_core+0x2ad/0x3a4
[11625828.444777]  [<ffffffff8199f7b6>] ? rest_init+0x7a/0x7a
[11625828.444782]  [<ffffffff8199f7d0>] ? kernel_init+0x1a/0xdc
[11625828.444787]  [<ffffffff819a4b8f>] ? ret_from_fork+0x3f/0x70
[11625828.444791]  [<ffffffff8199f7b6>] ? rest_init+0x7a/0x7a
[11625828.444796] ---[ end trace f03e18d053b95b97 ]---
[11625828.455101] x86/mm: Checked W+X mappings: FAILED, 6628 W+X pages found.
[11625828.568607] random: systemd urandom read with 72 bits of entropy available
[11625828.573500] systemd[1]: systemd 229 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[11625828.573549] systemd[1]: Detected virtualization xen.
[11625828.573559] systemd[1]: Detected architecture x86-64.
[11625828.615109] systemd[1]: No hostname configured.
[11625828.615127] systemd[1]: Set hostname to <localhost>.
[11625829.004801] systemd[1]: Listening on udev Control Socket.
[11625829.005616] systemd[1]: Created slice System Slice.
[11625829.005906] systemd[1]: Listening on Journal Audit Socket.
[11625829.006288] systemd[1]: Created slice system-dhcpcd.slice.
[11625829.006484] systemd[1]: Listening on LVM2 metadata daemon socket.
[11625829.006640] systemd[1]: Reached target Encrypted Volumes.
[11625829.006824] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[11625829.006972] systemd[1]: Reached target Remote File Systems.
[11625829.007186] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[11625829.007591] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[11625829.007918] systemd[1]: Created slice User and Session Slice.
[11625829.008065] systemd[1]: Reached target Slices.
[11625829.008252] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[11625829.008469] systemd[1]: Listening on Journal Socket.
[11625829.021201] systemd[1]: Mounting Debug File System...
[11625829.023289] systemd[1]: Starting File System Check on Root Device...
[11625829.025254] systemd[1]: Starting Setup Virtual Console...
[11625829.027070] systemd[1]: Mounting POSIX Message Queue File System...
[11625829.027409] systemd[1]: Listening on Journal Socket (/dev/log).
[11625829.038623] systemd[1]: Starting Apply Kernel Variables...
[11625829.040669] systemd[1]: Mounting Configuration File System...
[11625829.042145] systemd[1]: Mounting FUSE Control File System...
[11625829.042409] systemd[1]: Created slice system-dropbox.slice.
[11625829.043717] systemd[1]: Starting Journal Service...
[11625829.043946] systemd[1]: Listening on udev Kernel Socket.
[11625829.044220] systemd[1]: Created slice system-serial\x2dgetty.slice.
[11625829.053871] systemd[1]: Created slice system-getty.slice.
[11625829.059369] systemd[1]: Listening on Process Core Dump Socket.
[11625829.059573] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[11625829.059640] systemd[1]: Reached target Paths.
[11625829.079182] systemd[1]: Mounted FUSE Control File System.
[11625829.079295] systemd[1]: Mounted POSIX Message Queue File System.
[11625829.079368] systemd[1]: Mounted Configuration File System.
[11625829.079441] systemd[1]: Mounted Debug File System.
[11625829.080188] systemd[1]: Started Apply Kernel Variables.
[11625829.195466] systemd[1]: Started File System Check on Root Device.
[11625829.200181] systemd[1]: Started Journal Service.
[11625829.233603] EXT4-fs (xvda): Cannot change data mode on remount
[11625829.285728] systemd-journald[1577]: Received request to flush runtime journal from PID 1
[11625829.537080] Adding 524284k swap on /dev/xvdb.  Priority:-1 extents:1 across:524284k SSFS
[11625831.010967] random: nonblocking pool is initialized
[11625836.132071] perl[2691]: segfault at 5 ip 00007f6b5b153c0e sp 00007ffe1cb818b0 error 4 in Socket6.so[7f6b5b150000+5000]
[11625836.132117] audit: type=1701 audit(1461826518.800:2): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=2691 comm="perl" exe="/usr/bin/perl" sig=11
[11625836.161373] audit: type=1325 audit(1461826518.830:3): table=filter family=2 entries=0
[11625836.161394] audit: type=1325 audit(1461826518.830:3): table=mangle family=2 entries=0
[11625836.161406] audit: type=1325 audit(1461826518.830:3): table=nat family=2 entries=0
[11625836.161415] audit: type=1325 audit(1461826518.830:3): table=raw family=2 entries=0
[11625836.161425] audit: type=1325 audit(1461826518.830:3): table=security family=2 entries=0
[11625836.161536] audit: type=1325 audit(1461826518.830:3): table=filter family=10 entries=0
[11625836.161548] audit: type=1325 audit(1461826518.830:3): table=mangle family=10 entries=0
[11625836.161557] audit: type=1325 audit(1461826518.830:3): table=raw family=10 entries=0
[11625844.063285] audit_printk_skb: 12 callbacks suppressed
[11625844.063297] audit: type=1006 audit(1461826526.730:4): pid=2801 uid=0 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=1 res=1
[11625844.088797] audit: type=1006 audit(1461826526.757:5): pid=2803 uid=0 old-auid=4294967295 auid=1000 old-ses=4294967295 ses=2 res=1

Offline

#10 2016-04-28 07:36:31

beatbreaker
Member
Registered: 2009-07-13
Posts: 46

Re: System hosed after upgrade

Ok so, for posterity this has been fixed. It was a combination of a flag I had in my fstab file, and the Kernel version I was running - If I rolled back to an older Kernel (eg: 4.0.1) it worked also. No idea where the change in fstab was documented.

Broken

# cat /etc/fstab

# <file system>	<dir>	<type>	<options>	<dump>	<pass>
# UUID=c9bde213-64b9-4594-86e5-0539f5c79590
/dev/xvda           	/         	ext3      	rw,relatime,errors=continue,barrier=0,data=writeback	0 1

# UUID=cc161599-9eb3-4be4-8503-ee5ad269a864
/dev/xvdb           	none      	swap      	defaults  	0 0

tmpfs	/tmp	tmpfs	nodev,nosuid	0	0

# touch test
touch: cannot touch 'test': Read-only file system

# uname -a
Linux li382-54.members.linode.com 4.4.4-x86_64-linode64 #1 SMP x86_64 GNU/Linux

Fixed - note the removal of "data=writeback" in fstab

# cat /etc/fstab

# <file system>	<dir>	<type>	<options>	<dump>	<pass>
# UUID=c9bde213-64b9-4594-86e5-0539f5c79590
/dev/xvda           	/         	ext3      	rw,relatime,errors=continue,barrier=0	0 1

# UUID=cc161599-9eb3-4be4-8503-ee5ad269a864
/dev/xvdb           	none      	swap      	defaults  	0 0

tmpfs	/tmp	tmpfs	nodev,nosuid	0	0

# touch test

# uname -a
Linux li382-54.members.linode.com 4.4.4-x86_64-linode64 #1 SMP x86_64 GNU/Linux

Offline

Board footer

Powered by FluxBB