heavy hdd activity under high memory load, system unresponsive

whitesnow · 2015-03-08 09:02:27

Hi there,

don't know where to post this.

On a system without SWAP and on zfs root on top of luks encryption when available memory becomes low, the desktop - X-Window Xfce becomes unresponsive.

Sometimes this takes many minutes and it comes back, or I still can close an application with the hardly moving mouse, to free up some memory.
The hdd-led does blink heavily.
Sometimes the X-Window system dies.

Is there something I can do about this, is this expected behaviour?

On windows XP I did run whitout swap for quite a long time - years, and when available memory got low, usually just a programm died.

That way I did like much more, than the system getting slow because of using the swap heavily.

So what do I have?

Laptop Lenovo X200
Core2Duo P8600 @ 2.40GHz
4GB RAM
SSD
no Swap
LUKS cryptroot
ZFS on top of /dev/mapper/cryptroot

# uname -ra
Linux machine1 3.18.6-1-ARCH #1 SMP PREEMPT Sat Feb 7 08:44:05 CET 2015 x86_64 GNU/Linux

# lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Model name:            Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz
Stepping:              10
CPU MHz:               800.000
CPU max MHz:           2401.0000
CPU min MHz:           800.0000
BogoMIPS:              4790.58
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              3072K
NUMA node0 CPU(s):     0,1

# lspci
00:00.0 Host bridge: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)
00:02.1 Display controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)
00:03.0 Communication controller: Intel Corporation Mobile 4 Series Chipset MEI Controller (rev 07)
00:03.3 Serial controller: Intel Corporation Mobile 4 Series Chipset AMT SOL Redirection (rev 07)
00:19.0 Ethernet controller: Intel Corporation 82567LM Gigabit Network Connection (rev 03)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 (rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 03)
00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 (rev 03)
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 93)
00:1f.0 ISA bridge: Intel Corporation ICH9M-E LPC Interface Controller (rev 03)
00:1f.2 IDE interface: Intel Corporation 82801IBM/IEM (ICH9M/ICH9M-E) 2 port SATA Controller [IDE mode] (rev 03)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 03)
03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
04:00.0 Memory controller: Intel Corporation Turbo Memory Controller (rev 11)

# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=1967492k,nr_inodes=491873,mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
rpool/ROOT/rootfs on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
rpool/BACKUP on /BACKUP type zfs (rw,noatime,xattr,noacl)
rpool/BACKUP/boot on /BACKUP/boot type zfs (rw,noatime,xattr,noacl)
rpool/BACKUP/mail on /BACKUP/mail type zfs (rw,noatime,xattr,noacl)
/dev/sda4 on /boot type ext4 (rw,relatime,data=ordered)
rpool/HOME on /home type zfs (rw,noatime,xattr,noacl)
rpool/ROOT/rootfs/OPT on /opt type zfs (rw,noatime,xattr,noacl)
rpool/HOME/root on /root type zfs (rw,noatime,xattr,noacl)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=394404k,mode=700,uid=1000,gid=1000)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        1.8G        1.1G        108M        873M        1.5G
Swap:            0B          0B          0B

# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h6m with 0 errors on Tue Mar  3 21:23:16 2015
config:

	NAME         STATE     READ WRITE CKSUM
	rpool        ONLINE       0     0     0
	  cryptroot  ONLINE       0     0     0

errors: No known data errors
# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool    61G  25.7G  35.3G         -    32%    42%  1.00x  ONLINE  -

# arcstat.py 
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:29:42     0     0      0     0    0     0    0     0    0   757M  758M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:30:35     0     0      0     0    0     0    0     0    0   851M  853M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:31:44     0     0      0     0    0     0    0     0    0   743M  744M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:33:01     0     0      0     0    0     0    0     0    0   177M  172M  
# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        2.5G        137M        344M        1.1G        536M
Swap:            0B          0B          0B

that just to show that the zfs arc does resize up and down
closing some applications

# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        2.5G        137M        344M        1.1G        536M
Swap:            0B          0B          0B
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:35:05     0     0      0     0    0     0    0     0    0   164M  164M

# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:36:06     0     0      0     0    0     0    0     0    0   164M  166M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:36:11     0     0      0     0    0     0    0     0    0   165M  166M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:36:14     0     0      0     0    0     0    0     0    0   167M  167M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:36:15     0     0      0     0    0     0    0     0    0   169M  169M  
# arcstat.py
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
09:37:45     0     0      0     0    0     0    0     0    0   173M  178M  
# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        1.5G        1.4G        116M        878M        1.8G
Swap:            0B          0B          0B

and the system log

Mar 08 07:10:50 machine1 systemd-timesyncd[1078]: Using NTP server 217.79.181.50:123 (0.de.pool.ntp.org).
Mar 08 07:10:52 machine1 systemd[1314]: Time has been changed
Mar 08 07:10:52 machine1 systemd[1]: Time has been changed
Mar 08 07:10:52 machine1 systemd-timesyncd[1078]: interval/delta/delay/jitter/drift 32s/+1.379s/0.030s/0.520s/+25ppm
Mar 08 07:11:24 machine1 systemd-timesyncd[1078]: interval/delta/delay/jitter/drift 64s/+0.006s/0.040s/0.520s/+25ppm
Mar 08 07:12:28 machine1 systemd-timesyncd[1078]: interval/delta/delay/jitter/drift 128s/-0.008s/0.025s/0.520s/-5ppm
Mar 08 07:14:36 machine1 systemd-timesyncd[1078]: interval/delta/delay/jitter/drift 256s/+0.009s/0.035s/0.520s/+11ppm
Mar 08 07:18:53 machine1 systemd-timesyncd[1078]: interval/delta/delay/jitter/drift 512s/-0.001s/0.025s/0.520s/+10ppm
Mar 08 07:23:57 machine1 acpid[1084]: client 1346[1000:1000] has disconnected
Mar 08 07:23:58 machine1 systemd-logind[1109]: Suspend key pressed.
Mar 08 07:23:58 machine1 systemd-logind[1109]: Suspending...
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  sleep requested (sleeping: no  enabled: yes)
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  sleeping...
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  (enp0s25): device state change: unavailable -> unmanaged (reason 'sleeping') [20 10 37]
Mar 08 07:23:58 machine1 logger[14283]: SleepButton pressed
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  (wls1): device state change: activated -> unmanaged (reason 'sleeping') [100 10 37]
Mar 08 07:23:58 machine1 kernel: e1000e: enp0s25 NIC Link is Down
Mar 08 07:23:58 machine1 kernel: IPv6: ADDRCONF(NETDEV_UP): enp0s25: link is not ready
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  (wls1): deactivating device (reason 'sleeping') [37]
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  (wls1): canceled DHCP transaction, DHCP client pid 13916
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  (wls1): DHCPv4 state changed bound -> done
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  Removing DNS information from /usr/bin/resolvconf
Mar 08 07:23:58 machine1 kernel: wls1: deauthenticating from a0:f3:c1:a9:0c:63 by local choice (Reason: 3=DEAUTH_LEAVING)
Mar 08 07:23:58 machine1 kernel: cfg80211: Calling CRDA to update world regulatory domain
Mar 08 07:23:58 machine1 kernel: cfg80211: World regulatory domain updated:
Mar 08 07:23:58 machine1 kernel: cfg80211:  DFS Master region: unset
Mar 08 07:23:58 machine1 kernel: cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211: Calling CRDA for country: DE
Mar 08 07:23:58 machine1 kernel: cfg80211: Regulatory domain changed to country: DE
Mar 08 07:23:58 machine1 kernel: cfg80211:  DFS Master region: ETSI
Mar 08 07:23:58 machine1 kernel: cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (2400000 KHz - 2483500 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5150000 KHz - 5250000 KHz @ 80000 KHz, 200000 KHz AUTO), (N/A, 2000 mBm), (N/A)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5250000 KHz - 5350000 KHz @ 80000 KHz, 200000 KHz AUTO), (N/A, 2000 mBm), (0 s)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (5470000 KHz - 5725000 KHz @ 160000 KHz), (N/A, 2698 mBm), (0 s)
Mar 08 07:23:58 machine1 kernel: cfg80211:   (57000000 KHz - 66000000 KHz @ 2160000 KHz), (N/A, 4000 mBm), (N/A)
Mar 08 07:23:58 machine1 NetworkManager[1082]: <info>  NetworkManager state is now ASLEEP
Mar 08 07:23:58 machine1 kernel: IPv6: ADDRCONF(NETDEV_UP): wls1: link is not ready
Mar 08 07:23:58 machine1 dbus[1092]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Mar 08 07:23:58 machine1 systemd[1]: Starting Network Manager Script Dispatcher Service...
Mar 08 07:23:58 machine1 dbus[1092]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Mar 08 07:23:58 machine1 systemd[1]: Started Network Manager Script Dispatcher Service.
Mar 08 07:23:58 machine1 systemd[1]: Starting Sleep.
Mar 08 07:23:58 machine1 systemd[1]: Reached target Sleep.
Mar 08 07:23:58 machine1 systemd[1]: Starting Suspend...
Mar 08 07:23:58 machine1 systemd-sleep[14323]: Suspending system...
Mar 08 07:23:58 machine1 /usr/bin/gpm[1110]: *** info [mice.c(1990)]:
Mar 08 07:23:58 machine1 /usr/bin/gpm[1110]: imps2: Auto-detected intellimouse PS/2
Mar 08 07:23:59 machine1 systemd-logind[1109]: Lid closed.
Mar 08 07:23:59 machine1 logger[14334]: LID closed
Mar 08 07:24:01 machine1 kernel: PM: Syncing filesystems ... done.
Mar 08 07:24:01 machine1 kernel: PM: Preparing system for mem sleep
Mar 08 07:28:21 machine1 kernel: Freezing user space processes ... (elapsed 0.002 seconds) done.
Mar 08 07:28:21 machine1 kernel: Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Mar 08 07:28:21 machine1 kernel: PM: Entering mem sleep
Mar 08 07:28:21 machine1 kernel: Suspending console(s) (use no_console_suspend to debug)
Mar 08 07:28:21 machine1 kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Mar 08 07:28:21 machine1 kernel: sd 0:0:0:0: [sda] Stopping disk
Mar 08 07:28:21 machine1 kernel: e1000e: EEE TX LPI TIMER: 00000000
Mar 08 07:28:21 machine1 kernel: PM: suspend of devices complete after 595.345 msecs
Mar 08 07:28:21 machine1 kernel: PM: late suspend of devices complete after 13.265 msecs
Mar 08 07:28:21 machine1 kernel: ehci-pci 0000:00:1d.7: System wakeup enabled by ACPI
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1d.0: System wakeup enabled by ACPI
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1a.2: System wakeup enabled by ACPI
Mar 08 07:28:21 machine1 kernel: ehci-pci 0000:00:1a.7: System wakeup enabled by ACPI
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1a.0: System wakeup enabled by ACPI
Mar 08 07:28:21 machine1 kernel: PM: noirq suspend of devices complete after 13.370 msecs
Mar 08 07:28:21 machine1 kernel: ACPI: Preparing to enter system sleep state S3
Mar 08 07:28:21 machine1 kernel: PM: Saving platform NVS memory
Mar 08 07:28:21 machine1 kernel: Disabling non-boot CPUs ...
Mar 08 07:28:21 machine1 kernel: kvm: disabling virtualization on CPU1
Mar 08 07:28:21 machine1 kernel: smpboot: CPU 1 is now offline
Mar 08 07:28:21 machine1 kernel: ACPI: Low-level resume complete
Mar 08 07:28:21 machine1 kernel: PM: Restoring platform NVS memory
Mar 08 07:28:21 machine1 kernel: Enabling non-boot CPUs ...
Mar 08 07:28:21 machine1 kernel: x86: Booting SMP configuration:
Mar 08 07:28:21 machine1 kernel: smpboot: Booting Node 0 Processor 1 APIC 0x1
Mar 08 07:28:21 machine1 kernel: Disabled fast string operations
Mar 08 07:28:21 machine1 kernel: kvm: enabling virtualization on CPU1
Mar 08 07:28:21 machine1 kernel: CPU1 is up
Mar 08 07:28:21 machine1 kernel: ACPI: Waking up from system sleep state S3
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1a.0: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1a.2: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: ehci-pci 0000:00:1a.7: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: uhci_hcd 0000:00:1d.0: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: ehci-pci 0000:00:1d.7: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: PM: noirq resume of devices complete after 13.666 msecs
Mar 08 07:28:21 machine1 kernel: PM: early resume of devices complete after 0.279 msecs
Mar 08 07:28:21 machine1 kernel: mei_me 0000:00:03.0: irq 28 for MSI/MSI-X
Mar 08 07:28:21 machine1 kernel: usb usb4: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: usb usb5: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: usb usb3: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: snd_hda_intel 0000:00:1b.0: irq 29 for MSI/MSI-X
Mar 08 07:28:21 machine1 kernel: usb usb6: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: usb usb7: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: usb usb8: root hub lost power or was reset
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.0:pcie04: Timeout on hotplug command 0x1038 (issued 2840063 msec ago)
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.1:pcie04: Timeout on hotplug command 0x1038 (issued 2840063 msec ago)
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.2:pcie04: Timeout on hotplug command 0x1038 (issued 2840063 msec ago)
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.3:pcie04: Timeout on hotplug command 0x1038 (issued 2840063 msec ago)
Mar 08 07:28:21 machine1 kernel: sd 0:0:0:0: [sda] Starting disk
Mar 08 07:28:21 machine1 kernel: e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.2:pcie04: Device 0000:04:00.0 already exists at 0000:04:00, cannot hot-add
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.2:pcie04: Cannot add device at 0000:04:00
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.1:pcie04: Device 0000:03:00.0 already exists at 0000:03:00, cannot hot-add
Mar 08 07:28:21 machine1 kernel: pciehp 0000:00:1c.1:pcie04: Cannot add device at 0000:03:00
Mar 08 07:28:21 machine1 kernel: rtc_cmos 00:02: System wakeup disabled by ACPI
Mar 08 07:28:21 machine1 kernel: usb 1-6: reset high-speed USB device number 3 using ehci-pci
Mar 08 07:28:21 machine1 kernel: usb 4-2: reset full-speed USB device number 2 using uhci_hcd
Mar 08 07:28:21 machine1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Mar 08 07:28:21 machine1 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 08 07:28:21 machine1 kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded
Mar 08 07:28:21 machine1 kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
Mar 08 07:28:21 machine1 kernel: ata1.00: ACPI cmd ef/03:45:00:00:00:a0 (SET FEATURES) filtered out
Mar 08 07:28:21 machine1 kernel: ata1.00: ACPI cmd ef/03:0c:00:00:00:a0 (SET FEATURES) filtered out
Mar 08 07:28:21 machine1 kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out
Mar 08 07:28:21 machine1 kernel: ata1.00: configured for UDMA/133
Mar 08 07:28:21 machine1 kernel: tpm_tis 00:05: TPM is disabled/deactivated (0x6)
Mar 08 07:28:21 machine1 kernel: PM: resume of devices complete after 1037.690 msecs
Mar 08 07:28:21 machine1 kernel: PM: Finishing wakeup.
Mar 08 07:28:21 machine1 kernel: Restarting tasks ... 
Mar 08 07:28:21 machine1 kernel: ACPI: \_SB_.PCI0.SATA.SCND.MSTR: docking
Mar 08 07:28:21 machine1 kernel: ACPI: \_SB_.PCI0.SATA.SCND.MSTR: Unable to dock!
Mar 08 07:28:21 machine1 kernel: done.
Mar 08 07:28:21 machine1 systemd[1]: Time has been changed
Mar 08 07:28:21 machine1 systemd-timesyncd[1078]: System time changed. Resyncing.
Mar 08 07:28:21 machine1 systemd-timesyncd[1078]: Using NTP server 46.4.54.78:123 (0.de.pool.ntp.org).
Mar 08 07:28:21 machine1 systemd-timesyncd[1078]: Using NTP server 129.143.2.23:123 (0.de.pool.ntp.org).
Mar 08 07:28:21 machine1 systemd-timesyncd[1078]: Using NTP server 129.250.35.250:123 (0.de.pool.ntp.org).
Mar 08 07:28:21 machine1 systemd-logind[1109]: Lid opened.
Mar 08 07:28:21 machine1 systemd[1314]: Time has been changed
Mar 08 07:28:21 machine1 systemd-sleep[14323]: System resumed.
Mar 08 07:28:21 machine1 kernel: video LNXVIDEO:00: Restoring backlight state
Mar 08 07:28:21 machine1 logger[14425]: LID opened
Mar 08 07:28:21 machine1 logger[14429]: ACPI group/action undefined: processor / LNXCPU:00
Mar 08 07:28:21 machine1 logger[14431]: ACPI group/action undefined: processor / LNXCPU:01
Mar 08 07:28:21 machine1 systemd[1]: Starting Bluetooth.
Mar 08 07:28:21 machine1 systemd[1]: Reached target Bluetooth.
Mar 08 07:28:21 machine1 systemd[1]: Started Suspend.
Mar 08 07:28:21 machine1 systemd[1]: Unit sleep.target is not needed anymore. Stopping.
Mar 08 07:28:21 machine1 systemd[1]: Stopping Sleep.
Mar 08 07:28:21 machine1 systemd[1]: Stopped target Sleep.
Mar 08 07:28:21 machine1 systemd[1]: Starting Suspend.
Mar 08 07:28:21 machine1 systemd[1]: Reached target Suspend.
Mar 08 07:28:21 machine1 systemd[1]: Unit suspend.target is bound to inactive service. Stopping, too.
Mar 08 07:28:21 machine1 systemd[1]: Stopping Suspend.
Mar 08 07:28:21 machine1 systemd[1]: Stopped target Suspend.
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  wake requested (sleeping: yes  enabled: yes)
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  waking up...
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (enp0s25): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Mar 08 07:28:21 machine1 systemd-logind[1109]: Operation finished.
Mar 08 07:28:21 machine1 kernel: e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
Mar 08 07:28:21 machine1 kernel: e1000e 0000:00:19.0: irq 32 for MSI/MSI-X
Mar 08 07:28:21 machine1 kernel: iwlwifi 0000:03:00.0: L1 Disabled - LTR Disabled
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (enp0s25): preparing device
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Mar 08 07:28:21 machine1 kernel: iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1): preparing device
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  NetworkManager state is now DISCONNECTED
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1) supports 5 scan SSIDs
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: starting -> ready
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1): device state change: unavailable -> disconnected (reason 'supplicant-available') [20 30 42]
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: ready -> disconnected
Mar 08 07:28:21 machine1 NetworkManager[1082]: <info>  (wls1) supports 5 scan SSIDs
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Auto-activating connection 'WLAN-1'.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: disconnected -> inactive
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: starting connection 'WLAN-1'
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) scheduled...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) started...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: disconnected -> prepare (reason 'none') [30 40 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  NetworkManager state is now CONNECTING
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) scheduled...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) complete.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) starting...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: prepare -> config (reason 'none') [40 50 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: (wifi) access point 'WLAN-1' has security, but secrets are required.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: config -> need-auth (reason 'none') [50 60 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) complete.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) scheduled...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) started...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: need-auth -> prepare (reason 'none') [60 40 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) scheduled...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 1 of 5 (Device Prepare) complete.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) starting...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: prepare -> config (reason 'none') [40 50 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: (wifi) connection 'WLAN-1' has security, and secrets exist.  No new secrets needed.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Config: added 'ssid' value 'WLAN-1'
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Config: added 'scan_ssid' value '1'
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Config: added 'key_mgmt' value 'WPA-PSK'
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Config: added 'psk' value '<omitted>'
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 2 of 5 (Device Configure) complete.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Config: set interface ap_scan to 1
Mar 08 07:28:24 machine1 kernel: wls1: authenticate with a0:f3:c1:a9:0c:63
Mar 08 07:28:24 machine1 kernel: wls1: send auth to a0:f3:c1:a9:0c:63 (try 1/3)
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: inactive -> authenticating
Mar 08 07:28:24 machine1 kernel: wls1: send auth to a0:f3:c1:a9:0c:63 (try 2/3)
Mar 08 07:28:24 machine1 kernel: wls1: authenticated
Mar 08 07:28:24 machine1 kernel: wls1: associate with a0:f3:c1:a9:0c:63 (try 1/3)
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: authenticating -> associating
Mar 08 07:28:24 machine1 kernel: wls1: RX AssocResp from a0:f3:c1:a9:0c:63 (capab=0x431 status=0 aid=1)
Mar 08 07:28:24 machine1 kernel: wls1: associated
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: associating -> associated
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: associated -> 4-way handshake
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): supplicant interface state: 4-way handshake -> completed
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: (wifi) Stage 2 of 5 (Device Configure) successful.  Connected to wireless network 'WLAN-1'.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 3 of 5 (IP Configure Start) scheduled.
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 3 of 5 (IP Configure Start) started...
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): device state change: config -> ip-config (reason 'none') [50 70 0]
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  Activation (wls1) Beginning DHCPv4 transaction (timeout in 45 seconds)
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  dhclient started with pid 14438
Mar 08 07:28:24 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 3 of 5 (IP Configure Start) complete.
Mar 08 07:28:24 machine1 dhclient[14438]: DHCPREQUEST on wls1 to 255.255.255.255 port 67
Mar 08 07:28:26 machine1 acpid[1084]: client connected from 1346[1000:1000]
Mar 08 07:28:26 machine1 acpid[1084]: 1 client rule loaded
Mar 08 07:28:28 machine1 dhclient[14438]: DHCPREQUEST on wls1 to 255.255.255.255 port 67
Mar 08 07:28:28 machine1 dhclient[14438]: DHCPACK from 192.168.1.254
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    address 192.168.1.16
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    plen 24 (255.255.255.0)
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    gateway 192.168.1.254
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    server identifier 192.168.1.254
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    lease time -1
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    nameserver '217.68.161.141'
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>    nameserver '217.68.161.171'
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>  (wls1): DHCPv4 state changed unknown -> bound
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 5 of 5 (IPv4 Configure Commit) scheduled...
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 5 of 5 (IPv4 Commit) started...
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>  (wls1): device state change: ip-config -> ip-check (reason 'none') [70 80 0]
Mar 08 07:28:28 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 5 of 5 (IPv4 Commit) complete.
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  (wls1): device state change: ip-check -> secondaries (reason 'none') [80 90 0]
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  (wls1): device state change: secondaries -> activated (reason 'none') [90 100 0]
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  NetworkManager state is now CONNECTED_LOCAL
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  NetworkManager state is now CONNECTED_GLOBAL
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  Policy set 'WLAN-1' (wls1) as default for IPv4 routing and DNS.
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  Writing DNS information to /usr/bin/resolvconf
Mar 08 07:28:29 machine1 dhclient[14438]: bound to 192.168.1.16 -- renewal in 2147483647 seconds.
Mar 08 07:28:29 machine1 NetworkManager[1082]: <info>  (wls1): Activation: successful, device activated.
Mar 08 07:28:29 machine1 dbus[1092]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Mar 08 07:28:29 machine1 systemd[1]: Starting Network Manager Script Dispatcher Service...
Mar 08 07:28:29 machine1 dbus[1092]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Mar 08 07:28:29 machine1 systemd[1]: Started Network Manager Script Dispatcher Service.
Mar 08 07:28:56 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 4 of 5 (IPv6 Configure Timeout) scheduled...
Mar 08 07:28:56 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 4 of 5 (IPv6 Configure Timeout) started...
Mar 08 07:28:56 machine1 NetworkManager[1082]: <info>  (wls1): Activation: Stage 4 of 5 (IPv6 Configure Timeout) complete.


Mar 08 08:22:27 machine1 systemd-journal[15220]: Permanent journal is using 41.2M (max allowed 4.0G, trying to leave 4.0G free of 34.3G available → current limit 4.0G).
Mar 08 08:22:27 machine1 kernel: Core dump to |/usr/lib/systemd/systemd-coredump 6862 0 0 6 1425796589 systemd-journal pipe failed
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=killed, status=6/ABRT
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Flush Journal to Persistent Storage...
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Time has been changed
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service stop-sigabrt timed out. Terminating.
Mar 08 08:22:27 machine1 kernel: INFO: task panel-20-dateti:1449 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-20-dateti D 0000000000000000     0  1449   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880092927cb8 0000000000000082 ffff880093e75080 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880092927fd8 0000000000013640 ffff88007ce0d080 ffff880093e75080
Mar 08 08:22:27 machine1 kernel:  ffff880092927c18 ffffffff81192628 ffff880092857f00 00ff880092932c38
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81192628>] ? page_add_file_rmap+0x48/0x90
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dd8a8>] lock_rename+0xd8/0x100
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e0bf3>] SyS_renameat2+0x283/0x5f0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811b3719>] ? kmem_cache_free+0x199/0x1d0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e0f9e>] SyS_rename+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task xfce4-xkb-plugi:1452 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: xfce4-xkb-plugi D 0000000000000000     0  1452   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880092967bd8 0000000000000086 ffff880095441e30 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880092967fd8 0000000000013640 ffff880125e59e30 ffff880095441e30
Mar 08 08:22:27 machine1 kernel:  ffff880092967b38 ffffffff811e6d0a ffff880092967b38 0000000000000000
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dbbd4>] lookup_slow+0x34/0xc0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dcd02>] path_lookupat+0x6f2/0x850
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dd356>] filename_lookup.isra.27+0x26/0x80
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dfe93>] user_path_at_empty+0x63/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dff11>] user_path_at+0x11/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d441a>] vfs_fstatat+0x6a/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d449b>] vfs_stat+0x1b/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d4a99>] SyS_newstat+0x29/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task panel-16-cpufre:1472 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-16-cpufre D 0000000000000000     0  1472   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff8800913f7c18 0000000000000082 ffff880129d71e30 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff8800913f7fd8 0000000000013640 ffff880093e76eb0 ffff880129d71e30
Mar 08 08:22:27 machine1 kernel:  ffff880129d71e30 ffff8800913358e0 0000000000000001 0000000000000292
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ddc0b>] do_last.isra.35+0x24b/0xea0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811de922>] path_openat+0xc2/0x6e0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81188b7e>] ? handle_mm_fault+0x8ce/0x1110
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81183124>] ? tlb_flush_mmu_free+0x34/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dffc9>] do_filp_open+0x49/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ed657>] ? __alloc_fd+0xa7/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce41e>] do_sys_open+0x14e/0x250
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce53e>] SyS_open+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task panel-17-cpugra:1478 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-17-cpugra D 0000000000000000     0  1478   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880090117c18 0000000000000086 ffff880129d70a10 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880090117fd8 0000000000013640 ffff8800b6cb5080 ffff880129d70a10
Mar 08 08:22:27 machine1 kernel:  ffff880129d70a10 ffff880091335c60 0000000000000001 0000000000000292
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ddc0b>] do_last.isra.35+0x24b/0xea0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811de922>] path_openat+0xc2/0x6e0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81183124>] ? tlb_flush_mmu_free+0x34/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dffc9>] do_filp_open+0x49/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ed657>] ? __alloc_fd+0xa7/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce41e>] do_sys_open+0x14e/0x250
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce53e>] SyS_open+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task xfce4-sensors-p:1485 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: xfce4-sensors-p D 0000000000000000     0  1485   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880129df7db8 0000000000000082 ffff880129d73250 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880129df7fd8 0000000000013640 ffff8800b6cb5080 ffff880129d73250
Mar 08 08:22:27 machine1 kernel:  ffff880129d73250 ffffffff81186252 0000000000000161 ffff8800aac0c7a0
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81186252>] ? do_read_fault.isra.58+0x2a2/0x340
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef934>] ? mntput+0x24/0x40
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dc70d>] ? path_lookupat+0xfd/0x850
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df9e6>] do_unlinkat+0x166/0x350
Mar 08 08:22:27 machine1 kernel:  [<ffffffff8118c349>] ? vma_rb_erase+0x129/0x260
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e0536>] SyS_unlink+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task panel-23-batter:1486 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-23-batter D 0000000000000000     0  1486   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff88008f9d7c18 0000000000000086 ffff880090095080 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff88008f9d7fd8 0000000000013640 ffff8800b6cb5080 ffff880090095080
Mar 08 08:22:27 machine1 kernel:  ffff880090095080 ffff880091337160 0000000000000001 0000000000000292
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff810b29e8>] ? __wake_up+0x48/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ddc0b>] do_last.isra.35+0x24b/0xea0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811662ce>] ? release_pages+0x20e/0x2b0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811de922>] path_openat+0xc2/0x6e0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81183124>] ? tlb_flush_mmu_free+0x34/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81183d0d>] ? tlb_finish_mmu+0x4d/0x50
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dffc9>] do_filp_open+0x49/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ed657>] ? __alloc_fd+0xa7/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce41e>] do_sys_open+0x14e/0x250
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce53e>] SyS_open+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task panel-26-netloa:1488 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-26-netloa D 0000000000000000     0  1488   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff88008facbc18 0000000000000082 ffff880090091420 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff88008facbfd8 0000000000013640 ffff8800b6cb5080 ffff880090091420
Mar 08 08:22:27 machine1 kernel:  ffff880090091420 ffff880091337be0 0000000000000001 0000000000000292
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff810b29e8>] ? __wake_up+0x48/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ddc0b>] do_last.isra.35+0x24b/0xea0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811de922>] path_openat+0xc2/0x6e0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81183124>] ? tlb_flush_mmu_free+0x34/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dffc9>] do_filp_open+0x49/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ed657>] ? __alloc_fd+0xa7/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce41e>] do_sys_open+0x14e/0x250
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ce53e>] SyS_open+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service stop-sigterm timed out. Killing.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd-journald[14616]: File /var/log/journal/77dbe77418134ba68037e356ea1a09d5/system.journal corrupted or uncleanly shut down, renaming and replacing.
Mar 08 08:22:27 machine1 kernel: INFO: task xfce4-sensors-p:1485 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: xfce4-sensors-p D 0000000000000000     0  1485   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880129df7cb8 0000000000000082 ffff880129d73250 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880129df7fd8 0000000000013640 ffff8800b6cb5080 ffff880129d73250
Mar 08 08:22:27 machine1 kernel:  0000000000000000 0000000000000000 0000000290099440 ffffffff811df460
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811da682>] ? path_init+0x222/0x450
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dd8a8>] lock_rename+0xd8/0x100
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e0bf3>] SyS_renameat2+0x283/0x5f0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811b3719>] ? kmem_cache_free+0x199/0x1d0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e0f9e>] SyS_rename+0x1e/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=dumped, status=6/ABRT
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=127/n/a
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=dumped, status=6/ABRT
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd-journald[14644]: File /var/log/journal/77dbe77418134ba68037e356ea1a09d5/system.journal corrupted or uncleanly shut down, renaming and replacing.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!
Mar 08 08:22:27 machine1 kernel: Core dump to |/usr/lib/systemd/systemd-coredump 14644 0 0 6 1425797277 systemd-journal pipe failed
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=killed, status=6/ABRT
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 kernel: INFO: task panel-16-cpufre:1472 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-16-cpufre D 0000000000000000     0  1472   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff8800913f7bd8 0000000000000082 ffff880129d71e30 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff8800913f7fd8 0000000000013640 ffffffff81818540 ffff880129d71e30
Mar 08 08:22:27 machine1 kernel:  ffff8800913f7b38 ffffffff811e6d0a ffff8800913f7b38 0000000000000000
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dbbd4>] lookup_slow+0x34/0xc0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dcd02>] path_lookupat+0x6f2/0x850
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dd356>] filename_lookup.isra.27+0x26/0x80
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dfe93>] user_path_at_empty+0x63/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dff11>] user_path_at+0x11/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d441a>] vfs_fstatat+0x6a/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d449b>] vfs_stat+0x1b/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d4a99>] SyS_newstat+0x29/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 kernel: INFO: task panel-17-cpugra:1478 blocked for more than 120 seconds.
Mar 08 08:22:27 machine1 kernel:       Tainted: P           O   3.18.6-1-ARCH #1
Mar 08 08:22:27 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 08 08:22:27 machine1 kernel: panel-17-cpugra D 0000000000000000     0  1478   1390 0x00000000
Mar 08 08:22:27 machine1 kernel:  ffff880090117bd8 0000000000000086 ffff880129d70a10 0000000000013640
Mar 08 08:22:27 machine1 kernel:  ffff880090117fd8 0000000000013640 ffff8800b6cb5080 ffff880129d70a10
Mar 08 08:22:27 machine1 kernel:  ffff880090117b38 ffffffff811e6d0a ffff880090117b38 0000000000000000
Mar 08 08:22:27 machine1 kernel: Call Trace:
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e6d0a>] ? dput+0x2a/0x1c0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551599>] schedule+0x29/0x70
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81551a16>] schedule_preempt_disabled+0x16/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff81553155>] __mutex_lock_slowpath+0xd5/0x140
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815531db>] mutex_lock+0x1b/0x30
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dbbd4>] lookup_slow+0x34/0xc0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dcd02>] path_lookupat+0x6f2/0x850
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811e8c9c>] ? __d_lookup+0xac/0x180
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811df460>] ? getname_flags+0x30/0x130
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dd356>] filename_lookup.isra.27+0x26/0x80
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dfe93>] user_path_at_empty+0x63/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811ef7dc>] ? mntput_no_expire+0x2c/0x160
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811dff11>] user_path_at+0x11/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d441a>] vfs_fstatat+0x6a/0xd0
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d449b>] vfs_stat+0x1b/0x20
Mar 08 08:22:27 machine1 kernel:  [<ffffffff811d4a99>] SyS_newstat+0x29/0x60
Mar 08 08:22:27 machine1 kernel:  [<ffffffff815556e9>] system_call_fastpath+0x12/0x17
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service: main process exited, code=exited, status=203/EXEC
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-journald.service: Invalid argument
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed to run 'start' task: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
Mar 08 08:22:27 machine1 systemd[1]: Stopping Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: start request repeated too quickly for systemd-journald.service
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Mar 08 08:22:27 machine1 systemd[1]: Job systemd-journal-flush.service/start failed with result 'dependency'.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.service entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd[1]: start request repeated too quickly for systemd-journald.service
Mar 08 08:22:27 machine1 systemd[1]: Failed to start Journal Service.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald-dev-log.socket entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: Unit systemd-journald.socket entered failed state.
Mar 08 08:22:27 machine1 systemd[1]: systemd-journald.service failed.
Mar 08 08:22:27 machine1 systemd-udevd[846]: fork of child failed: Cannot allocate memory
Mar 08 08:22:27 machine1 systemd-udevd[846]: fork of child failed: Cannot allocate memory
Mar 08 08:22:27 machine1 kernel: Core dump to |/usr/lib/systemd/systemd-coredump 1816 1000 1000 7 1425797920 pavucontrol pipe failed
Mar 08 08:22:27 machine1 systemd-coredump[14563]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.0a1ebe766de041859b8e236ce6a70fa5.14549.1425796675000000.
Mar 08 08:22:27 machine1 systemd[1]: Starting Session c2 of user user1.
Mar 08 08:22:27 machine1 systemd[1]: Started Session c2 of user user1.
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Audit Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Socket.
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Socket.
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Socket (/dev/log).
Mar 08 08:22:27 machine1 systemd[1]: Listening on Journal Socket (/dev/log).
Mar 08 08:22:27 machine1 systemd[1]: Starting Journal Service...
Mar 08 08:22:27 machine1 systemd-journal[15220]: Journal started
Mar 08 08:22:27 machine1 systemd[1]: Starting Flush Journal to Persistent Storage...
Mar 08 08:22:27 machine1 systemd[1]: Started Journal Service.
Mar 08 08:22:27 machine1 sudo[15217]: pam_unix(sudo:session): session closed for user root
Mar 08 08:22:27 machine1 systemd[1]: Started Flush Journal to Persistent Storage.
Mar 08 08:22:31 machine1 acpid[1084]: client 14756[1000:1000] has disconnected
Mar 08 08:22:31 machine1 /usr/bin/gpm[1110]: *** info [mice.c(1990)]:
Mar 08 08:22:31 machine1 /usr/bin/gpm[1110]: imps2: Auto-detected intellimouse PS/2
Mar 08 08:22:41 machine1 acpid[1084]: client connected from 14756[1000:1000]
Mar 08 08:22:41 machine1 acpid[1084]: 1 client rule loaded
Mar 08 08:24:32 machine1 sudo[15254]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/coredumpctl
Mar 08 08:24:32 machine1 sudo[15254]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:24:51 machine1 sudo[15254]: pam_unix(sudo:session): session closed for user root
Mar 08 08:25:13 machine1 sudo[15257]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/journalctl
Mar 08 08:25:13 machine1 sudo[15257]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:27:01 machine1 sudo[15257]: pam_unix(sudo:session): session closed for user root
Mar 08 08:27:22 machine1 sudo[15271]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/journalctl
Mar 08 08:27:22 machine1 sudo[15271]: pam_unix(sudo:session): session opened for user root by (uid=0)

Mar 08 08:27:33 machine1 sudo[15271]: pam_unix(sudo:session): session closed for user root
Mar 08 08:27:34 machine1 acpid[1084]: client 14756[1000:1000] has disconnected
Mar 08 08:27:35 machine1 /usr/bin/gpm[1110]: *** info [mice.c(1990)]:
Mar 08 08:27:35 machine1 /usr/bin/gpm[1110]: imps2: Auto-detected intellimouse PS/2
Mar 08 08:27:42 machine1 acpid[1084]: client connected from 14756[1000:1000]
Mar 08 08:27:42 machine1 acpid[1084]: 1 client rule loaded
Mar 08 08:41:28 machine1 sudo[15387]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/zpool status rpool
Mar 08 08:41:28 machine1 sudo[15387]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:41:28 machine1 sudo[15387]: pam_unix(sudo:session): session closed for user root
Mar 08 08:52:41 machine1 sudo[15434]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/grub-mkconfig -o /boot/grub/grub--tt.cfg
Mar 08 08:52:41 machine1 sudo[15434]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:52:41 machine1 sudo[15434]: pam_unix(sudo:session): session closed for user root
Mar 08 08:53:11 machine1 sudo[15444]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/ln -s /dev/mapper/cryptroot /dev/cryptroot
Mar 08 08:53:11 machine1 sudo[15444]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:53:11 machine1 sudo[15444]: pam_unix(sudo:session): session closed for user root
Mar 08 08:53:14 machine1 sudo[15446]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/grub-mkconfig -o /boot/grub/grub--tt.cfg
Mar 08 08:53:14 machine1 sudo[15446]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:53:16 machine1 kernel: SGI XFS with ACLs, security attributes, realtime, no debug enabled
Mar 08 08:53:16 machine1 kernel: JFS: nTxBlock = 8192, nTxLock = 65536
Mar 08 08:53:16 machine1 kernel: ntfs: driver 2.1.31 [Flags: R/W MODULE].
Mar 08 08:53:16 machine1 kernel: raid6: sse2x1    3894 MB/s
Mar 08 08:53:16 machine1 kernel: raid6: sse2x2    2519 MB/s
Mar 08 08:53:16 machine1 kernel: raid6: sse2x4    4662 MB/s
Mar 08 08:53:16 machine1 kernel: raid6: using algorithm sse2x4 (4662 MB/s)
Mar 08 08:53:16 machine1 kernel: raid6: using ssse3x2 recovery algorithm
Mar 08 08:53:16 machine1 kernel: xor: measuring software checksum speed
Mar 08 08:53:16 machine1 kernel:    prefetch64-sse:  6709.200 MB/sec
Mar 08 08:53:16 machine1 kernel:    generic_sse:  5888.400 MB/sec
Mar 08 08:53:16 machine1 kernel: xor: using function: prefetch64-sse (6709.200 MB/sec)
Mar 08 08:53:16 machine1 kernel: Btrfs loaded
Mar 08 08:53:16 machine1 systemd[1]: Starting LVM2 metadata daemon...
Mar 08 08:53:16 machine1 systemd[1]: Started LVM2 metadata daemon.
Mar 08 08:53:16 machine1 os-prober[16147]: debug: running /usr/lib/os-probes/50mounted-tests on /dev/sda1
Mar 08 08:53:16 machine1 50mounted-tests[16170]: debug: mounted using GRUB fat filesystem driver
Mar 08 08:53:16 machine1 50mounted-tests[16171]: debug: running subtest /usr/lib/os-probes/mounted/05efi
Mar 08 08:53:16 machine1 05efi[16174]: debug: Not on UEFI platform
Mar 08 08:53:16 machine1 50mounted-tests[16175]: debug: running subtest /usr/lib/os-probes/mounted/10freedos
Mar 08 08:53:16 machine1 10freedos[16178]: debug: /dev/sda1 is a FAT partition (mounted by GRUB)
Mar 08 08:53:16 machine1 50mounted-tests[16181]: debug: running subtest /usr/lib/os-probes/mounted/10qnx
Mar 08 08:53:16 machine1 10qnx[16184]: debug: /dev/sda1 is not a QNX4 partition: exiting
Mar 08 08:53:16 machine1 50mounted-tests[16185]: debug: running subtest /usr/lib/os-probes/mounted/20macosx
Mar 08 08:53:16 machine1 macosx-prober[16187]: debug: /dev/sda1 is not an HFS+ partition: exiting
Mar 08 08:53:16 machine1 50mounted-tests[16188]: debug: running subtest /usr/lib/os-probes/mounted/20microsoft
Mar 08 08:53:16 machine1 20microsoft[16191]: debug: /dev/sda1 is a FAT partition (mounted by GRUB)
Mar 08 08:53:16 machine1 50mounted-tests[16200]: debug: running subtest /usr/lib/os-probes/mounted/30utility
Mar 08 08:53:16 machine1 30utility[16203]: debug: /dev/sda1 is a FAT partition (mounted by GRUB)
Mar 08 08:53:16 machine1 50mounted-tests[16208]: debug: running subtest /usr/lib/os-probes/mounted/40lsb
Mar 08 08:53:17 machine1 50mounted-tests[16210]: debug: running subtest /usr/lib/os-probes/mounted/70hurd
Mar 08 08:53:17 machine1 50mounted-tests[16212]: debug: running subtest /usr/lib/os-probes/mounted/80minix
Mar 08 08:53:17 machine1 50mounted-tests[16214]: debug: running subtest /usr/lib/os-probes/mounted/83haiku
Mar 08 08:53:17 machine1 83haiku[16217]: debug: /dev/sda1 is not a BeFS partition: exiting
Mar 08 08:53:17 machine1 50mounted-tests[16218]: debug: running subtest /usr/lib/os-probes/mounted/90linux-distro
Mar 08 08:53:17 machine1 50mounted-tests[16223]: debug: running subtest /usr/lib/os-probes/mounted/90solaris
Mar 08 08:53:17 machine1 50mounted-tests[16225]: debug: running subtest /usr/lib/os-probes/mounted/efi
Mar 08 08:53:17 machine1 os-prober[16233]: debug: running /usr/lib/os-probes/50mounted-tests on /dev/sda2
Mar 08 08:53:17 machine1 50mounted-tests[16239]: debug: /dev/sda2 is a LUKS partition; skipping
Mar 08 08:53:17 machine1 os-prober[16240]: debug: os detected by /usr/lib/os-probes/50mounted-tests
Mar 08 08:53:17 machine1 os-prober[16246]: debug: running /usr/lib/os-probes/50mounted-tests on /dev/sda3
Mar 08 08:53:17 machine1 kernel: EXT4-fs (sda3): VFS: Can't find ext4 filesystem
Mar 08 08:53:17 machine1 kernel: EXT4-fs (sda3): VFS: Can't find ext4 filesystem
Mar 08 08:53:17 machine1 kernel: EXT4-fs (sda3): VFS: Can't find ext4 filesystem
Mar 08 08:53:17 machine1 kernel: XFS (sda3): Invalid superblock magic number
Mar 08 08:53:17 machine1 kernel: FAT-fs (sda3): invalid media value (0xe2)
Mar 08 08:53:17 machine1 kernel: FAT-fs (sda3): Can't find a valid FAT filesystem
Mar 08 08:53:17 machine1 kernel: FAT-fs (sda3): invalid media value (0xe2)
Mar 08 08:53:17 machine1 kernel: FAT-fs (sda3): Can't find a valid FAT filesystem
Mar 08 08:53:17 machine1 kernel: VFS: Can't find a Minix filesystem V1 | V2 | V3 on device sda3.
Mar 08 08:53:17 machine1 kernel: hfsplus: unable to find HFS+ superblock
Mar 08 08:53:17 machine1 kernel: ufs: You didn't specify the type of your ufs filesystem
                                     
                                     mount -t ufs -o ufstype=sun|sunx86|44bsd|ufs2|5xbsd|old|hp|nextstep|nextstep-cd|openstep ...
                                     
                                     >>>WARNING<<< Wrong ufstype may corrupt your filesystem, default is ufstype=old
Mar 08 08:53:17 machine1 kernel: ufs: ufs_fill_super(): bad magic number
Mar 08 08:53:17 machine1 kernel: hfs: can't find a HFS filesystem on dev sda3
Mar 08 08:53:17 machine1 os-prober[16299]: debug: running /usr/lib/os-probes/mounted/05efi on mounted /dev/sda4
Mar 08 08:53:17 machine1 05efi[16302]: debug: Not on UEFI platform
Mar 08 08:53:17 machine1 os-prober[16303]: debug: running /usr/lib/os-probes/mounted/10freedos on mounted /dev/sda4
Mar 08 08:53:17 machine1 10freedos[16306]: debug: /dev/sda4 is not a FAT partition: exiting
Mar 08 08:53:17 machine1 os-prober[16307]: debug: running /usr/lib/os-probes/mounted/10qnx on mounted /dev/sda4
Mar 08 08:53:17 machine1 10qnx[16310]: debug: /dev/sda4 is not a QNX4 partition: exiting
Mar 08 08:53:17 machine1 os-prober[16311]: debug: running /usr/lib/os-probes/mounted/20macosx on mounted /dev/sda4
Mar 08 08:53:17 machine1 macosx-prober[16313]: debug: /dev/sda4 is not an HFS+ partition: exiting
Mar 08 08:53:17 machine1 os-prober[16314]: debug: running /usr/lib/os-probes/mounted/20microsoft on mounted /dev/sda4
Mar 08 08:53:17 machine1 20microsoft[16317]: debug: /dev/sda4 is not a MS partition: exiting
Mar 08 08:53:17 machine1 os-prober[16318]: debug: running /usr/lib/os-probes/mounted/30utility on mounted /dev/sda4
Mar 08 08:53:17 machine1 30utility[16321]: debug: /dev/sda4 is not a FAT partition: exiting
Mar 08 08:53:17 machine1 os-prober[16322]: debug: running /usr/lib/os-probes/mounted/40lsb on mounted /dev/sda4
Mar 08 08:53:17 machine1 os-prober[16324]: debug: running /usr/lib/os-probes/mounted/70hurd on mounted /dev/sda4
Mar 08 08:53:17 machine1 os-prober[16326]: debug: running /usr/lib/os-probes/mounted/80minix on mounted /dev/sda4
Mar 08 08:53:17 machine1 os-prober[16328]: debug: running /usr/lib/os-probes/mounted/83haiku on mounted /dev/sda4
Mar 08 08:53:17 machine1 83haiku[16331]: debug: /dev/sda4 is not a BeFS partition: exiting
Mar 08 08:53:17 machine1 os-prober[16332]: debug: running /usr/lib/os-probes/mounted/90linux-distro on mounted /dev/sda4
Mar 08 08:53:17 machine1 os-prober[16337]: debug: running /usr/lib/os-probes/mounted/90solaris on mounted /dev/sda4
Mar 08 08:53:17 machine1 sudo[15446]: pam_unix(sudo:session): session closed for user root
Mar 08 08:53:57 machine1 sudo[16350]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/cat /boot/grub/grub--tt.cfg
Mar 08 08:53:57 machine1 sudo[16350]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 08:53:57 machine1 sudo[16350]: pam_unix(sudo:session): session closed for user root
Mar 08 09:14:27 machine1 sudo[16500]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/zpool status
Mar 08 09:14:27 machine1 sudo[16500]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 09:14:27 machine1 sudo[16500]: pam_unix(sudo:session): session closed for user root
Mar 08 09:19:47 machine1 acpid[1084]: client 14756[1000:1000] has disconnected
Mar 08 09:19:48 machine1 /usr/bin/gpm[1110]: *** info [mice.c(1990)]:
Mar 08 09:19:48 machine1 /usr/bin/gpm[1110]: imps2: Auto-detected intellimouse PS/2
Mar 08 09:19:54 machine1 systemd[1]: Starting Getty on tty5...
Mar 08 09:19:54 machine1 systemd[1]: Started Getty on tty5.
Mar 08 09:19:55 machine1 systemd[1]: Starting Getty on tty6...
Mar 08 09:19:55 machine1 systemd[1]: Started Getty on tty6.
Mar 08 09:19:55 machine1 acpid[1084]: client connected from 14756[1000:1000]
Mar 08 09:19:55 machine1 acpid[1084]: 1 client rule loaded
Mar 08 09:44:45 machine1 sudo[16853]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/zpool list
Mar 08 09:44:45 machine1 sudo[16853]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 08 09:44:45 machine1 sudo[16853]: pam_unix(sudo:session): session closed for user root
Mar 08 09:52:32 machine1 sudo[16878]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/journalctl
Mar 08 09:52:32 machine1 sudo[16878]: pam_unix(sudo:session): session opened for user root by (uid=0)

whitesnow · 2015-03-15 14:30:51

Well no answer.

I did not find any hint so far either.

Now it is again in that state since hours.

I did do an update with

pacman -Syyu

.
That fininshed.

Later the same with

pacaur -Syyu

.

And now i don't see any output since 2,5 hours.
Beside the blinking hdd-LED, which that system luckily still has.

If I rembember right, the last time I did see on the screen was
12:40
now it is already 15:29.

Thanks in advance.

I have a

script

command running, or two, to save what I was doing on the console.
But to my experience that does not save if it is not properly exited by typing

exit

.
I already got some empty script logs.

Well, as far as I remember the zfs arc cache already went to a really small size something like 4 MB and 30 MB.
There is a tmpfs on that system.
And buffers/cache did show something in several 100 MB - GB range.

Last edited by whitesnow (2015-03-15 14:35:59)

lucke · 2015-03-15 15:02:06

Look at https://bbs.archlinux.org/viewtopic.php?id=190669, https://bbs.archlinux.org/viewtopic.php?id=190247 and https://bbs.archlinux.org/viewtopic.php?id=191346.

See if Alt+Sysrq+f with Magic SysRq enabled helps.

Having swap would really help you.

graysky · 2015-03-15 15:14:48

Try the bfs in the linux-ck package.

falconindy · 2015-03-15 16:12:16

graysky wrote:

Try the bfs in the linux-ck package.

There's no mention of anything CPU bound here. Why should he expect that changing the CPU scheduler has any effect?

Seems much simpler than this -- ZFS is memory hungry. Don't use ZFS with only 4GB of RAM. Your kernel backtraces point to slowness in the filesystem (dput).

graysky · 2015-03-15 16:48:48

@falconindy - Have you tried bfs? The responsiveness particularly on slower CPUs is remarkable.

falconindy · 2015-03-15 17:10:28

Have you read the original post? Why do you think changing the CPU scheduler will help when the problem appears to be I/O and memory bound? Do you have some alternate theory you'd like to share?

Last edited by falconindy (2015-03-15 17:12:15)

graysky · 2015-03-15 17:52:46

falconindy wrote:

Have you read the original post? Why do you think changing the CPU scheduler will help when the problem appears to be I/O and memory bound? Do you have some alternate theory you'd like to share?

I agree this issue in the first post seems to be a function of heavy I/O but I repeat: I have found the alternative queuing (due to BFS) to be superior under these circumstances particularly on older hardware. It's is a very simple experiment for the OP to try.

Last edited by graysky (2015-03-15 17:53:21)

whitesnow · 2015-03-16 18:25:27

Thanks a lot for the many responses.

So the first and most easy for me would be to just add swap. Or some more ram.
But still I'm new to arch linux. And better I learn now already, than later, when there might be really bad situations.

The linux-ck package I tried. But there I'm somehow lost how do I integrate the zfs?

I have the following packages regarding zfs:

zfs-git 0.6.3_r170_gd958324f_3.18.6_1-1
zfs-utils-git 0.6.3_r170_gd958324f_3.18.6_1-1

these should be from the following source:

[demz-repo-core]
SigLevel = Required DatabaseOptional TrustedOnly
Server = http://demizerone.com/$repo/$arch

And these updates are running well. That got updated already several times.

I did install just the linux-ck package.
with this repository:

[repo-ck]
Server = http://repo-ck.com/$arch

But there, I don't get the zfs support into the initramfs?

OK, the log looks like this:

>>> Updating module dependencies. Please wait ...
>>> Generating initial ramdisk, using mkinitcpio.  Please wait...
==> Building image from preset: /etc/mkinitcpio.d/linux-ck.preset: 'default'
  -> -k /boot/vmlinuz-linux-ck -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-ck.img
==> Starting build: 3.19.1-1-ck
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [encrypt]
  -> Running build hook: [zfs]
==> ERROR: module not found: `zavl'
==> ERROR: module not found: `znvpair'
==> ERROR: module not found: `zunicode'
==> ERROR: module not found: `zcommon'
==> ERROR: module not found: `zfs'
==> ERROR: module not found: `zpios'
==> ERROR: module not found: `spl'
==> ERROR: module not found: `splat'
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> ERROR: module not found: `zfs'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-ck.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux-ck.preset: 'fallback'
  -> -k /boot/vmlinuz-linux-ck -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-ck-fallback.img -S autodetect
==> Starting build: 3.19.1-1-ck
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
  -> Running build hook: [keyboard]
  -> Running build hook: [encrypt]
  -> Running build hook: [zfs]
==> ERROR: module not found: `zavl'
==> ERROR: module not found: `znvpair'
==> ERROR: module not found: `zunicode'
==> ERROR: module not found: `zcommon'
==> ERROR: module not found: `zfs'
==> ERROR: module not found: `zpios'
==> ERROR: module not found: `spl'
==> ERROR: module not found: `splat'
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> ERROR: module not found: `zfs'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-ck-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
( 95/162) upgrading linux-ck-core2-headers         [----------------------]   0%
( 95/162) upgrading linux-ck-core2-headers         [----------------------]   0%
( 95/162) upgrading linux-ck-core2-headers         [----------------------]   0%
( 95/162) upgrading linux-ck-core2-headers         [###-------------------]  14%
( 95/162) upgrading linux-ck-core2-headers         [###-------------------]  15%
( 95/162) upgrading linux-ck-core2-headers         [###-------------------]  16%
( 95/162) upgrading linux-ck-core2-headers         [######----------------]  29%
( 95/162) upgrading linux-ck-core2-headers         [#########-------------]  42%
( 95/162) upgrading linux-ck-core2-headers         [###########-----------]  53%
( 95/162) upgrading linux-ck-core2-headers         [##############--------]  66%
( 95/162) upgrading linux-ck-core2-headers         [################------]  75%
( 95/162) upgrading linux-ck-core2-headers         [###################---]  88%
( 95/162) upgrading linux-ck-core2-headers         [####################--]  95%
( 95/162) upgrading linux-ck-core2-headers         [######################] 100%

Well, back to the described behaviour:

It finally it did kill the x-Window-System, when it ran out of memory, I did run the update from there. It was compiling palemoon.
If I read it right from the log, that was 18:17, well some hours later, that does not really matter. The system was still alive but the whole X-Window System with the running compilation session therein was gone.

So there is no question that it was really low on memory.

After that, I did the update just from the console, no X-Window running at all.

Here is part of the log of free RAM. I have another log of the arcstat, the ram zfs was using. I'm not sure, if I get them time-syncronized.
It's all only in my spare time, there are other things to do as well.

	total	used	free	shared	buff/cache	available
Mem	3.800	1.200	887M	1.200	1.700	911
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.400	726M	1.200	1.700	751
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.500	585M	1.200	1.700	610
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.600	471M	1.200	1.700	496
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.700	405M	1.200	1.700	430
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.700	333M	1.200	1.700	357
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.500	627M	1.200	1.700	652
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.500	537M	1.200	1.700	562
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.600	487M	1.200	1.700	512
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.000	1.100	1.200	1.700	1100
Swap	0B	0B	0B			
						
	total	used	free	shared	buff/cache	available
Mem	3.800	1.100	1.000	1.200	1.700	1000
Swap	0B	0B	0B

OK, that for now.

Thanks a lot.

_____

edit:
Maybe I should add, this was just a different situation of running low on memory.
There was a webbrowser open, well with quite many tabs. using a lot of memory.
And that update with pacaur, that did invoke a compilation of the palemoon webbrowser - based on firefox.
But for this, I would have been happy, with this open webbrowser getting killed, the system being resposive. Or other tasks getting killed, the system reacting on user input.

I think for being on use as desktop system, it should still respond to keypresses or mousemovements.

Don't really know if there are other considerations for server use. But not beeing able to free some ram by killing some processes, don't really know if that is better on server or desktop.

Last edited by whitesnow (2015-03-16 18:47:55)

graysky · 2015-03-16 18:59:11

Demz doesn't supply zfs packages for built against linux-ck. You will need to build them yourself if you wanna use that kernel with zfs.

whitesnow · 2015-03-16 20:21:18

Ah, yes, thank you.

On this I still did not yet really understand how this works with the packages and the kernels.

I did build a linux kernel years before and a freebsd kernel before 1 or 2 years. But there it was mainly downloading the sources and creating/modifying a config file and issuing the commands to build.

Can you point me to a place, where it is explained how to build these zfs packages or other packages against the ck-kernel?

Thanks in advance.

teateawhy · 2015-03-17 00:01:10

Zfs from AUR should work.

Dark.Light · 2015-03-17 06:20:21

whitesnow wrote:

when available memory becomes low, the desktop - X-Window Xfce becomes unresponsive.

I think the root of the problem lies here. I've had similar experiences before.

Usually, when memory becomes low, swap comes to the rescue. But since in this case there is no swap, kernel will try its best to free up memory. This translates to killing processes (as you said, "sometimes X-windows systems dies.") I'm not sure why there is heavy hdd activity, but it might be something to do with encryption.

The best course of action would be to create a swapfile or avoid situations where there is high memory load...

Arch Linux

#1 2015-03-08 09:02:27

heavy hdd activity under high memory load, system unresponsive

#2 2015-03-15 14:30:51

Re: heavy hdd activity under high memory load, system unresponsive

#3 2015-03-15 15:02:06

Re: heavy hdd activity under high memory load, system unresponsive

#4 2015-03-15 15:14:48

Re: heavy hdd activity under high memory load, system unresponsive

#5 2015-03-15 16:12:16

Re: heavy hdd activity under high memory load, system unresponsive

#6 2015-03-15 16:48:48

Re: heavy hdd activity under high memory load, system unresponsive

#7 2015-03-15 17:10:28

Re: heavy hdd activity under high memory load, system unresponsive

#8 2015-03-15 17:52:46

Re: heavy hdd activity under high memory load, system unresponsive

#9 2015-03-16 18:25:27

Re: heavy hdd activity under high memory load, system unresponsive

#10 2015-03-16 18:59:11

Re: heavy hdd activity under high memory load, system unresponsive

#11 2015-03-16 20:21:18

Re: heavy hdd activity under high memory load, system unresponsive

#12 2015-03-17 00:01:10

Re: heavy hdd activity under high memory load, system unresponsive

#13 2015-03-17 06:20:21

Re: heavy hdd activity under high memory load, system unresponsive

Board footer