RAID 1 with failing hd

whoop · 2015-01-30 00:26:54

I have three raid 1 array's mirrored over two hard-disks (sata1 and sata2). Sata2 seems to be failing on a hardware level. It's not auto detecting in the bios and I get a srst failed erno=-16 at bootup.
The weird thing is that sata2 starts working after about ten minutes of power (so maybe it's a failing condensator somewhere)

The thing is my full raid 1 array arch os will not boot when sata2 is not detected by bios. It just fails.
Now I am expecting that today or tomorrow sata2 will be gone permanently so therefore I manually degraded the array's to just the partitions on sata1 untill I replace the drive or whatever else needs to be replaced, but I am wondering:

(How) Can I boot the system if I don't manually degrade the array's as I did and sata2 fails completely.
In other words(if I am not mistaken): have a full raid 1 system, shut it down, pull out one hard disk, make it boot somehow

I would expect this is possible, but I can't figure out how.
I have backups so I am willing to experiment...

alphaniner · 2015-01-30 01:09:15

It's kind of surprising that a degraded RAID would make your system unbootable. Exactly what errors are you seeing?

If worse comes to worst, you should be able to treat the functional drive as though it's not part of a RAID at all. But this would require having your system set up such that there are no direct references to the RAID.

Also, is there a specific reason you made three separate arrays rather than one array with eg. LVM on top?

whoop · 2015-01-30 01:21:24

alphaniner wrote:

It's kind of surprising that a degraded RAID would make your system unbootable. Exactly what errors are you seeing?

I degraded the array's once I was able to boot succesfully. Before that Systemd gave me srst failed erno=-16 on md1 and did not start X or login.
So is it possible to degrade the arrays manually when I got stuck at systemd?

alphaniner wrote:

If worse comes to worst, you should be able to treat the functional drive as though it's not part of a RAID at all. But this would require having your system set up such that there are no direct references to the RAID.

How do I do that?

alphaniner wrote:

Also, is there a specific reason you made three separate arrays rather than one array with eg. LVM on top?

I don't think so...

alphaniner · 2015-01-30 03:22:05

srst failed erno=-16 on md1

It doesn't make sense to see an error like that related to the RAID. Are you sure it wasn't related to one of the actual disks like this?

Before manually degrading the RAID, did you try physically disconnecting the suspect drive? That error could have caused boot to fail before mdadm even became involved.

Also based on the thread I linked, this could just be a case of a bad connection. Try replacing the SATA cable and replacing/reseating the drive power cable.

whoop · 2015-01-30 14:41:33

alphaniner wrote:

srst failed erno=-16 on md1
It doesn't make sense to see an error like that related to the RAID. Are you sure it wasn't related to one of the actual disks like this?
Before manually degrading the RAID, did you try physically disconnecting the suspect drive? That error could have caused boot to fail before mdadm even became involved.
Also based on the thread I linked, this could just be a case of a bad connection. Try replacing the SATA cable and replacing/reseating the drive power cable.

OK, I think I understand... So you are saying systemd boot stopped not because RAID was complaining but because sata2 was not functioning properly. Removing sata2 (making it not work at all) might let the system boot completely (with degraded RAID)...

I will have to test this... Thank you...

whoop · 2015-01-30 19:57:26

OK, if I fix the raid array's poweroff the machine, remove sata2 from the machine and boot I get:

fsck.ext2: Invalid argument while trying to open /dev/md1
/dev/md1:
The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem.
If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else),
then the superbloc is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>

ERROR:fsck faild on '/dev/md1'
:: mounting 'dev/md1' on real root
mount: you must specify the filesystem type
You are now being dropped into an emergency shell

I wouldn't know what to do now...
md1 is / which is ext4.

Last edited by whoop (2015-01-30 20:07:12)

alphaniner · 2015-01-30 20:06:38

I don't know what to tell you. It must have something to do with the particulars of your configuration. I made a somewhat similar setup in a VM, but using only two arrays: one is /boot other is an LVM PV. The system boots with one of the disks removed.

whoop · 2015-01-31 03:11:46

Configuration:

blkid
/dev/sda1: LABEL="System Reserved" UUID="92907A079079F1D9" TYPE="ntfs" PARTUUID="4a8476cc-01"
/dev/sda2: UUID="EC2E7D1B2E7CDFCE" TYPE="ntfs" PARTUUID="4a8476cc-02"
/dev/sdb1: UUID="9a6c7a1d-fc51-444c-ada1-7eeed3b0ec0e" TYPE="swap" PARTUUID="0003adbe-01"
/dev/sdb2: UUID="237a3e20-3553-3245-5241-990451489ef3" TYPE="linux_raid_member" PARTUUID="0003adbe-02"
/dev/sdb3: UUID="07b22e88-4c83-3c6b-1efc-0ada7de1260a" UUID_SUB="ee5ffeb2-ad56-4d38-2a80-6999049943a0" LABEL="archiso:1" TYPE="linux_raid_member" PARTUUID="0003adbe-03"
/dev/sdb4: UUID="8d193515-a857-8adb-749c-b969001c947b" UUID_SUB="17b7a0fc-68eb-a4cc-8242-3d7409bff820" LABEL="archiso:2" TYPE="linux_raid_member" PARTUUID="0003adbe-04"
/dev/md2: UUID="51af5fdf-9c50-4f38-8c61-95dd58a0e2ee" TYPE="ext4"
/dev/md1: UUID="a8b4b9a3-6eb7-40af-8952-0c1881cb28fb" TYPE="ext4"
/dev/md0: UUID="845ff300-ebe0-43be-b2ae-1ce64c499849" TYPE="ext2"

 cat /etc/fstab
# 
# /etc/fstab: static file system information
#
# <file system>	<dir>	<type>	<options>	<dump>	<pass>
#tmpfs		/tmp	tmpfs	nodev,nosuid	0	0
#/boot is on /dev/md0
UUID=845ff300-ebe0-43be-b2ae-1ce64c499849 /boot ext2 defaults 0 1
#root is on /dev/md1
UUID=a8b4b9a3-6eb7-40af-8952-0c1881cb28fb / ext4 defaults 0 1
#/home is on /dev/md2
UUID=51af5fdf-9c50-4f38-8c61-95dd58a0e2ee /home ext4 defaults 0 1
#swap is on /dev/sdb1
UUID=9a6c7a1d-fc51-444c-ada1-7eeed3b0ec0e swap swap defaults 0 0
#swap is on /dev/sdc1
UUID=d1e694cd-17f4-4c9a-a90b-474ad1adbe33 swap swap defaults 0 0

cat /etc/mdadm.conf
ARRAY /dev/md0 metadata=0.90 UUID=237a3e20:35533245:52419904:51489ef3
ARRAY /dev/md/1 metadata=1.2 name=archiso:1 UUID=07b22e88:4c833c6b:1efc0ada:7de1260a
ARRAY /dev/md/2 metadata=1.2 name=archiso:2 UUID=8d193515:a8578adb:749cb969:001c947b

cat /boot/syslinux/syslinux.cfg
# Config file for Syslinux -
# /boot/syslinux/syslinux.cfg
#
# Comboot modules:
#   * menu.c32 - provides a text menu
#   * vesamenu.c32 - provides a graphical menu
#   * chain.c32 - chainload MBRs, partition boot sectors, Windows bootloaders
#   * hdt.c32 - hardware detection tool
#   * reboot.c32 - reboots the system
#
# To Use: Copy the respective files from /usr/lib/syslinux to /boot/syslinux.
# If /usr and /boot are on the same file system, symlink the files instead
# of copying them.
#
# If you do not use a menu, a 'boot:' prompt will be shown and the system
# will boot automatically after 5 seconds.
#
# Please review the wiki: https://wiki.archlinux.org/index.php/Syslinux
# The wiki provides further configuration examples

DEFAULT arch
PROMPT 0        # Set to 1 if you always want to display the boot: prompt 
TIMEOUT 50
# You can create syslinux keymaps with the keytab-lilo tool
#KBDMAP de.ktl

# Menu Configuration
# Either menu.c32 or vesamenu32.c32 must be copied to /boot/syslinux 
UI menu.c32
#UI vesamenu.c32

# Refer to http://syslinux.zytor.com/wiki/index.php/Doc/menu
MENU TITLE Arch Linux
#MENU BACKGROUND splash.png
MENU COLOR border       30;44   #40ffffff #a0000000 std
MENU COLOR title        1;36;44 #9033ccff #a0000000 std
MENU COLOR sel          7;37;40 #e0ffffff #20ffffff all
MENU COLOR unsel        37;44   #50ffffff #a0000000 std
MENU COLOR help         37;40   #c0ffffff #a0000000 std
MENU COLOR timeout_msg  37;40   #80ffffff #00000000 std
MENU COLOR timeout      1;37;40 #c0ffffff #00000000 std
MENU COLOR msg07        37;40   #90ffffff #a0000000 std
MENU COLOR tabmsg       31;40   #30ffffff #00000000 std

# boot sections follow
#
# TIP: If you want a 1024x768 framebuffer, add "vga=773" to your kernel line.
#
#-*

LABEL arch
	MENU LABEL Arch Linux
	LINUX ../vmlinuz-linux
	APPEND root=/dev/md1 rw
	INITRD ../initramfs-linux.img

LABEL archfallback
	MENU LABEL Arch Linux Fallback
	LINUX ../vmlinuz-linux
	APPEND root=/dev/md1 rw 
	INITRD ../initramfs-linux-fallback.img

#LABEL windows
#        MENU LABEL Windows
#        COM32 chain.c32
#        APPEND hd0 1

LABEL hdt
        MENU LABEL HDT (Hardware Detection Tool)
        COM32 hdt.c32
 
LABEL reboot
        MENU LABEL Reboot
        COM32 reboot.c32
 
LABEL poweroff
        MENU LABEL Poweroff
        COM32 poweroff.c32

/dev/sdc is missing because of pulled sata

Last edited by whoop (2015-01-31 03:16:49)

Arch Linux

#1 2015-01-30 00:26:54

RAID 1 with failing hd

#2 2015-01-30 01:09:15

Re: RAID 1 with failing hd

#3 2015-01-30 01:21:24

Re: RAID 1 with failing hd

#4 2015-01-30 03:22:05

Re: RAID 1 with failing hd

#5 2015-01-30 14:41:33

Re: RAID 1 with failing hd

#6 2015-01-30 19:57:26

Re: RAID 1 with failing hd

#7 2015-01-30 20:06:38

Re: RAID 1 with failing hd

#8 2015-01-31 03:11:46

Re: RAID 1 with failing hd

Board footer