NAS reboots without warning during HDD operations

Lockheed · 2017-06-21 22:25:19

I have a small NUC-like HTPC with a DIY array of five 2.5” HDDs attached using a 2-port miniPCIe SATA card (the PC’s board has only one SATA port which is in use by a system SSD) connected to a SATA port multiplicator with 5 ports, out of which 4 are connected to disks and the fifth disk is connected directly to the second SATA port of miciPCIe card in the PC.
The array itself is powered independently from a separate power source which provides 5V/20A which should be plenty for five 2.5” HDDs.

This setup worked well for about 2 months. However, since several weeks the system started to reboot when there is a continuous disk operation on one or more HDDs in the external array. The reboot can happen anytime between 3-30 minutes of duration of the operation. This occurs always when there is a SnapRaid sync (because those take long time), and about 50% of times when there is a single large file being copied (over 1-2 GB).

I tried to investigate the reason for reboots, by:
- looking into system journal and into dmesg from just before the reboot occurred - there was absolutely nothing abnormal, which suggests the system experiences a sudden reboot for hardware reasons
- I run memtest on ram on this HTPC - the test passed successfully several times
- running mprime on the PC it for over 24 hours with the array attached - no reboot occurred

This would suggest that the array itself is causing the reboot, but how could this be possible, if it is connected to the PC with only two SATA cables (1: SATA miniPCIe -> SATA port multiplicator; 2: SATA miniPCIe -> directly to one of the HDDs), and the array itself has its own power source. Plus, even if there was some problem with the array, it should just impact HDDs, not the PC itself, right?
Maybe some electricity goes from the array through SATA cables to the PC, destabilising voltages and causing reboot?

In any case, at this point I am out of clear ideas about what could be the cause of those reboots, or even how could I investigate it further. I would appreciate any suggestions you might have.

fukawi2 · 2017-06-22 00:03:07

Post output of `lspci -v` and `uname -a` from the NAS please.

Lockheed · 2017-06-22 06:44:06

Certainly.

# uname -a
Linux nanoBox 4.11.5-1-ARCH #1 SMP PREEMPT Wed Jun 14 16:19:27 CEST 2017 x86_64 GNU/Linux

# lspci -v
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Root Complex
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Root Complex
	Flags: bus master, fast devsel, latency 0

00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Kabini [Radeon HD 8330] (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0123
	Flags: bus master, fast devsel, latency 0, IRQ 27
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=8M]
	I/O ports at f000 [size=256]
	Memory at feb00000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu

00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, fast devsel, latency 0, IRQ 37
	Memory at feb64000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 0
	Flags: fast devsel

00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1 (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 24
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff [size=4K]
	Memory behind bridge: fea00000-feafffff [size=1M]
	Prefetchable memory behind bridge: None
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [b0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1234
	Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1 (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 25
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff [size=4K]
	Memory behind bridge: None
	Prefetchable memory behind bridge: 00000000d0800000-00000000d08fffff [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [b0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1234
	Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 01) (prog-if 30 [XHCI])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, fast devsel, latency 0, IRQ 18
	Memory at feb68000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [50] Power Management version 3
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [a0] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Latency Tolerance Reporting
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40) (prog-if 01 [AHCI 1.0])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 19
	I/O ports at f140 [size=8]
	I/O ports at f130 [size=4]
	I/O ports at f120 [size=8]
	I/O ports at f110 [size=4]
	I/O ports at f100 [size=16]
	Memory at feb6e000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [60] Power Management version 3
	Capabilities: [70] SATA HBA v1.0
	Kernel driver in use: ahci
	Kernel modules: ahci

00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 39) (prog-if 10 [OHCI])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
	Memory at feb6d000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci-pci
	Kernel modules: ohci_pci

00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39) (prog-if 20 [EHCI])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
	Memory at feb6c000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
	Capabilities: [e4] Debug port: BAR=1 offset=00e0
	Kernel driver in use: ehci-pci
	Kernel modules: ehci_pci

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 39) (prog-if 10 [OHCI])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18
	Memory at feb6b000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci-pci
	Kernel modules: ohci_pci

00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39) (prog-if 20 [EHCI])
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 17
	Memory at feb6a000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
	Capabilities: [e4] Debug port: BAR=1 offset=00e0
	Kernel driver in use: ehci-pci
	Kernel modules: ehci_pci

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 3a)
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: 66MHz, medium devsel
	Kernel driver in use: piix4_smbus
	Kernel modules: i2c_piix4, sp5100_tco

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 02)
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, slow devsel, latency 32, IRQ 16
	Memory at feb60000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
	Subsystem: ZOTAC International (MCO) Ltd. Device b208
	Flags: bus master, 66MHz, medium devsel, latency 0

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 0
	Flags: fast devsel

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 1
	Flags: fast devsel

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 2
	Flags: fast devsel

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 3
	Flags: fast devsel
	Capabilities: [f0] Secure device <?>
	Kernel driver in use: k10temp
	Kernel modules: k10temp

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 4
	Flags: fast devsel
	Kernel driver in use: fam15h_power
	Kernel modules: fam15h_power

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Function 5
	Flags: fast devsel

01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
	Subsystem: ASMedia Technology Inc. Device 1060
	Flags: bus master, fast devsel, latency 0, IRQ 34
	I/O ports at e050 [size=8]
	I/O ports at e040 [size=4]
	I/O ports at e030 [size=8]
	I/O ports at e020 [size=4]
	I/O ports at e000 [size=32]
	Memory at fea10000 (32-bit, non-prefetchable) [size=512]
	Expansion ROM at fea00000 [disabled] [size=64K]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci
	Kernel modules: ahci

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
	Subsystem: ZOTAC International (MCO) Ltd. Device 8168
	Flags: bus master, fast devsel, latency 0, IRQ 35
	I/O ports at d000 [size=256]
	Memory at d0804000 (64-bit, prefetchable) [size=4K]
	Memory at d0800000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Endpoint, MSI 01
	Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
	Capabilities: [d0] Vital Product Data
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
	Kernel driver in use: r8169
	Kernel modules: r8169

R00KIE · 2017-06-22 21:52:38

From your description, having separate PSUs providing power to separate parts of the system, and having both sides connected only by the sata cables I would guess maybe some weird ground loop is causing the reboots.

I would try to connect the grounds of both power supplies and see if you can still trigger the problem. However if you are not very sure on how to do it maybe it's better you don't try it or you risk frying something.

Lockheed · 2017-06-24 17:30:07

R00KIE wrote:

From your description, having separate PSUs providing power to separate parts of the system, and having both sides connected only by the sata cables I would guess maybe some weird ground loop is causing the reboots.
I would try to connect the grounds of both power supplies and see if you can still trigger the problem. However if you are not very sure on how to do it maybe it's better you don't try it or you risk frying something.

I opened the array, vacuumed the dust and grounded the metal case to the ground connector inside the PSU AC cable (just stuck a stripped USB cable in there and put the plug into the PSU socket. The problem persists.

When you say to connect grounds of both PSUs, I wonder what would that do because from what I understand, the grounds of PSUs are not connected to the actual devices (NUC and array), because there are only two DC cables (black and red) going from the PSU to the device. If that's so, then what would grounding PSUs do for the devices?

ewaller · 2017-06-24 21:19:20

Be careful. Please.
Ground (Earth) is not necessarily the same ground used as signal reference. In a proper design, signal ground is tied to earth ground at (exactly) one point[1]. Avoid building ground loops and whatever you do, don't get tangled in up with the mains common or hot lines.

Edit:
[1] Here I am speaking of a lumped electronic system, not a factory building that has multiple motors and and need for lighting suppression requiring a well designed multipoint ground system.

Last edited by ewaller (2017-06-24 21:25:29)

R00KIE · 2017-06-24 22:45:50

If I understood your description of how you have things connected it looks somewhat like this:

HTPC with PSU1 - motherboard - PCIe card - SATA - HDDs - PSU2

What I was thinking about is the following, your pcie card and the sata signaling from the sata card to the HDDs use the ground from PSU1 as a reference. As ewaller says this ground is probably connected to earth or at least should be in some way(1). If the ground of PSU2 is not connected to the ground of PSU1 and/or to earth then the voltage provided by PSU2 might be floating (not the measured voltage between 5V/12V to ground but from 5V/12V/ground to earth). Here there are three options:

a) The ground in the sata cable is connected to the power ground inside the hard disks, I guess this is not ideal as you don't want to create a ground look though the sata cable but I suppose it should not cause the problem you are seeing.

b) There really is no connection between grounds. The signaling from the HDD to the pcie card will have as a reference the ground of PSU2. The common mode of this signaling voltage can be above of below what the pcie card expects or can handle and maybe somehow that triggers a fault that makes the machine reboot.

c) I'm very wrong and the problem is something else, but this was the only thing that came to mind after reading the description of your problem.

(1) I'm not sure if the output side of any PSU is connected to earth, any metal parts have to be connected to earth but from memory I don't recall if the output has any direct path to earth. I guess it ends up being connected to earth via the motherboard. When you mount the motherboard it is screwed to the chassis and I believe the contact points in the motherboard are connected to ground, the chassis has to be connected to earth since it is made of metal and so you make a connection there.

mich41 · 2017-06-25 10:58:31

Maybe software? Can you test the oldest kernel from /var/cache/pacman/pkg?

Lockheed · 2017-06-25 19:46:44

My building has no ground installation. Just two-line TN-C topology, meaning hot and neutral. The power sockets in the server room had no earth pins. So I now (i.e. after writing the fist post) replaced them and connected the earth pin to neutral, creating a bridge. This is as good as I can get with this house wiring.

So, previously there was no ground connection whatsoever, and things worked fine until few weeks ago. But now both PSUs are “grounded” to the same neutral. Problem persists, as per previous post.

That said, I do not see any grounding of the PC, as from what I can see there are only DC+ and - coming from the PSU to the PC.

As for the array which I constructed myself - it for sure has no connection to the PSU’s grounding, because I know for sure it only is connected using + and - DC wires. I now added a wire connecting the metal case of array to the grounding pin of the PSU’s AC cable.

I hope this clarifies some things.

@R00KIE

a)do you think SATA cable could let through some rouge current? Anyway, you mention that even if it is does, it should not cause this problem, so I will not explore it further for now.
b)As I mentioned, even though the array should be now grounded (or should I say “neutralized”?), I am not sure the PC is grounded at all (though its PSU is).
I don’t see how could it be, because there does not seem to be anything in the DC power plug (the PC is powered by a laptop-like PSU), and the metal case of the PC stands on rubber legs on a sheet of glass, so also no grounding there at all. See here: http://tech.firstpost.com/wp-content/up … 369198.jpg

Anyhow, problem persists. And I think I have no way of pursuing the investigation on the PCIe card level because I can’t measure the voltage changes there.

mich41 wrote:

Maybe software? Can you test the oldest kernel from /var/cache/pacman/pkg?

I tried LTS kernel with version older than the kernel that worked without reboots. It made no difference therefore I excluded the kernel as suspect.

I hope my further explanations shed some more light on the problem and will allow for more informed guessing.

Last edited by Lockheed (2017-06-25 19:50:41)

ewaller · 2017-06-25 20:22:15

The Earth ground in buildings is primarily for safety. It is not intended to carry any current whatsoever in no fault modes. In the event a mains circuit connection fails, safety can only be ensured through "double insulation" so one one fails there is a fall back, or through surrounding all mains with a metal enclosure that is tied to Earth ground. In the event of a failure, the mains touch the metal housing and short tripping the current limiting device or the ground fault isolator. In North America, the neutral is tied to ground at the panel. As the neutral could be carrying 10 to 20 Amps, and if the resistance of the wire is on the order of 200 to 400 milliohms, one could potentially see several volts of DC bias and/or noise in the neutrals at the device. Also, Earth grounds that are daisy chained back to the panel have ample opportunity to pick up noise from motors, microwave ovens, transmitters, etc... A secondary advantage to an Earth ground and metal enclosures is that it improves the unwanted electromagnetic emissions from a device and improves the electromagnetic susceptibility for a device. This includes static discharge.

Computers use signals that are increasing small. Modern logic is often 3.3V. Core voltages are down around 1.25V. Differential signals are measured in the 10s of mV and can withstand common mode voltages measured in single digit volts.
Engineers work very hard to design quite power supplies that have low "ground" noise. Note that I speak here of signal ground. In modern systems, signal ground is never, ever bonded to Earth let alone Neutral. Modern systems will have a filter network, generally consisting of a series ferrite bead or an inductor if a DC path from signal ground to Earth ground is required. If only an AC path is required, there will be a blocking capacitor will be used. Some systems completely float the signal ground with perhaps several 100K Ohms tieing them together just to keep large voltages from being developed.

So, feel free to tie the signal grounds of your power supplies together, but DO NOT TIE THE SIGNAL GROUND TO COMMON. You will kill something or someone.

Last edited by ewaller (2017-06-25 20:23:01)

Lockheed · 2017-06-25 22:28:17

I am well aware of the benefits of real ground. However, this is not possible in this building.

I am not sure what you mean by "common", but if it means the same as "neutral", the non-hot of the two AC wires, then of course this is not optimal, but is the legal and electric norm here in Poland (220V, by the way) when the wiring in the building is the old type with no separate ground wire. This is called "zeroing" here and is considered better practice than leaving the plug without any earth pin connection.

Last edited by Lockheed (2017-06-25 22:28:48)

R00KIE · 2017-06-26 01:17:10

Just to make it clear, when I refer to ground I'm always talking about the ground or reference point on the output/low voltage side of the PSUs, stay away from messing with mains.

seth · 2017-06-26 06:32:33

Stupid question: did you already try to detach the sata cable from the single disk to check whether that makes a major impact (since it's the "dumb" connection between the array power and the board), iow that your're dealing with AC issues at all?

Lockheed · 2017-06-26 10:30:02

seth wrote:

Stupid question: did you already try to detach the sata cable from the single disk to check whether that makes a major impact (since it's the "dumb" connection between the array power and the board), iow that your're dealing with AC issues at all?

It's a lot of work to test that, but I suppose I will have to do that.

seth · 2017-06-26 12:40:35

Not sure why that's a lot of work, but maybe it's sufficitent to get some protection inbetween?
Eg. another multiplidcator or a sata - esata - sata chain?

R00KIE · 2017-06-26 13:26:42

I'd say it's not just the one disk, it's all of them if they get power from a different PSU than the one powering the motherboard and pcie card.

Arch Linux

#1 2017-06-21 22:25:19

NAS reboots without warning during HDD operations

#2 2017-06-22 00:03:07

Re: NAS reboots without warning during HDD operations

#3 2017-06-22 06:44:06

Re: NAS reboots without warning during HDD operations

#4 2017-06-22 21:52:38

Re: NAS reboots without warning during HDD operations

#5 2017-06-24 17:30:07

Re: NAS reboots without warning during HDD operations

#6 2017-06-24 21:19:20

Re: NAS reboots without warning during HDD operations

#7 2017-06-24 22:45:50

Re: NAS reboots without warning during HDD operations

#8 2017-06-25 10:58:31

Re: NAS reboots without warning during HDD operations

#9 2017-06-25 19:46:44

Re: NAS reboots without warning during HDD operations

#10 2017-06-25 20:22:15

Re: NAS reboots without warning during HDD operations

#11 2017-06-25 22:28:17

Re: NAS reboots without warning during HDD operations

#12 2017-06-26 01:17:10

Re: NAS reboots without warning during HDD operations

#13 2017-06-26 06:32:33

Re: NAS reboots without warning during HDD operations

#14 2017-06-26 10:30:02

Re: NAS reboots without warning during HDD operations

#15 2017-06-26 12:40:35

Re: NAS reboots without warning during HDD operations

#16 2017-06-26 13:26:42

Re: NAS reboots without warning during HDD operations

Board footer