You are not logged in.

Pages: 1

#1 2020-11-18 09:14:14

joanmanel
Member
Registered: 2012-11-06
Posts: 250

Computer rebooting randomly

Hi, I have a computer which I mostly use for Deep Learning, and since the last 3-4 weeks, it is rebooting randomly. I connect to this computer throush SSH and I keep my sessions active using Tmux.

You can see here my reboot logs:

last reboot | less

reboot   system boot  5.9.8-arch1-1    Wed Nov 18 05:47   still running
reboot   system boot  5.9.8-arch1-1    Tue Nov 17 09:22   still running
reboot   system boot  5.9.8-arch1-1    Tue Nov 17 04:53 - 09:22  (04:28)
reboot   system boot  5.9.8-arch1-1    Sat Nov 14 16:52 - 09:22 (2+16:29)
reboot   system boot  5.9.6-arch1-1    Mon Nov  9 16:08 - 16:47 (5+00:38)
reboot   system boot  5.9.6-arch1-1    Mon Nov  9 12:09 - 16:07  (03:58)
reboot   system boot  5.9.6-arch1-1    Mon Nov  9 10:35 - 16:07  (05:32)
reboot   system boot  5.8.14-arch1-1   Mon Nov  9 09:57 - 10:22  (00:25)
reboot   system boot  5.8.14-arch1-1   Mon Nov  9 01:00 - 09:57  (08:56)
reboot   system boot  5.8.14-arch1-1   Thu Oct 22 09:09 - 09:57 (18+00:47)

You can see that it rebooted today at 5.47am. I don't understand why it says that the previous boot is "still running".

You can see here my journals before the boot: https://termbin.com/qcug

And my journal after the boot (still the active session): https://termbin.com/jzrbb

As I said, I mostly use this computer to do Deep Learning. The strange thing is that previously, when the GPU failed, the system would not reboot, the learning would fail and the gpu would be off, but the system would not reboot.

The training I am doing is also not that "hard", you can see here the output of nvidia-smi:

Wed Nov 18 09:12:08 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:01:00.0 Off |                  N/A |
| 18%   57C    P2    78W / 280W |   9677MiB / 11176MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       505      G   /usr/lib/Xorg                       9MiB |
|    0   N/A  N/A       532      G   /usr/bin/gnome-shell                3MiB |
|    0   N/A  N/A      1308      C   python                           9659MiB |
+-----------------------------------------------------------------------------+

As you can see the memory is not even full, the GPU fan is at 18%, temperature is OK, power is low.

I really have no idea what is wrong, and any help would be very welcome.

Offline

#2 2020-11-18 09:33:28

ponyrider
Member
Registered: 2014-11-18
Posts: 112

Re: Computer rebooting randomly

I suspect a hardware issue

It could be
1. Temperature
2. Ram
3. PSU

and check in that order

you listed the temp of the gpu, but not the cpu? if you are sure nothing got too hot, check the ram with Memtest86 to be sure you dont have a defective stick

also this is very helpful

# previous boot, highest priority
journalctl -b -1 -p3

Last edited by ponyrider (2020-11-18 09:36:43)

Offline

#3 2020-11-18 09:47:15

joanmanel
Member
Registered: 2012-11-06
Posts: 250

Re: Computer rebooting randomly

ponyrider wrote:

I suspect a hardware issue

It could be
1. Temperature
2. Ram
3. PSU

and check in that order

you listed the temp of the gpu, but not the cpu? if you are sure nothing got too hot, check the ram with Memtest86 to be sure you dont have a defective stick

also this is very helpful

# previous boot, highest priority
journalctl -b -1 -p3

Trying your command, I only get some errors that happened at boot time, far away from when the computer rebooted:

Nov 17 09:22:43 auchentoshan kernel: x86/cpu: VMX (outside TXT) disabled by BIOS
Nov 17 09:22:44 auchentoshan kernel: bcma-pci-bridge 0000:05:00.0: bus0: No SPROM available
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:44 auchentoshan kernel: Bluetooth: hci0: BCM: firmware Patch file not found, tried:
Nov 17 09:22:44 auchentoshan kernel: Bluetooth: hci0: BCM: 'brcm/BCM20702A1-0b05-180a.hcd'
Nov 17 09:22:44 auchentoshan kernel: Bluetooth: hci0: BCM: 'brcm/BCM-0b05-180a.hcd'
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:44 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:44 auchentoshan kernel: bcma-pci-bridge 0000:05:00.0: bus0: HT force timeout
Nov 17 09:22:45 auchentoshan kernel: bcma-pci-bridge 0000:05:00.0: bus0: PLL enable timeout
Nov 17 09:22:45 auchentoshan kernel: b43-phy0 ERROR: FOUND UNSUPPORTED PHY (Analog 12, Type 11 (AC), Revision 1)
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Couldn't find mci handler
Nov 17 09:22:45 auchentoshan kernel: EDAC sbridge: Failed to register device with error -19.

Offline

#4 2020-11-18 11:41:36

ponyrider
Member
Registered: 2014-11-18
Posts: 112

Re: Computer rebooting randomly

all that says it that you haven't installed whatever broadcom device you have, i dont think the EDAC messages are relevant. check your ram!

Offline

#5 2020-11-18 12:14:24

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 77,692

Re: Computer rebooting randomly

I don't understand why it says that the previous boot is "still running".

Do you have a prallel windows installation?

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#6 2020-11-18 12:24:03

joanmanel
Member
Registered: 2012-11-06
Posts: 250

Re: Computer rebooting randomly

seth wrote:

I don't understand why it says that the previous boot is "still running".

Do you have a prallel windows installation?

There is a W10 installation in another disk, which hasnt been used or booted in years. This is a computer that I only ssh in, no screen attached. grub boots straight to linux.

nevertheless, here is my fdisk:

Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EARX-00P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A27CEA4D-2950-45FB-9C13-EF648AFBC658

Device       Start        End    Sectors   Size Type
/dev/sda1     2048     923647     921600   450M Windows recovery environment
/dev/sda2   923648    1128447     204800   100M EFI System
/dev/sda3  1128448    1161215      32768    16M Microsoft reserved
/dev/sda4  1161216 1039831039 1038669824 495.3G Microsoft basic data


Disk /dev/sdb: 111.79 GiB, 120034123776 bytes, 234441648 sectors
Disk model: SanDisk SD8SBBU1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 13338038-1275-414F-8A49-C9B8B1AB7721

Device         Start       End   Sectors  Size Type
/dev/sdb1       2048 115345407 115343360   55G Linux filesystem
/dev/sdb2  115345408 234441614 119096207 56.8G Linux filesystem


Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST1000DM003-1ER1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F7B1D413-343C-4F16-8BCA-647C515E4008

Device     Start        End    Sectors   Size Type
/dev/sdc1   2048 1953525134 1953523087 931.5G Linux filesystem

Offline

#7 2020-11-18 12:25:44

joanmanel
Member
Registered: 2012-11-06
Posts: 250

Re: Computer rebooting randomly

ponyrider wrote:

all that says it that you haven't installed whatever broadcom device you have, i dont think the EDAC messages are relevant. check your ram!

Is there any way to know if the RAM is the one giving me problems?

The Deep Learning stuff I am doing doesnt use that much RAM, it mostly loads from disk. You can see from top:

top - 12:25:26 up  6:37,  3 users,  load average: 4.70, 4.95, 4.96
Tasks: 253 total,   7 running, 246 sleeping,   0 stopped,   0 zombie
%Cpu(s): 39.9 us,  6.8 sy,  0.0 ni, 53.2 id,  0.0 wa,  0.1 hi,  0.0 si,  0.0 st
MiB Mem :  96488.8 total,  84517.3 free,   7680.5 used,   4291.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  87678.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                
  38650 me    20   0   30.4g   4.0g 183416 R  85.4   4.3   0:02.57 python                                                                                                                 
  38652 me    20   0   30.4g   4.0g 183444 R  85.4   4.3   0:02.57 python                                                                                                                 
  38654 me    20   0   30.4g   4.0g 183428 R  85.4   4.3   0:02.57 python                                                                                                                 
  38656 me    20   0   30.4g   4.0g 183444 R  85.4   4.3   0:02.57 python                                                                                                                 
  38658 me    20   0   30.4g   4.0g 183436 R  85.4   4.3   0:02.57 python                                                                                                                 
  38660 me    20   0   30.4g   4.0g 183440 R  85.4   4.3   0:02.57 python                                                                                                                 
   7893 me    20   0   29.6g   5.0g   1.7g S  26.9   5.3  63:05.72 python

(I have replaced my name with "me")

Offline

#8 2020-11-18 12:29:19

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 77,692

Re: Computer rebooting randomly

The disk doesn't matter - boot windows and ensure it's not hibernating (notably because of fast-boot, 3rd link in my signature)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

#9 2020-11-18 12:35:30

joanmanel
Member
Registered: 2012-11-06
Posts: 250

Re: Computer rebooting randomly

seth wrote:

The disk doesn't matter - boot windows and ensure it's not hibernating (notably because of fast-boot, 3rd link in my signature)

I see, but if windows hasnt been booted in 2-3 years, and until recently everything was going OK, I would assume windows is not the problem? Why would boot the system now after so long?

Offline

#10 2020-11-18 12:42:01

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 77,692

Re: Computer rebooting randomly

ACPI voodoo - there's no guarantee that this is the case but *iff* the BIOS believes there's a hibernated system, weird things can be expected.
You need to rule out every possible cause and de-hibernating windows is going to be quicker than eg. running memtest86 (a reasonable amount of cycles will be counted in days)

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Offline

Pages: 1

Board footer

Powered by FluxBB