Disks doesn't enter standby mode

mrpaul · 2022-03-15 17:42:48

I've set up a local NAS containing one SSD (/dev/sda) for arch, 4 HDDs in a RAID5 (/dev/sd[cdef] -> mounted at /mnt/data) and a fifth HDD (/dev/sdb -> mounted at /mnt/backup) for Backups.
Additionally, I send my HDDs (/dev/sd[bcdef]) to standby after 15 minutes using hdparm ( I do not use /dev/sdX, I use /dev/disk/by-id instead). This works well so far.

Now I configured two Backups with rsnapshot:
Backup 1 to backup my arch system (sda3) to the RAID (/mnt/data/unsaved).
Backup 2 to backup personal data from the RAID (/mnt/data/saved) to the backup disk (/mnt/backup).

Running only Backup 1 works. After doing the backup, all the RAID drives enter the standby mode and disks stop spinning.
Running only Backup 2 doesn't work as expected. The backup is created but not all RAID disk enter the standby mode again. A random number of them stucks in active/idle mode. I checked this with smartctl -i -n standby /dev/sdX and hdparm -C /dev/sd[bcdef].

Now I'm confused, what could be the reason. iotop didn't show anything useful.

So my questions are:
Is it possible, that rsnapshot does not release the source disks properly?
How can I figure out the process, that prevents the disk to enter standby mode?

Thanks four your support.

seth · 2022-03-15 20:58:35

Can you -y or -Y them and do they stay there (unless expected access)?
Does the backup/2 typically last longer than 15 minutes? Can you reproduce this w/ a shorter interaction?
Does a subsequent access (not rsnapshot, just read some bytes from the device, not the raid, into /dev/null or so) "sanitize" the situation?
Can you actually also cause this by just reading some files from the raid into /dev/null or so?

mrpaul · 2022-03-19 09:29:08

seth wrote:

Does the backup/2 typically last longer than 15 minutes? Can you reproduce this w/ a shorter interaction?

Taking a look into the logfile shows, that this is not the case:

[2022-03-19T10:01:53] /usr/bin/rsnapshot -c /etc/rsnapshot_nas.conf beta: started
[2022-03-19T10:01:53] echo 143316 > /var/run/rsnapshot_nas.pid
[2022-03-19T10:02:00] /usr/bin/rm -rf /mnt/backup/beta.6/
[2022-03-19T10:02:01] mv /mnt/backup/beta.5/ /mnt/backup/beta.6/
[2022-03-19T10:02:01] mv /mnt/backup/beta.4/ /mnt/backup/beta.5/
[2022-03-19T10:02:01] mv /mnt/backup/beta.3/ /mnt/backup/beta.4/
[2022-03-19T10:02:01] mv /mnt/backup/beta.2/ /mnt/backup/beta.3/
[2022-03-19T10:02:01] mv /mnt/backup/beta.1/ /mnt/backup/beta.2/
[2022-03-19T10:02:01] /usr/bin/cp -al /mnt/backup/beta.0 /mnt/backup/beta.1
[2022-03-19T10:02:01] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /mnt/data/gesichert/ /mnt/backup/beta.0/data/
[2022-03-19T10:02:25] touch /mnt/backup/beta.0/
[2022-03-19T10:02:25] rm -f /var/run/rsnapshot_nas.pid
[2022-03-19T10:02:25] /usr/bin/rsnapshot -c /etc/rsnapshot_nas.conf beta: completed successfully

But even if the backup itself would take more than 15 minutes, this shouldn't have any drawback, because the 15 minute timeout should start after the last access. Or isn't this the case?

All other suggestions I will test after the next scheduled backup.

mrpaul · 2022-03-19 12:10:39

seth wrote:

Can you -y or -Y them and do they stay there (unless expected access)?

Yes, after forcing the disk to standby using hdparm -y , they spin down and stay in standby mode until I access the RAID again.

seth · 2022-03-19 12:54:46

this shouldn't have any drawback, because the 15 minute timeout should start after the last access.

https://www.youtube.com/watch?v=5RyYrs5tu60

I suspect that the idle timer gets thrown off by the raid condition - you access the raid but ultimately never the drive and the timemout doesn't get triggered (pure speculation)
So the idea is that

seth wrote:

Can you actually also cause this by just reading some files from the raid into /dev/null or so?

causes the exact same problem and

seth wrote:

Does a subsequent access (not rsnapshot, just read some bytes from the device, not the raid, into /dev/null or so) "sanitize" the situation?

would trigger the timeout and the drive drops into standby 15 minutes afterwards.

mrpaul · 2022-03-19 17:49:26

seth wrote:

Can you actually also cause this by just reading some files from the raid into /dev/null or so?
causes the exact same problem and

I'm not so familiar with linux, so when you say "read some files to /dev/null", do you mean something like this:

 cat /mnt/data/some/file > /dev/null

When doing this, then
- using the auto completition (tab key) already wakes up some of the RAID drives before the command is executed -> one drive became active, rest keept in standby
- not all, only some drives wake up (I think this is caused by the RAID chunk size, because my test file may was to small to get spread over multiple disks) -> executing the command woke up one additional drive, so two of four drives have been active
- finally all drives enter the standby mode and spin down the disks -> all four drives in standby

I'm also a little bit confused about the disk state, when checking it with hdparm -C:
- as long as I access a drive, hdparm tells me active/idle
- checking again a few seconds (10s i.e.) after file access, hdparm says "unknown" and smartctl tells me "power mode was: IDLE_B"
- after the waiting time (15min), then hdparm says "standby"
Where does this "unknown" state come from? Was is meant with "was IDLE_B"? Is this a special behavior of the Seagate IronWolf disks?

So back to your question: reading a file does not cause the same behavior as a backup with rsnapshot.

seth wrote:

Does a subsequent access (not rsnapshot, just read some bytes from the device, not the raid, into /dev/null or so) "sanitize" the situation?

Could you please give me a suggested command?
How can I access data from a drive without accessing the RAID? For me this sounds like a read command to an arbitrary byte address of the disk. But how to achieve this?

seth · 2022-03-21 20:02:57

do you mean something like this

Or just "cp", yes - the main point is to access the raid "somehow"

Could you please give me a suggested command?

dd if=/dev/sdXn -of /dev/null -count 1# sdXn  being a drive in the raid that doesn't go to standby

So back to your question: reading a file does not cause the same behavior as a backup with rsnapshot

Does "lsof" reveal some stale open file handles after the rsnapshot run?

mrpaul · 2022-03-22 16:26:07

By now I'm totally confused:
I only did the investigations reported in the previous posts and didn't changed anything in the rsnapshot config.
But since two days, the backup behaves as expected. All drives enter standby mode. It's magic...

seth wrote:

Does "lsof" reveal some stale open file handles after the rsnapshot run?

Thanks for your support.
I will do both, reading from the drive directly as mentioned in your last post and check open files with lsof, as soon as the backup will show it's strange behavior again.
At the moment, I do not really trust my system...

mrpaul · 2022-03-23 15:53:48

seth wrote:

Does "lsof" reveal some stale open file handles after the rsnapshot run?

lsof | grep /mnt

shows no entries.

lsof | grep /dev

shows some entries, but none of them points to /dev/sdX.

using lsof with the partition device number from lsblk

lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda         8:0    0 111,8G  0 disk  
├─sda1      8:1    0   512M  0 part  /boot
├─sda2      8:2    0     8G  0 part  [SWAP]
└─sda3      8:3    0    40G  0 part  /
sdb         8:16   0   2,7T  0 disk  
└─sdb1      8:17   0   2,7T  0 part  /mnt/backup
sdc         8:32   1   3,6T  0 disk  
└─sdc1      8:33   1   3,6T  0 part  
  └─md127   9:127  0  10,9T  0 raid5 /mnt/data
sdd         8:48   1   3,6T  0 disk  
└─sdd1      8:49   1   3,6T  0 part  
  └─md127   9:127  0  10,9T  0 raid5 /mnt/data
sde         8:64   1   3,6T  0 disk  
└─sde1      8:65   1   3,6T  0 part  
  └─md127   9:127  0  10,9T  0 raid5 /mnt/data
sdf         8:80   1   3,6T  0 disk  
└─sdf1      8:81   1   3,6T  0 part  
  └─md127   9:127  0  10,9T  0 raid5 /mnt/data

gives

lsof | grep 8,49
smbd         606            root    5u  a_inode               0,14        0      10363 [eventpoll:7,15,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50]

I did this for all of the device numbers 9:127, 8:32, 8:33, 8:48, 8:49, 8:64, 8:65, 8:80, 8:81.
The only entry I received is this previous listed entry, that is also there, even if all drives are in standby.

seth · 2022-03-23 22:44:32

"lsof | grep 8,49" is a false positive for "…44,46,47,48,49,50"

Is this maybe coincidental and the drive simply does it's internal SMART stuff (unrelated to the OS)
Does it staandby "later™"?

mrpaul · 2022-03-30 14:16:48

seth wrote:

Is this maybe coincidental and the drive simply does it's internal SMART stuff (unrelated to the OS)
Does it staandby "later™"?

No, the backup is scheduled for 1pm but disk are still active at 5pm.

But I observed following behavior:
- if there are any changes in the folder to backup, then the disks doesn't enter standby mode again
- if there are no changes, then the backup is executed as expected
- the problem only occurs, if the RAID contains the backup source: Backing up my filesystem to RAID works, backing up my data from the RAID fails.

Arch Linux

#1 2022-03-15 17:42:48

Disks doesn't enter standby mode

#2 2022-03-15 20:58:35

Re: Disks doesn't enter standby mode

#3 2022-03-19 09:29:08

Re: Disks doesn't enter standby mode

#4 2022-03-19 12:10:39

Re: Disks doesn't enter standby mode

#5 2022-03-19 12:54:46

Re: Disks doesn't enter standby mode

#6 2022-03-19 17:49:26

Re: Disks doesn't enter standby mode

#7 2022-03-21 20:02:57

Re: Disks doesn't enter standby mode

#8 2022-03-22 16:26:07

Re: Disks doesn't enter standby mode

#9 2022-03-23 15:53:48

Re: Disks doesn't enter standby mode

#10 2022-03-23 22:44:32

Re: Disks doesn't enter standby mode

#11 2022-03-30 14:16:48

Re: Disks doesn't enter standby mode

Board footer