You are not logged in.

#1 2015-06-07 12:39:43

while
Member
Registered: 2010-05-14
Posts: 35

Resize of RAID5 array frozen

I recently added a new disk to my RAID5 array and started growing it. Absent minded as I am I rebooted the server during this reshape process I as another program was hung and blocking some ports. Thinking of it now it might have been because the array hung but I cannot be sure.

After this reboot the reshape process has frozen at `28%`. I can no longer mount the array, stop it or anything it just seem to have frozen up.

Here is some info on the array:

    # mdadm -D /dev/md0
 
    /dev/md0:
            Version : 1.2
      Creation Time : Sat Mar 28 17:31:15 2015
         Raid Level : raid5
         Array Size : 5860063744 (5588.59 GiB 6000.71 GB)
      Used Dev Size : 2930031872 (2794.30 GiB 3000.35 GB)
       Raid Devices : 4
      Total Devices : 4
        Persistence : Superblock is persistent
    
      Intent Bitmap : Internal
    
        Update Time : Sun Jun  7 11:04:28 2015
              State : clean, reshaping 
     Active Devices : 4
    Working Devices : 4
     Failed Devices : 0
      Spare Devices : 0
    
             Layout : left-symmetric
         Chunk Size : 256K
    
     Reshape Status : 28% complete
      Delta Devices : 1, (3->4)
    
               Name : ocular:0  (local to host ocular)
               UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
             Events : 344582
    
        Number   Major   Minor   RaidDevice State
           4       8       17        0      active sync   /dev/sdb1
           1       8       49        1      active sync   /dev/sdd1
           3       8       65        2      active sync   /dev/sde1
           5       8       33        3      active sync   /dev/sdc1 

and /proc/mdstat shows

    Personalities : [raid6] [raid5] [raid4]
    md0 : active raid5 sdb1[4] sdc1[5] sde1[3] sdd1[1]
          5860063744 blocks super 1.2 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
          [=====>...............]  reshape = 28.6% (840259584/2930031872) finish=524064.9min speed=66K/sec
          bitmap: 3/22 pages [12KB], 65536KB chunk
    
    unused devices: <none>

Trying to mount the array just hangs

# mount /dev/md0 /mnt/storage/

And the same if I try to stop the array

# mdadm -S /dev/md0

I have also tried growing it down to 3 devices again but it is busy with the last reshape:

# mdadm --grow /dev/md0 --raid-devices=3
    mdadm: /dev/md0 is performing resync/recovery and cannot be reshaped

I tried to mark the new drive as faulty to see if the reshape would stop but to no avail. It works to mark it as failed but nothing happens.

I also tried to run a check instead of a reshape (as I read somewhere this fixed a similar problem) but the device is busy

    # echo check>/sys/block/md0/md/sync_action
    -bash: echo: write error: Device or resource busy

What does this mean? I'm on really thin ice here with no idea what to do so any help is greatly appreciated.

Pretty sure the reboot was not the cause of the problem. It seems to be some problem with the reshape that causes the array to hang. I get these errors in dmesg:

    [  360.625322] INFO: task md0_reshape:126 blocked for more than 120 seconds.
    [  360.625351]       Not tainted 4.0.4-2-ARCH #1
    [  360.625367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    [  360.625394] md0_reshape     D ffff88040af57a58     0   126      2 0x00000000
    [  360.625397]  ffff88040af57a58 ffff88040cf58000 ffff8800da535b20 00000001642a9888
    [  360.625399]  ffff88040af57fd8 ffff8800da429000 ffff8800da429008 ffff8800da429208
    [  360.625401]  0000000096400e00 ffff88040af57a78 ffffffff81576707 ffff8800da429000
    [  360.625403] Call Trace:
    [  360.625410]  [<ffffffff81576707>] schedule+0x37/0x90
    [  360.625428]  [<ffffffffa0120de9>] get_active_stripe+0x5c9/0x760 [raid456]
    [  360.625432]  [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
    [  360.625436]  [<ffffffffa01246e0>] reshape_request+0x5b0/0x980 [raid456]
    [  360.625439]  [<ffffffff81579053>] ? schedule_timeout+0x123/0x250
    [  360.625443]  [<ffffffffa011743f>] sync_request+0x28f/0x400 [raid456]
    [  360.625449]  [<ffffffffa00da486>] ? is_mddev_idle+0x136/0x170 [md_mod]
    [  360.625454]  [<ffffffffa00de4ba>] md_do_sync+0x8ba/0xe70 [md_mod]
    [  360.625457]  [<ffffffff81576002>] ? __schedule+0x362/0xa30
    [  360.625462]  [<ffffffffa00d9e54>] md_thread+0x144/0x150 [md_mod]
    [  360.625464]  [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
    [  360.625468]  [<ffffffffa00d9d10>] ? md_start_sync+0xf0/0xf0 [md_mod]
    [  360.625471]  [<ffffffff81093418>] kthread+0xd8/0xf0
    [  360.625473]  [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
    [  360.625476]  [<ffffffff8157a398>] ret_from_fork+0x58/0x90
    [  360.625478]  [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170

Also, looking at CPU usage md0_raid5 seems to be having problems:

     PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND
     125 root      20   0    0.0m   0.0m 100.0  0.0  35:57.44 R  `- md0_raid5
     126 root      20   0    0.0m   0.0m   0.0  0.0   0:00.06 D  `- md0_reshape

Could this be why the reshape has stopped?

Is it possible to revert to using 3 drives again without losing data?

Offline

#2 2015-06-07 12:53:36

frostschutz
Member
Registered: 2013-11-15
Posts: 1,211

Re: Resize of RAID5 array frozen

Which kernel are you using? `uname -a`

Can you show `mdadm --examine /dev/sd*` if that does not hang?

Is it possible to revert to using 3 drives again without losing data?

Yes, but it's a pain.

Since you already seem to be in a deadlock situation, you might not be able to avoid a reboot. Maybe reboot using a different kernel just in case, or something like the SystemRescueCD, maybe it will be able to resume the operation normally.

Although it may be better not to resume it if you failed one of your drives. If you did that you're looking at a grow operation without redundancy and if it proceeds that way you lose your ability to revert so you can only hope it runs through & none of the remaining disks are bad & you're able to restore redundancy.

Offline

#3 2015-06-07 13:20:29

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

Sure. Here is the info;

Linux ocular 4.0.4-2-ARCH #1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015 x86_64 GNU/Linux

And here is the mdadm examine:

/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4d
     Array UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
           Name : ocular:0  (local to host ocular)
  Creation Time : Sat Mar 28 17:31:15 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
     Array Size : 8790095616 (8382.89 GiB 9001.06 GB)
  Used Dev Size : 5860063744 (2794.30 GiB 3000.35 GB)
    Data Offset : 262144 sectors
     New Offset : 260608 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : b0b8a3c9:9a11499b:65e16567:6d40095e

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 2520575232 (2403.81 GiB 2581.07 GB)
  Delta Devices : 1 (3->4)

    Update Time : Sun Jun  7 14:07:18 2015
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 47269a16 - correct
         Events : 344588

         Layout : left-symmetric
     Chunk Size : 256K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
           Name : ocular:0  (local to host ocular)
  Creation Time : Sat Mar 28 17:31:15 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
     Array Size : 8790095616 (8382.89 GiB 9001.06 GB)
  Used Dev Size : 5860063744 (2794.30 GiB 3000.35 GB)
    Data Offset : 262144 sectors
     New Offset : 260608 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : b42d1b98:db70d93e:a60cd400:03de5f0e

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 2520575232 (2403.81 GiB 2581.07 GB)
  Delta Devices : 1 (3->4)

    Update Time : Sun Jun  7 14:07:18 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 2f3372a - correct
         Events : 344588

         Layout : left-symmetric
     Chunk Size : 256K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
           Name : ocular:0  (local to host ocular)
  Creation Time : Sat Mar 28 17:31:15 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860064143 (2794.30 GiB 3000.35 GB)
     Array Size : 8790095616 (8382.89 GiB 9001.06 GB)
  Used Dev Size : 5860063744 (2794.30 GiB 3000.35 GB)
    Data Offset : 262144 sectors
     New Offset : 260608 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 081357be:722dfa8f:81dbfbea:981e0740

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 2520575232 (2403.81 GiB 2581.07 GB)
  Delta Devices : 1 (3->4)

    Update Time : Sun Jun  7 14:07:18 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 961bc882 - correct
         Events : 344588

         Layout : left-symmetric
     Chunk Size : 256K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4d
     Array UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
           Name : ocular:0  (local to host ocular)
  Creation Time : Sat Mar 28 17:31:15 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860064143 (2794.30 GiB 3000.35 GB)
     Array Size : 8790095616 (8382.89 GiB 9001.06 GB)
  Used Dev Size : 5860063744 (2794.30 GiB 3000.35 GB)
    Data Offset : 262144 sectors
     New Offset : 260608 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : be585918:766c2ef1:b67c083b:de9cac20

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 2520575232 (2403.81 GiB 2581.07 GB)
  Delta Devices : 1 (3->4)

    Update Time : Sun Jun  7 14:07:18 2015
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 82046cc0 - correct
         Events : 344588

         Layout : left-symmetric
     Chunk Size : 256K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

I already tried rebooting again and it resumes from the same position. With the failed disk present again.

Last edited by while (2015-06-07 14:02:26)

Offline

#4 2015-06-07 13:36:27

frostschutz
Member
Registered: 2013-11-15
Posts: 1,211

Re: Resize of RAID5 array frozen

It says "bad blocks present" for /dev/sdb1 and /dev/sde1, can you check SMART data of all disks if there's any issue? mdadm --examine-badblocks /dev/???

Otherwise it does not look like you failed a disk; so, barring any bad block issues, I'd try to reboot and hope it will resume the grow operation normally.

If you actually do have bad blocks it may be the source of your issue. The bad block list is a rather new feature...

Last edited by frostschutz (2015-06-07 13:38:26)

Offline

#5 2015-06-07 14:18:26

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

Seems to be badblocks as you said.  Here is the output:

Bad-blocks on /dev/sdb1:
          1680781608 for 408 sectors
          1683782656 for 8 sectors
          1688501712 for 512 sectors
          1688502224 for 240 sectors
          4315347104 for 128 sectors
          5729057784 for 128 sectors
          5729063200 for 32 sectors
          5729065008 for 8 sectors
          5729066808 for 128 sectors
          5729074040 for 8 sectors
          5729075840 for 8 sectors
          5729077648 for 128 sectors
          5729081376 for 8 sectors
          5729083176 for 312 sectors
          5729084984 for 32 sectors
          5729086784 for 512 sectors
          5729087296 for 64 sectors
          5729088592 for 512 sectors
          5729089104 for 80 sectors
          5729090400 for 88 sectors
          5729092200 for 16 sectors
          5729094008 for 8 sectors
          5729095808 for 512 sectors
          5729096320 for 512 sectors
          5729096832 for 72 sectors
          5729097616 for 384 sectors
          5729099424 for 8 sectors
          5729101224 for 128 sectors
          5729103032 for 16 sectors
          5729104840 for 512 sectors
          5729105352 for 512 sectors
          5729105864 for 184 sectors
          5729106648 for 8 sectors
          5729110376 for 512 sectors
          5729110888 for 504 sectors
          5729112184 for 16 sectors
          5729113992 for 112 sectors
          5729115792 for 128 sectors
          5729117600 for 128 sectors
          5729119408 for 512 sectors
          5729119920 for 112 sectors
          5729121208 for 128 sectors
          5730507008 for 8 sectors
          5730508816 for 128 sectors
          5730510616 for 40 sectors
          5730512424 for 8 sectors
          5730514224 for 128 sectors
          5730517968 for 112 sectors
          5730518792 for 512 sectors
          5730519304 for 512 sectors
          5730519816 for 320 sectors
          5730520600 for 8 sectors
          5730521584 for 176 sectors
          5730536016 for 8 sectors
          5730537824 for 128 sectors
          5730539624 for 8 sectors
          5730543240 for 8 sectors
          5730550584 for 8 sectors
          5730552384 for 128 sectors
          5730553208 for 8 sectors
          5730554192 for 16 sectors
          5730555016 for 8 sectors
          5730556816 for 128 sectors
          5730557800 for 8 sectors
          5730558624 for 72 sectors
          5730559608 for 440 sectors
          5730561408 for 144 sectors
          5730562360 for 8 sectors
          5730563216 for 512 sectors
          5730563728 for 8 sectors
          5730598592 for 16 sectors
          5730600392 for 512 sectors
          5730600904 for 512 sectors
          5730601416 for 88 sectors
          5730604008 for 8 sectors
          5730605800 for 336 sectors
          5730607608 for 128 sectors
          5730608592 for 512 sectors
          5730609104 for 32 sectors
          5730609408 for 128 sectors
          5730610400 for 32 sectors
          5730611216 for 128 sectors
          5730613024 for 24 sectors
          5730614008 for 8 sectors
          5730614824 for 128 sectors
          5730615816 for 32 sectors
          5730616632 for 512 sectors
          5730617144 for 248 sectors
          5730617616 for 8 sectors
          5730618440 for 72 sectors
          5730620248 for 512 sectors
          5730620760 for 128 sectors
          5730621224 for 128 sectors
          5730623976 for 128 sectors
          5730625784 for 512 sectors
          5730626296 for 160 sectors
          5730627584 for 512 sectors
          5730628096 for 512 sectors
          5730628608 for 48 sectors
          5730629392 for 24 sectors
          5730631200 for 144 sectors
          5730633008 for 128 sectors
          5730634808 for 256 sectors
          5730636616 for 384 sectors
          5730638424 for 120 sectors
          5730640224 for 128 sectors
          5730642032 for 184 sectors
          5730643840 for 512 sectors
          5730644352 for 48 sectors
          5730649280 for 8 sectors
          5730652992 for 8 sectors
          5730653016 for 104 sectors
          5730685624 for 8 sectors 
Bad-blocks on /dev/sde1:
          1680781608 for 408 sectors
          1683782656 for 8 sectors
          1688501712 for 512 sectors
          1688502224 for 240 sectors
          4315347104 for 128 sectors
          5729057784 for 128 sectors
          5729063200 for 32 sectors
          5729065008 for 8 sectors
          5729066808 for 128 sectors
          5729074040 for 8 sectors
          5729075840 for 8 sectors
          5729077648 for 128 sectors
          5729081376 for 8 sectors
          5729083176 for 312 sectors
          5729084984 for 32 sectors
          5729086784 for 512 sectors
          5729087296 for 64 sectors
          5729088592 for 512 sectors
          5729089104 for 80 sectors
          5729090400 for 88 sectors
          5729092200 for 16 sectors
          5729094008 for 8 sectors
          5729095808 for 512 sectors
          5729096320 for 512 sectors
          5729096832 for 72 sectors
          5729097616 for 384 sectors
          5729099424 for 8 sectors
          5729101224 for 128 sectors
          5729103032 for 16 sectors
          5729104840 for 512 sectors
          5729105352 for 512 sectors
          5729105864 for 184 sectors
          5729106648 for 8 sectors
          5729110376 for 512 sectors
          5729110888 for 504 sectors
          5729112184 for 16 sectors
          5729113992 for 112 sectors
          5729115792 for 128 sectors
          5729117600 for 128 sectors
          5729119408 for 512 sectors
          5729119920 for 112 sectors
          5729121208 for 128 sectors
          5730507008 for 8 sectors
          5730508816 for 128 sectors
          5730510616 for 40 sectors
          5730512424 for 8 sectors
          5730514224 for 128 sectors
          5730517968 for 112 sectors
          5730518792 for 512 sectors
          5730519304 for 512 sectors
          5730519816 for 320 sectors
          5730520600 for 8 sectors
          5730521584 for 176 sectors
          5730536016 for 8 sectors
          5730537824 for 128 sectors
          5730539624 for 8 sectors
          5730543240 for 8 sectors
          5730550584 for 8 sectors
          5730552384 for 128 sectors
          5730553208 for 8 sectors
          5730554192 for 16 sectors
          5730555016 for 8 sectors
          5730556816 for 128 sectors
          5730557800 for 8 sectors
          5730558624 for 72 sectors
          5730559608 for 440 sectors
          5730561408 for 144 sectors
          5730562360 for 8 sectors
          5730563216 for 512 sectors
          5730563728 for 8 sectors
          5730598592 for 16 sectors
          5730600392 for 512 sectors
          5730600904 for 512 sectors
          5730601416 for 88 sectors
          5730604008 for 8 sectors
          5730605800 for 336 sectors
          5730607608 for 128 sectors
          5730608592 for 512 sectors
          5730609104 for 32 sectors
          5730609408 for 128 sectors
          5730610400 for 32 sectors
          5730611216 for 128 sectors
          5730613024 for 24 sectors
          5730614008 for 8 sectors
          5730614824 for 128 sectors
          5730615816 for 32 sectors
          5730616632 for 512 sectors
          5730617144 for 248 sectors
          5730617616 for 8 sectors
          5730618440 for 72 sectors
          5730620248 for 512 sectors
          5730620760 for 128 sectors
          5730621224 for 128 sectors
          5730623976 for 128 sectors
          5730625784 for 512 sectors
          5730626296 for 160 sectors
          5730627584 for 512 sectors
          5730628096 for 512 sectors
          5730628608 for 48 sectors
          5730629392 for 24 sectors
          5730631200 for 144 sectors
          5730633008 for 128 sectors
          5730634808 for 256 sectors
          5730636616 for 384 sectors
          5730638424 for 120 sectors
          5730640224 for 128 sectors
          5730642032 for 184 sectors
          5730643840 for 512 sectors
          5730644352 for 48 sectors
          5730649280 for 8 sectors
          5730652992 for 8 sectors
          5730653016 for 104 sectors
          5730685624 for 8 sectors

The other two seems to have no bad blocks.

I already tried rebooting again and it resumes from the same position (still stick in the same position). With the failed disk present again. That's why its back.

What does this imply? Are the disks in bad shape? One of them is new and the other is old (only a few months though) and they are of different fabrications so it seems strange to me that both would be broken.

Last edited by while (2015-06-07 14:19:06)

Offline

#6 2015-06-07 14:23:13

frostschutz
Member
Registered: 2013-11-15
Posts: 1,211

Re: Resize of RAID5 array frozen

The bad blocks list looks identical for both so I think it got cloned for some reason or other.

I don't have much experience with the bad block log, but I think it likely you hit a bug of some kind. Taking this bad blocks at face value your RAID would be broken because with two disks having bad blocks in the same positions your RAID would be dead [for those sectors] since you only have one disk redundancy.

Did you check SMART? smartctl -a for all disks and if you didn't run a self-test recently do a smartctl -t long and see if the disks pass.

Offline

#7 2015-06-07 14:37:27

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

No, I didn't even have it installed. I'm running the tests now on both disks.  Ill post the result as soon as its done.

Offline

#8 2015-06-07 20:17:24

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

Takes a lont time to run the extended tests. Here is the results:

sdb

smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.0.4-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1ER166
Serial Number:    Z500TQ59
LU WWN Device Id: 5 000c50 07a00147c
Firmware Version: CC25
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Jun  7 22:14:26 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (  25)	The self-test routine was aborted by
					the host.
Total time to complete Offline 
data collection: 		(   89) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 342) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x1085)	SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       216517664
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       15
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       538608
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       37
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       15
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   052   044   045    Old_age   Always   In_the_past 48 (Min/Max 43/50 #240)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       1
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       68
194 Temperature_Celsius     0x0022   048   056   000    Old_age   Always       -       48 (0 27 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       31h+24m+45.066s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       7774670316
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2541026610

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        37         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sdc

smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.0.4-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1ER166
Serial Number:    Z500TPSE
LU WWN Device Id: 5 000c50 079ffde12
Firmware Version: CC25
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jun  8 07:47:08 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(   89) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 333) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x1085)	SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   099   006    Pre-fail  Always       -       9204656
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       14
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.0.4-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1ER166
Serial Number:    Z500TPSE
LU WWN Device Id: 5 000c50 079ffde12
Firmware Version: CC25
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jun  8 07:47:08 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(   89) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 333) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x1085)	SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   099   006    Pre-fail  Always       -       9204656
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       14
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       125014
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       46
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       14
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   098   098   000    Old_age   Always       -       2
190 Airflow_Temperature_Cel 0x0022   052   046   045    Old_age   Always       -       48 (Min/Max 45/54)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       124
194 Temperature_Celsius     0x0022   048   054   000    Old_age   Always       -       48 (0 26 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       30h+24m+05.235s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1682280669
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1017214

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        42         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sdd

smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.0.4-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0E8VXW6
LU WWN Device Id: 5 0014ee 6afab7ec7
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jun  8 07:49:46 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39120) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 393) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   186   182   021    Pre-fail  Always       -       5691
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1681
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       22
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       556
194 Temperature_Celsius     0x0022   116   095   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1679         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sde

smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.0.4-2-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0F55REK
LU WWN Device Id: 5 0014ee 6afacdc98
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jun  8 07:51:08 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39540) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 397) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   182   177   021    Pre-fail  Always       -       5900
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1680
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       22
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       556
194 Temperature_Celsius     0x0022   116   091   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1672         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

All seems to be fine as far as I can see.

Last edited by while (2015-06-08 05:53:55)

Offline

#9 2015-06-08 19:01:38

frostschutz
Member
Registered: 2013-11-15
Posts: 1,211

Re: Resize of RAID5 array frozen

Right. Not sure what else can be done here, if the grow does not resume in either ArchLinux or SystemRescueCD.

The painful method: https://raid.wiki.kernel.org/index.php/ … erlay_file

Use that to create two independent sets of overlays. Like snapshots they will let you play with data without actually changing anything on the disks (if you're careful to use the overlays exclusively).

One set will represent your RAID after the grow, and on it you will find the first 28% (or whatever) of your data; the other will represent the RAID before the grow and it holds the remaining 72%. There will be a certain overlap zone that should be identical on both, since the RAID-after-grow has more capacity (spread over 4 disks instead of 3) so if it already holds 28% of the data, it won't have overwritten 28% of the old representation. So you might also be able to merge at 25%/75%. Don't take these values literally, I'm not 100% sure how to interpret the grow progress bar or reshape pos'n.

Commands to create those two arrays should roughly look like this (you might have to adapt these):

mdadm --create /dev/md44 --assume-clean --metadata=1.2 --level=5 --chunk=256 --layout=ls --data-offset=260608 --raid-devices=4 /disk/overlayA_sdb1 /disk/overlayA_sdd1 /disk/overlayA_sde1 /disk/overlayA_sdc1
mdadm --create /dev/md33 --assume-clean --metadata=1.2 --level=5 --chunk=256 --layout=ls --data-offset=262144 --raid-devices=3 /disk/overlayB_sdb1 /disk/overlayB_sdd1 /disk/overlayB_sde1

You have to get the settings (metadata version, offset, chunksize, disk order, etc.) entirely correct, please check if I understood the examine output correctly. Was sdc1 the disk you added in the grow operation?

With this /dev/md33 should represent your old RAID with 3 disks and /dev/md44 is the new one with 4 disks.

If that theory is correct there should be identical data to be found around the reshape position (2403.81GiB):

dd if=/dev/md44 bs=1M skip=$((2400*1024)) count=1 | hexdump -C
dd if=/dev/md44 bs=1M skip=$((2400*1024)) count=1 | md5sum
dd if=/dev/md33 bs=1M skip=$((2400*1024)) count=1 | md5sum

The hexdump is to check that it's not zero but more distinct/random looking data. Zeroes you can find anywhere so that's not good to go by.

If everything worked out so far you can get the whole device using device mapper linear.

cutoff=$((2400*1024*1024*1024/512))
total=$(blockdev --getsz /dev/md33)
dmsetup create md44md33 --table "0 $cutoff linear 9:44 0
$cutoff $total linear 9:33 $cutoff"

And then see if /dev/mapper/md44md33 is mountable and check as many files as you can (large files physically located before and after the cutoff point).

That's the theory anyways. It might take a few tries to get it right.

Offline

#10 2015-06-08 22:38:56

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

This is awesome. Thank you. I will try it if my current approach is not working.

I got a tip in the linux-raid mailing list that this could be a bug related to badblocks in the kernel. I applied a patch to fix it and am waiting for the kernel to compile now. Ill report back as soon as ive tried it out.

Offline

#11 2015-06-09 07:09:22

while
Member
Registered: 2010-05-14
Posts: 35

Re: Resize of RAID5 array frozen

Seems like the kernel fix did it. I patched the kernel using the following patch:

--- a/drivers/md/raid5.c        2015-06-08 23:05:02.808214213 +0200
+++ b/drivers/md/raid5.c        2015-06-08 23:05:47.601355604 +0200
@@ -3855,7 +3855,7 @@
         */
        if (s.failed > conf->max_degraded) {
                sh->check_state = 0;
-               sh->reconstruct_state = 0;
+               //sh->reconstruct_state = 0;
                if (s.to_read+s.to_write+s.written)
                        handle_failed_stripe(conf, sh, &s, disks, &s.return_bi);
                if (s.syncing + s.replacing)

After rebooting the reshape is continuing fine.

Offline

Board footer

Powered by FluxBB