Booting Windows 7 corrupts external Western Digital HDD if GPT is used

vk · 2015-06-14 15:27:10

I have recently formatted an old 2TB external HDD (Western Digital, "Green") into ext4 under Arch Linux (using GPT as its partitioning scheme). I've been using this HDD for backups for some time, when all of a sudden it stopped mounting. lsblk would no longer list its partition (sdh1), only the disk itself, as if it were unpartitioned. I have googled how to fix the problem using gdisk. gdisk says this about the HDD:

~ ❯❯❯ sudo gdisk /dev/sdh
GPT fdisk (gdisk) version 1.0.0
Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): v
Problem: main GPT header's backup LBA pointer (3907029167) doesn't
match the backup GPT header's current LBA pointer (3907024895).
The 'e' option on the experts' menu may fix this problem.
Problem: main GPT header's last usable LBA pointer (3907029134) doesn't
match the backup GPT header's last usable LBA pointer (3907024862)
The 'e' option on the experts' menu can probably fix this problem.
Problem: Disk is too small to hold all the data!
(Disk size is 3907024896 sectors, needs to be 3907029168 sectors.)
The 'e' option on the experts' menu may fix this problem.
Problem: GPT claims the disk is larger than it is! (Claimed last usable
sector is 3907029134, but backup header is at
3907029167 and disk size is 3907024896 sectors.
The 'e' option on the experts' menu will probably fix this problem
Identified 4 problems!

Using expert commands 'e' followed by 'w', the partition was restored, but I wanted to figure out the reason for this corruption. Turns out that each time I boot into Windows 7 (it's on a separate internal drive), the poor external HDD would be corrupted. While it's not difficult to rerun the gdisk commands again, I would prefer to not have the problem in the first place. Apart from never booting into Windows or physically disconnecting the external HDD beforehand, does anyone have any ideas on how to prevent this? Also, what may be the cause?

Some notes:
- I don't have to log into a user account under Windows to cause corruption, waiting until the login screen shows up is enough;
- Arch itself is installed to an internal SSD formatted to ext4. Windows doesn't corrupt it, thank God, so maybe it has something to do with the fact that it's an external HDD;
- when Windows boots, there is a long (~15 seconds) pause between when the Windows logo disappears and the login screen appears. During this pause, there is no output to the monitor. This pause only happens if this external HDD is connected. At the end of the pause, there is a short period of disk activity on the HDD (indicated by a LED), so I would assume it's during this moment that corruption happens;
- here's also the output of parted regarding the HDD:

~ ❯❯❯ sudo parted /dev/sdh
GNU Parted 3.2
Using /dev/sdh
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Error: Invalid argument during seek for read on /dev/sdh
Retry/Ignore/Cancel? I
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK
Model: WD Ext HDD 1021 (scsi)
Disk /dev/sdh: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Last edited by vk (2015-06-25 17:05:33)

ewaller · 2015-06-14 16:27:56

vk wrote:

never booting into Windows

This, of course, is the canonical solution

You might look at your boot loader and see if it has the ability to hide a drive. Grub has that ability. I know it used to work on BIOS system, I've no idea if it works with recent excretions from Redmond using EFI.

Last edited by ewaller (2015-06-14 16:28:17)

ewaller · 2015-06-14 16:34:45

In Windows, you might also try disabling the drive in the device manager, or you might use the usb drive letter manager and configure the drive not to mount.

Oh, by the way, Welcome to Arch Linux

rlees85 · 2015-06-14 16:46:47

I think I know your problem. I have had this before. I partitioned and formatted a disk under Linux attached via USB. Whenever I would then go and attach it to Windows or even Linux but without the USB adapter (direct to SATA bus) it would corrupt.

~ ❯❯❯ sudo parted /dev/sdh
GNU Parted 3.2
Using /dev/sdh
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Error: Invalid argument during seek for read on /dev/sdh
Retry/Ignore/Cancel? I
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK
Model: WD Ext HDD 1021 (scsi)
Disk /dev/sdh: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

A 2TB Western Digital that is NOT advanced format seems very unlikely. Does your external enclosure support eSATA instead? Some USB controllers do not properly report physical sector sizes - causing corruption.

Last edited by rlees85 (2015-06-14 16:47:04)

dcdriving · 2015-06-14 17:13:55

Further to previous answers, try having a look at disabling chkdsk from running when windows boots. This is my memory going way back I'm afraid though but there may be an issue where windows sees the drive as "dirty" and as it doesn't fix it, never sets the "dirty bit" back again so tries to fix it each boot. Perhaps linux fsck would set the same bit to clean and stop windows from trying to repair it but don't know enough about it? Make sure you have a backup of that backup though!

Its just a thought but look at something along these lines perhaps http://answers.microsoft.com/en-us/wind … d7131ec643

vk · 2015-06-15 12:41:45

Thanks for the suggesions, everyone.

rlees85 wrote:

A 2TB Western Digital that is NOT advanced format seems very unlikely. Does your external enclosure support eSATA instead? Some USB controllers do not properly report physical sector sizes - causing corruption.

According to this http://superuser.com/questions/443435/h … -hard-disk answer, it's a "feature" of WD external drives. It must have something to do with the corruption, but it can't be the sole reason. I have a second external HDD, completely the same in every regard, but formatted into NTFS (EDIT: I was wrong, it was also MBR instead of GPT, which is the reason why it worked). It, too, reports its sector size as 512B, but never becomes corrupted.

dcdriving wrote:

try having a look at disabling chkdsk from running when windows boots.

Unfortunately, it doesn't seem to be because of chkdsk. Turning it off doesn't help, and also chkdsk is supposed to write some output to log files if it runs, but there are no mentions of chkdsk recently.

ewaller wrote:

You might look at your boot loader and see if it has the ability to hide a drive. Grub has that ability.

It seems to be the best suggestion right now, thanks.

ewaller wrote:

Oh, by the way, Welcome to Arch Linux

Thanks!

Wish I understood what the hell Windows does to the drive... Judging from gdisk output, it seems that all the pointers/sizes in main GPT header acquire additional offset by 4272 sectors, so that the total drive size becomes larger than it is.

Last edited by vk (2015-06-25 17:09:23)

rlees85 · 2015-06-15 15:34:24

His output:

Sector size (logical/physical): 512B/4096B

Vs. Your output:

Sector size (logical/physical): 512B/512B

What make / model is your other drive? And the output of that command?

vk · 2015-06-15 16:14:40

The problematic drive, after fixing it with gdisk:

Model: WD Ext HDD 1021 (scsi)
Disk /dev/sdh: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2000GB 2000GB ext4

The other ("good") external drive:

Model: WD Ext HDD 1021 (scsi)
Disk /dev/sdi: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary ntfs

(Btw, it's probably unrelated, but gdisk issues warnings for this NTFS disk, saying that "Secondary partition table overlaps the last partition by 33 blocks!" However, I experienced no problems using it)

They're both external Western Digital "Elements", 2TB, bought around the same time in 2012, I think.

I assume that if I were to format the "good" drive to ext4, Windows would start corrupting it, too, because the "problematic" drive was a NTFS drive in its previous life, and I could use it fine both in Arch and in Windows.

Last edited by vk (2015-06-15 16:18:09)

frostschutz · 2015-06-15 16:25:19

parted /dev/sdh toggle 1 lvm

Linux doesn't care much about the partition type but you should pick one that Windows tends to ignore.

Apart from that make sure no stray filesystem headers are left that could get misinterpreted.

Last edited by frostschutz (2015-06-15 16:26:54)

rlees85 · 2015-06-15 17:28:55

I notice your start point of 1049kb is weird too... normally partitions should be aligned at 1mb (1024kb, or 2048 sectors).Not sure if that has anything to do with it (shouldn't do, in theory). I think if it was me, I would try finding out for sure if the drives are advanced format or not. If they are, then I am sure this is the problem. Linux is clearly thinking that they are not. Windows may have a different opinion and will cause the exact problems your having.

If they are indeed not advanced format, I would re-create the partition, using something like FDISK (if you want mbr) or GDISK (if you want gpt). I would make sure that the first partition is aligned at 1024kb/2048 sectors (default in both tools). Also make sure that the partition type is set correctly.

Of course they maybe a way to fix this "not" properly that may work... so I am just saying if it were me. If none of the above work I am all out its a weird one!

nomorewindows · 2015-06-15 18:43:56

Windows 95 always wanted to format unreadable floppy disks, maybe it is an automatic "feature" in Windows 7.

frostschutz · 2015-06-15 19:07:13

rlees85 wrote:

I notice your start point of 1049kb is weird too...

For some weird reason parted uses power of 1000 by default so 1MiB comes down to 1049kb by parted's reckoning.

parted /dev/disk unit mib print free

or

parted /dev/disk unit s print free

vk · 2015-06-25 17:02:04

An update:
I experimented with it a little bit and found that the filesystem doesn't matter at all. What actually triggers Windows to go crazy is the GPT on the disk. It's enough to create GPT (doesn't matter if it's created using parted or gdisk) without even creating any partitions, and the corruption will occur. Not sure what is the other trigger, whether it's the fact that it's a USB drive, the fact that WD drives purposefully lie about their sector size, or that the drive is SCSI. Anyway, it seems that ext4 with a MBR works just fine. I read a bit on GPT's advantages over MBR, and it seemed to me that most of them aren't important for my use case, because I don't intend to boot from this drive, and I'm only going to create a single partition. So for now, I decided to go the easy way and just stick with MBR. Also, the saddening thing is that even if I found exactly what causes the Windows bug, I don't believe for a second that it is actually ever going to be fixed, I guess the bug would just live on the Microsoft's bug tracker forever.

Last edited by vk (2015-06-25 17:10:20)

frostschutz · 2015-06-25 18:34:08

If there is corruption, it means Windows is writing - when it doesn't have any business writing to an unpartitioned disk without asking the user.

It's odd that you have this issue though, I've used GPT and Windows dual boot before and did not have issues (after setting the correct partition types as Windows ignores). And all my disks are WD...

If you have time to go into this deeper, can you tell us what the corruption looks like? Like make an image of the first/last 100MB of the disk before rebooting into windows and then afterwards cmp -l or hexdump -C | diff -u with the corrupted data?

It would be interesting if it was a malware signature...

The only thing I can think of with GPT is that it sadly depends on sector sizes so the on disk format of GPT looks different for a 4K sector disk than it does for a 512 byte sector disk, so if the OS detects the sector size wrong it would do odd things to the partition table. MSDOS has weakness there as well, it's too bad they did not make a proper solution for GPT...

vk · 2015-06-25 18:54:18

frostschutz wrote:

If there is corruption, it means Windows is writing - when it doesn't have any business writing to an unpartitioned disk without asking the user.
It's odd that you have this issue though, I've used GPT and Windows dual boot before and did not have issues (after setting the correct partition types as Windows ignores). And all my disks are WD...
If you have time to go into this deeper, can you tell us what the corruption looks like? Like make an image of the first/last 100MB of the disk before rebooting into windows and then afterwards cmp -l or hexdump -C | diff -u with the corrupted data?
It would be interesting if it was a malware signature...
The only thing I can think of with GPT is that it sadly depends on sector sizes so the on disk format of GPT looks different for a 4K sector disk than it does for a 512 byte sector disk, so if the OS detects the sector size wrong it would do odd things to the partition table. MSDOS has weakness there as well, it's too bad they did not make a proper solution for GPT...

Are your WD drives internal or external? Thanks for the suggestion about creating images, I'll give it a try, I'm curious about the result.

Last edited by vk (2015-06-26 08:44:32)

vk · 2015-06-26 18:00:44

Ok, so I've done the disk dumping/diffing. I compared the first 5 MB and the last 5MB. The last 5MB are the same, which means that Windows only touches the beginning (and it's consistent with how gdisk can fix the issue by restoring the table from the backup; IIUC, the backup is at the end of the drive, correct me if I'm wrong). 8 bytes are changed at the beginning of the drive. The output of cmp -l:

529 121 171
530 340 351
531 231 65
532 153 254
545 377 257
546 167 210
561 336 216
562 167 210

Output of hexdump -C after diffing:

Original GPT created from Arch:

00000210 51 e0 99 6b 00 00 00 00 01 00 00 00 00 00 00 00 |Q..k............|
00000220 ff 77 e0 e8 00 00 00 00 22 00 00 00 00 00 00 00 |.w......".......|
00000230 de 77 e0 e8 00 00 00 00 3a 73 eb 6e 4d b0 08 45 |.w......:s.nM..E|

GPT after corruption by Windows:

00000210 79 e9 35 ac 00 00 00 00 01 00 00 00 00 00 00 00 |y.5.............|
00000220 af 88 e0 e8 00 00 00 00 22 00 00 00 00 00 00 00 |........".......|
00000230 8e 88 e0 e8 00 00 00 00 3a 73 eb 6e 4d b0 08 45 |........:s.nM..E|

So, the byte transformations are (linux->windows):

51 e0 99 6b -> 79 e9 35 ac
ff 77 e0 e8 -> af 88 e0 e8
de 77 e0 e8 -> 8e 88 e0 e8

If I understand the conversion correctly, in decimal it is:

1'805'246'545 -> 2'889'214'329 (+1'083'967'784)
3'907'024'895 -> 3'907'029'167 (+4'272)
3'907'024'862 -> 3'907'029'134 (+4'272)

which is consistent with what gdisk reported (that drive is larger than it should be by 4272 sectors; not sure about the meaning of the big number).

EDIT: I checked with Wikipedia; the first modified sequence of bytes is CRC32 of the header (so no wonder the difference is huge), and the next two are "Current LBA (location of this header copy)" and "Backup LBA (location of the other header copy)". So it seems that all Windows does is increase pointers to current and backup LBA by 4272 and then modify the checksum. Where it gets this 4272 number from, damned if I know.

Last edited by vk (2015-06-26 19:29:35)

vk · 2015-06-27 12:39:21

I was thinking about the idea that the problem is, like several people suggested, fundamentally triggered by the WD's "clever" decision to lie about the sector size. It may be a bit of a stretch, because the sizes aren't exactly the same, but anyway, the difference between the actual sector size and the reported size is 4096 / 512 = 8. And if we divide the magical number 4272 by this difference, the result is 534. The bytes modified are in 522-569 range. Not an exact match, but still, the number 534 fits nicely into the range. So maybe Windows decides that the offsets are wrong because it somehow sees the actual sector size, and decides to modify the offsets to reflect the "real" position?

vk · 2015-06-27 18:55:28

Some more experiments:
- if under Linux I create GPT, partition the drive and format it to NTFS, Windows can see and use the drive after it "fixes" (corrupts) it;
- if I create MBR under Linux and try to convert the drive to GPT under Windows, it will fail after a while with a nonspecific error message like "Unable to find the requested sector" (or something to this effect, my Windows isn't in English);
- Western Digital has a Windows/Mac-only dedicated tool for formatting their external drives. It always fails for this drive, which is just pathetic;
- Windows always sees the drive as a local drive, even though it's external. When the drive still had its factory formatting, it was correctly recognized as external;
- if MBR is created for this drive, gdisk complains that

Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
Identified 1 problems!

Nevertheless, the drive seems to mount/work okay.

Last edited by vk (2015-06-27 18:57:52)

vk · 2015-06-30 08:46:58

Ok, seems like I'll have to go the MBR route after all. Thanks for suggestions, everybody.

severach · 2015-07-01 07:06:00

Windows 95 formatting bad disks was a horrible feature. Fortunately we had write protect notches to curb this bad behavior.

Simple suggestions:

Create the GPT partition in Windows with diskpart and see if Linux can format and use it? Does Windows create the partition with different numbers? Does GPT fdisk for Windows create the partition with the same numbers?

GPT partition 50% of the disk with Linux. If Windows doesn't mess with it you should be able to find a reasonable over provision percentage that doesn't discard much free space and keeps Windows away.

gdisk recovery option (h) hybrid MBR.

How to correct 512-byte sector MBR on a 4096-byte sector disk?

vk · 2015-07-02 17:24:22

severach wrote:

Windows 95 formatting bad disks was a horrible feature. Fortunately we had write protect notches to curb this bad behavior.
Simple suggestions:
Create the GPT partition in Windows with diskpart and see if Linux can format and use it? Does Windows create the partition with different numbers? Does GPT fdisk for Windows create the partition with the same numbers?
GPT partition 50% of the disk with Linux. If Windows doesn't mess with it you should be able to find a reasonable over provision percentage that doesn't discard much free space and keeps Windows away.
gdisk recovery option (h) hybrid MBR.
How to correct 512-byte sector MBR on a 4096-byte sector disk?

Thanks for suggestions, I'll try them.

Arch Linux

#1 2015-06-14 15:27:10

Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#2 2015-06-14 16:27:56

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#3 2015-06-14 16:34:45

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#4 2015-06-14 16:46:47

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#5 2015-06-14 17:13:55

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#6 2015-06-15 12:41:45

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#7 2015-06-15 15:34:24

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#8 2015-06-15 16:14:40

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#9 2015-06-15 16:25:19

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#10 2015-06-15 17:28:55

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#11 2015-06-15 18:43:56

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#12 2015-06-15 19:07:13

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#13 2015-06-25 17:02:04

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#14 2015-06-25 18:34:08

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#15 2015-06-25 18:54:18

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#16 2015-06-26 18:00:44

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#17 2015-06-27 12:39:21

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#18 2015-06-27 18:55:28

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#19 2015-06-30 08:46:58

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#20 2015-07-01 07:06:00

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

#21 2015-07-02 17:24:22

Re: Booting Windows 7 corrupts external Western Digital HDD if GPT is used

Board footer