Fixing Sun Oracle (HGST) 8TB drives reporting 7.86TB

Resources Hardware
1

Preface

This guide provides general guidance and is based on my Unraid setup. Specific commands for tools, time frame and final settings depend on your setup. Following this guide will require a low level format of your hard drive. You will need a 4K compatible SAS controller. Once formatted the drive will only be compatible with other 4K sector drives in traditional RAID (TrueNAS, Unraid cache), but will mix fine with other 512 formatted drives in Unraid parity pools. Proceed with caution with any low level format as you there is a (low) risk of bricking the drive. Always backup your data before proceeding with any maintenance.

Introduction

Recently 8TB drives have hit a sweet spot in pricing, touching the magical $10/TB. Many are are 8TB HGST SAS drives from Sun/Oracle Setups.

I recently purchased five 8TB drives when they were on sale for $80 each. I threw them in my test Unraid box and they work fine, but only reported 7.86TB full capacity BEFORE any format. My “new” 8TB Drives were really 7.86TB drives.

As they were, the drives worked fine, Unraid was happy to use them as 7.86TB hard drives and I could have been on my way, but I had a problem. I wanted to use the two of the HGST drives as parity, and then blend in the remainder of the HGST drives along with some 8TB SATA drives I have. Unraid did not like this. As the SATA drives reported a full 8TB RAW capacity, the HGST drives could not be used a parity. Well, poop.

Identifying the Challenge

In Unraid, drives should report their RAW storage size(3TB, 300GB, etc), as shown across the interface anywhere SIZE is displayed. Right away, I noticed my HGST 8TB drives showed 7.9TB (rounded to two digits).

When I looked at the attributes of the drive, I saw this:

User Capacity: 7,865,536,647,168 bytes [7.86 TB]

This is not correct for a full 8TB drive as it should report the following:

User capacity: 8,001,563,222,016 bytes [8.00 TB]

Something was going on.

What Causes the Challenge

After reading the Interwebs, I found this is a common challenge with some (not all) Sun/Oracle HGST drives of 8TB flavor. As received from eBay seller, the HGST drives were all formatted at 512e drives. Good, I should have a full 8TB RAW space. No, not the case. Seems these drives have special firmware. When a 512e format is applied, they work just fine, but have some sort of parity bits applied. This accounts for the missing 140GB of space.

Hope for a Full 8TB

As the parity built into the drives firmware was taking 140GB, I first thought I may just be able to remove that parity protections. The HGST drives I received did have protection enabled, but removing it did not change space. No amount of formatting with 512e restored the full drive space.

After more reading, found this special parity feature of the drives firmware does not seem to apply when the drive is formatted as 4K. This ultimately worked.

How to Get back those Missing GBs

To format a drive from 512 to 4096 (4K) sectors, you will need a few things:

General Procedure

  • Confirm SG3 Utils is is installed on your formatting server.
  • Identify a drives current size reporting with command: sg_readcap /dev/sXX (whre sXX is the targeted device on your system)
  • Confirm the misreported drive space on the 8TB drive and the drive is formatted ad 512. Will look something like this:

User Capacity: 7,865,536,647,168 bytes [7.86 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes

  • Run command: sg_format --format --size=4096 /dev/sXX
  • Wait many, many hours (12+)
  • When format is complete, shutdown then power up server.
  • Check capacity again: sg_readcap /dev/sXX
  • Check UnRaid reporting of drive space:

Vendor: HGST
Product: H7280A520SUN8.0T
User capacity: 8,001,563,222,016 bytes [8.00 TB]
Logical block size: 4096 bytes

Considerations and Wrap Up

I was able to restore a full 8TB RAW data to my five drives with a 4K format and my Unraid is happy.
Note, this will not apply to all HGST 8TB drives, so make sure to check capacities when testing your eBay purchases.

Also, I only recommend you do this if you really, really want your full 8TB of space (I did). As received, my 8TB drives would have been happy working at 7.86TB drives in Unraid. There is a chance you could goof up your drive on a bad format, so proceed with caution.

I you do decide to try this, please consult the Internet and feel free to message me in this forum or on Discord.

And, as always, check your backups.

5 Likes
2

Just a heads up, you definitely do not need to change sector size to resolve this. The overhead is caused by Protection Information, which is several bytes that follow each block. Reformatting with a different sector size without a flag setting the PI type means you incidentally disabled it and formatted the whole disk - the default behavior of a format with sg_format is no PI.

The resize flag in sg_format is a non-destructive means of fixing this if the drive does not currently have protection information appended to sectors, but still reports a touch low (indicated in SMART report). You’ll then need to grow the FS.

If it does have PI, you can then reformat without PI (but maintaining the same sector size) to get full use of the drive’s actual capacity.

1 Like
3

So that would be why my 10TB SAS drive only reports as 9.79TB and states “formatted with type 1 protection, 8 bytes of protection information per logical block” in the smart data on unraid.

image
image784Ă—283 16.1 KB

Can a resize be done in SG3 if the drive is currently running in a parity protected unraid? or would the best route to take be to move data off the drive and then format it without PI?

4

Unfortunately the only way to get rid of PI is to format, sg_format --resize --count=-1 would only work if it’s not formatted with protection information but still underreports size. You would have to rebuild parity after growing the FS following this operation.

Default sg_format --format /dev/xyz behavior is to use the same block size and to disable PI, though you need to include explicit block size with --size as well. However, it’s unbelievably slow on huge disks - you can and should try adding --ffmt=1, which is a fast format option supported by HGST SAS drives.

After you’ve got your data safe, give sg_format --format --ffmt=1 --quick --wait /dev/<driveID> a shot.

1 Like
5

Like I said in my original post, not all HGST drives need this drastic a measure, and I was offering my experience for those at SB that may have gotten in on the recent 8TB deal (sure there were a few).

Many Sun/Oracle/EMC drives seem to have PI turned on, and need it disabled to claw back the space, while sticking with a 512 format. Learned a whole bunch of cool stuff about Protection Information in this blog post: tales in IT from the help desk.: Formatted with Type 2 Protection, huh?

This seems to be a strange symptom of the $80 8TB eBay drives from a few weeks back. I’ve seen others forums where people do exactly as you say, turning off PI, doing the resize, all while sticking with 512 on SUN HGST 8TB He8 drives, same(ish) model as mine. I tried all that, more than a few times. No go. So, I formatted 4K and all is well.

If you think you have a solution to fix, I’m sure the people in following forum would love your assistance. I was not the only one having trouble with these $80 drives. :slight_smile:

1 Like
6

Did you do a --resize afterwards in your initial attempts?
To my knowledge, @mmbtrumpet has the exact same drives from the same seller and a simple --format + --resize took care of it. I had the 10TB equivalents from the same seller and followed the same steps.

7

Yes. Removal of PI then a resize (count -1 iirc). I even tried different SAS card. In no way an expert in SAS drives, but I did try multiple things, and my Google Fu is pretty strong. I would be curious if anyone actually got the 8TB/$80;drives to show full space with 512 format. If so, I will amend or remove my post.

8

He did mention an STH thread in discord, he probably went the 4K route then. I’ll do some digging and see what I can find regarding the drives.

9

Sorry I’m late to the party, I didn’t know I had been tagged.

I did not have to change the block size to 4K, mine appear to be the same model but were from the first group buy from Rhino not the $80 deal everyone recently got on eBay.

I used the command sg_format --format --fmtpinfo=0 /dev/s. It took about 12 hours to reformat each drive. After it finished I ran the resize command and they showed the correct size. Below is a screenshot that shows when the format finished I ran sg_readcap and it still showed 7.87tb, then I ran the resize command and it said successful, ran the sg_readcap again and it showed the correct size.

8tb resize
8tb resize1264Ă—598 26.2 KB

I referenced these sites.

10

Great info. I read all those same posts too. I will give it a shot on one of my 8TB drives. Mine are from the $80 deal. If your commands work, it will make the Server The Home thread I posted earlier very happy. Thanks for the update.

11

I had a nightmare with these drives. I bought these in two different group buys. I had two from the first group buy as dual parity drives in unraid and they were showing 7.87tb. I thought it was no big deal. I went to add another drive and it showed it was 7.87tb as well. I had already pre-cleared it months ago so I added it to the array and started the array then I tried to format it. Unraid kept saying it was formatting the drive then it would go back and say unmountable: unsupported parition layout. I looked through the logs and it seemed like it was trying to change the partition layout so I rebooted. After I rebooted Unraid suddenly showed that drive as the correct size (8tb) but then was saying it was the wrong drive. I ended up doing a new config and re-assigning all of my drives. I used two drives I had already resized as the parity drives and then rebuilt parity. Through all of this I figured out that the first batch I got had a serial number that were 0015* and the second batch were 0017*. The first batch had protection turned on and the second batch did not. The first batch I had to reformat to remove the protection and then used the resize command, the second batch I was able to just run the resize command.

These are the commands I used.

To reformat the drives and remove protection.
sg_format --format --fmtpinfo=0 /dev/sXX

To resize the drives.
sg_format --resize --count=-1 /dev/sXX

I hope all of my hours of frustration can help someone else.

12

Just to update, I had the same experience as @mmbtrumpet with the non-Sun firmware drives, but the Sun-flashed 10TBs did have the same issue as @SpamMeDeeper and the rest of those threads - every iteration of protection and block size had no effect on the available space with 512b sectors. Unfortunately, I don’t have any of the same capacity of either, so I can’t just dump the firmware and attempt flashing them.
I used hugo, which is the HGST tool for managing drives in quantity, for testing purposes. It also gets around any issues sg_format has with the Sun firmware’s potential mode sense issues. I can upload binaries if there’s any interest. I may put them on a Windows bench in the future with an HBA, as their more involved in-house drive management tools are for Windows only, interestingly enough.

A few things worth noting:

  • If the drive is actually 512-byte sectors (not 520 -8 for PI), you can use fastformat to change the block size to 4k in about 15 seconds as opposed to waiting for zeroes to be written to the entire disk surface. All it does is multiply the sector boundary addresses. Otherwise, there will be errors galore because the new sectors don’t align like the command thought, and you’ll have to deal with the long format time (you’re also welcome to attempt it if you’re not sure if you have PI or not, worth a shot to save some time). Add --ffmt=1 to your sg_format line, or --fastformat for hugo.

  • Reformatting to 4k sectors does not prevent the drives from being usable on LSI SAS2008-based controllers, so this is safe with a 9201/9210/9211 card. The compatibility issue with SAS2008 and 4Kn drives is actually related to the card’s oprom and boot support.

3 Likes
13

Is it possible to do this on Unraid?
If you need sg3-utils…can it be installed in Unraid from the link below by using installpkg
http://sg.danny.cz/sg/sg3_utils.html#__RefHeading___Toc611_3724309425

Also…
@Riggi
Is this the correct command including the fastformat:
sg_format --format --ffmt=1 --fmtpinfo=0 /dev/sXX

14

sg3_utils is included in the nerdpack plugin, which is going to be the easiest way to install the necessary tools. You can also follow JDM’s instructions in the video linked in the OP here, but the nerdpack way is just a couple of clicks.

Your command will work for a fast format of the same block size as presently configured. The --fmtpinfo=0 is already default, so that particular flag doesn’t do anything in this instance, but is nice for documentation purposes.

1 Like
15

Ahhh, why didn’t I look at nerdpack. Pretty obvious…alright.
Thanks

16

Does this result mean the fast format was a no go for me? And I should try the non fast-format command?

Format unit:
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in cdb
  Descriptor type: Sense key specific: Field pointer:
        Error in Command: byte 4 bit 1
  Descriptor type: Field replaceable unit code: 0x23
  Descriptor type: Vendor specific [0x80]
    f8 23
Format unit command: Illegal request, type: sense key, apart from Invalid opcode
FORMAT UNIT failed

Running non fast_format…seems to be working. Chugging along ultra slow.

17

Not working for me after doing the long reformat command below…then rebooting. Showing terminal copy/paste. Still doesn’t report 8.0 TB…rebooted again just for good measure.
This is a HGST Ultrastar SAS 8TB…bought four of em…same result
sg_format --format --fmtpinfo=0 /dev/sde

î‚° sg_readcap /dev/sde
READ CAPACITY (10) indicates device capacity too large
  now trying 16 byte cdb variant
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=15362376263 (0x393ab4247), Number of logical blocks=15362376264
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
   Lowest aligned LBA=0
Hence:
   Device size: 7865536647168 bytes, 7501160.3 MiB, 7865.54 GB, 7.87 TB
 ⚡  ~ 
î‚° sg_format --resize --count=-1 /dev/sde
    HGST      H7280B520SUN8.0T  A374   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: 001740S3AW1L        7SH3AW1L
      LU name: 5000cca2523e75b8
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=15362376264 [0x393ab4248]
  Block size=512 [0x200]
Resize operation seems to have been successful
 ⚡  ~ 
î‚° sg_readcap /dev/sde
READ CAPACITY (10) indicates device capacity too large
  now trying 16 byte cdb variant
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=15362376263 (0x393ab4247), Number of logical blocks=15362376264
   Logical block length=512 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
   Lowest aligned LBA=0
Hence:
   Device size: 7865536647168 bytes, 7501160.3 MiB, 7865.54 GB, 7.87 TB

Update:
Now doing the long reformat again but including the --size=4096 flag

Update 2:
That worked. I needed that 4096 flag

1 Like
18

Good luck on long format. The resize did not work for me either. I have plans to test again with 512 formatting as posted in this thread on my spare drive, but have not found time yet. Of course, the 4096 size sector format should work just fine.

19

The first long reformat command I did…it at least removed the whole protection thing. I was then able to do the fast format command @Riggi mentioned…it was successful but it didn’t help me. The resize command didn’t help me either.

So that is why I’m back to a long format except with the 4096 flag now.

20

Same experience I had with the $80/8TB drives. The 4K format brought me to full 8TB :slight_smile: