Advanced format disks and partitioning

msdobrescu · 03-16-2017, 01:50 AM

Hello,

Years after Advanced Format disks were launched on the market, I'm still having issues partitioning.

I have Sabayon up to date and always used GParted for partitioning.

Still, after reading documentation, I see the default GParted approach not matching it, meaning it should detect and partition/format disks according to their specs, i.e. it should recognize 4096 vs 512 sector size disks and treat them as such. Or I've misunderstood something.

I have added some disks to some external unit, accessible through USB. After entering GParted, creating one partition per disk, I see by fdisk:

Code:

Disk /dev/sdd: 3,7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DCB81A82-AFE2-11E5-8258-806E6F6E6963

Device      Start        End    Sectors  Size Type
/dev/sdd1  264192 7814035455 7813771264  3,7T Microsoft basic data

or

Code:

Disk /dev/sdi: 2,3 TiB, 2500495958016 bytes, 4883781168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 6D64D5B7-127B-4A88-913F-FC07F06E1FFA

Device     Start        End    Sectors  Size Type
/dev/sdi2   2048 4883779583 4883777536  2,3T Linux filesystem

or an internal drive in the same PC:

Code:

Disk /dev/sde: 5,5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 4A639AD4-5AA4-4024-A715-8B6ACC418F88

Device     Start         End     Sectors  Size Type
/dev/sde1   2048 11721043967 11721041920  5,5T Linux filesystem

So, I see several things here:

Is this information telling me it is using 512 bytes for the minimum unit?
Shouldn't be 4096 by default when partitioned with GParted?
Code:
```
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
```
I have no other partition on /dev/sdi, yet there is 2 for that single partition.
Shouldn't be 1. meaning /dev/sdi1 instead of /dev/sdi2?
Is there a way to use GParted to set 4096 instead of 512? (assuming I copy all data on some other disk, re-partition and re-format the disks and copy data back...) I don't think is it otherwise possible...

Well, I have large files on these, probably the space wouldn't be affected much.

BTW, I use ext4 partitions only.

Thank you.

rknichols · 03-16-2017, 09:49 AM

Any recent partitioning tool will default to aligning partitions on 1 MiB boundaries. That alignment is suitable for physical sector sizes of 512 bytes or 4096 bytes, as well as for SSDs with large (perhaps 256KB) erase blocks.

The 512-byte logical sector size is the smallest addressable unit on the disk. Blocks of 8 logical sectors map to a single 4096-byte physical sector. You cannot change the way the disk is addressed, but write operations need to be to properly aligned 4096-byte blocks to avoid a huge performance penalty. As long as the partitions are aligned to such blocks, the OS will take care of the rest.

The partition table has discrete slots. A partition can have any number up to the maximum 128 (the minimum required by EFI). If you set up a disk with partitions 5, 17, and 23, then those are what you will have.

jailbait · 03-16-2017, 09:52 AM

The things that you want to specify are specified in the mkfs.ext4 command, not gparted.

--------------------------
Steve Stites

msdobrescu · 03-16-2017, 10:06 AM

Quote:

Originally Posted by jailbait

The things that you want to specify are specified in the mkfs.ext4 command, not gparted.

--------------------------
Steve Stites

I've thought gparted will use mkfs.ext4 and set its proper parameters.

msdobrescu · 03-16-2017, 10:09 AM

According to this:

Code:

Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

What does this mean?
Are the files stored in chunks of 512 or 4096 bytes?

I did not setup the partition to be "2". Could it be set to "1" without re-creating it?

rknichols · 03-16-2017, 10:33 AM

Quote:

Originally Posted by msdobrescu

According to this:

Code:

Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

What does this mean?
Are the files stored in chunks of 512 or 4096 bytes?

I did not setup the partition to be "2". Could it be set to "1" without re-creating it?

Files should be stored in chunks of 4096 bytes, or else write performance will be very bad. The default block size for the filesystem is 4096 bytes.

I don't see a way in gparted to shift that partiton entry to the first slot, and I would be very reluctant to delete and re-create the partition with any GUI tool. You need to be precise. You can use fdisk (a GPT-aware version) and very easily print the current partition table, the delete partition 2 and create partition 1 with exactly the same locations. There is really not much reason to do that. Normally you refer to partitions by their label or UUID, since several things can cause disks to be detected in a different order at boot time.

jailbait · 03-16-2017, 05:24 PM

Quote:

Originally Posted by msdobrescu

I've thought gparted will use mkfs.ext4 and set its proper parameters.

Yes, but if you want to give mkfs.ext4 your own preferred parameters then you have to run mkfs.ext4 yourself after you finish with gparted and set the parameters your way. See:

man mkfs.ext4

for the list of options supported by mkfs.ext4.

---------------------
Steve Stites

mrmazda · 03-16-2017, 07:42 PM

Quote:

Originally Posted by msdobrescu

According to this:

Code:

Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

What does this mean?
Are the files stored in chunks of 512 or 4096 bytes?

Seagate explains advanced formatting.

Short summary: The operating system recognizes 512 bytes as the size of a sector. Inside the disk, groups of 512 byte OS sectors are grouped into 4096 byte HD sectors, managed by the HD's own firmware. To minimize the firmware's overhead, especially when writing, OS tools need to take into account what the HD is doing internally, thus dictating appropriate "alignment" when partitioning and various other disk I/O activities.

rknichols · 03-16-2017, 08:17 PM

Quote:

Originally Posted by mrmazda

Short summary: The operating system recognizes 512 bytes as the size of a sector. ...

That was no doubt true when it was written, but modern OSs can also work with disks with a 4096-byte logical sector size (4096 logical / 4096 physical).

mrmazda · 03-16-2017, 08:42 PM

Quote:

Originally Posted by rknichols

That was no doubt true when it was written, but modern OSs can also work with disks with a 4096-byte logical sector size (4096 logical / 4096 physical).

What they can do and what they do do isn't necessarily the same. Obviously there has been for some time development ongoing because of 4k sector existence, but existing 1k block size partitions and disks with 255/63 partitioning aren't changing just because 4k support exists at some level in kernel, driver and tools repertoire. No gain deviating from 512b sectors is apparent for users heavily dependent on many small files.

msdobrescu · 03-17-2017, 02:00 AM

Quote:

Originally Posted by mrmazda

Seagate explains advanced formatting.

Short summary: The operating system recognizes 512 bytes as the size of a sector. Inside the disk, groups of 512 byte OS sectors are grouped into 4096 byte HD sectors, managed by the HD's own firmware. To minimize the firmware's overhead, especially when writing, OS tools need to take into account what the HD is doing internally, thus dictating appropriate "alignment" when partitioning and various other disk I/O activities.

Thanks. My trouble here is to understand the two lines output by fdisk. I think your statement relates to this:

Code:

Sector size (logical/physical): 512 bytes / 4096 bytes

But what this mean:

Code:

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Is the OS set to bypass the 512 bytes emulation? Does it treat the disk as Advanced Format by writing 4096 bytes directly, instead of 512 bytes?

mrmazda · 03-17-2017, 03:03 AM

Quote:

Originally Posted by msdobrescu

what this mean:

Code:

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Is the OS set to bypass the 512 bytes emulation? Does it treat the disk as Advanced Format by writing 4096 bytes directly, instead of 512 bytes?

In effect, yes. Optimal throughput is achieved by making requests in chunks that are 4096 bytes (or integer multiples thereof) (properly aligned), which avoids any internal translation overhead.

rknichols · 03-17-2017, 08:25 AM

Quote:

Originally Posted by mrmazda

What they can do and what they do do isn't necessarily the same. Obviously there has been for some time development ongoing because of 4k sector existence, but existing 1k block size partitions and disks with 255/63 partitioning aren't changing just because 4k support exists at some level in kernel, driver and tools repertoire. No gain deviating from 512b sectors is apparent for users heavily dependent on many small files.

There are disk drives on the market that are "Advanced 4Kn Format" (4K native). They have no 512-byte emulation layer and directly address the 4K physical sector size. They have been supported in Linux since kernel version 2.6.31. One interesting effect that that FAT32 filesystems can be up to 16TB on such drives.

msdobrescu · 03-29-2017, 12:10 PM

Thank you, the thing are clear now.