Hello.

I have a system which records tons of data runnning on Linux 2.6.27.39 and have a problem on hard drive.
The USB drive is 3TB, 4096-bytes per sector and formatted as ext3 with GPT and single partition.
It looks fine when I plugged and format it, but the capacity of the drive is very different as shown below.
/dev/sdb1 is the 3TB drive.

# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 782683440 201904 782481536 0% /mnt/hdd2

I unplugged the drive and replugged it, more trouble happens.

--- kernel message on console ---
usb 1-1.1: new high speed USB device using emma-ehci and address 19
usb 1-1.1: configuration #1 chosen from 1 choice
scsi16 : SCSI emulation for USB Mass Storage devices
scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sdb: unknown partition table
--- kernel message on console ---

This drive is formatted as following.

--- command log ---
# parted /dev/sdb
(parted) mktable gpt
mktable gpt
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags

(parted) mkpart
mkpart
Partition name? []? record
record
File system type? [ext2]?

Start? 0
0
End? -1s
-1s
Warning: You requested a partition from 0.00B to 3001GB.
The closest location we can manage is 24.6kB to 3001GB.
Is this still acceptable to you?
Yes/No? y
y
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? i
i
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 24.6kB 3001GB 3001GB record

(parted) q
q
Information: You may need to update /etc/fstab.

# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
--- command log ---

I formatted this drive on the Windows machine using disk formatting utility as GPT 512byte/sector.
And then, I connect the drive to this Linux machine and format as ext3.
Parted shows as follows and unplug/replug works fine as following.

--- command log ---
# parted /dev/sdb print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 524kB 3001GB 3001GB ext3
-----
scsi13 : SCSI emulation for USB Mass Storage devices
scsi 13:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
-----
# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 2929892400 347928 2929544472 0% /mnt/hdd2
--- command log ---

I think every funny things are caused by 4096-byte/sector since 512-byte/sector gets rid of these problems.
How can I make 512byte/sector formatting on this Linux machine?
I've looked the man pages but no manual suggests it.

Thanks in advance.

I am surprised to see that you only partitioned the disk, never attempted to create a file system. The file system determines the block size, not the partition table.

Anyway, you can't create a file system with a block size less than 1 kB in ext3. And 1 kB limits you to disk of 2 TB. If you could create it somehow anyway, you'll be overflowing the sector counter which causes any kind of problem you can think of.
http://en.wikipedia.org/wiki/Ext3

The xfs filesystem allows sector sizes down to 512 bytes and xfs is especially suited for systems with large amounts of small files. But I was not able to find the maximum partition size for 512 byte blocks. You might want to check on that.

jlinkels

Hi, jlinkels.

Thanks for your reply.

>I am surprised to see that you only partitioned the disk, never attempted to create a file system.

Sure I do create a filesystem as my previous post says.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1

I believe that blocksize of -b option determines is different from sector size because parted reports the sector size as following even after made a filesystem as above.

>Sector size (logical/physical): 512B/512B
>Sector size (logical/physical): 4096B/4096B

And I want to make hard drive sector to 512byte without using Windows utility.

Please advise.

Sorry, overlooked that.

Well, it shows both. Which is true?

This is an explanation of sectors and blocks: http://en.wikipedia.org/wiki/Disk_sector. You can not change the sector size unless you low-level format a hard disk, and you should not attempt to do that.

Besides, for your problem of creating many small files you want a small block size. Altough the files must be really small in order to worry about block misalignment on a 3TB disk.

Furthermore, etx3 can not (NOT) create blocks below 1024. Please refer to the man page. If you want to create 512 blocks, use xfs. Have you tried?

jlinkels

Hi, jlinkels.

Thanks again.

>Well, it shows both. Which is true?

Both are true.
>Sector size (logical/physical): 512B/512B
I believe this report should be "512B/4096B" because all 3TB drive in the market has 4096byte sector.
This report is made as following.
- Format this drive on windows using DiskFormatter which can change the sector size. (GPT, NTFS, 512B/sector).
- Plug it to Linux and re-format with mkfs.ext3.
(I prefer this without using Windows.)

>Sector size (logical/physical): 4096B/4096B
- As my first post, using parted and mkfs.ext3.

>If you want to create 512 blocks, use xfs. Have you tried?

I'm sorry but no, because this Linux has no xfs feature.

Please advise.

You bet it does!
But you have to install it. I don't know which distro you are running, but your package manager should be able to provide it. It is one of the oldest journalling file systems.
On Debian it is called xfsprogs.
Not to be confused with xfs (X font server)
After you installed the package you can either run mkfs.xfs or mkfs -t xfs.
The man page will be present as well, now it is not.

jlinkels

Hi, jlinkels.

Thanks again.

>But you have to install it.

This is an embedded platform and I have very very limited way to extend its features.
Are there any tools or utilities which works like windows disk formatter utility does?

Please advise.

That is interesting reading matter, thanks.

Would have been good if you told in the first place you had an embedded system where next to nothing can be installed. That would have saved me a lot of typing, because if you don't know how to add a file system driver, it is not likely that I would know better.

Anyway, ext3 doesn't format below 1k block sizes. Hopefully jeet13's post sheds some light on the partitioning problem.

jlinkels

>scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
>sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
>sd 16:0:0:0: [sdb] Write Protect is off
>sd 16:0:0:0: [sdb] Assuming drive cache: write through
> sdb: unknown partition table

I found the reason of this 'sdb: unknown partition table' message.
According to Wikipedia, GUID partition contains its signature "EFI PART" at head of partition table.
http://en.wikipedia.org/wiki/GUID_Partition_Table

Suppose hard drive data has its own address in sequence, say MBR is 0x0000 to 0x01FF and next data is 0x0200.
On the 512B/sector drive contains "EFI PART" at 0x0200 while 4KiB/sector drive contains at 0x1000.
----
# hexdump -C /dev/sdb
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001c0 01 00 ee fe ff ff 01 00 00 00 75 14 aa 2b 00 00 |..........u..+..|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00001010 34 7f 76 bb 00 00 00 00 01 00 00 00 00 00 00 00 |4.v.............|
----

Linux kernel checks drive's partition signature only at 0x0200 and reports that this drive doesn't have EFI partition.
Parted writes GPT at 'LBA1', but it is different place on 512B/sect drive and 4KiB/sect drive.
And Linux kernel checks only at 0x0200.

I believe this problem is caused by 4KiB sector and this problem is only resolved by changing the sector size which this drive reports to Linux (maybe need a SCSI command?).
Could anybody tell me the Linux tool which can change the logical sector size ?

Thanks.

I finally found some kernel patches to fix this issue.
Following files are patched and works fine.
fs/partitions/efi.c
fs/partitions/efi.h
fs/partitions/msdos.c
fs/ntfs/super.c

These modifications are already included in the latest kernel (probably since 2.6.34).
But I can not find any utility on the Linux to change HDD logical sector size.