LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 11-08-2011, 07:58 AM   #1
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Rep: Reputation: 0
How can I make 512byte/sector format on 4KiB/sector drive?


Hello.

I have a system which records tons of data runnning on Linux 2.6.27.39 and have a problem on hard drive.
The USB drive is 3TB, 4096-bytes per sector and formatted as ext3 with GPT and single partition.
It looks fine when I plugged and format it, but the capacity of the drive is very different as shown below.
/dev/sdb1 is the 3TB drive.

# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 782683440 201904 782481536 0% /mnt/hdd2

I unplugged the drive and replugged it, more trouble happens.

--- kernel message on console ---
usb 1-1.1: new high speed USB device using emma-ehci and address 19
usb 1-1.1: configuration #1 chosen from 1 choice
scsi16 : SCSI emulation for USB Mass Storage devices
scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sdb: unknown partition table
--- kernel message on console ---

This drive is formatted as following.

--- command log ---
# parted /dev/sdb
(parted) mktable gpt
mktable gpt
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags

(parted) mkpart
mkpart
Partition name? []? record
record
File system type? [ext2]?

Start? 0
0
End? -1s
-1s
Warning: You requested a partition from 0.00B to 3001GB.
The closest location we can manage is 24.6kB to 3001GB.
Is this still acceptable to you?
Yes/No? y
y
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? i
i
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 24.6kB 3001GB 3001GB record

(parted) q
q
Information: You may need to update /etc/fstab.

# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
--- command log ---

I formatted this drive on the Windows machine using disk formatting utility as GPT 512byte/sector.
And then, I connect the drive to this Linux machine and format as ext3.
Parted shows as follows and unplug/replug works fine as following.

--- command log ---
# parted /dev/sdb print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 524kB 3001GB 3001GB ext3
-----
scsi13 : SCSI emulation for USB Mass Storage devices
scsi 13:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
-----
# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 2929892400 347928 2929544472 0% /mnt/hdd2
--- command log ---

I think every funny things are caused by 4096-byte/sector since 512-byte/sector gets rid of these problems.
How can I make 512byte/sector formatting on this Linux machine?
I've looked the man pages but no manual suggests it.

Thanks in advance.
 
Old 11-08-2011, 12:41 PM   #2
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,105

Rep: Reputation: 495Reputation: 495Reputation: 495Reputation: 495Reputation: 495
I am surprised to see that you only partitioned the disk, never attempted to create a file system. The file system determines the block size, not the partition table.

Anyway, you can't create a file system with a block size less than 1 kB in ext3. And 1 kB limits you to disk of 2 TB. If you could create it somehow anyway, you'll be overflowing the sector counter which causes any kind of problem you can think of.
http://en.wikipedia.org/wiki/Ext3

The xfs filesystem allows sector sizes down to 512 bytes and xfs is especially suited for systems with large amounts of small files. But I was not able to find the maximum partition size for 512 byte blocks. You might want to check on that.

jlinkels
 
Old 11-08-2011, 06:17 PM   #3
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Hi, jlinkels.

Thanks for your reply.

>I am surprised to see that you only partitioned the disk, never attempted to create a file system.

Sure I do create a filesystem as my previous post says.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1

I believe that blocksize of -b option determines is different from sector size because parted reports the sector size as following even after made a filesystem as above.

>Sector size (logical/physical): 512B/512B
>Sector size (logical/physical): 4096B/4096B

And I want to make hard drive sector to 512byte without using Windows utility.

Please advise.

Last edited by delorean-bf; 11-08-2011 at 06:50 PM.
 
Old 11-08-2011, 06:43 PM   #4
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,105

Rep: Reputation: 495Reputation: 495Reputation: 495Reputation: 495Reputation: 495
Quote:
Originally Posted by delorean-bf View Post
Sure I do create a filesystem as my previous post says.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
Sorry, overlooked that.

Quote:
Originally Posted by delorean-bf View Post
>Sector size (logical/physical): 512B/512B
>Sector size (logical/physical): 4096B/4096B
Well, it shows both. Which is true?

This is an explanation of sectors and blocks: http://en.wikipedia.org/wiki/Disk_sector. You can not change the sector size unless you low-level format a hard disk, and you should not attempt to do that.

Besides, for your problem of creating many small files you want a small block size. Altough the files must be really small in order to worry about block misalignment on a 3TB disk.

Furthermore, etx3 can not (NOT) create blocks below 1024. Please refer to the man page. If you want to create 512 blocks, use xfs. Have you tried?

jlinkels
 
Old 11-08-2011, 07:12 PM   #5
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Hi, jlinkels.

Thanks again.

>Well, it shows both. Which is true?

Both are true.
>Sector size (logical/physical): 512B/512B
I believe this report should be "512B/4096B" because all 3TB drive in the market has 4096byte sector.
This report is made as following.
- Format this drive on windows using DiskFormatter which can change the sector size. (GPT, NTFS, 512B/sector).
- Plug it to Linux and re-format with mkfs.ext3.
(I prefer this without using Windows.)

>Sector size (logical/physical): 4096B/4096B
- As my first post, using parted and mkfs.ext3.

>If you want to create 512 blocks, use xfs. Have you tried?

I'm sorry but no, because this Linux has no xfs feature.

Please advise.

Last edited by delorean-bf; 11-08-2011 at 07:14 PM.
 
Old 11-08-2011, 08:21 PM   #6
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,105

Rep: Reputation: 495Reputation: 495Reputation: 495Reputation: 495Reputation: 495
Quote:
Originally Posted by delorean-bf View Post
I'm sorry but no, because this Linux has no xfs feature.
You bet it does!
But you have to install it. I don't know which distro you are running, but your package manager should be able to provide it. It is one of the oldest journalling file systems.
On Debian it is called xfsprogs.
Not to be confused with xfs (X font server)
After you installed the package you can either run mkfs.xfs or mkfs -t xfs.
The man page will be present as well, now it is not.

jlinkels
 
Old 11-08-2011, 08:39 PM   #7
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Hi, jlinkels.

Thanks again.

>But you have to install it.

This is an embedded platform and I have very very limited way to extend its features.
Are there any tools or utilities which works like windows disk formatter utility does?

Please advise.
 
Old 11-09-2011, 06:43 AM   #8
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Lenny/Squeeze/Wheezy/Sid
Posts: 4,105

Rep: Reputation: 495Reputation: 495Reputation: 495Reputation: 495Reputation: 495
Quote:
Originally Posted by jeet13 View Post
Am just learning so maybe it can help you forums.debian.net/viewtopic.php?f=7&t=54354
That is interesting reading matter, thanks.

Quote:
Originally Posted by delorean-bf
I have a system which records tons of data runnning on Linux 2.6.27.39 and have a problem on hard drive.
Would have been good if you told in the first place you had an embedded system where next to nothing can be installed. That would have saved me a lot of typing, because if you don't know how to add a file system driver, it is not likely that I would know better.

Anyway, ext3 doesn't format below 1k block sizes. Hopefully jeet13's post sheds some light on the partitioning problem.

jlinkels
 
Old 11-10-2011, 06:10 AM   #9
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
>scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
>sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
>sd 16:0:0:0: [sdb] Write Protect is off
>sd 16:0:0:0: [sdb] Assuming drive cache: write through
> sdb: unknown partition table

I found the reason of this 'sdb: unknown partition table' message.
According to Wikipedia, GUID partition contains its signature "EFI PART" at head of partition table.
http://en.wikipedia.org/wiki/GUID_Partition_Table

Suppose hard drive data has its own address in sequence, say MBR is 0x0000 to 0x01FF and next data is 0x0200.
On the 512B/sector drive contains "EFI PART" at 0x0200 while 4KiB/sector drive contains at 0x1000.
----
# hexdump -C /dev/sdb
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001c0 01 00 ee fe ff ff 01 00 00 00 75 14 aa 2b 00 00 |..........u..+..|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00001010 34 7f 76 bb 00 00 00 00 01 00 00 00 00 00 00 00 |4.v.............|
----

Linux kernel checks drive's partition signature only at 0x0200 and reports that this drive doesn't have EFI partition.
Parted writes GPT at 'LBA1', but it is different place on 512B/sect drive and 4KiB/sect drive.
And Linux kernel checks only at 0x0200.

I believe this problem is caused by 4KiB sector and this problem is only resolved by changing the sector size which this drive reports to Linux (maybe need a SCSI command?).
Could anybody tell me the Linux tool which can change the logical sector size ?

Thanks.
 
Old 11-17-2011, 10:12 PM   #10
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
I finally found some kernel patches to fix this issue.
Following files are patched and works fine.
fs/partitions/efi.c
fs/partitions/efi.h
fs/partitions/msdos.c
fs/ntfs/super.c

These modifications are already included in the latest kernel (probably since 2.6.34).
But I can not find any utility on the Linux to change HDD logical sector size.
 
Old 10-14-2012, 08:35 PM   #11
percy.yang
LQ Newbie
 
Registered: Oct 2012
Posts: 2

Rep: Reputation: Disabled
Quote:
Originally Posted by delorean-bf View Post
I finally found some kernel patches to fix this issue.
Following files are patched and works fine.
fs/partitions/efi.c
fs/partitions/efi.h
fs/partitions/msdos.c
fs/ntfs/super.c

These modifications are already included in the latest kernel (probably since 2.6.34).
But I can not find any utility on the Linux to change HDD logical sector size.
Hi delorean-bf, may I have some details about this solution? It's much better if you still have the patch, or you remeber it.
I have encounter the same problem as yours.Thanks!

percy
 
Old 10-15-2012, 07:52 AM   #12
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by percy.yang View Post
Hi delorean-bf, may I have some details about this solution? It's much better if you still have the patch, or you remeber it.
I have encounter the same problem as yours.Thanks!

percy
Hi, percy.
The cause of this problem is 'position of GPT'.
GNU parted writes GPT to 'LBA 1' which is placed at 0x0200 on 512B/sec HDD and at 0x1000 on 4KiB/sec HDD. And old kernel checks GPT only at 0x0200, so GPT is not found on 4KiB HDD.

The patch modifies the kernel to check the HDD sector size and reads GPT from apropreate position.
As I wrote, kernel 2.6.34 or later already has this modification.
You can make a patch to your kernel version to use diff. Of course, you have to verify the patched source do the right thing.
And on too old kernel, say 2.6.18, making patch is very hard - I couldn't.

Cheers.
 
Old 10-15-2012, 08:45 AM   #13
percy.yang
LQ Newbie
 
Registered: Oct 2012
Posts: 2

Rep: Reputation: Disabled
Quote:
Originally Posted by delorean-bf View Post
Hi, percy.
The cause of this problem is 'position of GPT'.
GNU parted writes GPT to 'LBA 1' which is placed at 0x0200 on 512B/sec HDD and at 0x1000 on 4KiB/sec HDD. And old kernel checks GPT only at 0x0200, so GPT is not found on 4KiB HDD.

The patch modifies the kernel to check the HDD sector size and reads GPT from apropreate position.
As I wrote, kernel 2.6.34 or later already has this modification.
You can make a patch to your kernel version to use diff. Of course, you have to verify the patched source do the right thing.
And on too old kernel, say 2.6.18, making patch is very hard - I couldn't.

Cheers.
Hi delorean-bf

I have modify those files, and the partition table is ok now. But there is still a problem, when mount fat, which does say:

"FAT: unable to read boot sector
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested"

so poor ~.~, my kernel is 2.6.30. Have you ever encountered that problem?

BRs
percy
 
Old 10-15-2012, 09:45 AM   #14
delorean-bf
LQ Newbie
 
Registered: Nov 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by percy.yang View Post
Hi delorean-bf

I have modify those files, and the partition table is ok now. But there is still a problem, when mount fat, which does say:

"FAT: unable to read boot sector
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested"

so poor ~.~, my kernel is 2.6.30. Have you ever encountered that problem?

BRs
percy
Sorry but I have not encountered these messages. So I don't have any idea about it.
FAT: ... is reported by fs/fat/inode.c.
sd 1:0:0:0: .... is reported by drivers/scsi/sd.c.

Hope this helps.
 
Old 11-20-2012, 09:33 AM   #15
Bruno_von_Troba
LQ Newbie
 
Registered: Nov 2012
Posts: 1

Rep: Reputation: Disabled
The only way I was able to proper setup 2TB disks - formatting them using Ubuntu Server installer (in this case 10.04). System, no longer says they are missaligned. And I could use them normally in Ubuntu.
 
  


Reply

Tags
format, hdd


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Advanced Format 4k sector drive with LVM megamojo Linux - Hardware 4 01-16-2012 09:28 AM
cannot read/write SD card after improper format (sector size 0) hedpe Linux - General 7 06-14-2011 04:02 AM
[SOLVED] Know if my hard-drive had bad sector qlands Linux - Newbie 8 10-23-2010 01:25 PM
Hard drive sector errors crathm Linux - Newbie 3 12-09-2008 05:59 AM
Bad Sector Hard Drive tsigane Linux - Hardware 16 08-15-2004 03:07 AM


All times are GMT -5. The time now is 01:34 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration