Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
|
11-08-2011, 07:58 AM
|
#1
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Rep:
|
How can I make 512byte/sector format on 4KiB/sector drive?
Hello.
I have a system which records tons of data runnning on Linux 2.6.27.39 and have a problem on hard drive.
The USB drive is 3TB, 4096-bytes per sector and formatted as ext3 with GPT and single partition.
It looks fine when I plugged and format it, but the capacity of the drive is very different as shown below.
/dev/sdb1 is the 3TB drive.
# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 782683440 201904 782481536 0% /mnt/hdd2
I unplugged the drive and replugged it, more trouble happens.
--- kernel message on console ---
usb 1-1.1: new high speed USB device using emma-ehci and address 19
usb 1-1.1: configuration #1 chosen from 1 choice
scsi16 : SCSI emulation for USB Mass Storage devices
scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
sd 16:0:0:0: [sdb] Write Protect is off
sd 16:0:0:0: [sdb] Assuming drive cache: write through
sdb: unknown partition table
--- kernel message on console ---
This drive is formatted as following.
--- command log ---
# parted /dev/sdb
(parted) mktable gpt
mktable gpt
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
(parted) mkpart
mkpart
Partition name? []? record
record
File system type? [ext2]?
Start? 0
0
End? -1s
-1s
Warning: You requested a partition from 0.00B to 3001GB.
The closest location we can manage is 24.6kB to 3001GB.
Is this still acceptable to you?
Yes/No? y
y
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? i
i
Error: Partition(s) 16 on /dev/sdb have been written, but we have been unable to
inform the kernel of the change, probably because it/they are in use. As a
result, the old partition(s) will remain in use. You should reboot now before
making further changes.
Ignore/Cancel? i
i
(parted) print
print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Number Start End Size File system Name Flags
1 24.6kB 3001GB 3001GB record
(parted) q
q
Information: You may need to update /etc/fstab.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
--- command log ---
I formatted this drive on the Windows machine using disk formatting utility as GPT 512byte/sector.
And then, I connect the drive to this Linux machine and format as ext3.
Parted shows as follows and unplug/replug works fine as following.
--- command log ---
# parted /dev/sdb print
Model: BUFFALO External HDD (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 524kB 3001GB 3001GB ext3
-----
scsi13 : SCSI emulation for USB Mass Storage devices
scsi 13:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sd 13:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
sd 13:0:0:0: [sdb] 5860533168 512-byte hardware sectors (3000593 MB)
sd 13:0:0:0: [sdb] Write Protect is off
sd 13:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
-----
# df
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 118784 37740 81044 32% /
/dev/sdb1 2929892400 347928 2929544472 0% /mnt/hdd2
--- command log ---
I think every funny things are caused by 4096-byte/sector since 512-byte/sector gets rid of these problems.
How can I make 512byte/sector formatting on this Linux machine?
I've looked the man pages but no manual suggests it.
Thanks in advance.
|
|
|
11-08-2011, 12:41 PM
|
#2
|
LQ Guru
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,195
|
I am surprised to see that you only partitioned the disk, never attempted to create a file system. The file system determines the block size, not the partition table.
Anyway, you can't create a file system with a block size less than 1 kB in ext3. And 1 kB limits you to disk of 2 TB. If you could create it somehow anyway, you'll be overflowing the sector counter which causes any kind of problem you can think of.
http://en.wikipedia.org/wiki/Ext3
The xfs filesystem allows sector sizes down to 512 bytes and xfs is especially suited for systems with large amounts of small files. But I was not able to find the maximum partition size for 512 byte blocks. You might want to check on that.
jlinkels
|
|
|
11-08-2011, 06:17 PM
|
#3
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
Hi, jlinkels.
Thanks for your reply.
>I am surprised to see that you only partitioned the disk, never attempted to create a file system.
Sure I do create a filesystem as my previous post says.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
I believe that blocksize of -b option determines is different from sector size because parted reports the sector size as following even after made a filesystem as above.
>Sector size (logical/physical): 512B/512B
>Sector size (logical/physical): 4096B/4096B
And I want to make hard drive sector to 512byte without using Windows utility.
Please advise.
Last edited by delorean-bf; 11-08-2011 at 06:50 PM.
|
|
|
11-08-2011, 06:43 PM
|
#4
|
LQ Guru
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,195
|
Quote:
Originally Posted by delorean-bf
Sure I do create a filesystem as my previous post says.
# mkfs.ext3 -m 0 -b 4096 -T largefile4 -j -q /dev/sdb1
|
Sorry, overlooked that.
Quote:
Originally Posted by delorean-bf
>Sector size (logical/physical): 512B/512B
>Sector size (logical/physical): 4096B/4096B
|
Well, it shows both. Which is true?
This is an explanation of sectors and blocks: http://en.wikipedia.org/wiki/Disk_sector. You can not change the sector size unless you low-level format a hard disk, and you should not attempt to do that.
Besides, for your problem of creating many small files you want a small block size. Altough the files must be really small in order to worry about block misalignment on a 3TB disk.
Furthermore, etx3 can not ( NOT) create blocks below 1024. Please refer to the man page. If you want to create 512 blocks, use xfs. Have you tried?
jlinkels
|
|
|
11-08-2011, 07:12 PM
|
#5
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
Hi, jlinkels.
Thanks again.
>Well, it shows both. Which is true?
Both are true.
>Sector size (logical/physical): 512B/512B
I believe this report should be "512B/4096B" because all 3TB drive in the market has 4096byte sector.
This report is made as following.
- Format this drive on windows using DiskFormatter which can change the sector size. (GPT, NTFS, 512B/sector).
- Plug it to Linux and re-format with mkfs.ext3.
(I prefer this without using Windows.)
>Sector size (logical/physical): 4096B/4096B
- As my first post, using parted and mkfs.ext3.
>If you want to create 512 blocks, use xfs. Have you tried?
I'm sorry but no, because this Linux has no xfs feature.
Please advise.
Last edited by delorean-bf; 11-08-2011 at 07:14 PM.
|
|
|
11-08-2011, 08:21 PM
|
#6
|
LQ Guru
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,195
|
Quote:
Originally Posted by delorean-bf
I'm sorry but no, because this Linux has no xfs feature.
|
You bet it does!
But you have to install it. I don't know which distro you are running, but your package manager should be able to provide it. It is one of the oldest journalling file systems.
On Debian it is called xfsprogs.
Not to be confused with xfs (X font server)
After you installed the package you can either run mkfs.xfs or mkfs -t xfs.
The man page will be present as well, now it is not.
jlinkels
|
|
|
11-08-2011, 08:39 PM
|
#7
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
Hi, jlinkels.
Thanks again.
>But you have to install it.
This is an embedded platform and I have very very limited way to extend its features.
Are there any tools or utilities which works like windows disk formatter utility does?
Please advise.
|
|
|
11-09-2011, 06:43 AM
|
#8
|
LQ Guru
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,195
|
Quote:
Originally Posted by jeet13
Am just learning so maybe it can help you forums.debian.net/viewtopic.php?f=7&t=54354
|
That is interesting reading matter, thanks.
Quote:
Originally Posted by delorean-bf
I have a system which records tons of data runnning on Linux 2.6.27.39 and have a problem on hard drive.
|
Would have been good if you told in the first place you had an embedded system where next to nothing can be installed. That would have saved me a lot of typing, because if you don't know how to add a file system driver, it is not likely that I would know better.
Anyway, ext3 doesn't format below 1k block sizes. Hopefully jeet13's post sheds some light on the partitioning problem.
jlinkels
|
|
|
11-10-2011, 06:10 AM
|
#9
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
>scsi 16:0:0:0: Direct-Access BUFFALO External HDD 0000 PQ: 0 ANSI: 5
>sd 16:0:0:0: [sdb] 732566646 4096-byte hardware sectors (3000593 MB)
>sd 16:0:0:0: [sdb] Write Protect is off
>sd 16:0:0:0: [sdb] Assuming drive cache: write through
> sdb: unknown partition table
I found the reason of this 'sdb: unknown partition table' message.
According to Wikipedia, GUID partition contains its signature "EFI PART" at head of partition table.
http://en.wikipedia.org/wiki/GUID_Partition_Table
Suppose hard drive data has its own address in sequence, say MBR is 0x0000 to 0x01FF and next data is 0x0200.
On the 512B/sector drive contains "EFI PART" at 0x0200 while 4KiB/sector drive contains at 0x1000.
----
# hexdump -C /dev/sdb
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001c0 01 00 ee fe ff ff 01 00 00 00 75 14 aa 2b 00 00 |..........u..+..|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00 |EFI PART....\...|
00001010 34 7f 76 bb 00 00 00 00 01 00 00 00 00 00 00 00 |4.v.............|
----
Linux kernel checks drive's partition signature only at 0x0200 and reports that this drive doesn't have EFI partition.
Parted writes GPT at 'LBA1', but it is different place on 512B/sect drive and 4KiB/sect drive.
And Linux kernel checks only at 0x0200.
I believe this problem is caused by 4KiB sector and this problem is only resolved by changing the sector size which this drive reports to Linux (maybe need a SCSI command?).
Could anybody tell me the Linux tool which can change the logical sector size ?
Thanks.
|
|
|
11-17-2011, 10:12 PM
|
#10
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
I finally found some kernel patches to fix this issue.
Following files are patched and works fine.
fs/partitions/efi.c
fs/partitions/efi.h
fs/partitions/msdos.c
fs/ntfs/super.c
These modifications are already included in the latest kernel (probably since 2.6.34).
But I can not find any utility on the Linux to change HDD logical sector size.
|
|
|
10-14-2012, 08:35 PM
|
#11
|
LQ Newbie
Registered: Oct 2012
Posts: 2
Rep:
|
Quote:
Originally Posted by delorean-bf
I finally found some kernel patches to fix this issue.
Following files are patched and works fine.
fs/partitions/efi.c
fs/partitions/efi.h
fs/partitions/msdos.c
fs/ntfs/super.c
These modifications are already included in the latest kernel (probably since 2.6.34).
But I can not find any utility on the Linux to change HDD logical sector size.
|
Hi delorean-bf, may I have some details about this solution? It's much better if you still have the patch, or you remeber it.
I have encounter the same problem as yours.Thanks!
percy
|
|
|
10-15-2012, 07:52 AM
|
#12
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
Quote:
Originally Posted by percy.yang
Hi delorean-bf, may I have some details about this solution? It's much better if you still have the patch, or you remeber it.
I have encounter the same problem as yours.Thanks!
percy
|
Hi, percy.
The cause of this problem is 'position of GPT'.
GNU parted writes GPT to 'LBA 1' which is placed at 0x0200 on 512B/sec HDD and at 0x1000 on 4KiB/sec HDD. And old kernel checks GPT only at 0x0200, so GPT is not found on 4KiB HDD.
The patch modifies the kernel to check the HDD sector size and reads GPT from apropreate position.
As I wrote, kernel 2.6.34 or later already has this modification.
You can make a patch to your kernel version to use diff. Of course, you have to verify the patched source do the right thing.
And on too old kernel, say 2.6.18, making patch is very hard - I couldn't.
Cheers.
|
|
|
10-15-2012, 08:45 AM
|
#13
|
LQ Newbie
Registered: Oct 2012
Posts: 2
Rep:
|
Quote:
Originally Posted by delorean-bf
Hi, percy.
The cause of this problem is 'position of GPT'.
GNU parted writes GPT to 'LBA 1' which is placed at 0x0200 on 512B/sec HDD and at 0x1000 on 4KiB/sec HDD. And old kernel checks GPT only at 0x0200, so GPT is not found on 4KiB HDD.
The patch modifies the kernel to check the HDD sector size and reads GPT from apropreate position.
As I wrote, kernel 2.6.34 or later already has this modification.
You can make a patch to your kernel version to use diff. Of course, you have to verify the patched source do the right thing.
And on too old kernel, say 2.6.18, making patch is very hard - I couldn't.
Cheers.
|
Hi delorean-bf
I have modify those files, and the partition table is ok now. But there is still a problem, when mount fat, which does say:
"FAT: unable to read boot sector
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested"
so poor ~.~, my kernel is 2.6.30. Have you ever encountered that problem?
BRs
percy
|
|
|
10-15-2012, 09:45 AM
|
#14
|
LQ Newbie
Registered: Nov 2010
Posts: 9
Original Poster
Rep:
|
Quote:
Originally Posted by percy.yang
Hi delorean-bf
I have modify those files, and the partition table is ok now. But there is still a problem, when mount fat, which does say:
"FAT: unable to read boot sector
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested
sd 1:0:0:0: [sda] Bad block number requested"
so poor ~.~, my kernel is 2.6.30. Have you ever encountered that problem?
BRs
percy
|
Sorry but I have not encountered these messages. So I don't have any idea about it.
FAT: ... is reported by fs/fat/inode.c.
sd 1:0:0:0: .... is reported by drivers/scsi/sd.c.
Hope this helps.
|
|
|
11-20-2012, 09:33 AM
|
#15
|
LQ Newbie
Registered: Nov 2012
Posts: 1
Rep:
|
The only way I was able to proper setup 2TB disks - formatting them using Ubuntu Server installer (in this case 10.04). System, no longer says they are missaligned. And I could use them normally in Ubuntu.
|
|
|
All times are GMT -5. The time now is 08:57 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|