LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-03-2012, 09:53 AM   #1
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Rep: Reputation: Disabled
Question 16x2TB RAID. What Large File System Format to Use?


I have come to this forum many times to find answers, but it seems I've come to the point where I need to ask some questions myself and not rely on what others have already asked. I have followed how-tos, read through man pages, and a lot of trial and error (mainly error). I need guidance.

I am setting up a brand new file server. It is running CentOS 6.2 and has a Promise Vtrak E610 array attached via dual fiber connection. The array is loaded with 16 2Tb (or rather 1.8Tb) drives, which I have been able to easily configure as a RAID with the Promise web app. My problem comes to formatting the RAID as a file system that I can get the most out of. I have been using Linux for years, but this is new territory for me.

I've discovered through playing around with fdisk and gnu parted that I can't just create a 25.46TB partition with any fs. We have an old server here that is running CentOS 4.4 and has a 4.8Tb Raid attached, and looking at the fstab I can see it is formatted to reiserfs. My current CentOS 6.2 system's kernel doesn't have reiserfs support right now, and it seems complicated to get it to that point. So my first question is what format should I use, and what's the easiest way to do it?

My second question is about the fiber connections. There are two fiber cables, and I can see the storage on both of them as /dev/sdb and /dev/sdc. If I make a change to sdb I see it also on sdc, because they are obviously the same device. How do I make proper use of these two connections, and when it comes time to mount the device do I use one or the other, or is there a way to combine them?

Thanks in advance, and as I said this is my first time posting. If there are any additional questions about my set up just ask. Right now it's a pretty vanilla setup and I'm not too attached to anything. Obviously the less changes to make the better, but I'll take any advice I can get.
 
Old 05-03-2012, 11:33 AM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
My vote is for XFS. I have a 12x3TB RAID6 (30TB array) running on an Adaptec 51245, and XFS handles it beautifully.
 
1 members found this post helpful.
Old 05-03-2012, 11:42 AM   #3
masterclassic
Member
 
Registered: Jun 2007
Distribution: Knoppix
Posts: 140

Rep: Reputation: 23
The web page
http://en.wikipedia.org/wiki/Comparison_of_file_systems
gives comparisons of a large number of file systems, including ext2/3/4, reiserfs, reiser4 and others that linux can use.
According to that tables, reiserfs is limited to 16TB (or 16TiB?), so it isn't the solution for the size you want.
ext3 and ext4 can go up to 25.46TiB and more, with the proper block size and the, I guess, proper kernel support. Perhaps you need 64 bit support too.
In any case, I think such a large filesystem isn't very common.

Another factor is the partition table. Perhaps you need GPT.

(I can't contribute on the second question).
 
1 members found this post helpful.
Old 05-03-2012, 12:21 PM   #4
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks. I think xfs is what I will go with. Now installing xfs support will be an adventure in and of it's self.

Quote:
Originally Posted by masterclassic View Post
The web page
http://en.wikipedia.org/wiki/Comparison_of_file_systems
gives comparisons of a large number of file systems, including ext2/3/4, reiserfs, reiser4 and others that linux can use.
According to that tables, reiserfs is limited to 16TB (or 16TiB?), so it isn't the solution for the size you want.
ext3 and ext4 can go up to 25.46TiB and more, with the proper block size and the, I guess, proper kernel support. Perhaps you need 64 bit support too.
In any case, I think such a large filesystem isn't very common.

Another factor is the partition table. Perhaps you need GPT.
You're right about a partition table. I apparently don't have one. As I said before, this is all new territory to me, so I'm not sure where to start. Every google search I do leads me to another thing I need to google. Can someone give me a starting point of how I can create a partition table? What is GPT?

Here's an output of fdisk -l:
Code:
Disk /dev/sdb: 28000.0 GB, 27999999885312 bytes
255 heads, 63 sectors/track, 3404139 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 28000.0 GB, 27999999885312 bytes
255 heads, 63 sectors/track, 3404139 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table
 
Old 05-03-2012, 12:29 PM   #5
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
fdisk uses the DOS partition table which is limited to 2TB. You need to switch the partition table to GPT in order to allow support for >2TB, which means you need to use parted.

This will switch it to GPT and create a single partition the full size
Code:
parted /dev/sdb
mklabel gpt
mkpart primary 0 -1
quit
Once the partition table is set up, you can use "mkfs.xfs /dev/sdb1" to make the filesystem.
 
1 members found this post helpful.
Old 05-03-2012, 12:36 PM   #6
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
fdisk uses the DOS partition table which is limited to 2TB. You need to switch the partition table to GPT in order to allow support for >2TB, which means you need to use parted.

This will switch it to GPT and create a single partition the full size
Code:
parted /dev/sdb
mklabel gpt
mkpart primary 0 -1
quit
Once the partition table is set up, you can use "mkfs.xfs /dev/sdb1" to make the filesystem.
Thanks, you don't know what a huge head start that gives me.

So I've created the partition, and then made the filesystem, but when I try to mount it I get:
mount: unknown filesystem type 'xfs'

I just yum installed xfsprogs, and I saw elsewhere online about a "xfsdump" but no package for that was available. Is this the missing piece of the puzzle?
 
Old 05-03-2012, 12:39 PM   #7
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
It still won't mount with xfsprogs installed? You installed the right one for your architecture right? xfsdump is available in the repo for my machine, but it's not installed. I only have xfsprogs installed.
 
Old 05-03-2012, 12:42 PM   #8
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Yeah, this is what was installed:

xfsprogs i686 3.1.1-6.el6

Weird, right?

I mean, I couldn't mkfs.xfs before, then I installed and it was there. You'd think it would also recognize it as a valid fs...
 
Old 05-03-2012, 12:49 PM   #9
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
I found this:
http://boardreader.com/thread/RESOLV...urildXoui.html

Are you running a 32 or 64-bit kernel?
 
1 members found this post helpful.
Old 05-03-2012, 01:01 PM   #10
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
I'm running 32. I guess I need to look into 64...

What is the easiest way to tell if my machine can even run 64?
 
Old 05-03-2012, 01:05 PM   #11
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
Unfortunately it looks like XFS only works in 64 for some reason on CentOS. You could try the ext4 filesystem instead (still need the GPT partition table, but you should be able to use mkfs.ext4 /dev/sdb1 to switch from XFS to ext4).

Otherwise you'd have to switch to 64-bit, which means reinstalling the OS from scratch. If you post the output of "cat /proc/cpuinfo" we'll be able to see if the hardware supports it, but unless this is a brand new machine that was just set up, it will probably be more worthwhile to just use a different filesystem type.
 
1 members found this post helpful.
Old 05-03-2012, 01:18 PM   #12
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Again, I want to thank you for your guidance, and I want to apologize if I sound incompetent... but for some reason file system label is still "xfs". I even tried using parted to remove the partition and just start over, but it's still there.

Code:
# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Promise VTrak E610f (scsi)
Disk /dev/sdb: 28.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  28.0TB  28.0TB  xfs          primary

(parted) rm 1
(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary 0 -1
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? ignore
(parted) quit
Information: You may need to update /etc/fstab.

# mkfs.ext4 /dev/sdb
mke2fs 1.41.12 (17-May-2010)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
mkfs.ext4: Size of device /dev/sdb too big to be expressed in 32 bits
        using a blocksize of 4096.
# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Promise VTrak E610f (scsi)
Disk /dev/sdb: 28.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  28.0TB  28.0TB  xfs          primary
 
Old 05-03-2012, 01:21 PM   #13
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
When you use mkfs, you need to give it the partition, not the whole device, IE:
Code:
mkfs.ext4 /dev/sdb1
rather than

Code:
mkfs.ext4 /dev/sdb
 
1 members found this post helpful.
Old 05-03-2012, 01:27 PM   #14
Jonus
LQ Newbie
 
Registered: May 2012
Posts: 15

Original Poster
Rep: Reputation: Disabled
Ok, thanks.

Tried it, still there.

Code:
[root@proj src]# mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
mkfs.ext4: Size of device /dev/sdb1 too big to be expressed in 32 bits
        using a blocksize of 4096.
[root@proj src]# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Promise VTrak E610f (scsi)
Disk /dev/sdb: 28.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      17.4kB  28.0TB  28.0TB  xfs          primary
Any way I can wipe the table other than 'rm 1' in parted? Cause that just removes the partition doesn't it?
 
Old 05-03-2012, 01:33 PM   #15
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,258

Rep: Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947Reputation: 1947
It looks like mkfs is erroring out due to the size of the partition:
Quote:
mkfs.ext4: Size of device /dev/sdb1 too big to be expressed in 32 bits using a blocksize of 4096.
Apparently ext3/4 is limited to 16TB with the default 4k blocks, yay!

I believe parted can change the block size, but I have no experience with that. 8k blocks should raise the limit to 32TB which would cover your partition.

Last edited by suicidaleggroll; 05-03-2012 at 01:35 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Which file system for a large lvm Carpo Slackware 15 04-03-2008 04:29 PM
large data transfer to raid results in system reboot captainkrypto Linux - Hardware 1 04-15-2006 08:47 PM
how large can a file system be? Wolf_Assassin Linux - Software 2 08-21-2004 03:31 PM
Large tar file taking huge disk space in ext3 file system pcwulf Linux - General 2 10-20-2003 08:45 AM
Large file system on redhat 7.3 stefanof Linux - Distributions 1 08-28-2002 09:25 AM


All times are GMT -5. The time now is 08:50 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration