After creating raid6 array, mkfs.jfs /dev/md0, then cannot mount /dev/md0

artagel · 12-04-2009, 11:01 AM

Hi,
I've build this new array several times, and I can't seem to get it to mount.
I've searched around, but all the other solutions I've found don't seem to work for me.

I have 14 1.5TB drives, named sda-n in /dev
I do an fdisk /dev/sdx, new part, primary, 1, largest size, then t, fd, then write.
I do this for all the drives..
All my drives look like this:

Quote:

Disk /dev/sdn: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x04000000

Device Boot Start End Blocks Id System
/dev/sdn1 1 182401 1465136001 fd Linux raid autodetect

At this time, in /dev I see all the devices and the partitions...

Next I do this:
mdadm --create --verbose /dev/md0 --leve=raid6 --chunk=16 --raid-devices=14 /dev/sd{a,b,c,d,e,f,g,h,i,j,k,l,m,n}

That returns:

Quote:

mdadm: layout defaults to left-symmetric
mdadm: size set to 1465138496K
mdadm: array /dev/md0 started.

Now the partitions are all gone in /dev but they still show up in fdisk -l.
/dev/md0 says this:

Quote:

Disk /dev/md0: 18003.6 GB, 18003621838848 bytes
255 heads, 63 sectors/track, 2188815 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/md0p1 1 182401 1465136001 fd Linux raid autodetect

But I don't see md0p1 in /dev

file -Ls /dev/md0 shows:

Quote:

/dev/md0: x86 boot sector; partition 1: ID=0xfd, starthead 1, startsector 63, 2930272002 sectors

cat /proc/mdstat shows:

Quote:

Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sdn[13] sdm[12] sdl[11] sdk[10] sdj[9] sdi[8] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda[0]
17581661952 blocks level 6, 16k chunk, algorithm 2 [14/14] [UUUUUUUUUUUUUU]
resync=PENDING

unused devices: <none>

mdadm --detail /dev/md0 shows:

Quote:

/dev/md0:
Version : 00.90
Creation Time : Fri Dec 4 08:46:06 2009
Raid Level : raid6
Array Size : 17581661952 (16767.18 GiB 18003.62 GB)
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 14
Total Devices : 14
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Fri Dec 4 08:46:06 2009
State : clean
Active Devices : 14
Working Devices : 14
Failed Devices : 0
Spare Devices : 0

Chunk Size : 16K

UUID : 556004d8:88ed9e8a:9959903a:56f84fa3 (local to host Gigantor)
Events : 0.1

Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 32 2 active sync /dev/sdc
3 8 48 3 active sync /dev/sdd
4 8 64 4 active sync /dev/sde
5 8 80 5 active sync /dev/sdf
6 8 96 6 active sync /dev/sdg
7 8 112 7 active sync /dev/sdh
8 8 128 8 active sync /dev/sdi
9 8 144 9 active sync /dev/sdj
10 8 160 10 active sync /dev/sdk
11 8 176 11 active sync /dev/sdl
12 8 192 12 active sync /dev/sdm
13 8 208 13 active sync /dev/sdn

I add the array to /etc/mdadm/mdadm.conf
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Quote:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 01 Dec 2009 05:59:57 -0800
# by mkconf $Id$
ARRAY /dev/md0 level=raid6 num-devices=14 metadata=00.90 UUID=556004d8:88ed9e8a:9959903a:56f84fa3

Next I do mkfs -t jfs /dev/md0
mkfs -t jfs /dev/md0

Quote:

mkfs.jfs version 1.1.12, 24-Aug-2007
Warning! All data on device /dev/md0 will be lost!

Continue? (Y/N) y
\

Format completed successfully.

17581661952 kilobytes total disk space.

Then I try to mount md0:

Quote:

mount -t jfs /dev/md0 /mnt/Robot
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

cat /proc/mdstat now shows resyncing

Quote:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdn[13] sdm[12] sdl[11] sdk[10] sdj[9] sdi[8] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda[0]
17581661952 blocks level 6, 16k chunk, algorithm 2 [14/14] [UUUUUUUUUUUUUU]
[>....................] resync = 0.1% (1743368/1465138496) finish=935.1min speed=26077K/sec

My Debian version:
Linux version 2.6.26-2-686 (Debian 2.6.26-19lenny2) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Wed Nov 4 20:45:37 UTC 2009

Can anyone help me figure out why this isn't mounting? I can't think of any other way to make this work.
Thanks for the help.
-Dan

artagel · 12-05-2009, 10:47 AM

So I installed strace, and I found an error at the end that says:
ioctl(3, BLKGETSIZE, 0xbfc83500) = -1 EFBIG (File too large)

Is my array too big!?

**EDIT:
I just recreated the array with only 13 disks, and now it's mountable. Why can't I do more than 13 disk? jfs is suppose to support like 4 petabytes...

Here is the whole trace:

Quote:

execve("/bin/mount", ["mount", "-t", "jfs", "/dev/md0", "/mnt/Robot"], [/* 18 vars */]) = 0
brk(0) = 0x82c7000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fa4000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=55278, ...}) = 0
mmap2(NULL, 55278, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f96000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libblkid.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320!\0\0004\0\0\0L"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=38020, ...}) = 0
mmap2(NULL, 40824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f8c000
mmap2(0xb7f95000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8) = 0xb7f95000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libuuid.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\17\0\0004\0\0\0\350"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=12912, ...}) = 0
mmap2(NULL, 15624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f88000
mmap2(0xb7f8b000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xb7f8b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libselinux.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260B\0\0004\0\0\0\314"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=95964, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f87000
mmap2(NULL, 101276, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f6e000
mmap2(0xb7f85000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb7f85000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/i686/cmov/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260e\1\0004\0\0\0\4"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1413540, ...}) = 0
mmap2(NULL, 1418864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e13000
mmap2(0xb7f68000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x155) = 0xb7f68000
mmap2(0xb7f6b000, 9840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f6b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/i686/cmov/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\n\0\0004\0\0\0H"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9680, ...}) = 0
mmap2(NULL, 12412, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e0f000
mmap2(0xb7e11000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7e11000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7e0e000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e0e730, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7f68000, 4096, PROT_READ) = 0
munmap(0xb7f96000, 55278) = 0
brk(0) = 0x82c7000
brk(0x82e8000) = 0x82e8000
open("/etc/selinux/config", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
statfs64("/selinux", 84, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=18578217, f_bfree=17708473, f_bavail=16764756, f_files=4718592, f_ffree=4603083, f_fsid={-753487364, 238370395}, f_namelen=255, f_frsize=4096}) = 0
open("/proc/mounts", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fa3000
read(3, "rootfs / rootfs rw 0 0\nnone /sys "..., 1024) = 440
read(3, ""..., 1024) = 0
close(3) = 0
munmap(0xb7fa3000, 4096) = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1282816, ...}) = 0
mmap2(NULL, 1282816, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7cd4000
close(3) = 0
umask(022) = 022
open("/dev/null", O_RDWR|O_LARGEFILE) = 3
close(3) = 0
getuid32() = 0
geteuid32() = 0
lstat64("/etc/mtab", {st_mode=S_IFREG|0644, st_size=380, ...}) = 0
readlink("/dev", 0xbfc814bb, 4096) = -1 EINVAL (Invalid argument)
readlink("/dev/md0", 0xbfc814bb, 4096) = -1 EINVAL (Invalid argument)
stat64("/sbin/mount.jfs", 0xbfc832e0) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
stat64("/sbin/mount.jfs", 0xbfc832b0) = -1 ENOENT (No such file or directory)
mount("/dev/md0", "/mnt/Robot", "jfs", MS_MGC_VAL, NULL) = -1 EINVAL (Invalid argument)
rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=2586, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fa3000
read(3, "# Locale name alias data base.\n# "..., 4096) = 2586
read(3, ""..., 4096) = 0
close(3) = 0
munmap(0xb7fa3000, 4096) = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/util-linux-ng.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "mount: wrong fs type, bad option,"..., 119mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error) = 119
write(2, "\n"..., 1
) = 1
stat64("/dev/md0", {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 0), ...}) = 0
open("/dev/md0", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 3
ioctl(3, BLKGETSIZE, 0xbfc83500) = -1 EFBIG (File too large)
write(2, " In some cases useful info "..., 85 In some cases useful info is found in syslog - try
dmesg | tail or so
) = 85
write(2, "\n"..., 1
) = 1
exit_group(32) = ?

AwesomeMachine · 12-05-2009, 08:31 PM

I'm glad it works now. There are some limits to how many physical disks can be in a single array. But you can combine arrays into logical volumes (LV). You might consider making 2 arrays, and making a single logical volume out of both arrays.

artagel · 12-06-2009, 12:57 AM

I've seen some posts about making 2 arrays into a volume. I'll look it up. Thanks.

artagel · 12-06-2009, 08:30 AM

The problem was that I was using 32bit debian. I install 64bit debian and a 14 disk array had no issue mounting.
Weird....