Very long mount times of large ext4 filesystems

szafran81 · 09-14-2012, 06:53 AM

Hi,

I'm having problems with the mount times of larger filesystems (I use ext4).

Info:

Code:

root@NAS:~# time mount /dev/vdc /mnt/tempik

real	0m49.696s
user	0m0.000s
sys	0m49.339s
root@NAS:~# time mount /dev/vdb1 /mnt/nowy

real	1m26.682s
user	0m0.004s
sys	1m25.949s
root@NAS:~# uname -a
Linux NAS 3.2.0-31-generic #50-Ubuntu SMP Fri Sep 7 16:16:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Both those devs have ~10TBs. The first one is "normal" 32bit fs, and the second one is a 64bit fs (I'm using standard v1.42 e2fsprogs). Smaller drives (eg. 2TB) are mounted without problems in a reasonable time (0-2secs). I need at least one of those devs to be mounted during boot (they're not boot drives, nor root fs - they just contain lots of data that some of my startup apps need), but with those mount times I'm getting timeouts during boot. All filesystems are clean.

..a few hours after installation...

I've resized one of the fs to 16TB, and now the mount times are even more abstract:

Code:

root@NAS:~# time mount /dev/vdb1 /mnt/nowy

real	4m19.015s
user	0m0.000s
sys	4m17.964s

I can even live with the OSs boot time of >4min, but how to configure it to wait for that fs to mount properly instead of giving a timeout error during boot, and giving up the mount process. Is there a way to set that timeout ? (in grub, or maybe fstab?)

sundialsvcs · 09-14-2012, 07:21 AM

All that I can offer here is speculation and possible work-arounds ...

Let's say that you don't mount the file systems "during boot," but rather that you do it in a one-time startup script that is attached to this runlevel. Perhaps such a script could first mount the filesystem, then launch the remainder of the startup-time applications. (Per-application scripts could also check to see if the required filesystems are mounted yet, and wait-loop a certain number of cycles until they are. These would be used when you wanted to re-start one of these applications after initial booting.)

In any case, if you could mount the filesystems from the command-line, perhaps using the -v ("verbose") command-line switch, you might be able to deduce from this what is actually going wrong with regards to this drive. This behavior, in my mind, "obviously shouldn't be happening," but from the boot-cycle you can't really see it. Maybe "verbose" will give you that clue.

It would also be interesting to observe what does (or doesn't ...) happen if you dismount and re-mount the drive after startup. Does it take the same (excessive) amount of time? Or, not? Would be good to know that.

szafran81 · 09-14-2012, 11:21 AM

Thank you for your reply.
I was at work while writing the post, so I didn't have time to be more specific.
Here goes the whole story:
The machine i question is my home server/NAS. For some time It had Ubuntu 11.x (don't remember exactly if it was 04 or 10), and then on Ubuntu (using VMWare) I've had running Windows 7 (since unfortunatelly I do need some apps that work only on under Win). About 2 weeks ago I've decided to use Proxmox VE (http://proxmox.com/products/proxmox-ve), and to put on it Ubuntu 12.04.1 and Windows 7. And then it all began.

HW is:
Some Gigabyte motherboard with Phenom x6 3,2GHz and 16GB of RAM and some Radeon integrated video card.
The motherboard has 6 SATA3 and 2 SATA2 ports, in addition to that there's a LSI chip based RAID card (2SAS / 8SATA2 ports).
And now for the HDDs:
System (Proxmox, and the VMs): 3x Seagate 500GB (RAID0)
Storage (other stuff): 12x WD 2TB (RAID5) (waiting for the 13th drive to make it RAID6)

Info from Proxmox VE (this is based on Debian Squeeze):

Code:

root@szafran:/var# uname -a
Linux szafran 2.6.32-14-pve #1 SMP Tue Aug 21 08:24:37 CEST 2012 x86_64 GNU/Linux
root@szafran:/var# resize2fs
resize2fs 1.41.12 (17-May-2010)

Both RAIDs are software mdadm, and both are assembled under Proxmox (Debian). Then on the RAID5 array there's an LVM (one VG; one PV; two LVs). Ubuntu uses the LVs (since it's the fastest config to run under VM).

To add to the first post, after a few hours of the 16TB fs mounted and copying data to it, it hang :/ The OS still said it was mounted etc, but the fs wasn't doing anything and not responding to anything (on both Ubuntu and Proxmox/Debian). Had to reboot the physical machine.

And now some more mounting info (under Ubuntu VM):

Code:

root@NAS:/mnt# blockdev --setra 202752 /dev/vdb
root@NAS:/mnt# blockdev --setra 202752 /dev/vdc

root@NAS:/mnt# time mount -v -t ext4 /dev/vdb1 nowy
/dev/vdb1 on /mnt/nowy type ext4 (rw)

real	4m56.898s
user	0m0.000s
sys	4m55.698s

root@NAS:/mnt# time umount nowy

real	0m0.134s
user	0m0.000s
sys	0m0.016s

root@NAS:/mnt# time mount -v -t ext4 /dev/vdb1 nowy
/dev/vdb1 on /mnt/nowy type ext4 (rw)

real	4m56.916s
user	0m0.000s
sys	4m55.822s

root@NAS:/mnt# time umount nowy

real	0m0.133s
user	0m0.000s
sys	0m0.016s

root@NAS:/mnt# time mount -v -t ext4 /dev/vdc1 nowy2
/dev/vdc1 on /mnt/nowy2 type ext4 (rw)

real	0m14.379s
user	0m0.000s
sys	0m11.829s

root@NAS:/mnt# time umount nowy2

real	0m0.113s
user	0m0.000s
sys	0m0.012s

root@NAS:/mnt# time mount -v -t ext4 /dev/vdc1 nowy2
/dev/vdc1 on /mnt/nowy2 type ext4 (rw)

real	0m12.323s
user	0m0.000s
sys	0m11.729s

root@NAS:/mnt# time umount nowy2

real	0m0.446s
user	0m0.000s
sys	0m0.004s

vdb1 is 16TB with 9,7TB used.
vdc1 is 4TB and it's just free space.
Both fs were created using:

Code:

mkfs -O 64bit,extent,has_journal,uninit_bg,sparse_super,dir_index,large_file,flex_bg -t ext4 -T huge -b 4096 -v -m 0 -E stride=128,stripe_width=1536

As you can from the above - even after setting some cache for the drives (around 800MB per dev), and subsequent mounts - the mount times are horrible.
After that timed mounts I've run e2fsck -v -f on vdb1 - it still says that the fs is clean.
I've read somwhere that there's a problem with long mount times, but that supposed to be with the 3.4.X kernel, and I'm working on 3.2.X so it should be fine.

syg00 · 09-14-2012, 08:58 PM

How long to mount them on the host system rather than the guests ?.

szafran81 · 09-15-2012, 12:29 AM

on average a couple of seconds (but mostly under 1s) - the same goes for Ubuntu 11.10 installed under Proxmox. So it's Ubuntu 12.04.1's fault, but I really have no idea why :/

syg00 · 09-15-2012, 01:48 AM

If you can't get a handle on it, open a bug - or use another distro.

szafran81 · 09-15-2012, 02:16 AM

So, I can safely assume that there's no way of changing the mount timeout during boot ?
If so, then I think I'll make me a boot script as sundialsvcs proposed.

szafran81 · 09-15-2012, 04:07 AM

I've just re-installed Ubuntu 11.10 VM just to be sure (with exactly the same config that the 12.04.1 has). And on a clean install:

Code:

root@ubuntu-tempik:/mnt# time mount -v -t ext4 /dev/vdb1 nowy
/dev/vdb1 on /mnt/nowy type ext4 (rw)

real    0m0.668s
user    0m0.000s
sys     0m0.148s
root@ubuntu-tempik:/mnt# time mount -v -t ext4 /dev/vdc1 nowy2
/dev/vdc1 on /mnt/nowy2 type ext4 (rw)

real    0m0.314s
user    0m0.000s
sys     0m0.096s
root@ubuntu-tempik:/mnt# uname -a
Linux ubuntu-tempik 3.0.0-26-server #42-Ubuntu SMP Wed Sep 5 08:55:42 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

After that I've turned on the proposed updates, and updated:

Code:

root@ubuntu-tempik:/mnt# time mount -v -t ext4 /dev/vdb1 nowy
/dev/vdb1 on /mnt/nowy type ext4 (rw)

real    4m18.438s
user    0m0.000s
sys     4m17.432s
root@ubuntu-tempik:/mnt# uname -a
Linux ubuntu-tempik 3.2.0-30-generic #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

So it's the kernels fault. I wonder if I've turned on the proposed updates on the 12.04.1 to soon.
Is there a way for me to get back to the state before proposed updates, or do I need to reinstall a clean OS ?

EDIT:
I've just answered myself. On a clean install 12.04.1 the mount times are ok, but just after a kernel update everything goes slow.

szafran81 · 09-15-2012, 05:54 AM

I've reverted to kernel 3.2.0-29-generic, and everything is fine. I'll wait with kernel updates untill they fix it.

syg00 · 09-15-2012, 05:31 PM

If you don't open a bug it may never get fixed. Nothing jumped out at me on launchpad, so this may not be "known".
The Ubuntu folks will push it upstream in need.

szafran81 · 09-16-2012, 02:10 AM

The thing is I don't have any idea on how to open a bug (didn't really find any bugs that are new untill now).

syg00 · 09-16-2012, 03:18 AM

Have a read of this.

szafran81 · 09-16-2012, 03:25 AM

Thank you for your help. I'll try to dig in to it today.