mkdir slow

lqdime · 07-28-2009, 11:53 AM

I currently run a 6x500gb software raid5 on ext3 which is beginning to fill up.

/dev/md1 1kblocks: 2403601996 used: 2019513560 avail: 384088436 85% /save

I had it hovering around 1.5tb for a while until I recently filled it past 2tb. Since then, I've noticed that mkdirs seem to take up a lot of time.

This is a shared fileserver on a local network of 13 other computers. So at any given time, other people are accessing files over the network and I'll notice that it'll usually be transfering at ~1m/sec on average, 3-4m/s high and 10m/s peaks. The read/write is fine. Everything works great when I download onto it 10m/s from the internet while it's dishing out 10m/s to local users.

But whenever I use a mkdir on the raid, all other disk access stops for 10 seconds. Here's me doing a mkdir test.

http://img11.imageshack.us/img11/2286/mkdir.png

I'll notice that shortly after I create this directory, I have a small window of time in which I can mkdir test2, mkdir test3, etc and it won't take up any time at all.

Here's me running fibmap script on a recently created directory on the raid which I put 6 files into.

Code:

root@lanfear:~/fibmap# ./fibmap.pl /save/s/ps/1/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
    80  231.2MB  2959.95kB /save/s/ps/1/1.avi
    61  174.4MB  2927.02kB /save/s/ps/1/3.avi
    61  231.7MB  3889.31kB /save/s/ps/1/6.avi
    49  174.4MB  3645.63kB /save/s/ps/1/5.avi
    48  174.3MB  3719.08kB /save/s/ps/1/2.avi
    47  174.4MB  3800.17kB /save/s/ps/1/4.avi
Non-contiguous:
  Files 6 (1160.5MB, avg. 198055.67kB per file), blocks 346, average block 3434.49kB
No contiguous files found!

The first number is how many fragments the file is split into. Second # is the file size and 3rd # is the average size of each fragment.

As a comparison, I ran this script on a folder of 4 videos that are on my desktop (which is on a 2x250gb software raid1)

Code:

root@lanfear:~/fibmap# ./fibmap.pl ~/Desktop/temp/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
   316  623.4MB  2020.24kB /home/dime/Desktop/temp/ap.wmv
   276  524.3MB  1945.29kB /home/dime/Desktop/temp/p2.avi
   220  587.9MB  2736.22kB /home/dime/Desktop/temp/dh.mp4
   165  327.9MB  2034.74kB /home/dime/Desktop/temp/im.wmv
Non-contiguous:
  Files 4 (2063.5MB, avg. 528247.11kB per file), blocks 977, average block 2162.73kB
No contiguous files found!

So, it doesn't seem like the raid is anymore fragmented than my normal system drive, which is sitting pretty at:
/dev/md0 1k blocks: 241251392 used: 58511572 avail: 170581436 26% /

fibmap.pl is a script from here: http://www2.lut.fi/~ilonen/ext3_fragmentation.html

catkin · 07-29-2009, 12:48 PM

Hello lqdime

Is the parent directory large and busy, especially busy with new files being created?

I believe:

directory reads lock the directory against writes, allowing multiple read access.
directory writes takes an exclusive write lock, disallowing any reads or writes.

If I'm correct then, when you run mkdir, it queues for the exclusive write lock, waiting for all/any directory reads to complete and disallowing any new directory read or write requests. For a large and busy directory this could take a "long" time and would block opening files in the directory (for both read and write). If the usage of files in the directory meant files were not held open but re-opened frequently then file opens would be queued until the exclusive write lock is granted, the new file-or-directory created and the exclusive write lock released. This would have the effect of stopping the usual file I/O on files in that directory.

I have a feeling (a forgotten memory?) that the effect is more pronounced when creating a new direcory than when creating a new file but I can't imagine why that would be so.

Best

Charles

lqdime · 07-31-2009, 12:25 AM

Hi. Thanks for the reply!

Indeed, the directory /save/s/ is rather large and has 11,765 items, totalling 1333.7 GB

However, I've noticed that the same issue when doing a mkdir in other subdirectories as well, such as the /save/o/ directory, which has 8,332 items, totalling 122.0 GB. The 2.5tb raid is mounted on /save/.

My issue is that the /o/ directory hasn't changed in months while the /s/ directory has grown 250 gigs+ in the last month. But these issues affect the entire raid when a month ago, they didn't affect anything.

I'm more worried this may a limitation on ext3 or software raids. Is there a certain % of space used I shouldn't exceed?

catkin · 07-31-2009, 07:52 AM

I don't know (and can't find) enough information about what happens during mkdir to help much but ...

Having so many entries in a directory will be slower than having more, smaller directories containing the same number of files because ...

The system may pull the whole directory into memory before working on it -- a lot of I/O.

If the volume is nearly full (say > 85%), that huge directory will become fragmented, requiring a lot of seeks and so taking a longer time.

Maybe mkdir results in the whole directory structure being written back to disk to ensure directory integrity-- again onerous on a fragmented file system, especially as the file system may write a copy of the directory before deleting the old one, again having to find and write to many locations to do so.

These theories are general to any file system, not RAID-specific.

Sorry this is all speculation and not written from a position of real knowledge but nobody with that knowledge is replying yet.

lqdime · 08-06-2009, 02:34 AM

I've since moved the server offline while I install a new one with a larger capacity. Here is an example of a mkdir after a fresh reboot with the computer offline and no processes accessing the raid.

dime@lanfear:/save/s/d/3$ time mkdir NFOs

real 0m34.539s
user 0m0.000s
sys 0m0.136s

I've seen other times when it's taken a full minute also with no other processes accessing the raid. So this is simply a matter of ext3 or software raid.