I currently run a 6x500gb software raid5 on ext3 which is beginning to fill up.
/dev/md1 1kblocks: 2403601996 used: 2019513560 avail: 384088436 85% /save
I had it hovering around 1.5tb for a while until I recently filled it past 2tb. Since then, I've noticed that mkdirs seem to take up a lot of time.
This is a shared fileserver on a local network of 13 other computers. So at any given time, other people are accessing files over the network and I'll notice that it'll usually be transfering at ~1m/sec on average, 3-4m/s high and 10m/s peaks. The read/write is fine. Everything works great when I download onto it 10m/s from the internet while it's dishing out 10m/s to local users.
But whenever I use a mkdir on the raid, all other disk access stops for 10 seconds. Here's me doing a mkdir test.
I'll notice that shortly after I create this directory, I have a small window of time in which I can mkdir test2, mkdir test3, etc and it won't take up any time at all.
Here's me running fibmap script on a recently created directory on the raid which I put 6 files into.
As a comparison, I ran this script on a folder of 4 videos that are on my desktop (which is on a 2x250gb software raid1)
/dev/md0 1k blocks: 241251392 used: 58511572 avail: 170581436 26% /
fibmap.pl is a script from here: http://www2.lut.fi/~ilonen/ext3_fragmentation.html
Hello lqdime :)
Is the parent directory large and busy, especially busy with new files being created?
I have a feeling (a forgotten memory?) that the effect is more pronounced when creating a new direcory than when creating a new file but I can't imagine why that would be so. :twocents:
Hi. Thanks for the reply!
Indeed, the directory /save/s/ is rather large and has 11,765 items, totalling 1333.7 GB
However, I've noticed that the same issue when doing a mkdir in other subdirectories as well, such as the /save/o/ directory, which has 8,332 items, totalling 122.0 GB. The 2.5tb raid is mounted on /save/.
My issue is that the /o/ directory hasn't changed in months while the /s/ directory has grown 250 gigs+ in the last month. But these issues affect the entire raid when a month ago, they didn't affect anything.
I'm more worried this may a limitation on ext3 or software raids. Is there a certain % of space used I shouldn't exceed?
I don't know (and can't find) enough information about what happens during mkdir to help much but ...
Having so many entries in a directory will be slower than having more, smaller directories containing the same number of files because ...
The system may pull the whole directory into memory before working on it -- a lot of I/O.
If the volume is nearly full (say > 85%), that huge directory will become fragmented, requiring a lot of seeks and so taking a longer time.
Maybe mkdir results in the whole directory structure being written back to disk to ensure directory integrity-- again onerous on a fragmented file system, especially as the file system may write a copy of the directory before deleting the old one, again having to find and write to many locations to do so.
These theories are general to any file system, not RAID-specific.
Sorry this is all speculation and not written from a position of real knowledge but nobody with that knowledge is replying yet.
I've since moved the server offline while I install a new one with a larger capacity. Here is an example of a mkdir after a fresh reboot with the computer offline and no processes accessing the raid.
dime@lanfear:/save/s/d/3$ time mkdir NFOs
I've seen other times when it's taken a full minute also with no other processes accessing the raid. So this is simply a matter of ext3 or software raid.
Hi... I'm having the exact same problem. So I was wondering if you already got a solution for this problem?
Unfortunately, I don't have anything to update yet. I am in the process of building another file server so I may be able to use this one for testing soon if I get the time.
I have the exact same problem on an external (USB2.0) WD MyBook with 2*1TB raid0. The ext3 volume is filled 90%. Folders with big sub-structures are more likely to have this problem. What can I do to help find out whats wrong?
I realize this is an old thread, I'm just wondering if anybody has found a resolution yet.
I have a Fedora 12 machine with a 12TB ext3 RAID6 (3ware 9750-8i with 8 2TB Hitachis). It's sitting at 91% capacity, a little over 1TB free. File creation, editing, reading are all very fast, but mkdir takes an exceptionally long time, usually 20+ seconds.
|All times are GMT -5. The time now is 11:13 PM.|