LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 07-28-2009, 11:53 AM   #1
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Rep: Reputation: 0
mkdir slow


I currently run a 6x500gb software raid5 on ext3 which is beginning to fill up.

/dev/md1 1kblocks: 2403601996 used: 2019513560 avail: 384088436 85% /save

I had it hovering around 1.5tb for a while until I recently filled it past 2tb. Since then, I've noticed that mkdirs seem to take up a lot of time.

This is a shared fileserver on a local network of 13 other computers. So at any given time, other people are accessing files over the network and I'll notice that it'll usually be transfering at ~1m/sec on average, 3-4m/s high and 10m/s peaks. The read/write is fine. Everything works great when I download onto it 10m/s from the internet while it's dishing out 10m/s to local users.

But whenever I use a mkdir on the raid, all other disk access stops for 10 seconds. Here's me doing a mkdir test.



I'll notice that shortly after I create this directory, I have a small window of time in which I can mkdir test2, mkdir test3, etc and it won't take up any time at all.

Here's me running fibmap script on a recently created directory on the raid which I put 6 files into.

Code:
root@lanfear:~/fibmap# ./fibmap.pl /save/s/ps/1/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
    80  231.2MB  2959.95kB /save/s/ps/1/1.avi
    61  174.4MB  2927.02kB /save/s/ps/1/3.avi
    61  231.7MB  3889.31kB /save/s/ps/1/6.avi
    49  174.4MB  3645.63kB /save/s/ps/1/5.avi
    48  174.3MB  3719.08kB /save/s/ps/1/2.avi
    47  174.4MB  3800.17kB /save/s/ps/1/4.avi
Non-contiguous:
  Files 6 (1160.5MB, avg. 198055.67kB per file), blocks 346, average block 3434.49kB
No contiguous files found!
The first number is how many fragments the file is split into. Second # is the file size and 3rd # is the average size of each fragment.

As a comparison, I ran this script on a folder of 4 videos that are on my desktop (which is on a 2x250gb software raid1)

Code:
root@lanfear:~/fibmap# ./fibmap.pl ~/Desktop/temp/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
   316  623.4MB  2020.24kB /home/dime/Desktop/temp/ap.wmv
   276  524.3MB  1945.29kB /home/dime/Desktop/temp/p2.avi
   220  587.9MB  2736.22kB /home/dime/Desktop/temp/dh.mp4
   165  327.9MB  2034.74kB /home/dime/Desktop/temp/im.wmv
Non-contiguous:
  Files 4 (2063.5MB, avg. 528247.11kB per file), blocks 977, average block 2162.73kB
No contiguous files found!
So, it doesn't seem like the raid is anymore fragmented than my normal system drive, which is sitting pretty at:
/dev/md0 1k blocks: 241251392 used: 58511572 avail: 170581436 26% /

fibmap.pl is a script from here: http://www2.lut.fi/~ilonen/ext3_fragmentation.html
 
Old 07-29-2009, 12:48 PM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Hello lqdime

Is the parent directory large and busy, especially busy with new files being created?

I believe:
  • directory reads lock the directory against writes, allowing multiple read access.
  • directory writes takes an exclusive write lock, disallowing any reads or writes.
If I'm correct then, when you run mkdir, it queues for the exclusive write lock, waiting for all/any directory reads to complete and disallowing any new directory read or write requests. For a large and busy directory this could take a "long" time and would block opening files in the directory (for both read and write). If the usage of files in the directory meant files were not held open but re-opened frequently then file opens would be queued until the exclusive write lock is granted, the new file-or-directory created and the exclusive write lock released. This would have the effect of stopping the usual file I/O on files in that directory.

I have a feeling (a forgotten memory?) that the effect is more pronounced when creating a new direcory than when creating a new file but I can't imagine why that would be so.

Best

Charles
 
Old 07-31-2009, 12:25 AM   #3
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Hi. Thanks for the reply!

Indeed, the directory /save/s/ is rather large and has 11,765 items, totalling 1333.7 GB

However, I've noticed that the same issue when doing a mkdir in other subdirectories as well, such as the /save/o/ directory, which has 8,332 items, totalling 122.0 GB. The 2.5tb raid is mounted on /save/.

My issue is that the /o/ directory hasn't changed in months while the /s/ directory has grown 250 gigs+ in the last month. But these issues affect the entire raid when a month ago, they didn't affect anything.

I'm more worried this may a limitation on ext3 or software raids. Is there a certain % of space used I shouldn't exceed?
 
Old 07-31-2009, 07:52 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
I don't know (and can't find) enough information about what happens during mkdir to help much but ...

Having so many entries in a directory will be slower than having more, smaller directories containing the same number of files because ...

The system may pull the whole directory into memory before working on it -- a lot of I/O.

If the volume is nearly full (say > 85%), that huge directory will become fragmented, requiring a lot of seeks and so taking a longer time.

Maybe mkdir results in the whole directory structure being written back to disk to ensure directory integrity-- again onerous on a fragmented file system, especially as the file system may write a copy of the directory before deleting the old one, again having to find and write to many locations to do so.

These theories are general to any file system, not RAID-specific.

Sorry this is all speculation and not written from a position of real knowledge but nobody with that knowledge is replying yet.
 
Old 08-06-2009, 02:34 AM   #5
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
I've since moved the server offline while I install a new one with a larger capacity. Here is an example of a mkdir after a fresh reboot with the computer offline and no processes accessing the raid.

dime@lanfear:/save/s/d/3$ time mkdir NFOs

real 0m34.539s
user 0m0.000s
sys 0m0.136s

I've seen other times when it's taken a full minute also with no other processes accessing the raid. So this is simply a matter of ext3 or software raid.
 
Old 11-25-2009, 11:47 AM   #6
kscholte
LQ Newbie
 
Registered: Nov 2009
Posts: 1

Rep: Reputation: 0
slow mkdir

Hi... I'm having the exact same problem. So I was wondering if you already got a solution for this problem?
 
Old 11-26-2009, 02:32 AM   #7
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Unfortunately, I don't have anything to update yet. I am in the process of building another file server so I may be able to use this one for testing soon if I get the time.
 
Old 01-10-2010, 04:25 AM   #8
Sunday87
LQ Newbie
 
Registered: Jan 2010
Posts: 1

Rep: Reputation: 0
I have the exact same problem on an external (USB2.0) WD MyBook with 2*1TB raid0. The ext3 volume is filled 90%. Folders with big sub-structures are more likely to have this problem. What can I do to help find out whats wrong?
 
Old 11-29-2010, 12:36 PM   #9
suicidaleggroll
Senior Member
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 3,210

Rep: Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140Reputation: 1140
I realize this is an old thread, I'm just wondering if anybody has found a resolution yet.

I have a Fedora 12 machine with a 12TB ext3 RAID6 (3ware 9750-8i with 8 2TB Hitachis). It's sitting at 91% capacity, a little over 1TB free. File creation, editing, reading are all very fast, but mkdir takes an exceptionally long time, usually 20+ seconds.
 
  


Reply

Tags
ext3, mdadm, raid5


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
mkdir -p ? shanenin Linux - Software 4 10-19-2011 12:41 PM
I don't know. mkdir? Bobzilla2639716495 Linux - General 15 12-03-2007 12:22 PM
mkdir options sravanth.svk Linux - Newbie 4 10-26-2006 01:31 AM
mkdir help... Linux~Powered Linux - Software 2 06-13-2004 12:27 PM
mkdir help ITJedi Linux - Software 1 05-14-2004 06:19 PM


All times are GMT -5. The time now is 01:16 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration