LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 07-28-2009, 10:53 AM   #1
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Rep: Reputation: 0
mkdir slow


I currently run a 6x500gb software raid5 on ext3 which is beginning to fill up.

/dev/md1 1kblocks: 2403601996 used: 2019513560 avail: 384088436 85% /save

I had it hovering around 1.5tb for a while until I recently filled it past 2tb. Since then, I've noticed that mkdirs seem to take up a lot of time.

This is a shared fileserver on a local network of 13 other computers. So at any given time, other people are accessing files over the network and I'll notice that it'll usually be transfering at ~1m/sec on average, 3-4m/s high and 10m/s peaks. The read/write is fine. Everything works great when I download onto it 10m/s from the internet while it's dishing out 10m/s to local users.

But whenever I use a mkdir on the raid, all other disk access stops for 10 seconds. Here's me doing a mkdir test.



I'll notice that shortly after I create this directory, I have a small window of time in which I can mkdir test2, mkdir test3, etc and it won't take up any time at all.

Here's me running fibmap script on a recently created directory on the raid which I put 6 files into.

Code:
root@lanfear:~/fibmap# ./fibmap.pl /save/s/ps/1/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
    80  231.2MB  2959.95kB /save/s/ps/1/1.avi
    61  174.4MB  2927.02kB /save/s/ps/1/3.avi
    61  231.7MB  3889.31kB /save/s/ps/1/6.avi
    49  174.4MB  3645.63kB /save/s/ps/1/5.avi
    48  174.3MB  3719.08kB /save/s/ps/1/2.avi
    47  174.4MB  3800.17kB /save/s/ps/1/4.avi
Non-contiguous:
  Files 6 (1160.5MB, avg. 198055.67kB per file), blocks 346, average block 3434.49kB
No contiguous files found!
The first number is how many fragments the file is split into. Second # is the file size and 3rd # is the average size of each fragment.

As a comparison, I ran this script on a folder of 4 videos that are on my desktop (which is on a 2x250gb software raid1)

Code:
root@lanfear:~/fibmap# ./fibmap.pl ~/Desktop/temp/
Configurator ran OK; FIBMAP is 1; BLOCK_SIZE is 4096
   316  623.4MB  2020.24kB /home/dime/Desktop/temp/ap.wmv
   276  524.3MB  1945.29kB /home/dime/Desktop/temp/p2.avi
   220  587.9MB  2736.22kB /home/dime/Desktop/temp/dh.mp4
   165  327.9MB  2034.74kB /home/dime/Desktop/temp/im.wmv
Non-contiguous:
  Files 4 (2063.5MB, avg. 528247.11kB per file), blocks 977, average block 2162.73kB
No contiguous files found!
So, it doesn't seem like the raid is anymore fragmented than my normal system drive, which is sitting pretty at:
/dev/md0 1k blocks: 241251392 used: 58511572 avail: 170581436 26% /

fibmap.pl is a script from here: http://www2.lut.fi/~ilonen/ext3_fragmentation.html
 
Old 07-29-2009, 11:48 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Hello lqdime

Is the parent directory large and busy, especially busy with new files being created?

I believe:
  • directory reads lock the directory against writes, allowing multiple read access.
  • directory writes takes an exclusive write lock, disallowing any reads or writes.
If I'm correct then, when you run mkdir, it queues for the exclusive write lock, waiting for all/any directory reads to complete and disallowing any new directory read or write requests. For a large and busy directory this could take a "long" time and would block opening files in the directory (for both read and write). If the usage of files in the directory meant files were not held open but re-opened frequently then file opens would be queued until the exclusive write lock is granted, the new file-or-directory created and the exclusive write lock released. This would have the effect of stopping the usual file I/O on files in that directory.

I have a feeling (a forgotten memory?) that the effect is more pronounced when creating a new direcory than when creating a new file but I can't imagine why that would be so.

Best

Charles
 
Old 07-30-2009, 11:25 PM   #3
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Hi. Thanks for the reply!

Indeed, the directory /save/s/ is rather large and has 11,765 items, totalling 1333.7 GB

However, I've noticed that the same issue when doing a mkdir in other subdirectories as well, such as the /save/o/ directory, which has 8,332 items, totalling 122.0 GB. The 2.5tb raid is mounted on /save/.

My issue is that the /o/ directory hasn't changed in months while the /s/ directory has grown 250 gigs+ in the last month. But these issues affect the entire raid when a month ago, they didn't affect anything.

I'm more worried this may a limitation on ext3 or software raids. Is there a certain % of space used I shouldn't exceed?
 
Old 07-31-2009, 06:52 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
I don't know (and can't find) enough information about what happens during mkdir to help much but ...

Having so many entries in a directory will be slower than having more, smaller directories containing the same number of files because ...

The system may pull the whole directory into memory before working on it -- a lot of I/O.

If the volume is nearly full (say > 85%), that huge directory will become fragmented, requiring a lot of seeks and so taking a longer time.

Maybe mkdir results in the whole directory structure being written back to disk to ensure directory integrity-- again onerous on a fragmented file system, especially as the file system may write a copy of the directory before deleting the old one, again having to find and write to many locations to do so.

These theories are general to any file system, not RAID-specific.

Sorry this is all speculation and not written from a position of real knowledge but nobody with that knowledge is replying yet.
 
Old 08-06-2009, 01:34 AM   #5
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
I've since moved the server offline while I install a new one with a larger capacity. Here is an example of a mkdir after a fresh reboot with the computer offline and no processes accessing the raid.

dime@lanfear:/save/s/d/3$ time mkdir NFOs

real 0m34.539s
user 0m0.000s
sys 0m0.136s

I've seen other times when it's taken a full minute also with no other processes accessing the raid. So this is simply a matter of ext3 or software raid.
 
Old 11-25-2009, 10:47 AM   #6
kscholte
LQ Newbie
 
Registered: Nov 2009
Posts: 1

Rep: Reputation: 0
slow mkdir

Hi... I'm having the exact same problem. So I was wondering if you already got a solution for this problem?
 
Old 11-26-2009, 01:32 AM   #7
lqdime
LQ Newbie
 
Registered: Jun 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Unfortunately, I don't have anything to update yet. I am in the process of building another file server so I may be able to use this one for testing soon if I get the time.
 
Old 01-10-2010, 03:25 AM   #8
Sunday87
LQ Newbie
 
Registered: Jan 2010
Posts: 1

Rep: Reputation: 0
I have the exact same problem on an external (USB2.0) WD MyBook with 2*1TB raid0. The ext3 volume is filled 90%. Folders with big sub-structures are more likely to have this problem. What can I do to help find out whats wrong?
 
Old 11-29-2010, 11:36 AM   #9
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,569

Rep: Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129Reputation: 2129
I realize this is an old thread, I'm just wondering if anybody has found a resolution yet.

I have a Fedora 12 machine with a 12TB ext3 RAID6 (3ware 9750-8i with 8 2TB Hitachis). It's sitting at 91% capacity, a little over 1TB free. File creation, editing, reading are all very fast, but mkdir takes an exceptionally long time, usually 20+ seconds.
 
Old 04-30-2018, 02:16 AM   #10
jaatavaresf
LQ Newbie
 
Registered: Apr 2018
Posts: 3

Rep: Reputation: Disabled
Has anyone here identified the root cause of this?
It has passed 8 years since the last reply.

I have exactly the same issue described issue.
My config: ext3 FS, 5.6T, raid5, 4 drives with 2T.

Not too many small files or big directories. I think this happens since I started having some content in this raid5 setup.
It may be due to linux raid5 or some missing tuning flag in ext3 FS.

Thanks a lot.
 
Old 05-01-2018, 04:11 PM   #11
jaatavaresf
LQ Newbie
 
Registered: Apr 2018
Posts: 3

Rep: Reputation: Disabled
I also saw this in machine logs .....
Code:
May  1 17:53:17 machine kernel: [ 9240.652160] kjournald     D 00000000     0  6662      2 0x00000000
May  1 17:53:17 machine kernel: [ 9240.652250]  c3bda200 00000046 00000000 00000000 00000000 c1443e40 c1443e40 c143f354
May  1 17:53:17 machine kernel: [ 9240.652482]  c3bda3bc c8a88e40 00000001 00000000 00000000 00000000 00000000 00000000
May  1 17:53:17 machine kernel: [ 9240.652714]  c8a84354 c3bda3bc 00214f57 f704a200 c1006f50 00000000 00000000 00000000
May  1 17:53:17 machine kernel: [ 9240.652945] Call Trace:
May  1 17:53:17 machine kernel: [ 9240.652981]  [<c1006f50>] ? __switch_to+0xcf/0x141
May  1 17:53:17 machine kernel: [ 9240.653042]  [<f84f1b45>] ? journal_commit_transaction+0x117/0xd5a [jbd]
May  1 17:53:17 machine kernel: [ 9240.653077]  [<c10319eb>] ? finish_task_switch+0x34/0x95
May  1 17:53:17 machine kernel: [ 9240.653112]  [<c128016e>] ? schedule+0x78f/0x7dc
May  1 17:53:17 machine kernel: [ 9240.653146]  [<c104ab8a>] ? autoremove_wake_function+0x0/0x2d
May  1 17:53:17 machine kernel: [ 9240.653180]  [<c1041b38>] ? lock_timer_base+0x19/0x35
May  1 17:53:17 machine kernel: [ 9240.653213]  [<c1041d46>] ? try_to_del_timer_sync+0x4f/0x56
May  1 17:53:17 machine kernel: [ 9240.653261]  [<f84f48fa>] ? kjournald+0xb9/0x1e0 [jbd]
May  1 17:53:17 machine kernel: [ 9240.653295]  [<c104ab8a>] ? autoremove_wake_function+0x0/0x2d
May  1 17:53:17 machine kernel: [ 9240.653341]  [<f84f4841>] ? kjournald+0x0/0x1e0 [jbd]
May  1 17:53:17 machine kernel: [ 9240.653374]  [<c104a958>] ? kthread+0x61/0x66
May  1 17:53:17 machine kernel: [ 9240.653406]  [<c104a8f7>] ? kthread+0x0/0x66
May  1 17:53:17 machine kernel: [ 9240.653439]  [<c1008d87>] ? kernel_thread_helper+0x7/0x10
and I saw that the same happens with another ext3 FS in the same machine, which has 6.6T in an LVM.

Is this related to ext3 journaling? journaling size?

This machine is a debian squeeze with kernel 2.6.32-5-686-bigmem.

More details on one of the FS....
Code:
# tune2fs -l /dev/mapper/vg--internal--disks-home4
tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          8165e278-3763-440d-9512-63c77f95a185
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              366288896
Block count:              1465128960
Reserved block count:     36000
Free blocks:              18422440
Free inodes:              366047482
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      674
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              128
Filesystem created:       Fri Nov  9 01:54:53 2012
Last mount time:          Tue May  1 15:20:39 2018
Last write time:          Tue May  1 15:20:39 2018
Mount count:              5
Maximum mount count:      26
Last checked:             Mon Apr 30 12:45:46 2018
Check interval:           15552000 (6 months)
Next check after:         Sat Oct 27 13:45:46 2018
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      79f626f4-63ee-44c4-9555-a230ba2f2c4a
Journal backup:           inode blocks
#
Why "signed_directory_hash" and not unsigned?
I saw unsigned in another FS from another machine. What is the difference? I haven't found documentation on this.

Thanks.
 
Old 05-02-2018, 11:43 PM   #12
jaatavaresf
LQ Newbie
 
Registered: Apr 2018
Posts: 3

Rep: Reputation: Disabled
I think I found the solution...

Everytime mkdir hangs, I receive a kernel core dump about ext3 journal.
I removed journal from the FS and then it started working really fast.

I found a discussion on signed and unsigned directory hash where there is a bug related to this and related to machine architectures.

And I found the e2fsck -D parameter. This parameter was the default in the past for fsck, but now it isn't I don't know why.

I run e2fsck with -D, then I added journal back (it is impossible to live with a 5T FS without journal. It takes 5hs for an fsck), and then I run again e2fsck with -Dfy ... No problems anymore.

mkdir is not slow anymore. I am going to do just the e2fsck -D to the other 6.6T because it is slow on mkdir too and throwing kernel core dumps too.

I strange thing is that fsck does not check anymore for directory structures by default, ......
 
1 members found this post helpful.
  


Reply

Tags
ext3, mdadm, raid5


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
mkdir -p ? shanenin Linux - Software 4 10-19-2011 11:41 AM
I don't know. mkdir? Bobzilla2639716495 Linux - General 15 12-03-2007 11:22 AM
mkdir options sravanth.svk Linux - Newbie 4 10-26-2006 12:31 AM
mkdir help... Linux~Powered Linux - Software 2 06-13-2004 11:27 AM
mkdir help ITJedi Linux - Software 1 05-14-2004 05:19 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:51 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration