LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-01-2009, 04:28 AM   #1
BeacoN
Member
 
Registered: Nov 2008
Distribution: Linux Mint
Posts: 56

Rep: Reputation: 17
ext filesystem limits?


I know I know, there's tons of Q/A out there on the ext3 filesystem. I have found these, yet none seem to be congruent with what's happened to me with the filesystem.
I have tons of duplicate files I want to sort through-so I wrote a BASH script to compute md5s and create a file for each md5-in the file is the node, full path, and sha1 for each file that produced that md5, like....

cat ab3652ad12bff038e2dc713ba83fd9a3.txt
314561***/home/me/file.txt***9a5eef2a9f28024d66069d609caa5d743a604c1f
240196***/home/you/other.txt***9a5eef2a9f28024d66069d609caa5d743a604c1f

I first ran the script to place all files in one directory-once it got over 100,000 files it was way too slow so I stopped it.
Then I tried creating 6 levels of directories, one for each of the first 6 characters in the hash, and this ended up giving me an 'out of memory' error even though df -h showed that I had about 4 gigs left. I know 16^6 is a huge amount of directories, but I haven't found documentation anywhere placing a limit on the number of directories you can have in the filesystem (I'd really like to know how it works). All I've found is that a file can be up to 16GB in size and most sites say 'unlimited' for number of directories. Anybody know why I got this second error?
p.s. a friend suggested using a db to store this info, I'm looking into that, but are there any suggestions how to do it in BASH w/o a db? maybe a different filesystem?
thanks for sharing your knowledge,
-Brian
 
Old 07-01-2009, 04:37 AM   #2
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
Are you sure the out of memory problem is not in your script instead of in the filesystem?
 
Old 07-01-2009, 04:40 AM   #3
BeacoN
Member
 
Registered: Nov 2008
Distribution: Linux Mint
Posts: 56

Original Poster
Rep: Reputation: 17
script?

I wouldn't think so, the script simply processes one file at a time, I run it with the find command....

find ./ -type f -exec /home/me/myscript '{}' \;

so it runs a new script for every file.
 
Old 07-01-2009, 04:45 AM   #4
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
What is the exact error message? You are not including /proc and there are no symlink loop, right?

Last edited by Agrouf; 07-01-2009 at 04:46 AM.
 
Old 07-01-2009, 04:47 AM   #5
BeacoN
Member
 
Registered: Nov 2008
Distribution: Linux Mint
Posts: 56

Original Poster
Rep: Reputation: 17
error msg

I'm not definite-I was frustrated and deleted the whole directory quickly, I do remember it said some thing like 'out of memory'-looked akin to what I've gotten when df -h shows 100% usage on /, although like I stated df -h showed I had about 4 gigs left on /.
 
Old 07-01-2009, 04:48 AM   #6
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
You are not including /proc and there are no symlink loop, right?
 
Old 07-01-2009, 09:44 AM   #7
BeacoN
Member
 
Registered: Nov 2008
Distribution: Linux Mint
Posts: 56

Original Poster
Rep: Reputation: 17
/proc

No, it's always in a subfolder of /media.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
ext3 filesystem limits Fionnbharr Linux - General 1 01-10-2009 05:44 PM
LXer: Kernel: Ext 4 Filesystem Moves Beyond Developer Status LXer Syndicated Linux News 0 10-14-2008 11:00 PM
ext3 filesystem limits poblano Red Hat 14 02-12-2008 05:41 PM
Filesystem caching limits LinuxGeek Linux - General 0 04-14-2006 06:46 AM
EXT-fs error (device hda3) in start_transaction: Readonly filesystem kiwibird Linux - Hardware 2 10-21-2005 11:15 AM


All times are GMT -5. The time now is 07:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration