LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-22-2012, 11:50 AM   #1
kevin.watson
LQ Newbie
 
Registered: Oct 2012
Posts: 4

Rep: Reputation: Disabled
How to Overcome 32,000 Subdirectory Limit on Ubuntu Server 11.04 Natty


I have a server (Ubuntu 11.04 Natty with reiserFS) that holds and serves product images, that's it's only job. There are several applications that are pulling images from this server.

Images are automatically put into directories according to their unique identifier during a nightly batch job. Unfortunately, I now have too many product ID's (over 32,000). Other applications rely on the directory structure staying intact as it is now. I cannot update the file system because it is a Mediatemple (ve) server and I don't have access to control the hardware.

I need to find a solution that allows me to get over the 32,000 subdirectory limit and haven't been successful thus far.

I can modify the directory structure in which the images are loaded to further subdivide and avoid the issue, but as I mentioned, I still need to maintain my directory structure for the production_images folder.

Here is a solution I thought about but not sure how to implement:

Have all images loaded and subdivided in a folder called MB_Sync and them have a symlinked folder called production_images. Any directories added to MB_Sync would be linked in production_images. It would be based on the following structure:

MB_Sync
Category
Product ID
Language
image.jpg
production_images (symlinked folder)
Product ID
Language
image.jpg
How can I create the symlink to pull from the category folders in MB_Sync and just give me the Product ID directory in production_images?
 
Old 10-22-2012, 12:07 PM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
I know that ext3 has a hard limit on 32.000 sub-directories, I did not know and cannot find any refference that this is also true for reiserFS.

The ext4 file system has increased this limit to 64.000, but this might not be an option (it is if you have backups).

If changing the FS is out of the question then I suppose you do need to be creative and use sym-links.

Last edited by druuna; 10-22-2012 at 12:08 PM. Reason: fixed typo
 
Old 10-22-2012, 01:26 PM   #3
kevin.watson
LQ Newbie
 
Registered: Oct 2012
Posts: 4

Original Poster
Rep: Reputation: Disabled
I have not found any documentation on reiserFS subdirectory limits either but I created a script to create empty directories and it errored out at 31998 consistently. I cannot alter the filesystem in any way.
 
Old 10-23-2012, 09:35 AM   #4
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 7,170

Rep: Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208Reputation: 2208
Simply use a subdirectory structure within category based on substrings of the product-ID.

For instance, if you search for Microsoft issue-ID "K1234567," you'll see the URL structure is something like "K/12/345/K1234567."

First simply attempt to create the file. If you can't, create each subdirectory as-needed and try again.

Yes, this will require some programming changes, but you need to do those anyway.
 
Old 10-23-2012, 06:39 PM   #5
mreff555
Member
 
Registered: Sep 2011
Location: Philly
Distribution: Gentoo
Posts: 470

Rep: Reputation: Disabled
I'm not sure symbolic links would fix the problem since the limitation is the file system itself and not the drive.

I know you said you could not change the directory structure, but you could save the creation of many directories by prepending criteria to filenames

eg.
instead of
/ProductID/english/image.jpg

you could do this.
/ProductID/en-image.jpg
 
Old 10-24-2012, 02:08 AM   #6
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387Reputation: 2387
Quote:
Originally Posted by mreff555
I'm not sure symbolic links would fix the problem since the limitation is the file system itself and not the drive.
Although I am not able to find anything about this limitation (see my previous post #2), I do know that the limitation for ext3 is 31998 sub-directories per directory (source: Wiki Ext3)
Quote:
A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links.
If reiserFS is anything like ext3 the total number of files (/directories) that can be created is a lot higher and depends on settings during the creation of the file system.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Natty Ubuntu - Natty Narwhal Inspired Theme for Nokia Smartphones LXer Syndicated Linux News 0 06-10-2011 11:30 AM
LXer: The Perfect Server - Ubuntu Natty Narwhal (Ubuntu 11.04) ISPConfig 2 LXer Syndicated Linux News 0 05-12-2011 03:00 PM
LXer: How to upgrade from Ubuntu 10.04, 10.10 to Ubuntu 11.04 Natty Narwhal | Desktop & Server LXer Syndicated Linux News 0 04-01-2011 05:00 AM
Limit size of a subdirectory carlmarshall Linux - Newbie 6 07-16-2008 12:52 AM
How to overcome the 2GB Ftp limit in redhat 7.2 running kernel 2.4.7-10 Ryleco Linux - Software 0 09-07-2004 10:23 AM


All times are GMT -5. The time now is 07:08 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration