I have a server (Ubuntu 11.04 Natty with reiserFS) that holds and serves product images, that's it's only job. There are several applications that are pulling images from this server.

Images are automatically put into directories according to their unique identifier during a nightly batch job. Unfortunately, I now have too many product ID's (over 32,000). Other applications rely on the directory structure staying intact as it is now. I cannot update the file system because it is a Mediatemple (ve) server and I don't have access to control the hardware.

I need to find a solution that allows me to get over the 32,000 subdirectory limit and haven't been successful thus far.

I can modify the directory structure in which the images are loaded to further subdivide and avoid the issue, but as I mentioned, I still need to maintain my directory structure for the production_images folder.

Here is a solution I thought about but not sure how to implement:

Have all images loaded and subdivided in a folder called MB_Sync and them have a symlinked folder called production_images. Any directories added to MB_Sync would be linked in production_images. It would be based on the following structure:

MB_Sync

Category

Product ID

Language

image.jpg

production_images (symlinked folder)

Product ID

Language

image.jpg

How can I create the symlink to pull from the category folders in MB_Sync and just give me the Product ID directory in production_images?

I know that ext3 has a hard limit on 32.000 sub-directories, I did not know and cannot find any refference that this is also true for reiserFS.

The ext4 file system has increased this limit to 64.000, but this might not be an option (it is if you have backups).

If changing the FS is out of the question then I suppose you do need to be creative and use sym-links.

I have not found any documentation on reiserFS subdirectory limits either but I created a script to create empty directories and it errored out at 31998 consistently. I cannot alter the filesystem in any way.

Simply use a subdirectory structure within category based on substrings of the product-ID.

For instance, if you search for Microsoft issue-ID "K1234567," you'll see the URL structure is something like "K/12/345/K1234567."

First simply attempt to create the file. If you can't, create each subdirectory as-needed and try again.

Yes, this will require some programming changes, but you need to do those anyway.

I'm not sure symbolic links would fix the problem since the limitation is the file system itself and not the drive.

I know you said you could not change the directory structure, but you could save the creation of many directories by prepending criteria to filenames

eg.
instead of
/ProductID/english/image.jpg

you could do this.
/ProductID/en-image.jpg

Although I am not able to find anything about this limitation (see my previous post #2), I do know that the limitation for ext3 is 31998 sub-directories per directory (source: Wiki Ext3)
If reiserFS is anything like ext3 the total number of files (/directories) that can be created is a lot higher and depends on settings during the creation of the file system.