Filesystem structure for website with a large amount of uploads
 

I'm working on a web project for a client, and it's going to rely heavily on user uploads. The current issue I'm struggling with is how exactly to design the filesystem structure. The reason I'm coming here is to see if there are any downsides to having say.... 50,000 subdirectories under an uploads/ directory? What about 1,000,000? Is there a limit to directories in a linux filesystem? I'm of course going to segment the directories into chunks of x number of uploaded files, but am currently determining what x number is. Any advice on filesystem design if you were creating a youtube/flickr type site that could possibly have millions of uploads?

Thanks

Hi

I don't think there's a limit, but most filesystems start to slow down when you get a few thousand items in a directory. And it gets worse the more files you have. ReiserFS is an exception - it doesn't have this problem. Maybe other filesystems too? I don't know.

Not long ago I wrote some PHP to overcome this problem. Every file gets a numerical ID in the database. I divide that ID by 1000, so a typical path would be
"./uploads/".intval($id/1000)."/".($id % 1000).".file".

Why 1000? Well it's easier to find stuff manually. If I want to find file number 12345, I can just look at "./uploads/12/345.file". The actual filenames are put in a database, because people use all kinds of characters in filenames, and I didn't want them in the filesystem.

Hope this helps.