Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back > Forums > Linux Forums > Linux - General
User Name
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.


  Search this Thread
Old 07-06-2007, 10:53 AM   #1
LQ Newbie
Registered: Jun 2003
Posts: 18

Rep: Reputation: 0
Filesystem structure for website with a large amount of uploads

I'm working on a web project for a client, and it's going to rely heavily on user uploads. The current issue I'm struggling with is how exactly to design the filesystem structure. The reason I'm coming here is to see if there are any downsides to having say.... 50,000 subdirectories under an uploads/ directory? What about 1,000,000? Is there a limit to directories in a linux filesystem? I'm of course going to segment the directories into chunks of x number of uploaded files, but am currently determining what x number is. Any advice on filesystem design if you were creating a youtube/flickr type site that could possibly have millions of uploads?

Old 07-07-2007, 04:44 AM   #2
Senior Member
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,293

Rep: Reputation: 335Reputation: 335Reputation: 335Reputation: 335

I don't think there's a limit, but most filesystems start to slow down when you get a few thousand items in a directory. And it gets worse the more files you have. ReiserFS is an exception - it doesn't have this problem. Maybe other filesystems too? I don't know.

Not long ago I wrote some PHP to overcome this problem. Every file gets a numerical ID in the database. I divide that ID by 1000, so a typical path would be
"./uploads/".intval($id/1000)."/".($id % 1000).".file".

Why 1000? Well it's easier to find stuff manually. If I want to find file number 12345, I can just look at "./uploads/12/345.file". The actual filenames are put in a database, because people use all kinds of characters in filenames, and I didn't want them in the filesystem.

Hope this helps.


filesystem, linux, web, youtube

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
generate large amount of traffic data Mr_C Linux - Networking 3 03-09-2006 11:38 PM
HP laserjet 6p stalls with large amount of data simjii Mandriva 0 11-10-2005 03:32 PM
Large amount of threads left unanswered... J_K9 LQ Suggestions & Feedback 35 07-24-2005 10:55 AM
A large amount of problems DaBlade Linux - General 4 12-04-2004 05:47 AM
vsftp large file uploads fail randybell Linux - Newbie 2 09-28-2004 10:17 PM > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:54 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration