LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 07-06-2007, 10:53 AM   #1
DGath
LQ Newbie
 
Registered: Jun 2003
Posts: 18

Rep: Reputation: 0
Filesystem structure for website with a large amount of uploads


I'm working on a web project for a client, and it's going to rely heavily on user uploads. The current issue I'm struggling with is how exactly to design the filesystem structure. The reason I'm coming here is to see if there are any downsides to having say.... 50,000 subdirectories under an uploads/ directory? What about 1,000,000? Is there a limit to directories in a linux filesystem? I'm of course going to segment the directories into chunks of x number of uploaded files, but am currently determining what x number is. Any advice on filesystem design if you were creating a youtube/flickr type site that could possibly have millions of uploads?

Thanks
 
Old 07-07-2007, 04:44 AM   #2
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,148

Rep: Reputation: 236Reputation: 236Reputation: 236
Hi

I don't think there's a limit, but most filesystems start to slow down when you get a few thousand items in a directory. And it gets worse the more files you have. ReiserFS is an exception - it doesn't have this problem. Maybe other filesystems too? I don't know.

Not long ago I wrote some PHP to overcome this problem. Every file gets a numerical ID in the database. I divide that ID by 1000, so a typical path would be
"./uploads/".intval($id/1000)."/".($id % 1000).".file".

Why 1000? Well it's easier to find stuff manually. If I want to find file number 12345, I can just look at "./uploads/12/345.file". The actual filenames are put in a database, because people use all kinds of characters in filenames, and I didn't want them in the filesystem.

Hope this helps.
 
  


Reply

Tags
filesystem, linux, web, youtube


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
generate large amount of traffic data Mr_C Linux - Networking 3 03-09-2006 11:38 PM
HP laserjet 6p stalls with large amount of data simjii Mandriva 0 11-10-2005 03:32 PM
Large amount of threads left unanswered... J_K9 LQ Suggestions & Feedback 35 07-24-2005 10:55 AM
A large amount of problems DaBlade Linux - General 4 12-04-2004 05:47 AM
vsftp large file uploads fail randybell Linux - Newbie 2 09-28-2004 10:17 PM


All times are GMT -5. The time now is 12:16 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration