LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 09-15-2008, 01:03 PM   #1
jsstevenson
LQ Newbie
 
Registered: Jul 2006
Location: Scotland
Distribution: Gentoo
Posts: 21

Rep: Reputation: 0
Is it more efficient to store files flat or in folders?


Hello...

I have a folder in which I store letters that I have written, creatively called 'letters'. Within this folder I have more folders for each person or company that I communicate with, which obviously contain the relevant files/letters.

I have however, over the last number of months, started to ignore this folder structure and have begun to store letters directly within the letters folder without dropping them into sub-folders.

In order to make this manageable I have changed how I name files and now employ a specific naming routine... in the case of letters, I use the following naming convention:

recipient-date_of_letter_(year.month.day)-subject.extension

This method of a more-or-less flat file structure seems very natural to me.

Taking this to extremes this could result in a single partition having say 100,000 files stored flatly instead of within a hundred (for arguments sake) or so folders/directories.

The question is, is a flat file structure less, equally, or more efficient than a directory based structure, putting aside the delay that would occur as a graphical file manager attempts to display the contents of a flat structure with so many files?

Any thoughts on the efficiency and any pros and cons would be very welcome.

Thanks
Jamie.
 
Old 09-15-2008, 01:24 PM   #2
MS3FGX
Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 351Reputation: 351Reputation: 351Reputation: 351
It depends quite a bit on which filesystem you are using. EXT2/3 in general don't perform as well with (very) large numbers of files as more advanced filesystems like XFS and JFS; ReiserFS specifically is supposed to be very good with large directory trees comprised of many small files.

That said, 100K files isn't really a lot in the grand scheme of things; there are mail servers out there which are storing many times that number of files. All of the filesystems should be able to cope with that number, and the real world performance impact will likely be very small. I would still look into using XFS or ReiserFS, but that is more personal taste than anything.
 
Old 09-15-2008, 02:43 PM   #3
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
If i am not mistaken, putting files into folders simply means that the file entries have codes added to associate them with directories (folders)---which are actually files themselves. The associaton of files with folders does not affect how much space is used.

I would organize thing into folders based on what make you most efficient on the user side.
 
Old 09-16-2008, 07:55 AM   #4
SlowCoder
Member
 
Registered: Oct 2004
Location: Southeast, U.S.A.
Distribution: Fedora (Desktop), CentOS (Server), Knoppix (Diags)
Posts: 934

Rep: Reputation: 38
Quote:
Originally Posted by pixellany View Post
I would organize thing into folders based on what make you most efficient on the user side.
Bingo.

I also tend to place like files in single folders, and name them in such a manner that I can type 3-4 characters into my file browser and quickly find the file I want. I find that this method is the fastest and most efficient for me.
 
Old 09-16-2008, 12:21 PM   #5
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 4.0 Etch
Posts: 1,349

Rep: Reputation: 49
The main performance bottleneck may be whatever you're using to browse the files. While the file system driver may have no particular issues with thousands of files in one directory, your file browser might bog down.

This can be a particularly annoying performance issue if you're browsing over a network or over the internet.
 
Old 09-16-2008, 02:03 PM   #6
trickykid
Guru
 
Registered: Jan 2001
Posts: 24,133

Rep: Reputation: 197Reputation: 197
Possibly 100,000 loose files of different written letters and papers you've created over time? If I were you, I'd probably create me a database of these in MySQL or PostgreSQL, maybe SQLite even to organize these and to access them.
 
Old 09-16-2008, 08:33 PM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,269

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
The larger the num of files, the better it is to structure eg use sub-dirs. Its easier to manage.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LDAP authentication and flat files bx.s Linux - General 5 10-27-2006 04:18 AM
flat files or xml and css? trscookie Linux - General 1 01-17-2006 09:24 AM
Samba can create new files and folders but access denied in any new folders k.king Linux - Networking 2 01-15-2006 06:14 AM
Using flat files or tables with IPTables? tarballed Linux - Security 1 12-09-2004 04:08 PM
Flat files vs Databases coolman0stress Programming 5 09-08-2003 09:23 PM


All times are GMT -5. The time now is 03:05 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration