LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 05-15-2006, 04:15 PM   #1
calande
Member
 
Registered: Oct 2005
Distribution: Urubu
Posts: 159

Rep: Reputation: 15
Question Storing in text files vs. performance


Hi guys,

I need your opinion! I am creating a chat application for a large community of several million people. I need to find the best way to have a very good performance. I plan to store conversations in temporary text files, and when the chat is terminated, the system stores the conversation into a PostgreSQL DB and deletes the text file. There will be thousands of text files, and I am concerned with I/O overhead fron the HDDs. Maybe this won't be a problem if text files are stored in RAM while they are read/written. Do you know how Linux handles those files that are heavily read and written? Another way would be querying the DB directly but I would need a whole lot more hardware I guess.

What do you think is best to handle a lot of queries like this, and does Linux keep in RAM files that are heavily accessed (provided that there is enough RAM)?

Thank in advance!
 
Old 05-16-2006, 09:36 AM   #2
calande
Member
 
Registered: Oct 2005
Distribution: Urubu
Posts: 159

Original Poster
Rep: Reputation: 15
Any idea?
 
Old 05-16-2006, 11:06 AM   #3
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 4.0 Etch
Posts: 1,349

Rep: Reputation: 49
For best performance in a large scale chat application, it's best not to even receive the chat messages in the first place. Is there some reason it's not acceptable to design it so the clients directly communicate with each other?

Even if you use centralized communications, why bother storing conversations at all? If you're going to be deleting them at the end, then obviously you're not concerned with long term storage.
 
Old 05-16-2006, 02:22 PM   #4
calande
Member
 
Registered: Oct 2005
Distribution: Urubu
Posts: 159

Original Poster
Rep: Reputation: 15
Thank you Isaac, we are limited by the web technologies, I don't know how to have two users communicating with each other using just a browser (unless using an applet, but I don't want that anymore).

If they even could communicate directly, there would be other problems anyway (firewall blocking, etc...).

I need to centralize the chat so that conversations go through the server and so that I can store them because people need to be able to browse/search through previous chats.

Kind regards,
 
Old 05-16-2006, 02:26 PM   #5
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 4.0 Etch
Posts: 1,349

Rep: Reputation: 49
Which web technologies are you using? I'm not an expert in web application development, but I know enough to know that anyone giving you advice would benefit from knowing what you're using.
 
Old 05-16-2006, 02:50 PM   #6
calande
Member
 
Registered: Oct 2005
Distribution: Urubu
Posts: 159

Original Poster
Rep: Reputation: 15
Regular HTML, CSS, AJAX, PHP, Python, PostgreSQL, etc...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Convert DOS text files to UNIX text files ta0kira Linux - Software 7 03-15-2011 11:42 AM
Storing Sound (Media) files in MySQL 1kyle Suse/Novell 2 04-04-2006 09:44 AM
Storing BibTex bst files in home directory drben Linux - Software 1 11-01-2005 02:31 PM
storing text in a text file for a specified length of time. mrobertson Programming 7 08-02-2005 10:27 AM
Storing files from linux to NT buddy Linux - General 3 01-15-2003 05:36 PM


All times are GMT -5. The time now is 02:26 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration