Storing in text files vs. performance
 

Hi guys,

I need your opinion! I am creating a chat application for a large community of several million people. I need to find the best way to have a very good performance. I plan to store conversations in temporary text files, and when the chat is terminated, the system stores the conversation into a PostgreSQL DB and deletes the text file. There will be thousands of text files, and I am concerned with I/O overhead fron the HDDs. Maybe this won't be a problem if text files are stored in RAM while they are read/written. Do you know how Linux handles those files that are heavily read and written? Another way would be querying the DB directly but I would need a whole lot more hardware I guess.

What do you think is best to handle a lot of queries like this, and does Linux keep in RAM files that are heavily accessed (provided that there is enough RAM)?

Thank in advance! ;)

Any idea? :rolleyes:

For best performance in a large scale chat application, it's best not to even receive the chat messages in the first place. Is there some reason it's not acceptable to design it so the clients directly communicate with each other?

Even if you use centralized communications, why bother storing conversations at all? If you're going to be deleting them at the end, then obviously you're not concerned with long term storage.

Thank you Isaac, we are limited by the web technologies, I don't know how to have two users communicating with each other using just a browser (unless using an applet, but I don't want that anymore).

If they even could communicate directly, there would be other problems anyway (firewall blocking, etc...).

I need to centralize the chat so that conversations go through the server and so that I can store them because people need to be able to browse/search through previous chats.

Kind regards,

Which web technologies are you using? I'm not an expert in web application development, but I know enough to know that anyone giving you advice would benefit from knowing what you're using.

Regular HTML, CSS, AJAX, PHP, Python, PostgreSQL, etc...