Implementing a chat program and thus involving majority of networking concepts.
 

This is what I have thought:

1. Logging the chat messages (exchanged by the
   clients) on the server.
2. Registration of ALL clients on the server on
   their start up.
3. Client should be able to add another client on
   his list for chatting.
4. When a client logs in from other side, its
   friend client should be able to see it online.

Now, to add to this, there can be things like sharing
text/voice/video files etc, but then the focus will
be on compression majorly.

With the chat program, my intention is to learn the
majority of "networking" concepts.

What else, can be implemented (in this chat program)
which can brush up my "networking" concepts?

Honestly, it all depends on how far you want to go, Anisha. How far are you really looking to go? :p
You could always implement packet sniffing from another host, and learn how TCP and UDP works, along with how network packets are constructed. Besides your chat program, you also have the protocols themselves to learn about, how they operate, and why they operate.
And to be off topic, have you ever received the pictures of my son? :)

Cheers,

Josh

TBH, I know my question is vague.
Actually, it will be helpful if you or someone could
list down what are the programming possibilities for
this context.

Currently I have no idea how to answer your question.

I have been reading theory for quite sometime now, and
am bored, want to get my hands dirty with programming.

Okay, packet sniffing is one thing, I'll look into it.
Thanks.

anything, else that pops in your mind, something which
you can expect from a "real" chat program?

Yes, that too quite a long time ago. I already replied
to that mail, and also received an acknowledgement from
you a long time ago.

You'll probably find out that it is better to let clients worry about compression, and instead support just text and binary streams on the server. You'll most likely want to wrap the binary streams into packets (stream ID, sequence number, checksum, length, followed by length bytes of binary data) so that the server can distribute them to all members without having to parse the binary stream contents.

It is especially useful if you use UDP for binary streams. If the server keeps a copy of a binary stream packet for a short while, up to some time or memory limit, clients could receive packets asynchronously, out-of-order, and request resends only if they notice they missed a packet. You might wish to add "caching strategy" to binary streams, so that audio and video packets are not really cached on the server at all (since they can be omitted with just a small perceived glitch), but things like file transfers will completely break if you miss a packet in the middle.

Another thing is whether you wish to allow for client-to-client direct connections, bypassing the server altogether. It might be useful in some cases, for example when transferring files. On the other hand, the server then loses the opportunity to peek and record such communications.

Encryption is something I personally would like. You might wish to look at the OpenSSL library, and its support for certificates and/or public key infrastructure. After handshakes, you could use e.g. Blowfish or Twofish for fast symmetric encryption, using some largeish key (agreed upon during handshake).

(Public key based identification should be the simplest option. Consider the way authorized_keys work for SSH connections. Each server has their own (fixed) public key, defined in server configuration, and each client creates a new random key at startup. The server public key should be user-visible, for example as a hex string, so they can save it. Key exchange can be then done by each encrypting a random symmetric key using the others public key; only the private key will be able to decrypt it. User names and passwords used for identification are only sent afterwards.)

I would also recommend considering letting users contact either using a native binary protocol, and an XML-based one for browser-based (text-only) clients. Javascript can do asynchronous requests, but both the request and response must be XML. Being able to participate in discussions using only a browser (and a Javascript page) might make testing more fun.

Finally, it may be a good idea to consider writing the server end and the clients in different programming languages, especially if you are not developing this alone. It is very easy to fall victim of implicit conventions in a programming language, which will later on make porting very hard or impossible. Or to just use the same structures, and ignore all portability issues "for now". Having one side of the communications written in another language makes you keenly aware of the importance of making sure the communications follow agreed upon rules, and that the rules are easily followed in different languages. I have found that it is useful to first create simplified prototypes (server and client), and test the communication protocol before actual implementation or specifying the comms protocol in detail.

What's the point of reinventing the wheel? I'd recommend to do a research, find existing software that has functionality you want, and modify it.

I think there's no need for lengthy explanations - grab Qt 4, and check their "networking" examples. This will be enough to get you started. The only problem might be with encryption, but that also can be figured out in one or two days of research - there are multiple encryption libraries available. Sending data over network is not very difficult, as long as you don't go too far into details.

A multiplayer game. Something like battleship or checkers. Or battle city clone for multiple people (more than 2). Will be much more fun than chat program anyway.

Networking is one of those things that are easier to learn(well, to get basic understanding) by doing. There is limited number of operations for socket communication, so it'll be faster to search for tutorials instead of asking.

In my opinion, you're overdoing it. I think programmer's ideal goal is to find simplest, shortest and easiest solution that will take least amount of time to implement it but will yield maximum payment and will implement all features. In other words - implement the whole thing in 1 minute using one line of code and get paid few thousands of $s for it. You're looking for a way to create most sophisticated/advanced solution. While working this way is surely fun, such style is counter-productive, since it wastes development time and makes further maintenance more difficult. People don't need most complex and most advanced solution. They normally want solution that works as they want it, and it doesn't matter how complicated or advanced it is. Remember KISS principle. Adding functionality needlessly will doom any project.

The web is full of basic chat clients, there is zero need for yet another one.

There is, in my opinion, also zero need for yet another programmer who can do basic chat clients, but cannot handle the more advanced stuff.

However, if you strive for commercial success only, then you are absolutely right, and my suggestions are way too complicated and wasteful.

The KISS principle, at heart, means doing one thing and doing it well. It advocates keeping things simple, but not cutting corners with respect to quality. Your suggestion is to take something similar, stick your own label on it, and walk away. I stopped doing commercial work a few years ago, mostly due to the prevalence of that attitude: the belief that users want just something that looks like it works for now, most of the time; the main thing is to do it cheap and fast. Things like Gnome are losing the strengths I perceived them to have -- the configurability and the adaptability --, and instead are being simplified, to "please the masses".

I don't like cheap and rickety. I don't like simplified tools. I want my tools to work, and I want to adapt them to fit my workflow.

I would concede that having binary stream support in a chat client is overdoing it -- except that we still do not have an ubiquitous client for point-to-point file transfers. USB memory sticks are probably the most common method of moving files between two workstations, even if they are network-connected.

When I wrote my suggestion with regards to binary transfers, I was thinking about having a list of "published" files (not directories) in the client, with associated visibility flags and access control lists. If allowed, another client could either directly (if allowed by firewalls in between), or through the server (allowed through firewalls if using e.g. HTTP port, port 80) copy the files. There is real world need for something like this -- and in my mind, it would make the tool complete. Granted, the emphasis would be on file transfers, not on chatting.

So, although I understand your viewpoint, SigTerm, I respectfully disagree.

Thanks very much for the detailed reply, Nominal Animal.

By files do you mean "attachments" like .pdf/.doc etc.?
Will it not be better if I use TCP for file and text data
transfer, and UDP for video/voice transfer. I mean switching
between the protocols on run time based on the data?
Did I miss a point in your above quote?

I couldn't understand what you wanted to say in above quote.
How to participate in discussions using only a browser (and
a Javascript page)?
Any hints?
What am I supposed to read for this?

EDIT-
Ah! You meant this? http://chat.stackexchange.com/rooms/14/photography
EDIT-

You mean, creating dummy client-server programs and exchanging
messages between them for a trial?
Or you meant anything else by "test the communication protocol"?

The point is to "learn" networking.
The point is NOT to "chat" through a chat program.
The book "Unix network programming" describes all
sorts of functions etc. but simply reading them
doesn't help, and I am looking for ways to get them
implemented somewhere somehow.
The point is to face the actual programming problems.

I don't wish to spend time on developing GUI and the
complex game algorithms. If there are some special
networking conditions to be kept in mind for game
programming, I'd like to implement them here in the
chat program itself.

Qt provides wrap ups for all the basic functions, lately
I found it difficult to understand what exactly their
functions do underneath. Doing this through qt
might be helpful, once I get hold of the raw basics.

Offtopic: Yesterday I searched for Qt jobs in the major
areas in and around Delhi, and to my surprise only 3
companies listed!!

No! My suggestion is to use available ("off the shelf") components and libraries to combine them into software you want - instead of reinventing the wheel. Somebody has done it before, which means that existing work should be reused and improved.

Sounds very like ftp server. Or emule. Which includes chat program, by the way.

My point is that it is not a good idea to artificially increase difficulty of a task. Programming is often similar to rebuilding a plane mid-flight. You start with one requirements, and then they change many times. Selecting overly complex in the beginning will make it more difficult to adapt program to new requirements later, which means that everything that is not necessary should be avoided. See "YAGNI" on wikipedia.

Try to find an interesting problem that is related to networking, and that is not artificial challenge. "Chat program" is an artificial challenge, and it won't be very efficient for learning. How about remote logging library? I.e. make a library that provides logging routines similar to qDebug(), is configurable and can send logging messages to multiple servers depending on message priority. Plus a server that can receive messages from multiple programs. Similar to a chat program, but much more practical/useful.

On bottom level, there are socket(), connect(), send(), recv(), bind(), listen(), etc. They are platform-dependent. Search for documentation/tutorials for every function you don't know (there aren't many of them), and that should do the trick. Link I gave you in previous post had tutorials. It isn't something like C++ templates, so there shouldn't be much of a problem.

I don't know; it is something to test.

What I tried to suggest was to use a single protocol for binary data streams. Using the protocol I roughly outlined, a "caching policy" on the stream would tell the server how important individual packets in the stream are. After all, the only difference between an audio or video stream and a file transfer is the fact that file transfers will break if you skip a packet.

Using UDP for file transfers tends to be a bit faster, because it is inherently one way: send and forget. For TCP, each packet is acknowledged, usually before the next one is sent. Since packet loss is expected to be rare, instead of acknowledging each packet (like TCP does) the client could only request missing packets. The client could decide based on the buffer size it uses, and on the stream type, whether to request missing audio and video packets at all, although the server may not have copies any longer. The server backend would similarly use the "caching policy" on the stream to decide on how long to keep a copy of each packet sent to clients.

I guess text streams could use the same packet format, too. All you'd need is a better header structure, one that could specify the recipient of the text message (chatroom, person, recipient list).

Using a single protocol for all this makes your socket communications simpler, and easier to code and maintain.

Javascripts running on a web page can connect to any socket, but the content sent and received must be in XML. If one of the interfaces to the server backend is such an XML socket, you can write one interoperable client as a web page.

Being able to do something in real time between say a command-line client, and a web page showing up in the browser, tends to feel interesting. Making sure the data packets travel intact from one protocol to the other also happens to exercise the toughest bits to code in the server backend.

If you ignore the silly browser bugs, and concentrate on say open source browsers, you do not need a lot of Javascript to send and receive simple strings (encapsulated in some simple XML) and display them on the page.

That is one example. That page uses Google AJAX API to converse between the client browser (Javascript) and some server.

The parsing and handling of messages is something that has to be done carefully. If your server backend does a protocol change -- from custom binary transport to XML and vice versa -- it is a lot easier to detect any issues. If you just forward the packets, small accidental errors (like occasionally omitting the last byte or something) may be difficult to detect.

It is also much easier to get testers when "taking a quick peek" requires one to install nothing, just go to a specific page. The fact that functionality can be severely limited when using a browser is okay.

In my opinion, the days of one solitary developer creating a wonderful piece of software is well and truly over. Testing is not something you do afterwards; it is a natural part of the development process.

In my experience, the communications protocol, or actually its limitations and idiosyncracies, will largely shape the future of the application. I'm not sure if protocol design is something a programmer should nowadays learn, but I've seen too many poorly designed protocols already.

Consider HTTP for example. For a HTTP request, the first line must be of form method path HTTP/version followed by one or more header lines, and an empty line to signify the end of the request header. For HTTP/1.1, a Host: header is required; it tells the server which site (of the possibly many sites hosted on that single IP address) the client wants.

Now, this sounds perfect, right? Simple, text-based, quite easy to parse.

Well, there is a big problem when you add SSL/TLS (aka HTTPS) into the mix. The problem is that before the SSL/TLS handshake, the client should supply the site name first, in order for the server to select which certificate to show to the client. But the HTTP protocol does not allow that. To workaround this issue, quite complicated scenarios have been developed. They tend to work only in newish browsers, and there are a lot of ancient browsers still in use out there. The end result is that servers that have multiple independent secure sites on them, tend to still require multiple IP addresses, so that the certificate can be chosen based on the IP address the client used to connect, instead of just the host name.

It is therefore my belief that developing the communications protocol through diverse testing using different types of prototype clients and/or servers will yield a better result. In doing so, you might end up having to learn some of the weirder tools like tcpdump and strace to monitor the actual data transmitted, and the syscalls the process does.

Right. But since you are going to do some programming, why not try to write something that might prove to be useful, too?

Most code examples I've ever seen tend to ignore most error checking (or use assertions instead of gracefully failing or even retrying). Most programmers add that stuff only when forced to. Network is one environment where programmers tend to be forced to, because error conditions are common. Clients can be unexpectedly closed, connectivity may drop, and so on. If your communications routines are not careful, they may stall in some wait loop, waiting for a packet that will never come.

If you only create a test program, say a command-line chat client that only accepts ASCII text, you are never going to bother to test it completely enough to tell if your communications routines are actually robust, or just good enough. Testing something like that is just too dull. Some programmers tend to be satisfied at good enough; it is certainly a commercially viable approach.

Another point is that you will learn by doing. If you are creating something for just the sake of learning, the end result may be limited; if you are a typical human, you'll only follow the avenues that really interest you, neglecting the ones that do not. The interaction between users -- or at least testers -- and developers is important; if for nothing else, for the difference in viewpoints. Testers and users see different things in the software. Typically, what users find important, may not feel so to the developers, and vice versa.

There is also the importance of getting positive feedback when you develop something. We humans are, after all, social animals.

Finally, I must note that just like SigTerm pointed out, my approach is much more strict than what is considered sensible in the commercial world. If you see programming as just a career, you probably should follow SigTerm's advice above instead.

I personally don't need commercial quality programmers for anything; I need gurus that can do tools I can rely on, and develop things like OpenSSH and OpenLDAP and sed and grep and awk and the Linux kernel. I see software development in the future fused to customization and workflow optimization, and that will only happen with deep knowledge of the systems being worked on. Because I may be wrong, my advice in this thread may also be wrong. I obviously think I'm right, but I've been wrong before, so ..

Thanks you Nominal Animal, for the extremely helpful and detailed follow up.

I think I need to start a brain storming thread on the networking project ideas. ;-)