other than iptables firewall what are all the ways to protect a linux server from hackers and how to secure apache,ssh and ftp
This is a good question, one which has a multitude of answers, but not no one right answer. TB0ne's suggestion is ultimately the correct way to go, in as much as you will need to read, research, learn and apply. Getting started can be a difficult, especially as the subject of "how to secure a server" is so broad, which leads me to the most important question you need to ask: "what am I trying to protect against?" You mention "hackers" or as they are more appropriately known as "crackers" but even this isn't specific enough. The other major concept you need to understand is that security is an ongoing process, it isn't a set of settings, it isn't a firewall, and there isn't an application you can run that will keep them out. In the general sense, what you need to do is raise the bar to entry high enough that the average "hacker" and script kiddie goes elsewhere. I am going to borrow from an earlier post
of mine from today, which I encourage you to read.
Securing your server must be done in layers and as I mentioned it is a process. Part of the process is keeping watch on what is happening with your server. As I mentioned in that other post, you can use various intrusion detection tools such as Aide, Ossec, Samhain, Tripwire. These really should be installed at the onset on a known clean system. You can monitor the network traffic itself with an application like Snort. In both cases, you may want to get alerts regarding anomalies or changes in your system and can take action if something unexpected happens. You should monitor your log files regularly and Logwatch will help with this. You should operate with the least amount of privilege required and avoid using root except when necessary. You should use strong passwords and only open services to the public that you are required to. Limit the permissions on files and folders and consider using umask to set more restrictive default ones.
Lets discuss the applications you mention. SSH is pretty well known how to make it reasonably secure and it amounts to not allowing root access, use key based authentication, and short circuit attempts at entry. Apache is very dependent upon the application you are running, but you need to be careful of PHP, Perl or any other script. Make sure that any info that is received from a user is properly sanitized. Consider mounting /var and /tmp in separate partitions (so that an intruder can't overflow the entire drive) and set options in your mounting table to disallow binary execution, setuid, or device creation (note this is counter productive to chroot environments). Don't use web based configuration tools or if you do keep them private or require key based authentication. With regards to FTP, you should really reconsider it altogether. SFTP is a better choice as it has encrypted login, but unless you use key based auth, it is still subject to brute force passwords. Instead you might consider using SCP which works over SSH. You can run a vulnerability assessment against your system such as OpenVAS which will tell you what you are exposing to the world and if any of that information can be used against you. Lastly, you should keep things updated. The updates often times correct discovered exploits. All software has bugs and eventually someone discovers how to take advantage of them. Keeping your applications updated is the best defense against this.
If you were to apply these things, you would have a reasonably secure system, but the most important thing is that you continue to monitor what is happening and take corrective action when you see things are not as they should be.