I'm using squirrelmail. However, recently, I found that my first login always fail... 2nd log would be ok. could my server be hacked and the hacker has changed the login page to obtain my user passwords?

also, i find a log file called /var/log/forgemail... is this a Redhat default thing? Could someone be using my webserver to send spam mail?

What can I do to check the above? thanks!

I'm using squirrelmail. However, recently, I found that my first login always fail... 2nd log would be ok. could my server be hacked and the hacker has changed the login page to obtain my user passwords?
Try verifying your login and init scripts. To verify the integrity of rpm packages you can use:

rpm -Va

If you have tripwire or some other file alteration scanner installed, now would be a good time to run a check. Also make sure to download and run chkrootkit.

i find a log file called /var/log/forgemail... is this a Redhat default thing? Could someone be using my webserver to send spam mail?
As far as I am aware, that is not a standard Redhat thing. Given the name, I would think your guess is reasonably accurate. Did you try taking a look at it's contents? Also run: stat /var/log/forgemail to get creatiion and last modifcation dates. Use the creation date as a rough timeframe and go through all of your system and security logs looking for anything abnormal (including application errors/panics/etc). Checkout the output of last and lastb. Look at /etc/passwd to see if you have any new users or users other than root with a uid/gid of 0. Run netstat -pantu or lsof -i to get a list of what applications are listening on interfaces (look for any services which don't seem normal or that you don't remember running).

Take some time and do some reading in the security references thread by unSpawn at the top of the forum. In particular look at the links under "Compromise, breach of security, detection"