Thanks for pointing me towards the logs in /var/logs/secure. I saw an entry saying "Authentication refused: bad ownership or modes for directory"
I just had to fix the permissions for .ssh dir and the authorized_keys file as follows:
chmod 700 for my home directory (it was already 700 be default, which is good)
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Everything is working fine and i can connect to my box via key based authentication. Thanks for all your help