You could move your SSH port to something other than 22, for starters. In your sshd config file (on my systems it's /etc/ssh/sshd_config) you will want these entries:
Port 8945 # or something beside 22
To allow logins only from specific IPs, you can add this to sshd_config:
That will only allow SSH logins from any user from IP address 184.108.40.206.
You can lock it down even tighter by disallowing direct root logins, and specifying a user in the line above. For example, create a user "ridwan77", add them to the group wheel, and include this in your sshd_config file:
In this case, only ridwan77 will be allowed to login, and then only from IP address 220.127.116.11. You can include netblocks in the AllowUser with
You will have to restart sshd to make any changes effective. Restarting sshd will not kill your current session.