If your router isn't directing any traffic to the server, you're probably ok. But of course it would be safer to setup some iptables rules to firewall off any attempts at conenction from outside. I won't get into writing firewall rules, as I'm sure you could research those on your own, but suffice to say, I think you should try for a rule that says "if any traffic comes to me from an IP other than 192.168.x.x" then use -j DROP which means drop the packets.
But listen... if you're going to run a serious server and your concerns are that high, then you should definitely consider getting a professional grade router. If your router says "d-link" on the outside of the case, that's just not going to cut it. Consider something like SonicWALL, which can be expensive, but you'd get yourself some really strong firewalling capabilities.