Hello everyone,

I seem to be having one of the strangest issues i have come across in a while. I have a Dedicated server running CentOS 6.4, out of no where one day it seems i am totally locked out of my server outside of my house. This web server has been running fine for years, nothing on my router has changed and i havent made any real big changes to the web server. When everything was working as it should, outside of my house you were able to use FTP, HTTP, HTTPS, VPN, SSH, and a few other services i had open. As i said before, everything has been running smoothly for years, one day i went to check something on one of my websites (this box hosts 4 different domains), none of my domains worked, ftp didnt work, i had nothing. When i got home later that day i checked on my server, i was able to ping google from it fine, i was able to access my websites locally, i could use ssh and ftp locally, but once i tried to hit it by ip or domain, it would just time out. This made me think it was possibly a firewall issue, i disabled all my security for and still unable to get in. At this point i just about tried everything i was able to think of (doing more than was mentioned here), i backed up and reinstalled CentOS. On my first boot up of the fresh installed OS, Everything seemed to be working, i was able to get to a CentOS testpage when i hit any one of my domains or IP. FTP and SSH was working too. I thought that fixed it, but it looks like it didnt, only about 30min later i went back to my server to start restoring my files and getting it back to the way it was, before doing all that i gave everything another test. Punched in one of my domains to a browser and... page cannot be displayed, wonderful, so i tried to ping my server by ip and domain, both were unable to ping my server. So now im really scratching my head, i just reinstalled the OS, didnt update or install anything or make any changes, pure vanilla centos and my issues seem to have come right back over time. Now im thinking my router so i turned my firewall off and opened all ports ( i know this is bad but i was doing a test), even still i was unable to connect to my server. Luckily i have a 2nd router, i tried connecting that, setup a few of the port forwards i needed, and it looked like everything was working again, but i wanted to rule out the same thing as earlier. I came back to the server about an hour later, again i didnt install anything or make any changes, those pages that were accessible, are now no longer accessible. Theres something im doing wrong or something on the outside causing this, and im just clueless to what it is. If anyone has experienced this or may have any suggestions, any kind of help would be greatly appreciated.

Thanks in advance for the help and reading my long boring story.

nmap from the outside to your public IP and if you are running any of those services on odd ports then make sure to -P# the port.

ex: you are running ssh via port 222 instead of 22:

is it possible your ISP has changed policies and are now blocking you from running/hosting those services on their network?

Also when you are unable to access from the WAN are you still able to access from the LAN without issue? If that is the case, the issue is from your router out, not in.

Thank for the response,

I have tried this and namp comes back with no results.

I don't believe it to be my ISP, i pay extra for the ability to host a server.

I have actually tried just now to change the port-forward in my router from my CentOS server to my ArchLinux desktop, i have set up a quick apache webserver on my desktop and now tried to hit my domain, and it works!!

running nmap again now shows 2 services running, 443 for HTTPS, and 80 for HTTP, which are the two services i have running on my desktop. The strange thing is i also have many more ports open that are still pointed to the CentOS box which i can not access from the outside.

These new results are now leading me to believe its either network settings in my centos box or my iptables firewall. I dont think it would be iptables either since i stopped the service from automatically starting and i set it to allow all traffic. I am still unable to reach my server from any WAN. Id also like to add, that i have switched my server to another distribution and am still getting the same issues.

is it possible my mac addresses have been blacklisted? switching my router changed my ip address entirely, i have a dyndns and i have manually checked my DNS records on my domains they all point to the correct ip.

I have never been so lost on an issue like this before.


Edit: it's now 6pm, 6hours after I setup the small web server on my personal pc, which Is still accessible from the outside world. This is now leading me to believe some kind of DNS error or my server is putting up its own firewall other than iptables.
My DNS is currently google (8.8.8.8 8.8.4.4) set in the router.

Please help!

If nmap comes back blank then there is nothing there and the issue is either the ISP device or your router. then you have something odd with the router. If it is working now along with nmap, then there is your problem. nmap will NOT see your servers behind the router as it can not get past NAT. If it was blank before but is now responding, that means before the router was not properly configured, or was not properly responding. In either case it was the router causing issuesIf it were a DNS issue you would not be able to access the services via your LAN either. Also are you running DNS on your CentOS box? If not, then DNS will have nothing to do with visibility to a specific server behind your router. DNS will ONLY point to your WAN side IP. it can not get past NAT, see above with nmap. NAT is in and of it self a basic firewall by separating your WAN side IP and your LAN side PRIVATE computers from the rest of the world. It requires a ROUTER to negotiate past NAT.

again all of this information causes me to believe that your router is the issue just like my first post. If you can access your CentOS box via your LAN, but not your WAN and now we know for a fact that nmap was NOT working until you adjusted your router... hint hint hint... the issue is not your CentOS box, but the router of your ISP. With the information you have now provided I doubt it is your ISP.

money is on the router being the issue.

1 members found this post helpful.

The changes I made in my router were only changing the open port from my server to another machine. Nothing else in the router has changed. I am now getting results because I am no longer pointed to the centos server giving me the issue. If I go back into my router and change the port forward back to my centos server, my services go down again. It's not the router it's something on the server. I've tried two routers, two different brands, It's only when I have ports open on the server I can not make communication. Another reason why I think it's not the router.

Efficiency-wise there IMHO are some things you should not do (like re-install the OS just because you can't fix something right away) and some things you should do (like post network, firewall, router, service configuration, post actual diagnostics output).

- If you don't mind starting diagnostics from scratch start by reviewing your routers set up. Ensure it is set to your preferred configuration, ensure it does NAT (or DMZ) properly and ensure just one common port like TCP/80 is properly forwarded to your server.
- If your router allows you direct access to iptables then adding a "iptables -t filter -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j LOG" logging rule to the filter table INPUT chain right after the --state ESTABLISHED,RELATED rule may help during testing.
- If your router can log traffic (temporarily use one of your LAN hosts for remote syslog reception if the router allows remote syslogging) enable that. Do not tweak, reconfigure, change or otherwise modify your routers setting after this.
- Add the same logging rule to your servers firewall or otherwise ensure it logs inbound traffic.

- Resolve your DynDNS host name (here: "host.dom.ain") from a host outside your LAN and with default name server settings (meaning using defaults and not asking custom name servers like Google DNS or OpenDNS):
(Note these switches don't provide magic but aim to unclutter output only.)
*As a bonus the resulting IP address should have a proper PTR:
If the host name doesn't resolve to your routers IP address then you have a name server problem.
*If you don't have access to an external host use one of the gazillion web-based network tools.

- If "host.dom.ain" does resolve to your routers IP address then use a simple HEAD command to access the port from a host outside your LAN:
then check your router and servers firewall or syslog and your web servers access / error logs. (Note the User-Agent is only set for easy grepping your access / error logs, you could use the HEAD or GET if libwww-perl is available or telnet or netcat.)
*If you don't have access to an external host use one of the web-based header viewers like http://web-sniffer.net/.

- Don't post but attach as plain text your servers information *Do obfuscate IP addresses first but ensure a distinction can be made between public and LAN ones.

Thank you everyone for you help and patience so far, i am very grateful.

Last night before i went to bed i checked the services running on my system with "ntsysv", i turned on one service ntpd, rebooted the server and went to bed (left the router the way it was). This morning i checked on my server again, punched in of my domain names, and sure enough it brings me right to a my CentOS apache testpage. Tested a few other services i opened ports for, and it seems to be working. For the sake of troubleshooting, i ran everything unSpawn has mentioned. Here is the output along with the attached file.

i had to run these two commands on my server, my arch box doesnt recognize dig as a valid command.
Like you said to do this outside of my LAN, i ran this from my Work PC
everything seems to be Okay, could the possibility of the ntpd service not running cause an effect like this? Say if the date and time were out of sync?

thanks again guys for the help, right now it seems okay, i will be checking periodicity throughout the day if my site is still alive, it usually took some time before everything went down.

Attached Files

output.3370.txt (6.0 KB, 14 views)

Bumping this.

Still having this issue. I am starting to believe its either IPTables or a DNS problem. With iptables disabled and setup to allow all traffic, everything seems fine, now instead of a few hours i have been able to get about two days before all connections start getting dropped. I have also found for a quick TEMPORARY fix, i can run "#> ifdown eth0; ifup eth0; ifdown eth1; ifup eth1;" since i have two nics. once my nic cards come back online my websites are accessible by both IP and Domain Name. Im still convinced there is some kind of security other than iptables running that over time is kicking in to put the server into almost a "lockdown" state. I also have SELinux Disabled. Still scratching my head on this. When i get home from work later i might try switching the distro over to Debian 7 to see if the issue follows.

Howdy,
iptables is a constant allow or deny, unless you are using --limit in the config. If restarting your network is doing the job then we need to consider the possibility of a hardware problem. If you do an ifconfig what's the error count for the interface your web traffic is coming in on?

You have two interfaces, are you using the other one? Could we try swapping the ports (set eth1's config to eth0 and eth0's config to eth1, and switch the cables)?

In Ye Aulden Days the RH(EL?) ntpd package would modify the iptables rule set but AFAIK that isn't the case anymore (anyone confirm?). Unfortunately the OP never resolved and traced his FQDN from a host outside his LAN as I requested and neither was any machine / router firewall logging shown. This means the attached log file only confirms things work OK on the actual web server inside his LAN and remains inconclusive about remote networks and his router.

Im actually going to try and disable ntpd and see how things go, usually it seems after connections are getting dropped, if i leave the server alone, it eventually opens back up. My only other thought would be a DDoS attack or some other type of webserver flood. My /var/log/secure is clean, no incoming connections or even failed attempts at breaking in.

I still have yet to wipe and try debian 7, i may do that in a few days after i play around more with a few settings.

thanks again for the help so far

It sure is nice to talk about it but do realize that tells us zilch. Output from these tools / utils OTOH could help us help you:
route -n
ifconfig -a
iptables-save
logwatch