Can't make inbound connections, and servers won't activate (no ports available)

CJ Chitwood · 01-21-2008, 01:22 PM

Hello, yet again...

Story: I have (err, HAD) Apache 2.2.3 serving a website, www.test.tld. I wanted to set it up to serve on the same IP address the site devel.test.tld (using an actual real-life FQDN of course). So I set out to figure stuff out, and immediately went to Webmin.

This will be the final time I ever use Webmin as my limited understanding of it and the files it edits gets me in more trouble than manually editing the files themselves. But I digress.

Somehow, something I edited in Webmin borked the crap out of my system. I can no longer make connections on sockets. Although the Apache module in Webmin was the ONLY module I screwed around in, somehow or another I've managed to lock out all services on all ports. My DNS doesn't serve anymore, I can't ssh or telnet localhost, I can't even 'ping 127.0.0.1' and get a reply.

I have been searching for three hours last night and two this morning trying for the life of me to find out what can cause this. I have found nothing.

The original clue was when Apache said no ports were available:

Code:

{YOU ARE ROOT}
[myhostname][pts/1]
[root][/var]# apache2ctl start 
no listening sockets available, shutting down
Unable to open logs

{YOU ARE ROOT}
[myhostname][pts/1]
[root][/var]#

This actually wasn't the original error. The original error I forget, but it had something to do with my VirtualHosts etc. and I have since resolved that.

Still, as I started exploring, port 80 was NOT in use by anything, and that's the only port that I can find in any of the Apache configs I know of.

Code:

{YOU ARE ROOT}
[myhostname][pts/1]
[root][/var]# lsof -i :80

{YOU ARE ROOT}
[myhostname][pts/1]
[root][/var]#

I later tried various things, and eventually learned I could also not dig @127.0.0.1, I couldn't ping 127.0.0.1, I couldn't do anything to my own address, my own hostname, nor localhost.

It's almost like my firewall is blocking all, but I haven't messed with any of its rules in like 4 months, and even when I tell Firestarter to turn off the firewall, and confirm it's wide open, I still can't connect to myself (and of course, neither can anyone else).

I'm right now running /etc/init.d/portmap restart to no avail -- says restarting, but no activity.

Code:

{YOU ARE ROOT}
[myhostname][pts/0]
[root][/etc/apache2]# lsof -i
COMMAND     PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
portmap    2631 daemon    3u  IPv4    7599       UDP *:sunrpc 
portmap    2631 daemon    4u  IPv4    7600       TCP *:sunrpc (LISTEN)
named      2883   bind   20u  IPv6    8000       UDP *:domain 
named      2883   bind   21u  IPv6    8001       TCP *:domain (LISTEN)
named      2883   bind   22u  IPv4    8003       UDP *:32768 
named      2883   bind   23u  IPv6    8004       UDP *:32769 
named      2883   bind   24u  IPv4   37368       UDP myhostname.mydomain.tld:domain 
named      2883   bind   25u  IPv4   37369       TCP myhostname.mydomain.tld:domain (LISTEN)
named      2883   bind   26u  IPv4   63132       UDP 192.168.0.1:domain 
named      2883   bind   27u  IPv4   63133       TCP 192.168.0.1:domain (LISTEN)
mysqld     3032  mysql   13u  IPv4    8200       TCP *:mysql (LISTEN)
cupsd      3128   root    3u  IPv6  118763       TCP *:ipp (LISTEN)
cupsd      3128   root    4u  IPv4  118764       TCP *:ipp (LISTEN)
cupsd      3128   root    6u  IPv4  118767       UDP *:ipp 
dhcpd      3153   root    7u  IPv4    8522       UDP *:bootps 
master     3325   root   11u  IPv4    8861       TCP *:smtp (LISTEN)
master     3325   root   12u  IPv6    8862       TCP *:smtp (LISTEN)
miniserv.  3552   root    6u  IPv4    9348       TCP *:spare-NAT (LISTEN)
miniserv.  3552   root    7u  IPv4    9349       UDP *:spare-NAT 
dhclient   3622   root    7u  IPv4    9340       UDP *:bootpc 
pmap_dump 11153   root    3u  IPv4 1769795       TCP myhostname.mydomain.tld:729->myhostname.mydomain.tld:sunrpc (SYN_SENT)
xfce4-ses 11173     cj    4u  IPv6   12355       TCP *:47446 (LISTEN)
xfce4-ses 11173     cj    5u  IPv4   12356       TCP *:44370 (LISTEN)
sshd      24632   root    3u  IPv6 1036232       TCP *:ssh (LISTEN)
named     25753   bind   20u  IPv6 1051187       UDP *:domain 
named     25753   bind   22u  IPv4 1051190       UDP myhostname.mydomain.tld:domain 
named     25753   bind   24u  IPv4 1051192       UDP 192.168.0.1:domain 
named     25753   bind   26u  IPv4 1051194       UDP *:32808 
named     25753   bind   27u  IPv6 1051195       UDP *:32809 
named     25844   bind   20u  IPv6 1052081       UDP *:domain 
named     25844   bind   22u  IPv4 1052084       UDP myhostname.mydomain.tld:domain 
named     25844   bind   24u  IPv4 1052086       UDP 192.168.0.1:domain 
named     25844   bind   26u  IPv4 1052088       UDP *:32810 
named     25844   bind   27u  IPv6 1052089       UDP *:32811 

{YOU ARE ROOT}
[myhostname][pts/0]
[root][/etc/apache2]# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN     3032/mysqld         
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     2631/portmap        
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN     3552/perl           
tcp        0      0 0.0.0.0:44370           0.0.0.0:*               LISTEN     11173/xfce4-session 
tcp        0      0 192.168.0.1:53          0.0.0.0:*               LISTEN     2883/named          
tcp        0      0 192.168.10.9:53         0.0.0.0:*               LISTEN     2883/named          
tcp        0      0 0.0.0.0:631             0.0.0.0:*               LISTEN     3128/cupsd          
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN     3325/master         
tcp6       0      0 :::53                   :::*                    LISTEN     2883/named          
tcp6       0      0 :::22                   :::*                    LISTEN     24632/sshd          
tcp6       0      0 :::47446                :::*                    LISTEN     11173/xfce4-session 
tcp6       0      0 :::631                  :::*                    LISTEN     3128/cupsd          
tcp6       0      0 :::25                   :::*                    LISTEN     3325/master         
udp        0      0 0.0.0.0:32768           0.0.0.0:*                          2883/named          
udp        0      0 0.0.0.0:10000           0.0.0.0:*                          3552/perl           
udp        0      0 0.0.0.0:32808           0.0.0.0:*                          25753/named         
udp        0      0 0.0.0.0:32810           0.0.0.0:*                          25844/named         
udp        0      0 192.168.0.1:53          0.0.0.0:*                          25844/named         
udp        0      0 192.168.10.9:53         0.0.0.0:*                          25844/named         
udp        0      0 192.168.0.1:53          0.0.0.0:*                          25753/named         
udp        0      0 192.168.10.9:53         0.0.0.0:*                          25753/named         
udp        0      0 192.168.0.1:53          0.0.0.0:*                          2883/named          
udp        0      0 192.168.10.9:53         0.0.0.0:*                          2883/named          
udp        0      0 0.0.0.0:67              0.0.0.0:*                          3153/dhcpd          
udp        0      0 0.0.0.0:68              0.0.0.0:*                          3622/dhclient       
udp        0      0 0.0.0.0:111             0.0.0.0:*                          2631/portmap        
udp        0      0 0.0.0.0:631             0.0.0.0:*                          3128/cupsd          
udp6       0      0 :::32769                :::*                               2883/named          
udp6       0      0 :::32809                :::*                               25753/named         
udp6       0      0 :::32811                :::*                               25844/named         
udp6       0      0 :::53                   :::*                               25844/named         
udp6       0      0 :::53                   :::*                               25753/named         
udp6       0      0 :::53                   :::*                               2883/named          
raw        0      0 0.0.0.0:1               0.0.0.0:*               7          3153/dhcpd          
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     8891     3325/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     9141     3381/xfstt          /tmp/.font-unix/fs7101
unix  2      [ ACC ]     STREAM     LISTENING     9704     3721/X              /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     12319    11158/ssh-agent     /tmp/ssh-aKpuS11104/agent.11104
unix  2      [ ACC ]     STREAM     LISTENING     12357    11173/xfce4-session /tmp/.ICE-unix/11173
unix  2      [ ACC ]     STREAM     LISTENING     8895     3325/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     8899     3325/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     8903     3325/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     8907     3325/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     8911     3325/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     8915     3325/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     8919     3325/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     8923     3325/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     8927     3325/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     8931     3325/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     8935     3325/master         private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     8939     3325/master         private/uucp
unix  2      [ ACC ]     STREAM     LISTENING     8943     3325/master         private/ifmail
unix  2      [ ACC ]     STREAM     LISTENING     8947     3325/master         private/bsmtp
unix  2      [ ACC ]     STREAM     LISTENING     8951     3325/master         private/scalemail-backend
unix  2      [ ACC ]     STREAM     LISTENING     8955     3325/master         private/spamassassin
unix  2      [ ACC ]     STREAM     LISTENING     1637045  4224/gconfd-2       /tmp/orbit-cj/linc-1080-0-30d4525696fba
unix  2      [ ACC ]     STREAM     LISTENING     179599   25911/xmms          /tmp/xmms_cj.0
unix  2      [ ACC ]     STREAM     LISTENING     1637051  4209/iceape-bin     /tmp/orbit-cj/linc-1071-0-d649f989923a
unix  2      [ ACC ]     STREAM     LISTENING     12328    11165/dbus-daemon   @/tmp/dbus-zh7o7lCUdK
unix  2      [ ACC ]     STREAM     LISTENING     8503     3145/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     8868     3325/master         public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     12397    11201/gam_server    @/tmp/fam-cj-
unix  2      [ ACC ]     STREAM     LISTENING     8875     3325/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     8879     3325/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     8883     3325/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     8887     3325/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     118765   3128/cupsd          /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     9026     3352/saslauthd      /var/spool/postfix/var/run/saslauthd/mux
unix  2      [ ACC ]     STREAM     LISTENING     9567     3697/gdm            /tmp/.gdm_socket
unix  2      [ ACC ]     STREAM     LISTENING     8201     3032/mysqld         /var/run/mysqld/mysqld.sock

{YOU ARE ROOT}
[myhostname][pts/0]
[root][/etc/apache2]#

Recap:

Cannot ping, telnet, ssh, http, ftp, dig, or anything; to my own hostname, localhost, or 127.0.0.1. These services OUTBOUND work fine. That is to say, I can (obviously) load web pages on other servers, I can ssh out to other servers... just no connections from the outside (on my LAN) in.
No, my firewall is not blocking them.
hosts.allow and hosts.deny completely commented out -- effectively, they are blank. I've never touched them, in fact.

I'm at a real loss here. I also have to find a way to configure Apache. I told Synaptic to do a complete removal (before I realized this was a bigger-than-that problem) including config files, expecting that reinstalling them later would recreate a default set of config files.

Nope.

So I'm hoping maybe it's related. I don't see how.

Please, anything?

Thanks,

CJ

P.S. / Edit: In the "code" block above, the output of "netstat -nlp" shows a lot of 0.0.0.0 -- that *IS* normal, *RIGHT*?

ilikejam · 01-21-2008, 01:58 PM

Hi.

What does 'iptables -L' look like?

Dave

farslayer · 01-21-2008, 02:35 PM

0.0.0.0 indicates that the service is listening on all available addresses 127.0.0.1, 192.168.0.1, etc..

have you verified there is nothing wrong in your configuration using apachectl, or apache2ctl (depending on which version of apache you are running.)

Code:

it-etch:~# apache2ctl -t
[Mon Jan 21 15:30:19 2008] [warn] NameVirtualHost *:0 has no VirtualHosts
Syntax OK

If you want to remove apache and it's configuration files to start over from scratch, you would need to purge it rather than remove it..

aptitude purge apache2

remove un-installs the application but leaves the configuration files
purge un-installs the application AND all of it's configuration files

Apache2 Name based virtual hosting
more virtual hosting docs and examples

CJ Chitwood · 01-21-2008, 04:56 PM

Quote:

Originally Posted by ilikejam

Hi.

What does 'iptables -L' look like?

Dave

iptables -L

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- 192.168.10.1 anywhere tcp flags:!FIN,SYN,RST,ACK/SYN
ACCEPT udp -- 192.168.10.1 anywhere
ACCEPT 0 -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere limit: avg 10/sec burst 5
DROP 0 -- anywhere 255.255.255.255
DROP 0 -- anywhere 192.168.10.255
DROP 0 -- BASE-ADDRESS.MCAST.NET/8 anywhere
DROP 0 -- anywhere BASE-ADDRESS.MCAST.NET/8
DROP 0 -- 255.255.255.255 anywhere
DROP 0 -- anywhere default
DROP 0 -- anywhere anywhere state INVALID
LSI 0 -f anywhere anywhere limit: avg 10/min burst 5
INBOUND 0 -- anywhere anywhere
INBOUND 0 -- anywhere 192.168.0.1
INBOUND 0 -- anywhere myhostname.mydomain.tld
INBOUND 0 -- anywhere 192.168.0.255
LOG_FILTER 0 -- anywhere anywhere
LOG 0 -- anywhere anywhere LOG level info prefix `Unknown Input'

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT icmp -- anywhere anywhere limit: avg 10/sec burst 5
TCPMSS tcp -- anywhere anywhere tcp flags:SYN,RST/SYN TCPMSS clamp to PMTU
OUTBOUND 0 -- anywhere anywhere
ACCEPT tcp -- anywhere 192.168.0.0/24 state RELATED,ESTABLISHED
ACCEPT udp -- anywhere 192.168.0.0/24 state RELATED,ESTABLISHED
LOG_FILTER 0 -- anywhere anywhere
LOG 0 -- anywhere anywhere LOG level info prefix `Unknown Forward'

Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- myhostname.mydomain.tld 192.168.10.1 tcp dpt:domain
ACCEPT udp -- myhostname.mydomain.tld 192.168.10.1 udp dpt:domain
ACCEPT 0 -- anywhere anywhere
DROP 0 -- BASE-ADDRESS.MCAST.NET/8 anywhere
DROP 0 -- anywhere BASE-ADDRESS.MCAST.NET/8
DROP 0 -- 255.255.255.255 anywhere
DROP 0 -- anywhere default
DROP 0 -- anywhere anywhere state INVALID
OUTBOUND 0 -- anywhere anywhere
OUTBOUND 0 -- anywhere anywhere
LOG_FILTER 0 -- anywhere anywhere
LOG 0 -- anywhere anywhere LOG level info prefix `Unknown Output'

Chain INBOUND (4 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT 0 -- 192.168.10.0/24 anywhere
ACCEPT 0 -- 192.168.100.0/24 anywhere
ACCEPT 0 -- 192.168.0.0/24 anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:www
ACCEPT udp -- anywhere anywhere udp dpt:www
ACCEPT tcp -- anywhere anywhere tcp dpt:https
ACCEPT udp -- anywhere anywhere udp dpt:https
ACCEPT tcp -- anywhere anywhere tcp dpt:ntp
ACCEPT udp -- anywhere anywhere udp dpt:ntp
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT udp -- anywhere anywhere udp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpts:62778:gnut-NAT
ACCEPT udp -- anywhere anywhere udp dpts:62778:gnut-NAT
ACCEPT tcp -- anywhere anywhere tcp dpt:smtp
ACCEPT udp -- anywhere anywhere udp dpt:25
ACCEPT tcp -- anywhere anywhere tcp dpt:10025
ACCEPT udp -- anywhere anywhere udp dpt:10025
ACCEPT tcp -- 169.157.0.0/16 anywhere tcp dpts:ftp-data:ftp
ACCEPT udp -- 169.157.0.0/16 anywhere udp dpts:20:fsp
ACCEPT tcp -- anywhere anywhere tcp dpt

op3
ACCEPT udp -- anywhere anywhere udp dpt

op3
LSI 0 -- anywhere anywhere

Chain LOG_FILTER (5 references)
target prot opt source destination

Chain LSI (2 references)
target prot opt source destination
LOG_FILTER 0 -- anywhere anywhere
LOG tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5 LOG level info prefix `Inbound '
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/SYN
LOG tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5 LOG level info prefix `Inbound '
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,ACK/RST
LOG icmp -- anywhere anywhere icmp echo-request limit: avg 1/sec burst 5 LOG level info prefix `Inbound '
DROP icmp -- anywhere anywhere icmp echo-request
LOG 0 -- anywhere anywhere limit: avg 5/sec burst 5 LOG level info prefix `Inbound '
DROP 0 -- anywhere anywhere

Chain LSO (0 references)
target prot opt source destination
LOG_FILTER 0 -- anywhere anywhere
LOG 0 -- anywhere anywhere limit: avg 5/sec burst 5 LOG level info prefix `Outbound '
REJECT 0 -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTBOUND (3 references)
target prot opt source destination
ACCEPT icmp -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT 0 -- anywhere anywhere

Actually, I've tried it with and without Firestarter active. If I tell Firestarter to deactivate, then run iptables -L, I get this:

Code:

{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]# /etc/init.d/firestarter stop
Stopping the Firestarter firewall:done.

{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.

--- localhost ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3011ms


{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]#

CJ Chitwood · 01-21-2008, 05:07 PM

Quote:

Originally Posted by farslayer

0.0.0.0 indicates that the service is listening on all available addresses 127.0.0.1, 192.168.0.1, etc..

That's what I thought... Wanted to be sure. Thanks!

Quote:

Originally Posted by farslayer

have you verified there is nothing wrong in your configuration using apachectl, or apache2ctl (depending on which version of apache you are running.)

Code:

it-etch:~# apache2ctl -t
[Mon Jan 21 15:30:19 2008] [warn] NameVirtualHost *:0 has no VirtualHosts
Syntax OK

Code:

{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]# apache2ctl -t
Syntax OK

{YOU ARE ROOT}
[myhostname][pts/0]
[root][~]#

Quote:

Originally Posted by farslayer

If you want to remove apache and it's configuration files to start over from scratch, you would need to purge it rather than remove it..

Correct. In synaptic, there is a "remove completely" option that, according to the program, removes configuration files as well. I'm starting to think, however, that I would be well advised to change the overall structure that I'm given with the installation (or, WAS given BEFORE with the installation) so that I can more easily manage having multiple sites. If the two I want to do go well, I'm going to add one for myself as a subdomain.

Then again, at the minimal cost of a domain name, it should be easy enough to forward multiple domains to the same IP and let Apache sort them out.

But I digress again -- right now I just want to get it working. I'll then back up what I can and after that start tinkering with conffiles.

Thanks also for the examples. I had been reading up on them (and several others) around the time this problem started. I remember thinking I borked something in the configs, and tried reverting, knowing full well there was nothing wrong with the file after I had reverted, and still nothing -- that's about the time I started suspecting something deeper than Apache.

This isn't an Apache issue. It's closer to the O/S than that. I'm starting to suspect RPC, portmap, etc.

Maybe if I reinstall that......?

CJ Chitwood · 01-21-2008, 05:41 PM

Okay. I've reinstalled portmap. I don't know that it is what did anything, but at least now I can ping/ssh/webmin my localhost and 127.0.0.1, as well as my other assigned IP addresses. So now I know I have at least some ports working again. I can only assume portmap reinstall is what did it.

However, was it portmap, or is there a portmap config file somewhere that I know not of? And I still have issues with Apache, which I still think are coincidental. I'll start looking into getting it back up now.

However, I'm still very open to suggestions on what the problem could be. I've got that feeling that this isn't over yet.

Thanks

CJ

farslayer · 01-21-2008, 06:00 PM

Well from the man page for portmap that would make sense.. and no I don't see a config file for it anywhere. must have been stopped or corrupted in some fashion then ehh ?

Quote:

Portmap is a server that converts RPC program numbers into DARPA protocol
port numbers. It must be running in order to make RPC calls.

When an RPC server is started, it will tell portmap what port number it
is listening to, and what RPC program numbers it is prepared to serve.

CJ Chitwood · 01-22-2008, 07:53 PM

Quote:

Originally Posted by farslayer

Well from the man page for portmap that would make sense.. and no I don't see a config file for it anywhere. must have been stopped or corrupted in some fashion then ehh ?

Yeah, stopped I don't know about; I'd rebooted, changed runlevels, etc. multiple times to know avail. Corrupted, I could see sooner... although, what corrupted it? Or, was it something that simply prevented portmap from running correctly? I don't know. I reinstalled, and the source was the same .deb that was sitting in my apt archive, so it's the same version and same files, so it wasn't a versioning issue...

Dunno...