Weird loopback connection...

Djjules · 07-27-2006, 05:09 AM

Quote:

Originally Posted by blackhole54

Reinstallation might be a good idea. If you do so I would use different (and strong!) passwords.

Yep, I allready changed my root passwords to "stronger" one whether it's too late or not.

Quote:

Originally Posted by blackhole54

You never answered my question on post #5 about whether you allow ssh access on your server. If you don't need it I would turn it off completely. (This is generally a good security rule; disable any unneeded service. Each service is a potential security hole.) If you do need ssh, read up on how to offer it in the most secure way that meets your needs. There are a number of articles on the internet about this.

Umm, ssh access is blocked (at least) from Firestarter. I tested th ssh connection with putty with one of our workstations and Firestarter logged it as blocked.

Quote:

Originally Posted by blackhole54

Are the 127.0.0.1 connections on the workstations coming from their loopback interface, or are they coming in on ethernet? Am I correct that the workstations' only access to the Internet is through this server?

Well, my workstations are running on Windows XP's so... Damn, of course there is

127.0.0.1 connections on every workstation.

. Sorry 'bout that.

And YES you are correct that the workstations are connected to internet via this server.

Djjules · 07-27-2006, 05:37 AM

Quote:

Originally Posted by Capt_Caveman

IMO, I have yet to see anything indicative of a compromise. You can easily replicate the connection to 127.0.0.1 in the TIME_WAIT state by sending an email locally:

Code:

[root@localhost root]# netstat -nap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         
State       PID/Program name  
tcp        0      0 :::22                   0.0.0.0:*               LISTEN      1640/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1683/sendmail: acce           

[root@localhost root]# mail root@localhost
Subject: test
test
Cc: 
[root@localhost root]# netstat -nap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::22                   0.0.0.0:*               LISTEN      1640/sshd                
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1683/sendmail: acce   
tcp        0      0 127.0.0.1:1444          127.0.0.1:25            TIME_WAIT   -

Try sending an email locally to see if you can replicate the FS log and netstat output. Also you've never mentioned whether or not you looked at the mail logs. If you have some type of local mail problem, you could easily have connections sit there in TIME_WAIT. It's also entirely normal to see connections from loopback-to-loopback as well as from loopback to the machines external IP. Unless you are seeing connections to unknown IP addresses, then the traffic is not leaving your box.

Ok, I gotta admit that I've been very stupid with this. And I forgot to tell you that I have a script running by cron on every 10 minutes. And that script connects to email server.

But still: it shouldn't (and it don't) keep the connection open. It should only send an email and then quit and that's what it does. Is it normal that even sending an email 127.0.0.1 is connected to some weird private port? Shouldn't it be something like 25. I gotta get some documents how this loopback works anyways, since I don't seem to know even the basics.

Oh, and I looked through the email logs and didn't see anything out of order.

What about if the root "mailbox"? is full and can't receive any messages? How do I check this?

Capt_Caveman · 07-27-2006, 12:41 PM

Quote:

And I forgot to tell you that I have a script running by cron on every 10 minutes. And that script connects to email server.

If cron is sending an email every 10 min, then you should expect to see local connections to the mailer daemon on a fairly consistent basis. You'll have the actual mail being sent and then the connection in TIME_WAIT for awhile (~1 min) under normal cirumstances.

Quote:

But still: it shouldn't (and it don't) keep the connection open. It should only send an email and then quit and that's what it does.

Normally you'll have the connection remaining open for roughly 1 min, most of which will be in TIME_WAIT. If there is some problem with the mail being sent, then it could take even longer.

Quote:

Is it normal that even sending an email 127.0.0.1 is connected to some weird private port? Shouldn't it be something like 25.

Yes, it is normal and it *is* connected to port 25. The random port number you are seeing is the *source* port. The destination port is still port 25. That is the normal way in which most types of connections work. Opening a port less than 1024 requires root priviledges, so for normal users to establish connections, they need to use a non-priviledged source port above 1023 in order to open a listening socket (even for outgoing connections). Linux will assign source ports incrementally to socket requests starting with the port numbers listed in /proc/sys/net/ipv4/ip_local_port_range. So if you were to send out several emails in a row, each consecutive connection will get a different source port number (n, n+1, n+2, etc), but the destination port will still be port 25.

Quote:

What about if the root "mailbox"? is full and can't receive any messages? How do I check this?

If you login as root and check your mail, does it give you a warning about a full mailbox? An email every 10 min can definitely add up.

I would definitely recommend temporarily disabling the mail getting sent by cron every 10 min. You can just comment it out in your script. Then see if your weird mail issue is resolved. Again I still haven't seen anything suggesting a compromise. According to the netstat output this connection is local. It's not to a broadcast IP and it's not being tunnelled.

Djjules · 07-31-2006, 05:10 AM

Quote:

Originally Posted by Capt_Caveman

If cron is sending an email every 10 min, then you should expect to see local connections to the mailer daemon on a fairly consistent basis. You'll have the actual mail being sent and then the connection in TIME_WAIT for awhile (~1 min) under normal cirumstances.

Normally you'll have the connection remaining open for roughly 1 min, most of which will be in TIME_WAIT. If there is some problem with the mail being sent, then it could take even longer.

Yup, this I figured out...

Quote:

Originally Posted by Capt_Caveman

Yes, it is normal and it *is* connected to port 25. The random port number you are seeing is the *source* port. The destination port is still port 25. That is the normal way in which most types of connections work. Opening a port less than 1024 requires root priviledges, so for normal users to establish connections, they need to use a non-priviledged source port above 1023 in order to open a listening socket (even for outgoing connections). Linux will assign source ports incrementally to socket requests starting with the port numbers listed in /proc/sys/net/ipv4/ip_local_port_range. So if you were to send out several emails in a row, each consecutive connection will get a different source port number (n, n+1, n+2, etc), but the destination port will still be port 25.

So do You think it is normal that at the moment the port it is connected to is 6009/Xwindows?

I guess that it uses any port it gets without root priviledges...

Quote:

Originally Posted by Capt_Caveman

If you login as root and check your mail, does it give you a warning about a full mailbox? An email every 10 min can definitely add up.

I deleted all root mails. At least when typing mail -u root I get no mails for user root.

Quote:

Originally Posted by Capt_Caveman

I would definitely recommend temporarily disabling the mail getting sent by cron every 10 min. You can just comment it out in your script. Then see if your weird mail issue is resolved. Again I still haven't seen anything suggesting a compromise. According to the netstat output this connection is local. It's not to a broadcast IP and it's not being tunnelled.

I will disable the script for a moment and then check if something changes.

Capt_Caveman · 07-31-2006, 12:00 PM

Quote:

So do You think it is normal that at the moment the port it is connected to is 6009/Xwindows?

I guess that it uses any port it gets without root priviledges...

Depends. If the *source* port is 6009 and the destination port is still 25, then it is normal. For ports over 1023, the service/port combos listed in /etc/services are simply which services are commonly used with those port numbers, some of which have been registered with IANA. They are not an absolute rule though. The system will start with a specific port number and then simply increment, so it could use port 6010 in 10 minutes and then port 7588 tomorrow. Also remember that those port assignments are referring to the destination port number, not the source port.

Quote:

I will disable the script for a moment and then check if something changes.

Keep us informed.

nx5000 · 08-01-2006, 07:57 AM

I remembered reading in the changelogs of new kernels that now source ports are randomized and don't increment only. This is a big security fix.

Capt_Caveman · 08-01-2006, 09:51 AM

Quote:

Originally Posted by nx5000

I remembered reading in the changelogs of new kernels that now source ports are randomized and don't increment only. This is a big security fix.

Interesting. People have been complaining about that for quite some time now. Do you know when the change occurred?

nx5000 · 08-01-2006, 11:35 AM

Quote:

Originally Posted by Capt_Caveman

Interesting. People have been complaining about that for quite some time now. Do you know when the change occurred?

Yep interesting:

http://www.kernel.org/pub/linux/kern...angeLog-2.6.15

Quote:

commit 6df716340da3a6fdd33d73d7ed4c6f7590ca1c42
Author: Stephen Hemminger <shemminger@osdl.org>
Date: Thu Nov 3 16:33:23 2005 -0800

[TCP/DCCP]: Randomize port selection

This patch randomizes the port selected on bind() for connections
to help with possible security attacks. It should also be faster
in most cases because there is no need for a global lock.

Signed-off-by: Stephen Hemminger <shemminger@osdl.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@mandriva.com>

blackhole54 · 08-01-2006, 06:19 PM

Quote:

Originally Posted by nx5000

I remembered reading in the changelogs of new kernels that now source ports are randomized and don't increment only. This is a big security fix.

So does the kernel still use net.ipv4.ip_local_port_range to determine the range to pull those random port numbers from? If so, Djjules could compare the source ports observed against this range for a little added comfort. Or lack thereof.

Djjules · 08-02-2006, 05:11 AM

Ok, now the script have been disabled for 2 days - no visible changes. The connection remains (according to Firestarter) and still the same connection cannot be found with netstat -nap | fgrep 127.0.0.1.

Here you can find a screenshot of the Firestarter GUI, if it helps. There are two images in that folder from 2 different days.

Quote:

Originally Posted by blackhole54

So does the kernel still use net.ipv4.ip_local_port_range to determine the range to pull those random port numbers from? If so, Djjules could compare the source ports observed against this range for a little added comfort. Or lack thereof.

How can I generally do this?

nx5000 · 08-02-2006, 07:57 AM

Quote:

Originally Posted by blackhole54

So does the kernel still use net.ipv4.ip_local_port_range to determine the range to pull those random port numbers from? If so, Djjules could compare the source ports observed against this range for a little added comfort. Or lack thereof.

No clue.

Isn't net.ipv4.ip_local_port_range used only when SELinux is enabled?

I was not able to find the code of the above mentionned patch from Stephen.

blackhole54 · 08-02-2006, 09:31 AM

Quote:

Originally Posted by nx5000

Isn't net.ipv4.ip_local_port_range used only when SELinux is enabled

I don't have experience with the 2.6 kernel. In the 2.2 and 2.4 kernels this was the sysctl parameter that controlled the ports typically used by nopriviledged applications (I am guessing through bind(), but I've really never delved into it in that depth). It had nothing to do with SELinux. So if /etc/sysctl.conf had a line that said

net.ipv4.ip_local_port_range = 1024 4999

The first source port assigned would be 1024, the next 1025, etc up to 4999 and then it would cycle back. I've in fact modified a line like the above to read, e.g

net.ipv4.ip_local_port_range = 51024 54999

To move these ports away from some ports I was blocking liked 6000-6010.

@Djjules

If this still exists, you can check its value, as root, by giving the command

sysctl -a | grep port

My question was whether this still exists and whether, other than the randomization, it still has the same meaning it used to. If it does on both counts, then I would expect the connections Capt_Caveman is talking about to have source port numbers in this range.

I have glanced over the FireStarter manual. While I think I have somewhat of a grasp on the UI, I don't really have a feel for what it is doing behind the scenes. Specifically I don't know what the daemon is doing.

Also remember that FS lists both incoming and outgoing connections in that list (although a loopback is both). So it is not clear to me how to interpret port 6009. And I don't know why it doesn't show up with netstat.

nx5000 · 08-02-2006, 10:19 AM

To get the best documentation... look at the source:

->2.2:
http://lxr.linux.no/ident?v=2.2.26;i...cal_port_range
->2.4:
http://lxr.linux.no/ident?v=2.4.28;i...cal_port_range
->2.6:
http://www.linux-m32r.org/lxr/http/i...cal_port_range

There has been a lot of modification for the 2.6...

Djjules · 08-03-2006, 04:41 AM

Quote:

Originally Posted by blackhole54

If this still exists, you can check its value, as root, by giving the command

sysctl -a | grep port

My question was whether this still exists and whether, other than the randomization, it still has the same meaning it used to. If it does on both counts, then I would expect the connections Capt_Caveman is talking about to have source port numbers in this range.

I have glanced over the FireStarter manual. While I think I have somewhat of a grasp on the UI, I don't really have a feel for what it is doing behind the scenes. Specifically I don't know what the daemon is doing.

Also remember that FS lists both incoming and outgoing connections in that list (although a loopback is both). So it is not clear to me how to interpret port 6009. And I don't know why it doesn't show up with netstat.

With sysctl -a | grep port I got the following

Code:

net.ipv4.ip_local_port_range = 32768    61000

so it seems that the ports in between that range are normal. So this part is almost solved. The only thing that bothers me at the moment is that the port is now 6009...

Djjules · 08-04-2006, 01:15 AM

The weird connection is gone.

It just happened overnight.

I read a little bit if the Firestarter manual and blocked once again the outgoing traffic for 127.0.0.1 and for port 6009 yesterday.

So

I'm going to get the script running once again (after few weeks or so) and if this loopback starts haunting again, I'll figure this one out myself.

Many thnx to everyone.