Port 80/UDP

czezz · 02-03-2014, 04:01 PM

On my Linux Box I can see with iptraf outgoing traffic on port 80 UDP.
See screen shot under this link: https://www.dropbox.com/s/uq2r04akydbe0fm/80udp.PNG

This port is not used in my system and also by netstat I can see nothing.

Code:

# netstat -anop
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer
tcp        0      0 0.0.0.0:10022           0.0.0.0:*               LISTEN      2793/sshd        off (0.00/0/0)
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      6894/(squid)     off (0.00/0/0)
tcp        0    312 83.19.119.231:10022     84.75.41.161:63055      ESTABLISHED 6544/sshd: meeeee [ on (0.26/0/0)
tcp        0    300 83.19.119.231:10022     84.75.41.161:63042      ESTABLISHED 6514/sshd: meeeee [ on (0.46/0/0)
tcp6       0      0 :::10022                :::*                    LISTEN      2793/sshd        off (0.00/0/0)
udp        0      0 0.0.0.0:47914           0.0.0.0:*                           6894/(squid)     off (0.00/0/0)
udp        0      0 0.0.0.0:3130            0.0.0.0:*                           6894/(squid)     off (0.00/0/0)
udp        0      0 0.0.0.0:67              0.0.0.0:*                           2882/dhcpd       off (0.00/0/0)
raw        0      0 0.0.0.0:1               0.0.0.0:*               7           2882/dhcpd       off (0.00/0/0)

fuser gives no output as well

Code:

# fuser 80/udp

It wonders me what the heck is happening there.
I have blocked OUTPUT traffic on port 80 UDP with iptables:

Code:

iptables -A OUTPUT -p udp --dport 80 -j DROP

...but that did not stopped it at all.
Anyone have any clue what is going on here ?

My box is:

Code:

Slackware 12.2

# uname -a
Linux proxy 2.6.27.7-smp #2 SMP Thu Nov 20 22:32:43 CST 2008 i686 Intel(R) Pentium(R) 4 CPU 3.80GHz GenuineIntel GNU/Linux

Habitual · 02-03-2014, 04:38 PM

Do you have logging enabled on iptraf?
You can then check the log at /var/log/iptraf/file.log after it runs for a minute or 3.

Mine is at /var/log/iptraf/iface_stats_detailed-eth0.log

Edit: shouldn't that have been

Code:

fuser 80/udp

?

There's this that may reveal something?

Code:

netstat -p udp -launt | grep udp
udp        0      0 0.0.0.0:63606           0.0.0.0:*                           3307/skype          
udp        0      0 0.0.0.0:37              0.0.0.0:*                           2046/inetd          
udp        0      0 0.0.0.0:512             0.0.0.0:*                           2046/inetd          
udp        0      0 127.0.0.1:36775         0.0.0.0:*                           3307/skype

I'm sure there's a "correct" way to solicit only upd packets with those switches but I'm just trying to help you out, not be "correct".

czezz · 02-03-2014, 04:57 PM

Hi, thanks for answer.
Here is output you asked:

Code:

Mon Feb  3 23:50:08 2014; ******** TCP/UDP service monitor started ********

*** TCP/UDP traffic log, generated Mon Feb  3 23:51:54 2014

UDP/123: 3912 packets, 140832 bytes total, 10.62 kbits/s; 3912 packets, 140832 bytes incoming, 10.62 kbits/s; 52 packets, 1872 bytes outgoing, 0.13 kbits/s

UDP/80: 2349 packets, 84564 bytes total, 6.38 kbits/s; 0 packets, 0 bytes incoming, 0.00 kbits/s; 2349 packets, 84564 bytes outgoing, 6.38 kbits/s

UDP/22: 109 packets, 3924 bytes total, 0.29 kbits/s; 0 packets, 0 bytes incoming, 0.00 kbits/s; 109 packets, 3924 bytes outgoing, 0.29 kbits/s

UDP/53: 268 packets, 10292 bytes total, 0.78 kbits/s; 8 packets, 586 bytes incoming, 0.04 kbits/s; 260 packets, 9706 bytes outgoing, 0.73 kbits/s

TCP/443: 19 packets, 1332 bytes total, 0.11 kbits/s; 12 packets, 859 bytes incoming, 0.07 kbits/s; 7 packets, 473 bytes outgoing, 0.03 kbits/s

UDP/666: 29 packets, 1044 bytes total, 0.10 kbits/s; 0 packets, 0 bytes incoming, 0.00 kbits/s; 29 packets, 1044 bytes outgoing, 0.10 kbits/s


Running time: 106 seconds
Mon Feb  3 23:51:54 2014; ******** TCP/UDP service monitor stopped ********

You are also right about fuser - it should be fuser 80/udp(edited and corrected)

I think I got something more now.

It was very difficult to log in to the system with SSH as the traffic was really huge.
I have stopped NTP server as I noticed this in syslog and it helped immediately:

Code:

cat /var/log/syslog
Feb  3 00:07:55 mybox last message repeated 2 times
Feb  3 00:08:15 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 79.133.192.40
Feb  3 00:08:38 mybox last message repeated 79 times
Feb  3 00:09:33 mybox last message repeated 117 times
Feb  3 00:10:26 mybox last message repeated 115 times
Feb  3 00:11:23 mybox last message repeated 178 times
Feb  3 00:12:21 mybox last message repeated 92 times
Feb  3 00:13:31 mybox last message repeated 98 times
Feb  3 00:31:34 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 84.201.34.76
Feb  3 00:31:59 mybox last message repeated 21 times
Feb  3 08:10:15 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:11:09 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:12:09 mybox last message repeated 4 times
Feb  3 08:13:05 mybox last message repeated 3 times
Feb  3 08:15:29 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:17:32 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:27:15 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:30:57 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 103.25.202.54
Feb  3 08:31:41 mybox last message repeated 2 times
Feb  3 16:50:34 mybox ntpd[2553]: process_private: INFO_ERR_FMT: test 1 failed, pkt from 85.25.119.13

I think I also read incorrect screen output form iptraf. If I look now at tcpdump I see this.
It looks like someone/something floods me with UDP traffic comming from foreign addresses from port 80 to my local port 123.

Code:

# tcpdump -i eth0 -s0 -n -nn -N -p -tttt port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
2014-02-03 23:39:36.493346 IP 174.58.132.224.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:36.556847 IP 1.2.219.171.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:36.669153 IP 99.225.21.112.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:36.791598 IP 187.2.231.194.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.034541 IP 174.58.132.224.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.058172 IP 139.146.146.225.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.085260 IP 31.170.160.173.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.440323 IP 177.84.148.135.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.477727 IP 177.84.148.135.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.539066 IP 177.84.148.135.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:39:37.623584 IP 139.146.146.225.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
...

Code:

# tcpdump -i eth0 -s0 -n -nn -N -p -tttt port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
2014-02-03 23:58:44.958644 IP 31.186.242.82.5147 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.108765 IP 91.121.17.31.2302 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.134872 IP 31.170.160.173.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.137333 IP 80.86.90.242.28315 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.199409 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.200647 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.241549 IP 68.117.43.29.3074 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.243765 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.248684 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.253617 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.324314 IP 68.117.43.29.3074 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.375055 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
2014-02-03 23:58:45.400468 IP 67.176.204.78.80 > 83.19.119.231.123: NTPv2, Reserved, length 8
...

NOTE: this is happening on different 2 nodes in 2 different locations. Nodes are completly not related with each other Odd.

czezz · 02-03-2014, 06:03 PM

I think this might be (partly) an answer: http://www.cvedetails.com/cve/CVE-2013-5211/

Habitual · 02-03-2014, 06:08 PM

Quote:

Originally Posted by czezz

I think this might be (partly) an answer: http://www.cvedetails.com/cve/CVE-2013-5211/

Could be that you just found the source.

Quote:

Originally Posted by czezz

I have stopped NTP server as I noticed this in syslog and it helped immediately:

Is it now disabled or merely stopped?

Maybe your rule should be:

Code:

iptables -A INPUT -p udp --dport 139 -j DROP

to further discourage abuse?
Shutting down ntp is a start, but this may also be necessary.

czezz · 02-04-2014, 02:44 AM

NPTd is stopped fully: /etc/rc.d/rc.ntpd stop Additionally, I have blocked port 123 in my iptables.
Why do you want me to block port 139 ? There is no smbd/nmbd on my box.

I might have a bad understanding of iptraf output but it seems that there is a lot of traffic:
coming in to port 123
coming out from port 80

I have blocked both ports INPUT and OUTPUT with iptables and see no changes :/

pingwinowiewc · 02-04-2014, 06:23 AM

if you see no changes, ublock all the port ranges 120 - upwards....
read lpms from /dev/eth0, if none - set lpms to null in /etc/conf.d
then restart your system.
Should be fine.

Habitual · 02-04-2014, 08:11 AM

Quote:

Originally Posted by czezz

Why do you want me to block port 139 ?

I mistyped, I meant udp/123 (the listening ntp port) but if that's disabled then you only need to monitor further to determine if the activity has in fact subsided. Sorry about that.

czezz · 02-04-2014, 08:57 AM

Well... the activity has stopped in this sense that the eth0/internet link is not overloaded anymore.
I turned off NTP which was compromised and blocked ports.
netstat shows no unwanted connections - thats good.

However, iptraf and tcpdump shows a lot of traffic coming in to port 123 and coming out from port 80. Even though I blocked these ports (in and out going).
That is something I dont understand.

Habitual · 02-04-2014, 10:00 AM

Quote:

Originally Posted by czezz

However, iptraf and tcpdump shows a lot of traffic coming in to port 123 and coming out from port 80. Even though I blocked these ports (in and out going).
That is something I dont understand.

tcp or udp activity in those 2 programs?

czezz · 02-04-2014, 12:49 PM

udp - please have a look at my post #3

This is more detailed tcpdump output:

Code:

tcpdump -i eth0 -s0 -n -nn -N -p -tttt -vvv
2014-02-04 19:46:59.708028 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 37.187.56.104.22 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.763303 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 37.187.56.104.22 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.854033 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.860646 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.862112 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.867535 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.874687 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.876890 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.889456 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.904486 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:46:59.928135 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 88.238.101.118.5000 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8
        Reserved, Leap indicator:  (0), Stratum 0, poll 3s, precision 42
        Root Delay: 0.000000 [|ntp]
2014-02-04 19:47:00.544389 IP (tos 0x0, ttl 109, id 512, offset 0, flags [DF], proto: UDP (17), length: 36) 94.66.4.100.80 > 83.19.119.231.123: [udp sum ok] NTPv2, length 8

unSpawn · 02-04-2014, 03:15 PM

Quote:

Originally Posted by czezz

I turned off NTP which was compromised and blocked ports.

Do verify but it is unlikely your NTP daemon was compromised. Instead it was run without restrictions (for your LAN that's OK but you shouldn't run a public NTP service in the first place) and an old version to boot which allowed a CVE-2013-5211-style DoS attack to happen. Perps spoof the source address and ports so they're actually victims.

czezz · 02-05-2014, 02:51 PM

Thanks for reply.
I can see today that unwanted traffic is gone now.
This is my NTP config. Is there anything to improve ?

Code:

# cat /etc/ntp.conf
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

driftfile /etc/ntp/drift

restrict 0.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 1.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery
restrict 2.pool.ntp.org mask 255.255.255.255 nomodify notrap noquery

restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

unSpawn · 02-07-2014, 05:30 PM

Nothing I can see. Deny inbound requests to your LAN NTP service on your routers firewall?