UDP Packet loss

manikandanc · 07-30-2013, 05:25 AM

Hi,
I am trying to send an UDP datagram to a remote server. But, I am not getting any response. When I used wireshark, I observed that the packets are leaving the machine. But not received at the other end.

If I send the same data from some other machines, I get proper responses. So, I am concluding that, something is wrong at the sender side that makes the packets to be dropped/ignored or whatever happens.

This is how I send my data:
echo -en <MY DATA> | netcat -u <Remote Host> <Port>

Among the senders, I could not find any pattern. ie. This issue is observed on both SLES/RHEL machines and they are on the same subnet. Also, firewall is disabled on all the machines involved in this communication.

Any ideas on what might be causing the datagram to be dropped/ignored?

zhjim · 07-30-2013, 06:33 AM

do a traceroute of the package and see where it hangs.

manikandanc · 07-30-2013, 08:03 AM

Quote:

Originally Posted by zhjim

do a traceroute of the package and see where it hangs.

I was under ths assumption that traceroute is only for hosts and not ports. (googling though) One more thing is, I could ping the UDP Server from all the machines.

zhjim · 07-30-2013, 09:23 AM

traceroute as far as I know sends a TCP packet on a fixed port. Not sure which one. At least this is true for the windows version. The linux version can be given options which port to use or which protocol. Check out the man page.
Just that as you can ping the host traceroute might be of less value. But try it non the less.

Did you run wireshark on the remote host as well? If the package is not seen there then the network must block it on its way. If seen maybe /etc/hosts.deny has some entries? iptables is off as you stated.

manikandanc · 07-30-2013, 10:46 AM

Quote:

Originally Posted by zhjim

Did you run wireshark on the remote host as well? If the package is not seen there then the network must block it on its way. If seen maybe /etc/hosts.deny has some entries? iptables is off as you stated.

This is what is happening..

The packet is not seen on the other end. The other end is a Windows machine. (Even I tried running an UDP server on a linux machine. That too didn't receive any packets). Somehow, I think that the packet is constructed incorrectly or is being ignored.

I compared the packets (between the working ones and the non-working ones). Could not find any significant differences.

zhjim · 07-31-2013, 02:25 AM

Okay so we have a windows machine on the remote site and a linux machine on the local. What distribution is the linux one? Anything special? Anything special about the hardware of the local machine?

ping from local to remote works. Right? What about the other way around? From remote to local?
Can you send udp packets to another linux machine that can talk to the remote machine?

Please just for certainty give the output of

Code:

iptables-save

manikandanc · 07-31-2013, 05:39 AM

Quote:

Originally Posted by zhjim

Okay so we have a windows machine on the remote site and a linux machine on the local. What distribution is the linux one?

The sender of the packet is SLES and RHEL. (Have observed in both the flavours) Also, tried writing a small java app (UDP Server) on a linux machine. The packets didn't reach this machine as well.

Quote:

Originally Posted by zhjim

Anything special? Anything special about the hardware of the local machine?

All the machines are running on VMware. Had another VM running on the same machine where the packets didn't reach the UDP Server. But, that could reach the UDP Server.

After these two observations, I decided that some network related setting in the OS is the culprit.

Quote:

Originally Posted by zhjim

ping from local to remote works. Right? What about the other way around? From remote to local?

Ping works..

Quote:

Originally Posted by zhjim

Can you send udp packets to another linux machine that can talk to the remote machine?

The java app (UDP Server) that I wrote, was running on a linux machine that could reach the windows machine.

Quote:

Originally Posted by zhjim

Please just for certainty give the output of

Code:

iptables-save

Disabled firewall. So, it returned empty

This is the set of commands I executed and the observations...

Code:

# /etc/init.d/iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
# iptables-save 
# echo hello | nc -u <My IP> <Port>

zhjim · 07-31-2013, 07:50 AM

Quote:

Originally Posted by manikandanc

Had another VM running on the same machine where the packets didn't reach the UDP Server. But, that could reach the UDP Server.

What do you mean by this? ICMP (ping) is working but UDP is not?

I would definitely check the VMWare host for anything unusual. Maybe somethings blocking there. Cause if you can ping the machine we know that layer 2 and layer 3 work. So the problem must be somewhere around layer 4 of OSI. Don't know if you can give any configuration of the vmware host but that might be useful.

manikandanc · 07-31-2013, 09:24 AM

Quote:

Originally Posted by zhjim

What do you mean by this? ICMP (ping) is working but UDP is not?

I mean, I had two VMs on a host. One of them could reach the UDP server and the server responded back. But on the other VM, I am facing this issue. Both are of SLES. Currently, this machine have a hardware problem. Waiting to be repaired.

Quote:

Originally Posted by zhjim

I would definitely check the VMWare host for anything unusual. Maybe somethings blocking there. Cause if you can ping the machine we know that layer 2 and layer 3 work. So the problem must be somewhere around layer 4 of OSI. Don't know if you can give any configuration of the vmware host but that might be useful.

If you can let me know the kind of information you are looking for, I'll try to get them.

BTW. Thanks for your responses

manikandanc · 08-02-2013, 06:29 AM

I tried logging the udp packets that are going out...

Could see these messages in /var/log/messages:

Code:

Aug  2 21:54:01 myhost kernel: IPTables-SentIN= OUT=eth0 SRC=<src ip> DST=<dest ip> LEN=29 TOS=0x00 PREC=0x00 TTL=64 ID=35483 DF PROTO=UDP SPT=35662 DPT=1434 LEN=9 
Aug  2 21:54:01 myhost kernel: IPTables-SentIN= OUT=eth0 SRC=<src ip> DST=<dest ip> LEN=29 TOS=0x00 PREC=0x00 TTL=64 ID=35483 DF PROTO=UDP SPT=35662 DPT=1434 LEN=9

And this is what I have in my firewall as of now...

Code:

*filter
:INPUT ACCEPT [79393:113859316]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [22905:1291919]
:LOGGING - [0:0]
:LOGGING_IN - [0:0]
-A INPUT -j LOGGING_IN 
-A OUTPUT -j LOGGING 
-A LOGGING -p udp -j LOG --log-prefix "IPTables-Sent" 
-A LOGGING -p udp -j LOG --log-prefix "IPTables-Sent" 
-A LOGGING_IN -p udp -j LOG --log-prefix "IPTables-Recv" 
COMMIT
# Completed on Fri Aug  2 22:04:09 2013

nyshtyak · 08-02-2013, 08:29 AM

Hm. Cannot find if you answered obvious question: does your UDP arrive to recipient machine?

manikandanc · 08-02-2013, 09:34 AM

Quote:

Originally Posted by nyshtyak

Hm. Cannot find if you answered obvious question: does your UDP arrive to recipient machine?

Nope

The packet seems to leave the source machine but not received at the other end.

Lantzvillian · 08-02-2013, 11:13 AM

I quickly skimmed this thread, but I thought I saw that you are using VMs.. have you tried a physical machine and tested to see if that is being blocked? I have had VMs do very funny things at times with their virtual networks.

manikandanc · 08-02-2013, 12:43 PM

Quote:

Originally Posted by Lantzvillian

I quickly skimmed this thread, but I thought I saw that you are using VMs.. have you tried a physical machine and tested to see if that is being blocked? I have had VMs do very funny things at times with their virtual networks.

Yes.. Tried from a couple of physical machines but was not able to reproduce the issue. Also, there was another VM on the same host that could send UDP packets to the remote machine. I am not even sure whether I'll be able to replicate this issue in physical machines

zhjim · 08-05-2013, 09:24 AM

If you have sane ip's for all the involved virtual machines and no iptables rules that block or deny traffic its all VMWares fault. I'd definitely check the network options of the guest machines. Like where they are attached and how and alike.