LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 08-20-2019, 05:06 PM   #1
smashedcat
LQ Newbie
 
Registered: Aug 2019
Posts: 4

Rep: Reputation: Disabled
Question Unilateral Ping SSH/General Communication Issues


Oracle Linux(one 6.10 the other 7.6)

Let's call them servers A and B for simplicity.

Server B:
-Can ssh, ping, etc server A and everything seems like it's fine

Server A(communicating to server B)
-No ssh, no telnet, no nmap
-Can interact normally with other servers which all reside within the same network with no firewalls
-Can netcat over port 22

Server B interacts with all servers on the network normally. And Server A interacts with all other servers on the network with the exception of Server A.

A ping from Server A to Server B DOES return a result which takes(if it works, which most don't get a reply) 10-25 seconds, the time on the ping is very short however(under 2 ms).

Using tcpdump I see the pings from Server A to Server B, hit Server B and even the reply from Server B back to A, however it never reaches A. I thought this could be rc filtering, but it is set to off.

selinux, firewalld, IP tables are all disabled/empty.

Is there anything I could be missing here? Thanks for taking the time to read, if I can supply any details to make it easier, please let me know.
 
Old 08-20-2019, 05:19 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 21,948

Rep: Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811Reputation: 5811
Quote:
Originally Posted by smashedcat View Post
Oracle Linux(one 6.10 the other 7.6)
Let's call them servers A and B for simplicity.

Server B:
-Can ssh, ping, etc server A and everything seems like it's fine

Server A(communicating to server B)
-No ssh, no telnet, no nmap
-Can interact normally with other servers which all reside within the same network with no firewalls
-Can netcat over port 22

Server B interacts with all servers on the network normally. And Server A interacts with all other servers on the network with the exception of Server A.

A ping from Server A to Server B DOES return a result which takes(if it works, which most don't get a reply) 10-25 seconds, the time on the ping is very short however(under 2 ms). Using tcpdump I see the pings from Server A to Server B, hit Server B and even the reply from Server B back to A, however it never reaches A. I thought this could be rc filtering, but it is set to off.

selinux, firewalld, IP tables are all disabled/empty. Is there anything I could be missing here? Thanks for taking the time to read, if I can supply any details to make it easier, please let me know.
Are these virtual machines, or 'real' machines? Is serverA a new install, or existing and (if existing), have any updates taken place? Any networking changes between the boxes?
 
Old 08-20-2019, 06:01 PM   #3
Skaperen
Senior Member
 
Registered: May 2009
Location: WV, USA
Distribution: Xubuntu, Slackware, Amazon Linux
Posts: 1,903
Blog Entries: 21

Rep: Reputation: 126Reputation: 126
what does "No ssh, no telnet, no nmap" really mean? lack of the software or lack of response on these?
 
Old 08-20-2019, 06:12 PM   #4
smashedcat
LQ Newbie
 
Registered: Aug 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TB0ne View Post
Are these virtual machines, or 'real' machines? Is serverA a new install, or existing and (if existing), have any updates taken place? Any networking changes between the boxes?
Server A is virtual, Server B is physical.

Both were existing and working as expected, the only updates have been yum week updates to get current that I'm aware of. Unfortunately other people have access to these boxes, which leaves recent changes open ended.

No networking changes.
 
Old 08-20-2019, 06:13 PM   #5
smashedcat
LQ Newbie
 
Registered: Aug 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Skaperen View Post
what does "No ssh, no telnet, no nmap" really mean? lack of the software or lack of response on these?
I could have been more clear on that. In this case it means no response.

ssh times out as does telnet, nmap doesn't think Server B is even on(running from Server A).
 
Old 08-20-2019, 08:18 PM   #6
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,447

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Make sure you use IP addresses to exclude name resolution problems.

netcat gets connected to port 22. This means that the ssh client should get connected as well, but something prevents it from completing its protocol with the server. Did you try the ssh -v options (you can crank verbosity up to 4 v's, I think)?

telnet timing out doesn't shock me particularly. Not timing out would shock me

Since netcat gets connected, it's a mystery how nmap doesn't see server B. Perhaps you didn't use the right command line options for nmap.

I am a bit confused about the ping from A to B. You say that the ping takes several seconds, but the "time on the ping" is only 1-2 ms. What do you mean by that? You also say that the echo reply back to server A "never reaches A". How do you know this, and if it never reaches A, how can an echo request/reply take 1-2 ms rather than timing out?
When you trace the echo's, it would be interesting to see where the 10-25 seconds delay that you mention are spent.

Finally, how are the two servers connected to the network? Is server B a guest on Server A or somewhere else? Could there be a setting on B's host that causes trouble? Do you get different results when tracing traffic at the bridges or whatever other network interfaces that connect B to the network?

Last edited by berndbausch; 08-20-2019 at 08:20 PM. Reason: grammar
 
Old 08-20-2019, 10:10 PM   #7
smashedcat
LQ Newbie
 
Registered: Aug 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by berndbausch View Post
Make sure you use IP addresses to exclude name resolution problems.
I've tried both for thoroughness, but tend to stick with IPs.

Quote:
Originally Posted by berndbausch View Post
netcat gets connected to port 22. This means that the ssh client should get connected as well, but something prevents it from completing its protocol with the server. Did you try the ssh -v options (you can crank verbosity up to 4 v's, I think)?
Not sure I was aware of verbosity on ssh, I'll give that a go and report back.

Quote:
Originally Posted by berndbausch View Post
Since netcat gets connected, it's a mystery how nmap doesn't see server B. Perhaps you didn't use the right command line options for nmap.
I used -sS, I'll look more into that.

Quote:
Originally Posted by berndbausch View Post
I am a bit confused about the ping from A to B. You say that the ping takes several seconds, but the "time on the ping" is only 1-2 ms. What do you mean by that? You also say that the echo reply back to server A "never reaches A". How do you know this, and if it never reaches A, how can an echo request/reply take 1-2 ms rather than timing out?
I've setup tcpdump on the interface for both servers. For clarity, let's forget about that for just a moment though and say that I just do a normal ping from Server A. If I wait anywhere from a few seconds or maybe even a few minutes, I'll get a reply from that ping. In this case, let's say I waited 45 seconds and got a reply, the time on the ping is under 2ms(Random Example: Reply from 10.10.10.X: bytes=1500 time=2ms TTL=54). So I suppose, it could be that most of the pings are getting.. dropped,ignored, whatever the case may be and that ping does actually return in 2ms, but the first 5-50 pings never make it back.

On to the point of how I know. Tcpdump on both sides, I can see Server A sending out a ping, then Server B gets it and instantly replies to Server A, however on Server A I see no such reply ever making it.

So it is false for me to say the ping from Server A -> Server B never replies, it's just very rare.

Quote:
Originally Posted by berndbausch View Post
Finally, how are the two servers connected to the network? Is server B a guest on Server A or somewhere else? Could there be a setting on B's host that causes trouble? Do you get different results when tracing traffic at the bridges or whatever other network interfaces that connect B to the network?
Server A is a VM on a esxi host with permissions that match other VMs that are working, while Server B is a physical. There is one hop between the two, which is wide open, I'll need to dig around to find how to get any logging from it. But these should be grouped settings for Server A and Server B that match other workings connections, which makes me think that the Server B has something preventing connectivity, but I can't seem to understand how or even where to start looking based on the things I've ruled out.


While I don't know that it will help because we're troubleshooting the most basic things: Server A is a Cloudera Manager Host(Which includes 3 other hosts driving different services all grouped together with the same settings) and Server B is a Hadoop node(grouped with 10 others just like it with the same settings) and all previously working until heartbeats started failing in the service. Which led to realizations about the state of Server B.. or A.

Thanks for the insightful questions, I'll clarify anything I did a bad job of explaining in the morning.

Last edited by smashedcat; 08-20-2019 at 10:12 PM.
 
Old 08-20-2019, 10:58 PM   #8
permaroot
Member
 
Registered: Aug 2019
Location: Arden, NC
Distribution: Arch Linux
Posts: 129

Rep: Reputation: 47
Have you tried just sending one ping? Ping [ip address] -c 1 and seeing how often it fails? Or using trace route?

It sounds like it could be a routing problem? Maybe a routing loop that exceeds the TTL. How is your routing handled?

Pinging a non-used ip with nmap for me says “host seems down. If it’s up it’s blocking our pings.”

Last edited by permaroot; 08-20-2019 at 11:02 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] CUPS HPLIP communication failures under -current causing communication errors. bamunds Slackware 10 09-24-2016 06:02 PM
Strange Ping Issue - Can't ping localhost but can ping others on LAN code_slinger Linux - Networking 15 03-30-2015 02:39 PM
I cannot ping with command 'ping IP' address but can ping with 'ping IP -I eth0' sanketmlad Linux - Networking 2 07-15-2011 05:32 AM
Fault in KDE processes communication: Could not read network communication list Magnus Johansson MEPIS 0 03-30-2008 12:50 PM
Ping PING Ping uid0sd Linux - Networking 1 02-13-2007 06:50 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 01:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration