LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 01-15-2013, 08:26 AM   #1
treadwm
LQ Newbie
 
Registered: Sep 2011
Posts: 2

Rep: Reputation: Disabled
Routing Problems w/ Oracle linux-Exadata (Solved)


Thought I'd share a routing problem (and a resolution) we had with Oracle Exadata. For those that aren't familar with it, the Exadata is an Oracle "managed" solution. Basically, a 1/2 rack of storage and linux servers optimized to run oracle database. Ours uses their unbreakable kernel.

After a software install, we were unable to connect to the db on this server. Connections are made to a vip which point to a bonded interface on the server. At first, none of the vip addresses worked but the direct ip did. After a reboot of the servers, even that stopped working.

Tcpdump showed traffic reaching the interface but no response. No denial, just... dropped. After more troubleshooting we discovered a new error in the message log:

messages.1:Jan 9 16:59:14 exadb01 kernel: martian source 10.22.102.24 from 172.22.22.90, on dev bondeth0

Turns out that "martian source" is a cute way of saying the packet is unrouteable. Interesting. SO, the packets from the .90 server are hitting the primary ip (.24) and being considered unrouteable. Why? The default routes look good.

# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.22.100.0 * 255.255.254.0 U 0 0 0 eth0
10.22.102.0 * 255.255.254.0 U 0 0 0 bondeth0
172.22.10.0 * 255.255.254.0 U 0 0 0 eth2
default 10.22.102.1 0.0.0.0 UG 0 0 0 bondeth0

The answer is that Oracle enables policy based routing (pbr) by default on their kernels. Even if you don't make use of it, it must be properly configured. PBR requires a rule-* and a route-* file per interface. When oracle set up the server they did that. However, the interfaces were changed and added to over time. No one was aware of the additional configuration requirements.

The key point here is that it 'may' work for awhile without the rule/route files but you'll never know when it will fail. And it took us 3 days and the 'right' Oracle tech to fix. Once we added the files and restarted networking, it worked perfectly.

Here are some example files. Note real interfaces start with table 220 and bonded ones with 210 by convention. The files basically specify that traffic go back out whatever interface it came in on.
For more info, see http://www.pythian.com/news/36747/ad...olicy-routing/

Good luck and I hope this helps someone.

# cat rule-bondeth0
from 10.22.102.0/23 table 210
to 10.22.102.0/23 table 210

# cat route-bondeth0
10.22.102.0/23 dev bondeth0 table 210
default via 10.22.102.1 dev bondeth0 table 210

# cat route-eth0
10.22.100.0/23 dev eth0 table 220
default via 10.22.100.1 dev eth0 table 220

# cat rule-eth0
from 10.22.100.33 table 220
to 10.22.100.33 table 220
 
Old 01-15-2013, 08:52 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,823

Rep: Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636Reputation: 2636
Outstanding write-up, thank you for posting it.
 
Old 01-15-2013, 08:57 AM   #3
treadwm
LQ Newbie
 
Registered: Sep 2011
Posts: 2

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TB0ne View Post
Outstanding write-up, thank you for posting it.
No problem. It was one of those issues, that search turned up many similar questions and not many answers.
 
Old 02-20-2013, 12:28 PM   #4
LSeley
LQ Newbie
 
Registered: Jan 2013
Posts: 1

Rep: Reputation: Disabled
Thank you! We encountered this when we upgraded to 11.2.3.2.0/11.2.0.3. I stumbled across your post, and saved it, hours before we found the "martian source" messages so once we did we had a ready fix (as opposed to Oracle which still hasn't given us an RCA over a month later). My search was for “oracle Exadata vip troubleshooting”, how lucky was that?

Our symptoms: Everything upgraded just fine, crsctl reported that everything was up and running, but we couldn't connect to the databases from any other servers. tnspings from the other servers would either time out with TNS-12535 or ORA-12170. pings to the IPs and the scan listener always got “100% packet loss”.

This Exadata had been installed and configured by Oracle ACS, no changes had been made to the configuration after the initial install. We’d done some minor patching a year ago, no problems, and have rebooted everything a couple of times, again no problems. We started out on 11.2.2.3.2 so based on your post, and Marc Fielding’s blog, I would guess it was a configuration problem. It’s interesting that it took us 1.5 years to encounter the issue.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] [SOLVED]Force routing of packets coming from the same subnet genese Linux - Networking 2 12-29-2012 11:31 AM
[SOLVED] Linux Mint Gloria and Realtek RTL8168C(P)/8111C(P) Wireless Problems... Is It Solved? MexDeath Linux Mint 5 09-06-2009 06:42 PM
linux routing problems gabell27 Linux - Networking 2 04-26-2004 02:38 PM


All times are GMT -5. The time now is 02:48 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration