Visit Jeremy's Blog.
Go Back > Forums > Linux Forums > Linux - Networking
User Name
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.


  Search this Thread
Old 01-15-2013, 08:26 AM   #1
LQ Newbie
Registered: Sep 2011
Posts: 2

Rep: Reputation: Disabled
Routing Problems w/ Oracle linux-Exadata (Solved)

Thought I'd share a routing problem (and a resolution) we had with Oracle Exadata. For those that aren't familar with it, the Exadata is an Oracle "managed" solution. Basically, a 1/2 rack of storage and linux servers optimized to run oracle database. Ours uses their unbreakable kernel.

After a software install, we were unable to connect to the db on this server. Connections are made to a vip which point to a bonded interface on the server. At first, none of the vip addresses worked but the direct ip did. After a reboot of the servers, even that stopped working.

Tcpdump showed traffic reaching the interface but no response. No denial, just... dropped. After more troubleshooting we discovered a new error in the message log:

messages.1:Jan 9 16:59:14 exadb01 kernel: martian source from, on dev bondeth0

Turns out that "martian source" is a cute way of saying the packet is unrouteable. Interesting. SO, the packets from the .90 server are hitting the primary ip (.24) and being considered unrouteable. Why? The default routes look good.

# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface * U 0 0 0 eth0 * U 0 0 0 bondeth0 * U 0 0 0 eth2
default UG 0 0 0 bondeth0

The answer is that Oracle enables policy based routing (pbr) by default on their kernels. Even if you don't make use of it, it must be properly configured. PBR requires a rule-* and a route-* file per interface. When oracle set up the server they did that. However, the interfaces were changed and added to over time. No one was aware of the additional configuration requirements.

The key point here is that it 'may' work for awhile without the rule/route files but you'll never know when it will fail. And it took us 3 days and the 'right' Oracle tech to fix. Once we added the files and restarted networking, it worked perfectly.

Here are some example files. Note real interfaces start with table 220 and bonded ones with 210 by convention. The files basically specify that traffic go back out whatever interface it came in on.
For more info, see

Good luck and I hope this helps someone.

# cat rule-bondeth0
from table 210
to table 210

# cat route-bondeth0 dev bondeth0 table 210
default via dev bondeth0 table 210

# cat route-eth0 dev eth0 table 220
default via dev eth0 table 220

# cat rule-eth0
from table 220
to table 220
Old 01-15-2013, 08:52 AM   #2
LQ Guru
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 20,004

Rep: Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752Reputation: 4752
Outstanding write-up, thank you for posting it.
Old 01-15-2013, 08:57 AM   #3
LQ Newbie
Registered: Sep 2011
Posts: 2

Original Poster
Rep: Reputation: Disabled
Originally Posted by TB0ne View Post
Outstanding write-up, thank you for posting it.
No problem. It was one of those issues, that search turned up many similar questions and not many answers.
Old 02-20-2013, 12:28 PM   #4
LQ Newbie
Registered: Jan 2013
Posts: 1

Rep: Reputation: Disabled
Thank you! We encountered this when we upgraded to I stumbled across your post, and saved it, hours before we found the "martian source" messages so once we did we had a ready fix (as opposed to Oracle which still hasn't given us an RCA over a month later). My search was for “oracle Exadata vip troubleshooting”, how lucky was that?

Our symptoms: Everything upgraded just fine, crsctl reported that everything was up and running, but we couldn't connect to the databases from any other servers. tnspings from the other servers would either time out with TNS-12535 or ORA-12170. pings to the IPs and the scan listener always got “100% packet loss”.

This Exadata had been installed and configured by Oracle ACS, no changes had been made to the configuration after the initial install. We’d done some minor patching a year ago, no problems, and have rebooted everything a couple of times, again no problems. We started out on so based on your post, and Marc Fielding’s blog, I would guess it was a configuration problem. It’s interesting that it took us 1.5 years to encounter the issue.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] [SOLVED]Force routing of packets coming from the same subnet genese Linux - Networking 2 12-29-2012 11:31 AM
[SOLVED] Linux Mint Gloria and Realtek RTL8168C(P)/8111C(P) Wireless Problems... Is It Solved? MexDeath Linux Mint 5 09-06-2009 06:42 PM
linux routing problems gabell27 Linux - Networking 2 04-26-2004 02:38 PM > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 03:26 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration