I have a strange problem with icmp redirects causing my redhat 9 box to cache invalid routes. The first problem has to do with a poorly designed network and could be fixed with a reconfiguration of the router but this is not an option at the moment and does not explain the real issue.
I have a monitoring application that pings all of my routers every 5 minutes. When a wan link goes down the router sends an icmp redirect causing the monitoring system to send the echo requests out an invalid route. This invalid route is then cached in the kernel routing table and stays indefinitely until I manually clear the cache. This means that even when the wan link comes back up my monitoring server reports it as down due to the invalid route.
To further complicate the issue a traceroute from the monitoring server shows the request taking the invalid route yet I am able to telnet to the remote system successfully. It appears as though the invalid route is cached and used for icmp traffic only.
Has anyone seen this before? Any suggestions for troubleshooting? Your help is greatly appreciated