Doing a traceroute, I can see for example that my provider is using private IP's inside of their networks then the traffic hops onto a gateway and flows to/from the internet.

Here is how it hops.

# traceroute 4.2.2.2
traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets
1 192.168.1.1 (192.168.1.1) 0.418 ms 0.428 ms 0.470 ms
2 xx-xx-251-177.cpe.my-router (xx.xx.251.177) 1.131 ms 1.990 ms 2.125 ms (Public IP)
3 10.105.128.1 (10.105.128.1) 10.253 ms 12.318 ms 13.304 ms
4 192.168.41.65 (192.168.41.65) 13.520 ms 13.769 ms 13.729 ms
5 192.168.103.21 (192.168.103.21) 16.804 ms 24.680 ms 23.337 ms
6 xx-xx-xx-xx.to.Level3.net (4.28.80.133) 19.387 ms
**SNIP**
11 x.xx.xx.to.Level3.net (4.2.2.2) 43.663 ms 43.656 ms 49.026 ms

So, I can see the following.

1 my private gateway (or firewall)
2 then my public IP (on the router)
3,4,5 back to private IP's over my providers network for several hops and this is where I am making assumptions. For example, perhaps hop 3 is the amplifier on the local pole, then hop 4 is something else along the way, then finally hop 5 to my providers local NOC.
From there,
6 finally hits the internet and on to their provider which is obviously level3.

I didn't know you could flow public IP data over a private network or maybe I'm not understanding something. Are the private IP's being encapsulated or something then dropped off to the edge router?

Anyhow, then, when trying to traceroute a DNS server inside of my providers network, I get the following.

# traceroute xx.xx.x.50 (public IP)
traceroute to 24.116.2.50 (24.116.2.50), 30 hops max, 40 byte packets
1 192.168.1.1 (192.168.1.1) 0.590 ms 0.588 ms 0.626 ms
2 xx-xx-251-177.cpe.my-router (xx.xx.251.177) 1.261 ms 2.114 ms 2.484 ms (Public IP)
3 10.105.128.1 (10.105.128.1) 8.414 ms 9.267 ms 12.522 ms
4 192.168.41.65 (192.168.41.65) 13.508 ms 14.452 ms 14.664 ms
5 192.168.103.21 (192.168.103.21) 16.190 ms 17.014 ms 17.134 ms
6 * * *
Times out for good.

Am I diagnosing this correctly? I'm assuming they have disabled ICMP on devices starting at hop6.
Is there any other way to get more information using traceroute or something else?

Initially, as far as tools go, I find mtr (Matt's traceroute / My traceroute) and lft (layer four trace) very useful. Especially lft as this gets through a lot more firewalls and routers. This is because it will use TCP not UDP so so just send out a packet supposedly destined for a web server on port 80, so looks much more normal than a conventional traceroute.

Public IP's can route over private networks, it's all just numbers really, there's no technical difference at all. The IP addresses do not persist across the neighbouring devices, so there's no impact. The IP's you're seeing there are not necessarily the actual addresses involved in the routing operations though, but the address that each device arbitrarily decides to place in the ICMP TTL Expired packets, so in theory can be any IP at all.

I've been reading up on tracerouting using udp, tcp and icmp. All seem to have their limitations if devices are blocking the required ports or packet.

I also read that there are at least sometimes, ways around this. All of our devices for example do not allow icmp from public side but allow on private side but when we have problems, I cannot determine where the problems are because I don't always control all of the devices that we use. Therefore, I've been trying to find a way of being able to see the path from point A to point B using alternative tools.

You mention using port 80? Had not thought of that so I tried it but it gives me the exact same * results. I'll keep digging on your input however so thank you for that.

Does anyone know of any sort of application which runs on Centos which could simply respond to port 80.
Not a web server or anything heavy like that, just a simple, low resource tool which does nothing but respond in some very rudimentary manner to a standard port.

If so, then I could just install that on the various web servers at different locations and get a point A to point B test at least.

netcat ?

"I didn't know you could flow public IP data over a private network or maybe I'm not understanding something. Are the private IP's being encapsulated or something then dropped off to the edge router?"

The private IP's are being used as transit links internaly to your ISP. Traffic is not intended to be sourced from these transit links, they are purely for passing packets around, so do not need public addresses.

You are seeing them because traceroute is forcing the routers to send ICMP error messages so in this instance they are sourcing traffic. Because that traffic is never leaving your ISPs network they can find their way to you. ICMP messages from these transit links would be dropped if they found their way our of you ISP network to the internet.

Yes, I understand the concept of private IPs and public IPs but that wasn't really what I was asking about
Thank you however for your input.

"I didn't know you could flow public IP data over a private network"

So you didn't mean this then?

Sure I did but that wasn't my question. I answered that part in the same sentence .
Yes, they are encapsulated but then the main question remains.

>I didn't know you could flow public IP data over a private network or maybe I'm not understanding something.
>Are the private IP's being encapsulated or something then dropped off to the edge router?

If port 80 is what you want to "check" for up|down" indicator, try
or something like:
should also work.

It's actually the path, or hops that I need, in order to know where the connection stops to know who to contact in what region for example. Traceroute would be perfect but since too many providers block that traffic, I need another way of accomplishing this.

I brought up standard port 80 because I had been reading about folks using port 80 but once I read more about it and tested for myself, I found that this doesn't work either to show me where something might be down since it's only an end to end test.

traceroute uses UDP port 33434 by default, but you can specify both the source and destination port number using command line switches.

The problem is that it relies on ICMP "TTL Expired in Transit" messages being returned by the various hosts along the route. If these messages are blocked (which is certainly not a Best Practice), there's no way for traceroute or any other application to know the identity of each host along the way.

You can change application, protocol and port numbers all you like; if the "unreachable" messages are filtered, there's just no way to know where the packet was dropped.

>You can change application, protocol and port numbers all you like; if the "unreachable" messages are filtered,
>there's just no way to know where the packet was dropped.[/QUOTE]

Right, totally true. Which is why I posted to see if there might be some thoughts from others who have had to deal with this.
Trying to think outside the box, I've come up with a lot of dead ends. If most and more networks turn icmp and related tcp/udp ports off, how the heck is anyone going to troubleshoot.

*shrug* I don't know why people keep clinging on to ICMP and UDP for TCP services as there's tcptraceroute...

I did try that as well. I did mention having tried all of the icmp/tcp/udp based standard tools.
No need for a greater than thou attitude you know