Hi Everyone,

This is the 1st time i have posted here and i think this is the best section. Also i am a bit of a Linux noob, but have learned a great deal so far

What i am trying to do should not be that difficult but i am now totally fustrated and need someone to jump in and help me.

Background:
I have a sniffer (e.g Wireshark, Frontline Ethertest 10) and used it to capture some packets on another network. Through the sniffer application I have exported the Hex to create my own Hex file so that i have complete packets (1 per line). This Hex file is then to be loaded into my custom application so that i can replay those packets to test my telephone app.

Quick Diagram of Environment:
(Play Packets on LAN) ---> Linux Router ---> WAN or ppp0 ---> My Tel App


Goal:
I need to replay these packets to go out my WAN, so that i can develop my application. I must do it this way, because there is just no other way to do it.

Problem:
I have been down many paths and i'm now looking at the Linux kernel (i am using Ver 2.4). All i am trying to do is Route from LAN to WAN (or ppp0). Locally generated packets can go through and i can control their destination via IP tables and Routing, but when i throw my Hex file packets at the Linux router they get into the LAN, but i cannot see if go any further???? (Noob!!)

Extra :
I have learned about Linux Socket Filters, Tcpdump, Iptables, Promiscuous mode..... but i dont know what i dont know, so i am asking if anyone can help me with the kernel stuff incase i need to tweak some settings, filters or whatevers.

Thanks,

Rich.

Okay, few issues. If the packets (assuming ethernet) you grabbed include the ethernet header, then they can only ever be sent to the MAC of the destination host specified in the ethernet header. That is, after all, how they get sent to the specified host, unless they are ethernet broadcast packets.

If you are sending ill-formed packets on a WAN, normally a public network under the authority of someone higher up the networking food chain, you are likely to get some unwanted attention.

Routing of packets will, by definition, alter the packets, since they must become destined for a new host. It makes a lot more sense to generate packets in cooperation with the network stack, rather than trying to end-run around it. Many of the packets you capture probably only make any sense whatsoever in the context of a 'conversation' between two or more hosts, and simply barfing them out without the coordination of the other hosts will just do more harm than good.

If all of this still hasn't deterred you (and I hope it has, especially if you're on a network near me), then you have roughly three ways to coerce a kernel to put packets on the wire without using the TCP stack. You can create iptables/netfilter rules to transform &/or forward incoming packets. You can use the Raw Sockets protocol to send arbitrary ethernet datagrams (requires root privileges). You can write a kernel driver to receive data from userspace applications and put it on the wire.

So, why is it that there is just no other way to do what you think you want? If you need to exercise a device of some sort, you should be able to generate the appropriate data and send it using conventional methods.

--- rod.

Hi Rod,

Thanks for you reply, you seem like you much more than i do. And im sure that i am not on a network near you .

You are correct to assume ethernet and this is the only way to do what i want because i need to be able my test app over a WAN (VPN Connection), and dont have the hardware here to create new packets. To get where i am now with my app, i was testing by using a hub between my Tel app and my app that plays raw packets (sender app). But, i failed to see that when i replaced my non-intelligent hub with my intelligent network (Linux router), the packets failed to go anywhere!

This is when i started digging into what was wrong and how i can fix my sender app...

I understand what you said about the MAC addresses and that the packets go through changes in their header to get routed to their destination. And i would like to cooperate with the network stack, but i dont have the hardware to send the packets i need, so i am stuck with trying to play the captured packets locally. If i fix this, then i will still have my local test environment. Although i am stuck, this is a challenge and i would like to learn my way through to get over this and use the same tool to test, but i dont know enough to troubleshoot. So im just following my own ideas and finding pieces of info to research and its taking more time than i thought . But this is why i am looking for some help.

In my latest attempts to solve this, i have tried manually manipulating the packet header and verified the packets by running it through a sniffer. This pointed out some TCP and IP checksum errors and actually showed what it expected the value to be . I have also made sure on some tests, that the Destiaton MAC address was of the Linux Router, but the IP was not on the local network. Which i figured should cause the packet to use the default route and go out the gateway, but this did not work either.

The latest error that i found is from Wireshark, and it showed a TCP Out-of-Order segment and i am researching to see if this would stop the packets going through? Im assuming its a sequence thing, but not sure right now what i would need to do to number them properly, if that was all that was needed?

I believe that i am very close to understanding why the packets will not go through. When i get there, i can then figure out how to fix it, but i would *really* appreciate help to understand my problem.

BTW - Great link! A couple weeks ago i wrote a quick LSF and may be able to use this link to add to that .

Thanks!

Rich.

If you have a host capable of generating ethernet frames, then you have the hardware to create packets. What I will conclude you do not have is the means to create the appropriate payload data that goes into those packets. You seem to be seeing the effect to which I referred earlier of packets that are not generated as part of the TCP connection management process ('Out-of-Order', etc.). What you have done is record one side of a conversation, and then replayed it without the context of a corresponding peer, which is nonsensical.
Without having any details about your real intention, my suggestion is that you try to extract the payload data from your recorded session, and use conventional TCP sockets to replay those, after having built up the TCP connection in the usual way.
If you are using TCP, and your application is sensitive the differences between the nature of a LAN and a WAN configuration, then I would raise the question of whether TCP is the appropriate protocol to be using.
--- rod.

Hi Rod,

Sorry for the delay. I read what you said and was troubleshooting more, and i can say that i have made some progress

Before i comment on the progress.... i understand what you said about having half of the conversation between the 2 machines. This at any networking level is non-sensical because there needs to be control, such as acknowledgement, which is part of the conversation i do not have to play (in the reverse direction). However, this does not seem to mater in this instace, because the receiving end for those packets will always be a Sniffer. And this to the best of my knowledge will never reject packets and just take what ever is passing by on the network. The purpose of this sniffer is to decode the packets(custom decoder) and insert them in text form into a local Database (also custom code). I like to think about the sniffer simply as a Decoder.

For my progress.... after toying with the packet headers i was able to prove that i was able to successfully play specific packets through the WAN of my Linux Router. That is, only packets that had the:
- Destination MAC of the Router.
- Source MAC of the sending machine.
- Destination IP that was not on same subnet as the Router.
- Source IP of the sending machine.
- TCP and IP checksums correct.
I was also able to prove that any packet was able to get to the Linux Router, but did not manage to flow out of the WAN.

So i am now at the point where i need to understand the WAN side of the Router. I suspect some socket filter, but i dont know how to find or verify one is on the WAN. I have not tried the VPN yet, because i will add another variable and i want to prove that i can get all packets going through my WAN 1st.

If i create some custom code (e.g a Linux Socket Filter) to send the packet to the WAN via sockets. Do you think that the packet still get blocked?

Thank you very much for your comments so far. Just being able to bounce ideas with someone has helped very much

BTW i have made sure im not kicking this junk out to internet .

Rich.

If your Linux Router is based on netfilter rules (uses iptables), then it is probably a stateful router/firewall. This means that it probably understands the give and take of IP traffic that constitutes a connection, and rejects the likes of out-of-order or other non-conformant traffic. How was your router configured? I assume it is not of your own design, because if it was, you would already understand all of this. If you are designing some sort of system that must operate across all router configurations, then this will be a very large obstacle. I'm afraid I could not guide you on the likes of that.
If you have the option of specifying some router behavior, then I would try to figure out how to remove any rules that use 'ip_conntrack' (at least) module. These can can be identified at least in part by iptables rules that contain the 'state' parameter. These would be likely causes for dropping packets that do not make sense as part of a valid IP connection.
It seems probable that there will be other networking machinery across which your traffic must pass, and of course these will be out of your control or scrutiny.
I cannot keep from thinking that you are trying to do something that all of the forces of the universe are conspiring against, and that there must be some better, easier, and more conventional way to accomplish it.

--- rod.

Hi Rod,

I am all too familiar with iptables . With that i mean, i now know how to use it to create the rules in the different tables and chains. You are correct that the router is not of my own design, but i have downloaded the source code and tried looking at that to try and learn more, but have nothing new yet. But as for the design, as you know the rest, i think its a good idea to more clearly state what i am trying to do with this, because i have focused on my issue rather than the design itself.

Router Hardware: WRT54GL V1.1

Background:
There are 2 sites that are separated by WAN, our site and the client site. Each site will have a WRT54GL router that has 3rd party firmware on it (Open-WRT or DD-WRT) because it was planned to provide a VPN layer to extend the sniffer for our in-house Product named as "LS". At the Customer Site we can already use a HUB as a network Tap to provide a data stream for ourselves without interfering with their traffic. In-fact our old design was with the Sniffer + LS being connected to the HUB at the Customer site, but now we need to extend this design to reach our site over the WAN.

New Design
---------------

Client PCs
|
|
HUB ------> (Customer Site) WRT54GL Router (as VPN Client) ------> VPN over WAN ------> (Our Site) WRT54GL (as VPN Server) ------> LS with DB + Sniffer
|
|
Server

We have already covered that fact that the packets that i will be receiving via the HUB will not be addressed to the router (via MAC or IP). I have tried so far with both Open-WRT and DD-WRT firmwares and had little success. I can still only get the packets into the LAN area of the Router, but they do not pass through the WAN.

So to me, this still seems that it is more of a configuration problem that is at a linux/kernel level.

I also investigated what you said about needing to pass other routers on the internet, which was a very good point! But, i figured that if i am using a VPN, the overlaying VPN software would be controlling this via software at both ends, no? What i think i am trying to say is, that the VPN would circumvent the 'other' internet machinery because of the VPN. Do you agree?

Rod, thus far you have been my best resource by a long way. I have tried posting on other forums, but got little or nothing back.

Thank you again,

Rich.

Not too sure what you mean by 'the LAN area of the Router'. I assume the router has two ethernet ports: one on the LAN side and one on the WAN side. Are you saying that traffic from your customer's PC hosts is visible at the LAN-side port? If so, then using iptables rules to translate the IP address(es) to something that will end up at your site should do the trick. You will need to characterize the traffic such that you can formulate rules that match the packets. I don't see how the router will recognize that traffic not destined for it at the MAC level is to be captured and sent elsewhere. Routers don't typically use promiscuous mode to look at traffic.

To produce packets at your own site that are exactly the same as the ones produced at the customer site, you will have to use complementary rules at your own site router. The trick will be to coerce the second router to route IP traffic to a host whose MAC does not match that of the original destination host. Will the traffic of interest be sent to only one, or more than one MAC level targets? Since IP traffic gets wrapped with link-layer addressing, the IP stack will use MACs found with ARP for sending to IP hosts. This might make it easier to use MAC-level rules, especially if the original traffic is not IP oriented.

You are trying to make a router into something it really isn't. Since you already have your sniffer application, what about the possibility of running it on the router(s)?

I think that you need to analyze what elements of the traffic are modified by each of the components along its route. Since each modification is performed for the purpose of moving the traffic to the appropriate next 'box', you need to understand not just what transformation will be made along the way, but how you will be ale to undo that transformation at the destination, in order to re-transmit the data as it was captured.

I agree that using the VPN to wrap the traffic passing across the WAN should serve to insulate it from mangling by the WAN. That strategy seems correct to me. SSH tunneling may provide an alternate form of encapsulation.

--- rod.

Hi Rod,

Yes, you are correct, the Router does have more than one NIC and two are used for the LAN and WAN (They each have a different MAC Address). And yes, the packets do enter the the first NIC to pass into the router via its LAN (I called this the LAN area). And yes again , i too thought that iptables would have done the trick!!! The packets that i am trying to route will be either sending to, or receiving from port 5678, so this makes it easier to build iptables rules. However, i tried everything i knew for iptables and could not get it working.

When the router is flashed, its default settings are already set to have all of its LAN ports configured internally as a Virtual Lan 0 (vlan0). The WAN port is configured as a vlan1. I dont confess to understand it all right now, but it seems to be the only way to have it configured. There is also bridging too, but i believe that this is to connect the LAN and the Wireless LAN together. Each of the LAN ports are not completely isolated either, because they can talk to each other.

You bring up another good point!! About what the 2nd router will do to handle the inbound packets via the WAN if/when i can get the packets past the WAN of the 1st router. It would be great if i could set promiscuous mode on the WANs of both routers, but i have already tried something similar and it is more difficult than normal (e.g ifconfig vlan1 promisc). For the MAC level targets, the traffic of interest will only be flowing between 2 hosts.

We would love to run everything on the router, it does make perfect sense, but that is a long way off and not compatible right now. The Router is Linux and our software is developed for Windows.

It sounds like you are talking about some kind of network/protocol analyser. This too is a good idea, but i dont have one to use, unless i can find a software one???

It definately sounds like the VPN layer is the way to go, but SSH could be another idea. I would need to read into it more.

Thanks,

Rich.

Well, don't forget that just because a packet is visible to a host it will be received by the host. By 'received' I mean that it will make it past the lowest level driver, possibly even the hardware itself. Each packet on the LAN must, by definition, have a destination MAC, and this is the only way a packet can get received by the ethernet card (ignoring promiscuous mode and broadcasts for the moment). My strategy would be to use some iptables rules that simply log the presence/detection of packets of interest, just to confirm that the traffic you want to forward is actually present.

If the traffic is there, and is IP traffic, then forwarding it across the VLAN should be possible. I'm sorry I don't know exactly how to do that in a VPN scenario, especially when the VPN seem to be contrived from rules in the router. If it was a standard router, it would be as simple as a source NAT or MASQUERADE rule. Maybe the VLAN is just getting in the way, rather than helping. If the subject of this thread is it's only purpose, then in my opinion, it is not needed.

Reading between the lines a little, it sounds like you are expecting the router to handle all packets that are visible on its LAN input, whereas it will really only receive traffic specifically destined for it. Sorry if I'm mistaken about what you're expecting; just want to make sure we are talking apple/apples.

--- rod.

Hi Rod,

Agreed, there is more to transmitting a packet than just throwing it at a destination, there needs to be a well formed header that satisfies the MAC and IP layers.

That is a good point about iptables, and it is the one method i used to detect the presence of the packets during my testing so far. What i did was create some rules that didnt alter the packet, but they got triggered when the packet passed through. Otherwise it is difficult to know if the packet actually made it. My other way was with Tcpdump, but i had 1st figure out how to install Tcpdump on the Router, but when i did i could confirm that the packets were coming in.

The traffic that i saw was IP and i seem to remember that i could apply iptables rules on the LAN side ok, but they absolutely did not work when i used the rules to DNAT the packets out the WAN. If i could get the packets out of the WAN, then routing the packets over the VPN looks very easy, because the link created with the VPN is "ppp0" and this is an Interface.

The VLAN piece is adding confusion for me, but i think this is necessary and does not impact what i am trying to do. I have tried to remove the VLAN, but have not been able to do it and have a working router.

Yes i am expecting the Router to be able to manipulate/route the packets that it does receive. And because this is Linux based router, almost anything can be achieved through programing, but i think enough programing has already been done by the 3rd party firmare groups. For example, some features advertised in the firmware are Wireless repeating, bridging and vlans. I am quite sure that the Router is capable of doing what i need and the packets can get through the LAN ports, but i just need a way to figure out what is happening when it tries to go through the WAN.

Thanks,

Rich.

I hope you didn't miss my point about the distinction between traffic being 'visible' on a LAN port, and being received by the host. To conclude that the kernel will see and handle traffic using iptables rules based on the ability of Tcpdump to capture traffic would be incorrect. Tcpdump is a packet sniffer, and uses the network in promiscuous mode, meaning that it captures all traffic on the wire. This is not the same behavior of a router, with respect to capturing of packets on the wire. Don't forget that for the router to route (IP) traffic, the hosts that are sending the data must have knowledge that the router is a gateway. Traffic destined for the WAN side is still addressed to the router at the MAC level. This and broadcast traffic is the only traffic that the router will handle. A packet sniffer should handle all traffic.

--- rod.

No, or at least i dont think i missed you point on visible traffic... my understanding of it is (for the moment forgetting about broadcasts and multicasts), that all traffic which passes over the local connection (wire) is visible and not necessarily addressed to the host, and this is what a sniffer can hear (which would be everything on the wire). Then my definition of "Received traffic" would be that packets are addressed to the host and can be received normally without the NIC operating in a promiscuous mode.

Dont worry, i have not concluded the Router's ability based on Tcpdump because it does run in a promiscuous mode. And i think it actually does this via the socket, and not by directly setting the NIC as you could from the command line. Tcpdump only assisted me in the very beginning when i could not see any packets at all.

When you said that the machine that transmits the original packet on the network will need to know of the Routers existance as a Gateway in order to use it. Then it could be possible that the Router is discarding the packets that dont have 'its' MAC as the Destination MAC in the packet header, right?
Based on what i 'know' so far from testing. I can get any packet past the LAN Port on the Router, but only packets that contain the MAC of the Router as the Destination MAC in the packet header make it past the WAN Port of the Router!!
But i dont know how i would go and test that out?? I check the statistics and i dont think they are accurate because i dont see any packets dropping? So i wonder if something is broken, or if it is working differently to as i expect.

If the above were true, then to me this would sound like some filtering being applied (on the WAN's NIC) to only pick the packets that match the MAC correctly. So it may not know about any packets that do not match.
What do you think about this idea?

Thanks,

Rich.

Unicast traffic not addressed to the router's MAC will not be received by the router. This is basic ethernet behavior. You will not see them as dropped packets, since they never make it past the lowest level ethernet driver. In some cases, they will be discarded right on the ethernet hardware.

--- rod.

Hi Rod,

Ok, that does explain a few things and also agrees with the results of my testing so far.

In its current (default) setup, the LAN NIC for the router seems to be in a promiscuous mode and is working as i would like, to allow all packets. It would seem that if i could have this LAN NIC working this way, that i could get the WAN NIC working in a similar way too? It would be great if there was a setting for this.

If this becomes a custom solution, the key is to figure out how to get into the WAN NIC and update it to do what i need. Even if this means updating the source code or applying my own socket filter/sniffer.

Thanks,

Rich.