Skip down to the summary part if you're in a hurry!

I'm wondering if anyone might be able to help me here. I'm working on a research project and as part of our testbed we're setting up linux PCs (running the latest Fedora) that are meant to emulate traffic of multiple hosts across the network – basically creating virtual hosts on a single host PC.

Each linux PC is running multiple instances of traffic generating software, and is configured with multiple IP addresses on the eth0 interface (one IP address for each instance of the traffic software). Each instance of the traffic generator will create TCP traffic with a different source port. The goal is for this unique source port to allow for differentiation between multiple virtual hosts (and thus allow for the TCP traffic to use the correct source IP address). We do this by choosing an IP address based upon the source port of the TCP traffic.

Setting up the ability to receive traffic for multiple IP addresses on a single interface was not a problem. Sending traffic with different source IP addresses over a single interface is the issue that we are concerned about. The way in which I went about solving this problem appears to be the correct one.

I set up multiple routing tables, one for each IP address (or virtual host). All routing details in each table were the same except for the src IP address for the routes.

I then used iptables in an attempt to mark all TCP traffic that had a particular source ports



I then set up rules like so:

ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2
and so on...

Here is how this should have worked:

My traffic generator creates TCP traffic with source port 1232 to represent traffic generated by virtual host #2. iptables catches these packets and marks them with a '2'. The ip rule for fwmark 2 then uses table 2 to route this traffic. Table 2 sets the src IP address for all routes to the corresponding IP address of virtual host #2.

Summary:

Long story short, I couldn't get the marking to work. Rules using specific marks would be skipped, and further testing has proven to me that I am failing to mark this traffic. Here is an example of what I used for iptables to mark by source port:

iptables -t mangle -A OUTPUT -p tcp --source-port 1231 -j MARK --set-marks 1
iptables -t mangle -A OUTPUT -p tcp --source-port 1232 -j MARK --set-marks 2
and so on...

My question: Does anyone know how to mark TCP traffic originating within the PC by source port? It seems to me like the above should have worked. Since I'm new to iptables (and linux networking in general) I attempted this with every single hook (PREROUTING, INPUT, ETC) for the mangle table, all without any luck.

Any ideas, or alternate ways to do what I'm trying to do?


Thanks,
Jason

iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1231 -j SNAT --to-source ip.address.1
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1232 -j SNAT --to-source ip.address.2
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1233 -j SNAT --to-source ip.address.3
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1234 -j SNAT --to-source ip.address.4
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1235 -j SNAT --to-source ip.address.5
...
So long as the replies are coming back to the correct source ports, it should work..

Peter is right, you should use SNAT.

The reason for this is…

All of you packets are locally generated, so when each packet leaves an interface its source IP address will become that of the first IP address you have assigned to the NIC. The routing process will not change source or destination IP addresses, only source/destination MAC addresses. NAT will rewrite the source (SNAT) or destination (DNAT) IP address of a packet based on source port (and many other things) like Peter said.

Thanks for the advice guys. Was away for a few days and finally had the chance to try this today. Unfortunately, I didnt' have much luck. I used the suggested iptables commands and my receiving host is still showing the wrong source IP address in its logs. For example, I'll generate traffic with source port 1232, which should send from the source IP address of 192.168.1.27, and my logs on the receiving PC show traffic being received from 192.168.1.26 port 1232. Obviously I'm missing a piece of the puzzle here. Any suggestions?

In case it matters, my main routing table now looks as follows

192.168.1.32/29 dev eth0 scope link
192.168.1.16/29 dev eth0 scope link
192.168.1.24/29 dev eth0 scope link

eth0 is as follows:

inet 192.168.1.26 brd 192.168.1.31 scope global eth0
inet 192.168.1.27 brd 192.168.1.31 scope secondary eth0
inet 192.168.1.28 brd 192.168.1.31 scope secondary eth0

and iptables is set up using the following commands:

iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1231 -j SNAT --to-source 192.168.1.26
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1232 -j SNAT --to-source 192.168.1.27
iptables -t nat -A POSTROUTING -o eth0 -p tcp --sport 1233 -j SNAT --to-source 192.168.1.28

Appreciate the help!

Check your ifconfig command

Each of the possible sending ip numbers must be set on the interface
eth0
eth0:1
eth0:2
etc

So you're saying that when using SNAT, simply adding secondary IP addresses via the "ip addr add" command won't allow source address translation to work in this case? Any idea as to why?

I'll give this a try and let you know how things go.

The ip address can't be bound to if it doesn't exist..
And replies will ask for a mac address for that number, which doesn't exist..

I'm not familiar with the 'ip addr add' commans/script.
I do it with ifconfig directly..

"ip address add" is part of iproute2

I used ifconfig to make the aliases as you suggested and i still have the same problem (which I expected, because adding the aliases didn't seem like a major difference from simply adding secondary IP addresses). I'm going to research using SNAT in iptables to see if I'm missing something here. Let me know if you have any more ideas, or if you want me to include more information about my configuration.

Also, if you do happen to know how to mark TCP packets that originate from within the local host, let me know, because I can definitely get this working if I can just solve that problem.


thanks

This is very interesting, because I know this works correctly with DNAT. The only thing I could find that might relate is below:

You want to do Source NAT; change the source address of connections to something different. This is done in the POSTROUTING chain, just before it is finally sent out; this is an important detail, since it means that anything else on the Linux box itself (routing, packet filtering) will see the packet unchanged. It also means that the `-o' (outgoing interface) option can be used.

Just wanted to thank you guys for the help. The problem (as I expected) was a mistake on my part. I was under the impression that the traffic generator I was using worked with TCP packets by default. Turns out its default is UDP. So of course when I attempted to SNAT by the source port of TCP data, nothing happened. Now I've fixed the traffic generator to create TCP packets and everything is working fine. Thanks!