Thank you to kbp (for the knowledge of tools I wasn't aware of) and lithos (for pointing out the pattern that I was missing that leads to my following theory).
Here is a theory. Not sure how to address it and not sure I'm dead-on yet, but here goes:


Workstation in my LAN builds a packet for upload. It is encoded with the address 172.16.1.1
It gets sent to router2, which proceeds to masq it with the IP of 192.168.0.2
It gets sent to router1, which proceeds to masq it with the IP of 64.126.162.234
The packet gets sent to the upload server.
The packet is dismantled for review on the upload server.
The 'dismantler' (for lack of another term) digs through the envelopes to the underlying data
It then wants to respond to the send, but instead of going back out to the outer envelope to find the data it needs, it looks at the inner most envelope before the data (which would be the 192.168.0.2 address).
Since the upload server, residing behind a firewall, is in fact, in the 192.168.0.0 network, it decides it must have received the packet from a computer within its network and sends the response out to the local subnet.

Part of my reasoning for this is that I have tested from two other subnets behind router1 that would not have put the 192.168.0.0 envelope on the packet and they are both successful.
If this is infact a true theory, then what would I look to for fixing this? Would it be a misconfig of Apache? Routing on the upload server? Malformed envelope from router2?

No, the thing about NAT is that it should be transparent, there are certain protocols like ipsec that do have the client ip embedded within the data section so the server needs to know that NAT traversal has taken place. The envelope analogy isn't valid in this case, with NAT the original source ip in the packet header is overwritten with an external address and an entry is recorded in the translation table to allow return packets to be sent to the correct host.

Sorry, just noticed the client ip's in the log file, you shouldn't be seeing any 192.168.x.x addresses in the web server logs. Is the web server behind a firewall/router/NAT ?

I'm told it is behind a "Cisco firewall". I had another reason why my theory was invalid. I would be getting 100% failure rate rather than only on files over 250k. I'm exploring the idea that it is firmware related, but again why the hit-and-miss failures? Are there any settings in RHEL5 or iptables that would hinder large file transfer. I looked at large send offload but my netXtreme cards don't seen to be able to configure it.