Scenario:

Customer recently Migrated Clustered HANA DB Servers to Azure Cloud Platform but these are Physical Servers on Azure (Offering: Azure HLI). Usually these HLIs (HANA DB Servers) in Azure cannot be accessible directly, even not from Azure VNETs. In other Words, HLIs cannot access Servers hosted on other VNET or on Customer On-Prem Datacentres and vice-versa. So, customer has provisioned express route between HLIs and one of the Prod VNET. So, any Servers or VM's can access these HLIs. But other VNETs (like DEV, TEST) or from Customer Locations, these HLIs are not accessible.
*
So, to overcome the above problem, we have provisioned SLES 12 VM on Prod VNET and made it as IPTABLE Server for NAT Translations to solve two type of issues
*
a) When HLI Servers needs to access APPs/DBs (here, Hadoop Server).
b) When On-Prem Datacentre Servers needs to access HLIs.
*
So, in this case, we have assigned Natted IPs for On-Prem Servers and also HLIs as follows. Refer the Diagram attached
*
Original Subnet – 10.10.x.x

HADNA001 - 192.168.1.2
HADNA002 - 192.168.1.3
HADNA003 - 192.168.1.4
*
Original Subnet – 172.168.x.x

HANANA001 – 192.168.1.5
HANANA002 – 192.168.1.6
HANANA003 – 192.168.1.7
*
Problem:

Case 1: When SAP Support team of customer, started pulling data from Hadoop spark controller, they are seeing time out requests. As per them, they are seeing this only in case of High loads. For normal loads this is not happening.
Stages of Pulling Data:

1) Using HANA Studio which is installed on users Laptop.
2) Connect to any one of the HLI Server (In this case, HANANA001)
3) Configure loads and trigger loads from Hadoop Controller (HADNA001) via HLI (HANANA001)

And the request timeout is dynamic, it may happen 1 – 10 times in 4 hours of loads.
*
Case 2: It is observed that when we try to copy data using rsync with multiple parallel sessions from on-prem servers to HLI via iptable, after sometime sessions are getting stalled. This is not happening in case of rsync with 5 Sessions. If it is more than 5 Sessions, sessions are getting stalled.
The data we are trying to copy is 2 TB having 5 GB files each. This is the reason we are trying with multiple parallel sessions, assumption is with one go more 5 GB files will be copied.

We have tried capturing tcpdump and analysed the logs, we have seens only [F] and [R] flags which is not giving much information. And also there are other networking devices in between on-prem and HLI which are not shown in attached diagram and also we have no clue on those.

This is issue is not there with other iptable servers on different environment or VNET. Till, now customer has reported us few errors they can see in HANA Studio

1) SQL Error, 403 Request Time out
2) Unable to Create Virtual Table, Request Time out.

Hi,
Check your environment - it could be cause of the issue.
You probably connect your machines to some switches. Check if you have acl's that limit throughput e.g. in the permit policy.
Ceck if any other traffic shaping is on place.
Also verify that Nat really works - this depends on the protocols you use e.g. a multi session protocol that negotiates ports with its peer.
SQL error 403: check if tcp1433 is open.
Did you implement protocol inspection on your fw? As you mentioned there is other nw equipment in your data path - could there be another fw not configured properly. Is your private net/ip routed?
What is the bandwidth of your nw link - can the nw devices provide sufficient ram / throughput?
Do you use bonded nic's and have you checked the bonding mode?
Do you have endpoint security in place, sgt's, pam etc?
What about your mtu - especially when using encryption (macsec & co).
Good luck - Martin