Hi,
I am performing network bandwidth benchmarking between 2 nodes in different locations.
iperf and scp shows huge difference in Bandwidth.

iperf3 -i 5 -w 64M -t 60 -p 443 -c remote_host

scp 100G_file.img remote_host:/tmp/

iperf showed bandwidth of around 9.32 Gbits/sec, which is approximately 1.165 GB/sec
scp showed speed of just approximately 100-140MBps

So that is like scp is 10 times slower than iperf.
Any idea why is that so ?

One reason I could think of is that scp need to write to harddisk on remote host, which can slow things. While iperf doesn't really write anything to disk.
But again, that shouldn't really make a difference of 10x.

Anyone who has experience on the same, please share your inputs.

You are forgetting about the overhead from the encryption used in the connection with the scp, depending on the speed of the machines doing that, it could take considerable time.

It looks like you have a fast network. A HDD will definitely slow down the data transfer. You can try to cp 100G on that host without network to check.

Thanks guys. Looks like both the factors contributes to the delay i.e. encryption as well as slow disk (compared to network)

dd command shows disk speed significantly lower than network speed

dd if=/dev/zero of=100G.img bs=1024 count=102400000
102400000+0 records in
102400000+0 records out
104857600000 bytes (105 GB) copied, 330.768 s, 317 MB/s

scp tranfer speed would have been restricted to 317 MB/sec by the disk and the scp encryption would have brought it down further.