Hmm, according to the man page it should work:
Any file name may contain a host and user specification to indicate that
the file is to be copied to/from that host. Copies between two remote
hosts are permitted.
Have you tried it using public keys, so you don't have to enter a password? Either way should work, but it might narrow down the problem.