G'day,

Wanted to check in with others who have experience using s3cmd on aws s3.
I have 1TB data that I want to cp from one s3 bucket to another, both in the same region.
I'm testing a 50GB copy job.
Very Slow.

31 GB (345,000 Objects) copied in 10 hrs. 45 minutes.
This is slightly less than 1MB per second.

Is this typical speed/performance?
I find it hard to believe I'll need approximately 10 - 14 days to copy 1TB of data from one s3 bucket to another.

If it's not typical, please advise on anything I can try to increase copy speed and overall performance on s3.

Thanks for reading my post.

Honestly, I do s3 backups using s3cmd also, and it is painfully slow.
Always has been.

Thanks for the reality check.
My pain is real.
:-)

Happy peaceful T-Day to ya.

Wanted to follow up on the issue.
I found and tried a suggestion by this thoughtful soul: https://stackoverflow.com/users/622140/james

On centos 7: yum install awscli

After executing the aws configure set commands above, use the sync command as follows:
Result: 1TB copy from bucket A to bucket B in 6 hours; or, roughly 47.5 MB/s.
6 hours vs. 10 - 14 days using s3cmd - enormous improvement.
I observed the aws s3 sync command created more processes than I could count, used 10% - 32% cpu across all 8 cores on a Xeon cpu, and grabbed roughly 8 GB's of RAM.

James from stackoverflow achieved much higher data transfer rates up to 700MiB/s transferring files between 3 to 50 GB in size; whereas, I was transferring hundreds of thousands of files of much smaller size.
This may explain our speed differences.

Good Stuff, Maynard. Thanks!