LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 11-22-2017, 10:28 AM   #1
Sum1
Member
 
Registered: Jul 2007
Distribution: CentOS and Slackware and Gentoo
Posts: 313

Rep: Reputation: 29
s3cmd cp -r s3://bucket1/ s3://bucket2/ so slow it hurts


G'day,

Wanted to check in with others who have experience using s3cmd on aws s3.
I have 1TB data that I want to cp from one s3 bucket to another, both in the same region.
I'm testing a 50GB copy job.
Very Slow.

31 GB (345,000 Objects) copied in 10 hrs. 45 minutes.
This is slightly less than 1MB per second.

Is this typical speed/performance?
I find it hard to believe I'll need approximately 10 - 14 days to copy 1TB of data from one s3 bucket to another.

If it's not typical, please advise on anything I can try to increase copy speed and overall performance on s3.

Thanks for reading my post.
 
Old 11-22-2017, 11:27 AM   #2
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Honestly, I do s3 backups using s3cmd also, and it is painfully slow.
Always has been.
 
Old 11-22-2017, 01:23 PM   #3
Sum1
Member
 
Registered: Jul 2007
Distribution: CentOS and Slackware and Gentoo
Posts: 313

Original Poster
Rep: Reputation: 29
Quote:
Originally Posted by Habitual View Post
Honestly, I do s3 backups using s3cmd also, and it is painfully slow.
Always has been.
Thanks for the reality check.
My pain is real.
:-)

Happy peaceful T-Day to ya.
 
Old 11-23-2017, 10:19 PM   #4
Sum1
Member
 
Registered: Jul 2007
Distribution: CentOS and Slackware and Gentoo
Posts: 313

Original Poster
Rep: Reputation: 29
Wanted to follow up on the issue.
I found and tried a suggestion by this thoughtful soul: https://stackoverflow.com/users/622140/james

On centos 7: yum install awscli

Quote:
. . . I ran across documentation on how to get the AWS S3 CLI sync command to synchronize buckets with massive parallelization. The following commands will tell the AWS CLI to use 1,000 threads to execute jobs (each a small file or one part of a multipart copy) and look ahead 100,000 jobs:
Code:
aws configure set default.s3.max_concurrent_requests 1000
aws configure set default.s3.max_queue_size 100000
After executing the aws configure set commands above, use the sync command as follows:
Code:
aws s3 sync s3://source-bucket/source-path s3://destination-bucket/destination-path
Result: 1TB copy from bucket A to bucket B in 6 hours; or, roughly 47.5 MB/s.
6 hours vs. 10 - 14 days using s3cmd - enormous improvement.
I observed the aws s3 sync command created more processes than I could count, used 10% - 32% cpu across all 8 cores on a Xeon cpu, and grabbed roughly 8 GB's of RAM.

James from stackoverflow achieved much higher data transfer rates up to 700MiB/s transferring files between 3 to 50 GB in size; whereas, I was transferring hundreds of thousands of files of much smaller size.
This may explain our speed differences.
 
Old 11-25-2017, 03:01 PM   #5
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Good Stuff, Maynard. Thanks!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Encrypted offsite backup with EncFS, Amazon S3, and s3cmd LXer Syndicated Linux News 1 07-31-2016 08:39 PM
So new it hurts dougelephant Linux - Newbie 38 03-24-2006 06:01 PM
my arm hurts paddy General 11 02-15-2006 04:54 PM
So new it hurts TyBarton Linux - Newbie 5 10-25-2005 03:25 PM
I'm so newbie it hurts M2tM Linux - Newbie 10 01-07-2004 05:22 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration