LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices


Reply
  Search this Thread
Old 04-09-2015, 11:36 AM   #1
kha.t.nguyen
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
pipe mysqldump to S3


I'm using s3cmd version 1.5.2 which I understand now accepts stdin to send files to s3.

Google returned this link. I tested s3cmd with stdin and the 'echo hello' command works for me:
echo "hello" | s3cmd put - s3://my-bucket/folder/hello.txt --verbose

Now that I'm past that, I'm trying to do something similar with mysqldump. I would like to be able to send the *.gz file directly to S3 so that I do not use any disk space on the local machine:
mysqldump -uroot -pubuntu --databases test_database | gzip > backupdb.sql.gz | s3cmd put - s3://mybucket/folder/backupdb.sql.gz --verbose

The mysqldump command results in the following message:

INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
INFO: Forwarding request to us-east-1
ERROR: S3 error: The XML you provided was not well-formed or did not validate against our published schema


I'm not sure what I'm doing wrong. Any ideas are greatly appreciated.

Thanks!
 
Old 04-09-2015, 11:52 AM   #2
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
I don't think the "-" in "s3cmd put -" is going to do it, (could be wrong though)
I used
Code:
mysqldump <db> -uroot -pubuntu | gzip > backupdb.sql.gz | s3cmd put backupdb.sql.gz s3://mybucket/folder/ --verbose
and that worked fine.
 
Old 07-07-2015, 11:14 PM   #3
kha.t.nguyen
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Original Poster
Rep: Reputation: 0
Turns out the dash in the s3cmd is supported by the latest version of s3. I ended up using this to back up all the databases:
for x in `mysql -BN -uroot -pubuntu -e 'SHOW DATABASES'`; do mysqldump -u'root' -p'ubuntu' --single-transaction --databases $x --events --opt | gzip | s3cmd put - s3://my-bucket/mysql_backup_$x.sql.gz; done
 
Old 07-08-2015, 08:08 AM   #4
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by kha.t.nguyen View Post
Turns out the dash in the s3cmd is supported by the latest version of s3.
As is usually the case, no? Good job and well done.
 
Old 08-31-2018, 05:07 AM   #5
parmarth
LQ Newbie
 
Registered: Aug 2018
Posts: 1

Rep: Reputation: Disabled
Smile change to aws s3 cp

mysqldump --host=$HOST --user=$USER --password=$PASSWORD $DB_NAME --routines --single-transaction | gzip -9 | aws s3 cp - s3://bucket/database/filename.sql.gz

will directly store file to s3.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to ignore Pipe in Pipe delimited file? rohit_shinez Programming 29 08-13-2013 11:53 PM
[SOLVED] How to handle a broken pipe exception (SIGPIPE) in FIFO pipe? zyroot998 Programming 5 03-03-2011 08:10 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud

All times are GMT -5. The time now is 07:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration