LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-28-2013, 07:30 PM   #1
techux
Member
 
Registered: Mar 2013
Posts: 51

Rep: Reputation: Disabled
Lftp mirroring multiple folders


I need to backup several folders,files in my linux box.. I have read about lftp but the thing is every example I find is about local/path to remote/path.. ONE folder and SUBfolders..

is there any option to copy multiple folders ?

the idea is to execute a script..

lftp -u xx,yy ftpserver -e "mirror -R file/ remotefile/"

thanks
 
Old 04-28-2013, 11:33 PM   #2
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
Quote:
Originally Posted by techux View Post
I need to backup several folders,files in my linux box.. I have read about lftp but the thing is every example I find is about local/path to remote/path.. ONE folder and SUBfolders..

is there any option to copy multiple folders ?

the idea is to execute a script..

lftp -u xx,yy ftpserver -e "mirror -R file/ remotefile/"

thanks
rsync is your friend. lftp will not be secure and does not offer the flexibility that rsync will provide. use the right tool for the job, rsync is the tool for this job.

Code:
$ rsync -aviS /path/of/files/to/backup ssh user@REMOTE_IP:/path/to/put/backed/up/files
enjoy.
 
Old 04-29-2013, 08:20 AM   #3
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
the think is that for the moment we have to do the backup to a ftp server running windows. that is why I would like to use lftp. in the past I have used rsync but using a 2nd linux server.

thanks
 
Old 04-30-2013, 07:54 AM   #4
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
hmm try something along this line:

Code:
lftp -d -u <user_name>,<password> -e "set ftp:passive-mode on && mirror -R /full/path/of/files/to/mirror/do/not/forget/leading/slash/ /full/path/to/destination/ && quit" <URL/FTP/connection_server_info>  >> lftp.log
if you want to pass more then one command via -e option you have to use && to separate them.

p.s. also as you are dealing with MS FTP from a linux box, adding the passive can not hurt.
 
Old 04-30-2013, 02:16 PM   #5
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
thanks.

I am creating something like this:
Quote:
open -u xx,yy A.B.C.D
set ftpassive-mode on

echo "################################" >>lftp.out
echo "################################" >>lftp.out

!date >>lftp.out
mirror -R --verbose /root/test1
mirror -R --verbose /root/test2

echo "END OF TRANSFER" >> lftp.out
!date >>lftp.out
echo "#################################" >>lftp.out
exit
in the log file lftp.out only register the successful transfer if there is any.. but if there is any error because the path or something else, it doesnt show the failed transfer..

is there any way to show the failed transfer?

thanks
 
Old 04-30-2013, 08:28 PM   #6
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
you should be able to call both stderr and stdout with 2>&1 at the end of your lines that contain
Code:
 >> lftp.out 2>&1
 
Old 05-01-2013, 11:14 AM   #7
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by lleb View Post
you should be able to call both stderr and stdout with 2>&1 at the end of your lines that contain
Code:
 >> lftp.out 2>&1
thanks, that works...

this thing never end...now, 3 of the servers has 90MB in /etc and I would like to compress it before uploading to the ftp server...

I will try something meanwhile. if anyone know how, feel free to post it.


thanks
 
Old 05-01-2013, 11:22 AM   #8
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
actually, I can add this line to the script

Quote:
!tar -cvzf test1.tar.gz test1
and then use put test1.tar.gz

the thing is...

if this directory has a big size where it could take some seconds or minutes while it get compressed...does the script will wait until this tar finish to upload or will try to upload the tar file even while it is still compressing?
 
Old 05-01-2013, 11:46 AM   #9
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
Quote:
Originally Posted by techux View Post
actually, I can add this line to the script



and then use put test1.tar.gz

the thing is...

if this directory has a big size where it could take some seconds or minutes while it get compressed...does the script will wait until this tar finish to upload or will try to upload the tar file even while it is still compressing?
yes to both. bash is linear. in other words it goes line by line in order from top to bottom. it will not move onto the next line until the last is completed. few other things to keep in mind if you are doing this for backups you might want to both encrypt and create a md5sum for the file to be tested before/after of each of the steps as well as a file check after compression.

things are starting to get a bit larger, but if you wish to do that i can help to an extent. ive done a few of those types of scripts in the past. typically though ill just compress, verify the compression, encrypt and transmit. that is if i am backing up daily. if i am only backing up weekly or less often, ill add the md5sum check after encryption and verify it after transmission.
 
Old 05-04-2013, 07:00 PM   #10
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
thanks for your help, I appreciate it.

I think it wont go further. let see how this works. maybe I could try using samba to share the folders we need to backup..

thanks
 
Old 05-05-2013, 12:53 PM   #11
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
Quote:
Originally Posted by techux View Post
thanks for your help, I appreciate it.

I think it wont go further. let see how this works. maybe I could try using samba to share the folders we need to backup..

thanks
instead of samba, create the share on the MS system and mount it locally on the Linux system as a cifs mount point. you can move the files around much easier that way.

remember samba is not native for linux. cifs is not much better, but if you mount it locally then the linux box will treat it as a local drive not a network drive.

then you would be able to use rsync, but look up in the man page howto point to a local /tmp instead of the remote /tmp that rsync would typically create.
 
Old 05-05-2013, 03:00 PM   #12
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
Thanks.

I am still trying something with lftp..

I would like to keep some old backups in the server. so the idea is to create some variable in the file and use it for creating several directories in the ftp server. i.e backup_552013 backup_542013 ....

I can't get the variables to work.

What I do is create a script similar to this:

Quote:

DATE=$(date +%y%m%d)

echo ${DATE}

open -u xx,yy A.B.C.D
set ftpassive-mode on

echo "################################" >>lftp.out
echo "################################" >>lftp.out

!date >>lftp.out
mirror -R --verbose /root/test1
mirror -R --verbose /root/test2

echo "END OF TRANSFER" >> lftp.out
!date >>lftp.out
echo "#################################"
exit


then invoke it using:

lftp -f script in the console or cron

and get this error:
Quote:
Unknown command `DATE=$(date'.
${DATE}

how could I create a folder like backup_05-05-05-2013 in the ftp server with lftp? I am stuck with this right now..

thanks
 
Old 05-05-2013, 05:24 PM   #13
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
now I solved this issue about the date..

I'm just running the script.sh and no invoking it with lftp -f...


But the issue now is removing folders older than X day..

Quote:
rm_date=`/bin/date +%m-%d-%y -d '5 days ago'`

lftp -u xx,yy serverip

rm -r -f test/${rm_date}
and it doesnt work...

the other issue is if I connect directly to the ftp server I can list with ls the folder.. then type rm -rf folderName and it delete some of the files but not ALL.. onley some of them are hidden files but other arent....not sure why it doesnt delete ALL...
 
Old 05-05-2013, 08:44 PM   #14
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora
Posts: 2,454

Rep: Reputation: 440Reputation: 440Reputation: 440Reputation: 440Reputation: 440
try something like this instead:

Code:
find test/ -mtime +5 -exec rm -rf '{}' \;
the find command will continue to execute until it runs out of items to remove. instead of removing first (fyi rm -rf is the same as rf -r -f) you might want to first try it with an ls -laF:

Code:
find test/ -mtime +5 -exec ls -laF '{}' \;
This way you can print out the files and see if there are any that meet the date restrictions. If that works, then replace ls -laF with rm -rf
 
Old 05-05-2013, 09:53 PM   #15
techux
Member
 
Registered: Mar 2013
Posts: 51

Original Poster
Rep: Reputation: Disabled
I was doing that..

it get me the list of files.. but at the end I get this:


Quote:
/test/backup_04-30-13/root/scripts/rsyncscript
/test/backup_04-30-13/root/test
find: Access failed: 550 Failed to change directory. (/-mtime)
find: Access failed: 550 Failed to change directory. (/+5)
find: Access failed: 550 Failed to change directory. (/-exec)
find: Access failed: 550 Failed to change directory. (/ls)
find: Access failed: 550 Failed to change directory. (/-laF)
find: Access failed: 550 Failed to change directory. (/{})
find: Access failed: 550 Failed to change directory. (/
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
lftp copying folders charliehorse Linux - Software 1 03-31-2013 11:54 AM
mirroring system folders isaaclw Linux - Software 2 10-07-2010 03:59 AM
lftp mirroring hangs cannot open file for reading archangel_617b Linux - Software 3 05-22-2008 11:03 AM
lftp-client can't see folders/files on a wsftp server sigmaflex Linux - Software 4 10-25-2007 10:19 AM
Problem mirroring with lftp Deathspawner Linux - Software 1 11-22-2005 02:25 PM


All times are GMT -5. The time now is 01:05 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration