Transferring remote files via ssh,find,cpio locally

metallica1973 · 11-20-2011, 06:17 PM

I am trying to transfer my website using ssh(hostbasedauthentication) using:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include| cpio -oavc| gzip" > $backup_dir/fullwwwsite$date.cpio.gz

My issue that I am having is that cpio is not on the webserver and I dont have the permission to install it. So I assumed that I can just use cpio locally to do my dirty work.

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include" |cpio -oavc |gzip > $backup_dir/fullwwwsite$date.cpio.gz

I can see that the find stuff is not getting piped to my cpio locally and I get no such file errors:

Code:

cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory

What am I missing??

neonsignal · 11-21-2011, 03:02 AM

The problem is that the cpio is running on the local machine, not the remote one, so it is looking on the local machines for the files (and not finding them).

Is tar a possibility instead of cpio?

If not, you might have to copy the files over to the local machine before archiving them (perhaps using rsync over ssh or something similar).

Incidentally, you could probably run your own copy of cpio on the remote machine in your own home folder; you don't have to install it in a system directory and it doesn't have many dependencies.

metallica1973 · 11-21-2011, 01:09 PM

Thanks for the reply,

I am looking to see if my boss will let me install cpio on the remote machine. If not I will have to improvise. I will give the forum an update.

bigearsbilly · 11-22-2011, 05:29 AM

or rsync would be better I'd say.

if you have tar and netcat you can cheat like this:

target

Code:

nc -l port > blah.tar

source

Code:

tar cvf - blah | nc remote-machine port

or like

metallica1973 · 11-24-2011, 11:13 PM

So now I installed cpio on the webserver and using the include file works:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include| cpio -oavc| gzip" > $backup_dir/fullwwwsite$date.cpio.gz

but oddly when used with the exclude file which I have tested locally and it doesnt work:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include |grep -v -f exclude"| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz

I also tried which didnt work:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz

it still complains:

Code:

cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory

So its appears to be "cpio -oavc" which does not want to receive output via the webserver or locally. ???????

neonsignal · 11-24-2011, 11:24 PM

You seem to be moving the quotes around on the command you are sending over ssh. If the close quote happens before the cpio, then cpio will be run locally, and will not be able to find the files. The close quote must occur after the cpio.

metallica1973 · 11-25-2011, 12:11 AM

thanks for the reply,

That does work but like I said before, for some odd reason the exclude file is excluding everything. Here is the content of both the include and the exclude:

include

Code:

/var/www/vhosts/mywebsite/cgi-bin
/var/www/vhosts/mywebsite/httpdocs

exclude:

Code:

/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/

I run the command and it works but ignores the exclude file stuff, so from the webserver directly. This is how I test the command:

Code:

find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude

It does nothing but if I dont use the exclude file its fine.

So ultimately this is what I want to do.

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude | cpio -oavc | gzip " > mywesite.cpio.gz

neonsignal · 11-25-2011, 12:28 AM

I'm assuming the files 'include' and 'exclude' are located on the webserver in the testuser home directory?

metallica1973 · 11-25-2011, 08:36 AM

I have the include/exclude on both of them. Let me show you what is happening(it is strange).

So I will use the include/exclude on the webserver:

Code:

find /var/www/vhosts/mywebsite/ -depth | grep -f include

/var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key

find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude 

/var/www/vhosts/mywebsite.com/statistics: Permission denied
/var/www/vhosts/mywebsite/pd: Permission denied
/var/www/vhosts/mywebsite/conf: Permission denied

These denied folder are expect but as you can see the exclude file is excluding everything. There is nothing magical with my exclude file other then certain directories that I want to exclude. Both of the files have the same permissions:

Code:

-rw-r--r--  1 testuser testuser    91 Nov 24 09:50 include
-rw-r--r--  1 testuser testuser   224 Nov 24 09:55 exclude

cat include
/var/www/vhosts/mywebsite/cgi-bin
/var/www/vhosts/mywebsite/httpdocs

cat exclude

/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/

This works but I need to have the exclude do what I need it to do. This would work beautifully if I could somehow get my exclude file to work to eventually produce what I want:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude | cpio -oavc | gzip " > mywesite.cpio.gz

Here I have the include/exclude files on the local server:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude

/var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz

cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory

You can now see the difference. It in the first example that the exclude is the snag. In the second example it appears to be cpio. ??

metallica1973 · 11-25-2011, 10:46 AM

I guess I can just exclude my stuff via "! -path" option using find for the time being:

Code:

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth ! -path '*/httpdocs/downloads*' ! -path '*/httpdocs/data/testcompany*' ! -path '*/httpdocs/data/tmp*' ! -path '*/httpdocs/demo*'| cpio -oavc | gzip" > /media/backup_drive/wwwmywebsitefull.cpio.gz

but I would still like to understand my original problem

neonsignal · 11-25-2011, 08:08 PM

You have a blank line in your exclude file which causes it to match every line.

metallica1973 · 11-28-2011, 12:41 PM

Thanks for the response,

The only space that I had was in the bottom of the exclude:

Code:

/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/

I haven't testing it yet but if that is the case then I have to say, white spaces in files has caused me many headaches. I will test it out and let you know thanks.