LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-20-2011, 07:17 PM   #1
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Rep: Reputation: 60
Transferring remote files via ssh,find,cpio locally


I am trying to transfer my website using ssh(hostbasedauthentication) using:
Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include| cpio -oavc| gzip" > $backup_dir/fullwwwsite$date.cpio.gz
My issue that I am having is that cpio is not on the webserver and I dont have the permission to install it. So I assumed that I can just use cpio locally to do my dirty work.

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include" |cpio -oavc |gzip > $backup_dir/fullwwwsite$date.cpio.gz
I can see that the find stuff is not getting piped to my cpio locally and I get no such file errors:

Code:
cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory
What am I missing??

Last edited by metallica1973; 11-20-2011 at 08:54 PM.
 
Old 11-21-2011, 04:02 AM   #2
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Buster (Fluxbox WM)
Posts: 1,391
Blog Entries: 52

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
The problem is that the cpio is running on the local machine, not the remote one, so it is looking on the local machines for the files (and not finding them).

Is tar a possibility instead of cpio?

If not, you might have to copy the files over to the local machine before archiving them (perhaps using rsync over ssh or something similar).

Incidentally, you could probably run your own copy of cpio on the remote machine in your own home folder; you don't have to install it in a system directory and it doesn't have many dependencies.

Last edited by neonsignal; 11-21-2011 at 04:03 AM.
 
Old 11-21-2011, 02:09 PM   #3
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
Thanks for the reply,

I am looking to see if my boss will let me install cpio on the remote machine. If not I will have to improvise. I will give the forum an update.
 
Old 11-22-2011, 06:29 AM   #4
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: NetBSD, Void, Debian, Mint, Ubuntu, Puppy, Raspbian
Posts: 3,492

Rep: Reputation: 235Reputation: 235Reputation: 235
or rsync would be better I'd say.

if you have tar and netcat you can cheat like this:


target
Code:
nc -l port > blah.tar
source
Code:
tar cvf - blah | nc remote-machine port
or like
 
Old 11-25-2011, 12:13 AM   #5
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
So now I installed cpio on the webserver and using the include file works:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include| cpio -oavc| gzip" > $backup_dir/fullwwwsite$date.cpio.gz
but oddly when used with the exclude file which I have tested locally and it doesnt work:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth | grep -f include |grep -v -f exclude"| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz
I also tried which didnt work:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz
it still complains:

Code:
cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory
So its appears to be "cpio -oavc" which does not want to receive output via the webserver or locally. ???????

Last edited by metallica1973; 11-25-2011 at 12:47 AM.
 
Old 11-25-2011, 12:24 AM   #6
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Buster (Fluxbox WM)
Posts: 1,391
Blog Entries: 52

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
You seem to be moving the quotes around on the command you are sending over ssh. If the close quote happens before the cpio, then cpio will be run locally, and will not be able to find the files. The close quote must occur after the cpio.

Last edited by neonsignal; 11-25-2011 at 12:25 AM.
 
Old 11-25-2011, 01:11 AM   #7
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
thanks for the reply,

That does work but like I said before, for some odd reason the exclude file is excluding everything. Here is the content of both the include and the exclude:

include

Code:
/var/www/vhosts/mywebsite/cgi-bin
/var/www/vhosts/mywebsite/httpdocs
exclude:

Code:
/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/
I run the command and it works but ignores the exclude file stuff, so from the webserver directly. This is how I test the command:

Code:
find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude
It does nothing but if I dont use the exclude file its fine.

So ultimately this is what I want to do.

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude | cpio -oavc | gzip " > mywesite.cpio.gz
 
Old 11-25-2011, 01:28 AM   #8
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Buster (Fluxbox WM)
Posts: 1,391
Blog Entries: 52

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
I'm assuming the files 'include' and 'exclude' are located on the webserver in the testuser home directory?
 
Old 11-25-2011, 09:36 AM   #9
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
I have the include/exclude on both of them. Let me show you what is happening(it is strange).

So I will use the include/exclude on the webserver:

Code:
find /var/www/vhosts/mywebsite/ -depth | grep -f include

/var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key

find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude 

/var/www/vhosts/mywebsite.com/statistics: Permission denied
/var/www/vhosts/mywebsite/pd: Permission denied
/var/www/vhosts/mywebsite/conf: Permission denied
These denied folder are expect but as you can see the exclude file is excluding everything. There is nothing magical with my exclude file other then certain directories that I want to exclude. Both of the files have the same permissions:

Code:
-rw-r--r--  1 testuser testuser    91 Nov 24 09:50 include
-rw-r--r--  1 testuser testuser   224 Nov 24 09:55 exclude

cat include
/var/www/vhosts/mywebsite/cgi-bin
/var/www/vhosts/mywebsite/httpdocs

cat exclude

/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/


This works but I need to have the exclude do what I need it to do. This would work beautifully if I could somehow get my exclude file to work to eventually produce what I want:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth | grep -f include | grep -v -f exclude | cpio -oavc | gzip " > mywesite.cpio.gz
Here I have the include/exclude files on the local server:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude

/var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key
/var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key

sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite.com -depth" | grep -f include |grep -v -f exclude| cpio -oavc| gzip > $backup_dir/fullwwwsite$date.cpio.gz

cpio: /var/www/vhosts/mywebsite.com/httpdocs/data/customer/log/kwesthuba.log: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/chaz.key: Cannot stat: No such file or directory
cpio: /var/www/vhosts/mywebsite/httpdocs/data/customer/log/mob786.key: Cannot stat: No such file or directory
You can now see the difference. It in the first example that the exclude is the snag. In the second example it appears to be cpio. ??

Last edited by metallica1973; 11-25-2011 at 09:45 AM.
 
Old 11-25-2011, 11:46 AM   #10
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
I guess I can just exclude my stuff via "! -path" option using find for the time being:

Code:
sudo ssh -o "PasswordAuthentication no" -o "HostbasedAuthentication yes" -l testuser 192.168.3.1 "find /var/www/vhosts/mywebsite/ -depth ! -path '*/httpdocs/downloads*' ! -path '*/httpdocs/data/testcompany*' ! -path '*/httpdocs/data/tmp*' ! -path '*/httpdocs/demo*'| cpio -oavc | gzip" > /media/backup_drive/wwwmywebsitefull.cpio.gz
but I would still like to understand my original problem

Last edited by metallica1973; 11-25-2011 at 06:27 PM.
 
Old 11-25-2011, 09:08 PM   #11
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Buster (Fluxbox WM)
Posts: 1,391
Blog Entries: 52

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
You have a blank line in your exclude file which causes it to match every line.
 
Old 11-28-2011, 01:41 PM   #12
metallica1973
Senior Member
 
Registered: Feb 2003
Location: Washington D.C
Posts: 2,190

Original Poster
Rep: Reputation: 60
Thanks for the response,

The only space that I had was in the bottom of the exclude:

Code:
/var/www/vhosts/mywebsite/httpdocs/downloads/
/var/www/vhosts/mywebsite/httpdocs/data/caca/
/var/www/vhosts/mywebsite/httpdocs/data/tmp/
/var/www/vhosts/mywebsite/httpdocs/demo/
I haven't testing it yet but if that is the case then I have to say, white spaces in files has caused me many headaches. I will test it out and let you know thanks.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
transferring files Garrett85 Linux - Networking 37 07-10-2011 03:12 PM
Transferring Files via Reverse SSH metallica1973 Linux - Networking 9 02-08-2011 08:26 AM
Transferring file through ssh dstar101 Linux - Newbie 4 01-19-2008 06:08 AM
Transferring 40000+ files with FTP (mget) shows 0 files tim1235 Linux - Software 5 10-17-2004 07:06 PM
SSH Transferring Question Mark2020 *BSD 1 12-17-2002 09:26 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:03 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration