LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-28-2017, 12:23 PM   #1
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Rep: Reputation: 8
Error while RSYNC over SSH only specified files


I want to `RSYNC over SSH` files from Raspberry Pi (Raspbian) to NAS. `RSYNC` server and `SSH` servers are running on several RPi`s. I want to rsync every 10 min files not older than 3 days and each day, files not older then 30 days, depending on semaphore file.
I googled a lot on this forum and try to use the solution I found here. But it does not work. Below fragment of my script:

Code:
    path="/share/homes/bioama"
    port=777
    serwer="192.168.0.222"
    user="root"
    rpi="piotr-test"
    path_rpi="/home/pi/EXT"
    
    [ -f $path/Pobrane/flag_sync ] && days=30 || days=3
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" --files-from=<(/usr/bin/ssh -p$port -i $path/.ssh/id_rsa_bio1 $user@$serwer '/usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basename {} \;') $user@$serwer:$path_rpi/$rpi $path/
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" $user@$serwer:$path_rpi/LOGS $path/$rpi
    /bin/rm -f $path/Pobrane/flag_sync 2>/dev/null
But if I run that script I got such errors:

Code:
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basenamrsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
Can anybody tell me what I am doing wrong? Syncing over SSH without `find` works fine. Eaver variables are not passing correctly to `ssh` command.

Last edited by mackowiakp; 11-28-2017 at 12:55 PM.
 
Old 11-28-2017, 12:40 PM   #2
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,555

Rep: Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088
Quote:
Originally Posted by mackowiakp View Post
Code:
    path="/share/homes/bioama"
    port=777
    serwer="192.168.0.222"
    user="root"
    rpi="piotr-test"
    path_rpi="/home/pi/EXT"
    
    [ -f $path/Pobrane/flag_sync ] && days=30 || days=3
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" --files-from=<(/usr/bin/ssh -p$port -i $path/.ssh/id_rsa_bio1 $user@$serwer '/usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basename {} \;') $user@$serwer:$path_rpi/$rpi $path/
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" $user@  $serwer:$path_rpi/LOGS $path/$rpi
    /bin/rm -f $path/Pobrane/flag_sync 2>/dev/null
I haven't really examined that lengthy command in great detail, but I do see that you have variables "$path_rpi", "$rpi", and "$days" appearing within a single-quoted string, preventing those variables from being expanded.

Last edited by rknichols; 11-28-2017 at 12:43 PM.
 
Old 11-28-2017, 12:47 PM   #3
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
OK, I removed single quotation but still have syntax error, but variables are passing correctly:

Code:
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec barsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
 
Old 11-28-2017, 12:52 PM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
In the second rsync line, you seem to have a pair of spaces after @ which should not be there. Maybe that is the problem.

If not, try breaking it down into steps first before trying the process substitution.

You might try putting "set -e -x" near the beginning of your script to watch what it really is sending.

A bit trivial would be to eliminate the -exec clause and replace it with -printf "%f\n" instead. However, that won't solve the problem, only simplify the activity a little.
 
Old 11-28-2017, 12:58 PM   #5
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Ups. That "pair of spaces after @" is copy/paste error from terminal to forum. I just edited it. But the roblem is in the first line with "rsync". The second works properly.
 
Old 11-28-2017, 01:00 PM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Add the following to your script at the top below the #! line.

Code:
set -x
That will show each line as the system will see it, just prior to execution.
 
Old 11-28-2017, 01:05 PM   #7
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
The result below:

Code:
+ path=/share/homes/bioama
+ port=777
+ serwer=192.168.0.222
+ user=root
+ rpi=piotr-test
+ path_rpi=/home/pi/EXT
+ '[' -f /share/homes/bioama/Pobrane/flag_sync ']'
+ days=3
+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' '--files-from=<(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec basename {} ; )' root@192.168.0.222:/home/pi/EXT/piotr-test /share/homes/bioama/
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec barsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' root@192.168.0.222:/home/pi/EXT/LOGS /share/homes/bioama/piotr-test
receiving incremental file list

sent 24 bytes  received 89 bytes  226.00 bytes/sec
total size is 590  speedup is 5.22
+ /bin/rm -f /share/homes/bioama/Pobrane/flag_sync
 
Old 11-28-2017, 01:12 PM   #8
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Just out of curiosity, what happens if you swap

Code:
-exec basename {}
with

Code:
-printf "%f\n"
That will remove the need for escaping the semicolon by removing the need for the semicolon. It wil also remove the call to an external program basename
 
Old 11-28-2017, 01:17 PM   #9
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.8.2003
Posts: 5,442

Rep: Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070
The reported error is
Code:
error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
Why do you have "<(" at the beginning of your --files-from entry? I'm guessing that might be the syntax error...
Code:
--files-from=<(/usr/bin/ssh ...
In fact, as I read the man page
Code:
--files-from=FILE       read list of source-file names from FILE
I'm not sure I see why you have the ssh command in there at all...there should be filename on the right side of that entry

But I'm guessing (as I said)...providing food for thought, I hope...

Last edited by scasey; 11-28-2017 at 01:23 PM.
 
Old 11-28-2017, 01:31 PM   #10
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by Turbocapitalist View Post
Just out of curiosity, what happens if you swap

Code:
-exec basename {}
with

Code:
-printf "%f\n"
That will remove the need for escaping the semicolon by removing the need for the semicolon. It wil also remove the call to an external program basename
The result is simmilar:

Code:
+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' '--files-from=<(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -printf %f\n )' root@192.168.0.222:/home/pi/EXT/piotr-test /share/homes/bioama/
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -printf rsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
 
Old 11-28-2017, 01:34 PM   #11
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Code:
--files-from=<(/usr/bin/ssh ...
I take such example from this forum as proper usage. But maybe I am wrong...
 
Old 11-28-2017, 01:38 PM   #12
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by mackowiakp View Post
Code:
--files-from=<(/usr/bin/ssh ...
I take such example from this forum as proper usage. But maybe I am wrong...
Yes, it's correct process substitution

Does the clause run fine by itself?

Code:
/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name "*.csv" -printf "%f\n";
 
Old 11-28-2017, 01:43 PM   #13
mackowiakp
Member
 
Registered: Jun 2014
Location: Poland/Gdynia
Distribution: Mageia 7, SH4, Debian
Posts: 316

Original Poster
Rep: Reputation: 8
Yes, it works properly. Below fragment of output:

Code:
010.csvn171127_0310.csvn171128_0400.csvn171126_0900.csvn171128_1650.csvn171128_1040.csvn171125_2330.csvn171126_1330.csvn171125_2140.csvn171125_2320.csvn171127_0950.csvn171128_1020.csvn171126_1900.csvn171126_2340.csvn171128_1920.csvn171126_0750.csvn171128_1440.csvn171127_1840.csvn171128_0510.csvn171126_0600.csvn171127_1210.csvn171128_1100.csvn171128_1430.csvn171127_2320.csvn171127_1610.csvn171126_0820.csvn171128_0120.csvn171128_1320.csvn171128_1250.csvn171127_0650.csvn171126_0300.csvn171128_0620.csvn171125_2220.csvn171126_2210.csvn171127_0330.csvn171127_1410.csvn171125_2210.csvn171128_0420.csvn171127_0840.csvn171127_2200.csvn171127_0820.csvn171127_2120.csvn171127_0920.csvn171126_2100.csvn171128_1210.csvn171125_2340.csvn171126_1350.csvn171126_1600.csvn171128_1850.csvn171125_2310.csvn171128_0450.csvn171126_2110.csvn171128_1930.csvn171128_0350.csvn171128_0210.csvn171126_1150.csvn171128_1710.csvn171128_1220.csvn171127_0450.csvn171128_1110.csvn171127_1820.csvn171127_0410.csvn171126_1200.csvn171128_0900.csvn171125_2240.csvn171126_0130.csvn171126_1710.csvn171127_1320.csvn171128_0530.csvn171127_0340.csvn171127_0220.csvn171128_0920.csvn171127_1510.csvn171126_0800.csvn171128_0050.csvn171127_1930.csvn171126_0220.csvn171128_2020.csvn171126_1250.csvn171127_2340.csvn171127_2300.csvn171127_1140.csvn171127_1800.csvn171126_0650.csvn171127_1900.csvn171127_2020.csvn171127_1720.csvn171128_1530.csvn171127_0600.csvn171127_0910.csvn171128_1200.csvn171128_1630.csvn171127_0800.csvn171127_1230.csvn171126_0320.csvn171126_1210.csvn171127_2110.csvn171128_0300.csvn171126_0050.csvn171128_1720.csvn171128_0200.csvn171126_0440.csvn171126_1700.csvn171127_1310.csvn171128_0240.csvn171127_1950.csvn171127_1340.csvn171128_1800.csvn171127_0710.csvn171128_1950.csvn171128_1130.csvn171126_0930.csvn171125_2150.csvn171127_1240.csvn171126_0250.csvn171126_2030.csvn171128_1620.csvn171126_1940.csvn171127_0520.csvn171127_0020.csvn171127_1620.csvn171127_0350.csvn171128_2030.csvn171127_2010.csvn171127_1430.csvn171128_0410.csvn171126_0110.csvn171128_1030.csvn171126_1410.csvn171127_0100.csvn171127_2240.csvn171126_1020.csvn171128_1240.csvn171126_0740.csvn171127_0640.csvn171126_1920.csvn171127_1710.csvn171126_0450.csvn171127_0740.csvn171127_0200.csvn171128_1520.csvn171128_0110.csvn171126_0410.csvn171126_1820.csvn171127_1450.csvn171128_0600.csvn171126_1030.csvn171128_0230.csvn171126_1010.csvn171128_0850.csvn171127_2140.csvn171127_0610.csvn171128_0220.csvn171128_0310.csvn171127_2330.csvn171127_0150.csvn171128_1510.csvn171127_0130.csvn171126_2310.csvn171126_0640.csvn171128_1820.csvn171126_1420.csvn171126_0530.csvn171126_0520.csvn171126_0120.csvn171125_2200.csvn171126_0730.csvn171127_2050.csvn171126_1430.csvn171127_1810.csvn171127_0940.csvn171128_0340.csvn171126_1440.csvn171127_2130.csvn171126_0340.csvn171127_1150.csvn171127_0420.csvn171126_1910.csvn171126_2250.csvn171128_0830.csvn171128_1340.csvn171128_0330.csvn171127_2230.csvn171127_0900.csvn171128_0940.csvn171128_0550.csvn171128_0430.csvn171128_0650.csvn171126_2000.csvn171126_1850.csvn171126_0610.csvn
Strange thing is "n" letter between filenames.
 
Old 11-28-2017, 01:45 PM   #14
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,690
Blog Entries: 3

Rep: Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920Reputation: 2920
Quote:
Originally Posted by mackowiakp View Post
Strange thing is "n" letter between filenames.
The \n is somehow not escaped and becoming n instead.
 
Old 11-28-2017, 02:01 PM   #15
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.8.2003
Posts: 5,442

Rep: Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070Reputation: 2070
Quote:
Originally Posted by Turbocapitalist View Post
Yes, it's correct process substitution
As I said on another post today: I love the learnin' that happens here.

More food for thought: Then maybe escape the < and/or the ( and ) in the substitution, so that rsync doesn't misread them?...as it appears to (may)be causing a syntax error -- and/or quote the clause to the --files-from ?
-- also maybe escape the \ of the \n in the printf portion? "%f\\n" to get the \n to happen in the output.

Last edited by scasey; 11-28-2017 at 02:07 PM.
 
  


Reply

Tags
find, raspberrry pi 3 model b, rsync+ssh


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync -e ssh error protocol incompatibility osio Linux - Networking 13 11-06-2019 04:35 PM
RSync error syncing over ssh driving me crazy hazey11 Linux - Server 4 02-26-2012 02:17 PM
rsync using www as user gives ssh error sir-lancealot Linux - Server 6 02-11-2011 02:08 PM
Rsync error cannot do ssh and transfer file linuxbee1 Linux - Newbie 13 06-23-2008 04:41 PM
Is there a way to allow a user to rsync files to my server without being able to SSH abefroman Linux - Security 1 12-08-2007 06:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:32 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration