Error while RSYNC over SSH only specified files

mackowiakp · 11-28-2017, 12:23 PM

I want to `RSYNC over SSH` files from Raspberry Pi (Raspbian) to NAS. `RSYNC` server and `SSH` servers are running on several RPi`s. I want to rsync every 10 min files not older than 3 days and each day, files not older then 30 days, depending on semaphore file.
I googled a lot on this forum and try to use the solution I found here. But it does not work. Below fragment of my script:

Code:

    path="/share/homes/bioama"
    port=777
    serwer="192.168.0.222"
    user="root"
    rpi="piotr-test"
    path_rpi="/home/pi/EXT"
    
    [ -f $path/Pobrane/flag_sync ] && days=30 || days=3
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" --files-from=<(/usr/bin/ssh -p$port -i $path/.ssh/id_rsa_bio1 $user@$serwer '/usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basename {} \;') $user@$serwer:$path_rpi/$rpi $path/
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" $user@$serwer:$path_rpi/LOGS $path/$rpi
    /bin/rm -f $path/Pobrane/flag_sync 2>/dev/null

But if I run that script I got such errors:

Code:

rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basenamrsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]

Can anybody tell me what I am doing wrong? Syncing over SSH without `find` works fine. Eaver variables are not passing correctly to `ssh` command.

rknichols · 11-28-2017, 12:40 PM

Quote:

Originally Posted by mackowiakp

Code:

    path="/share/homes/bioama"
    port=777
    serwer="192.168.0.222"
    user="root"
    rpi="piotr-test"
    path_rpi="/home/pi/EXT"
    
    [ -f $path/Pobrane/flag_sync ] && days=30 || days=3
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" --files-from=<(/usr/bin/ssh -p$port -i $path/.ssh/id_rsa_bio1 $user@$serwer '/usr/bin/find $path_rpi/$rpi -maxdepth 1 -type f -mtime -$days -name *.csv -exec basename {} \;') $user@$serwer:$path_rpi/$rpi $path/
    sudo -u bioama /usr/bin/rsync -av -e "/usr/bin/ssh -i $path/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p$port" $user@  $serwer:$path_rpi/LOGS $path/$rpi
    /bin/rm -f $path/Pobrane/flag_sync 2>/dev/null

I haven't really examined that lengthy command in great detail, but I do see that you have variables "$path_rpi", "$rpi", and "$days" appearing within a single-quoted string, preventing those variables from being expanded.

mackowiakp · 11-28-2017, 12:47 PM

OK, I removed single quotation but still have syntax error, but variables are passing correctly:

Code:

rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec barsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]

Turbocapitalist · 11-28-2017, 12:52 PM

In the second rsync line, you seem to have a pair of spaces after @ which should not be there. Maybe that is the problem.

If not, try breaking it down into steps first before trying the process substitution.

You might try putting "set -e -x" near the beginning of your script to watch what it really is sending.

A bit trivial would be to eliminate the -exec clause and replace it with -printf "%f\n" instead. However, that won't solve the problem, only simplify the activity a little.

mackowiakp · 11-28-2017, 12:58 PM

Ups. That "pair of spaces after @" is copy/paste error from terminal to forum. I just edited it. But the roblem is in the first line with "rsync". The second works properly.

Turbocapitalist · 11-28-2017, 01:00 PM

Add the following to your script at the top below the #! line.

Code:

set -x

That will show each line as the system will see it, just prior to execution.

mackowiakp · 11-28-2017, 01:05 PM

The result below:

Code:

+ path=/share/homes/bioama
+ port=777
+ serwer=192.168.0.222
+ user=root
+ rpi=piotr-test
+ path_rpi=/home/pi/EXT
+ '[' -f /share/homes/bioama/Pobrane/flag_sync ']'
+ days=3
+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' '--files-from=<(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec basename {} ; )' root@192.168.0.222:/home/pi/EXT/piotr-test /share/homes/bioama/
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -exec barsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]
+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' root@192.168.0.222:/home/pi/EXT/LOGS /share/homes/bioama/piotr-test
receiving incremental file list

sent 24 bytes  received 89 bytes  226.00 bytes/sec
total size is 590  speedup is 5.22
+ /bin/rm -f /share/homes/bioama/Pobrane/flag_sync

Turbocapitalist · 11-28-2017, 01:12 PM

Just out of curiosity, what happens if you swap

Code:

-exec basename {}

with

Code:

-printf "%f\n"

That will remove the need for escaping the semicolon by removing the need for the semicolon. It wil also remove the call to an external program basename

scasey · 11-28-2017, 01:17 PM

The reported error is

Code:

error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]

Why do you have "<(" at the beginning of your --files-from entry? I'm guessing that might be the syntax error...

Code:

--files-from=<(/usr/bin/ssh ...

In fact, as I read the man page

Code:

--files-from=FILE       read list of source-file names from FILE

I'm not sure I see why you have the ssh command in there at all...there should be filename on the right side of that entry

But I'm guessing (as I said)...providing food for thought, I hope...

mackowiakp · 11-28-2017, 01:31 PM

Quote:

Originally Posted by Turbocapitalist

Just out of curiosity, what happens if you swap

Code:

-exec basename {}

with

Code:

-printf "%f\n"

That will remove the need for escaping the semicolon by removing the need for the semicolon. It wil also remove the call to an external program basename

The result is simmilar:

Code:

+ sudo -u bioama /usr/bin/rsync -av -e '/usr/bin/ssh -i /share/homes/bioama/.ssh/id_rsa_bio1 -oStrictHostKeyChecking=no -qx -p777' '--files-from=<(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -printf %f\n )' root@192.168.0.222:/home/pi/EXT/piotr-test /share/homes/bioama/
rsync: failed to open files-from file <(/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name *.csv -printf rsync error: syntax or usage error (code 1) at main.c(2472) [client=3.0.7]

mackowiakp · 11-28-2017, 01:34 PM

Code:

--files-from=<(/usr/bin/ssh ...

I take such example from this forum as proper usage. But maybe I am wrong...

Turbocapitalist · 11-28-2017, 01:38 PM

Quote:

Originally Posted by mackowiakp

Code:

--files-from=<(/usr/bin/ssh ...

I take such example from this forum as proper usage. But maybe I am wrong...

Yes, it's correct process substitution

Does the clause run fine by itself?

Code:

/usr/bin/ssh -p777 -i /share/homes/bioama/.ssh/id_rsa_bio1 root@192.168.0.222 /usr/bin/find /home/pi/EXT/piotr-test -maxdepth 1 -type f -mtime -3 -name "*.csv" -printf "%f\n";

mackowiakp · 11-28-2017, 01:43 PM

Yes, it works properly. Below fragment of output:

Code:

010.csvn171127_0310.csvn171128_0400.csvn171126_0900.csvn171128_1650.csvn171128_1040.csvn171125_2330.csvn171126_1330.csvn171125_2140.csvn171125_2320.csvn171127_0950.csvn171128_1020.csvn171126_1900.csvn171126_2340.csvn171128_1920.csvn171126_0750.csvn171128_1440.csvn171127_1840.csvn171128_0510.csvn171126_0600.csvn171127_1210.csvn171128_1100.csvn171128_1430.csvn171127_2320.csvn171127_1610.csvn171126_0820.csvn171128_0120.csvn171128_1320.csvn171128_1250.csvn171127_0650.csvn171126_0300.csvn171128_0620.csvn171125_2220.csvn171126_2210.csvn171127_0330.csvn171127_1410.csvn171125_2210.csvn171128_0420.csvn171127_0840.csvn171127_2200.csvn171127_0820.csvn171127_2120.csvn171127_0920.csvn171126_2100.csvn171128_1210.csvn171125_2340.csvn171126_1350.csvn171126_1600.csvn171128_1850.csvn171125_2310.csvn171128_0450.csvn171126_2110.csvn171128_1930.csvn171128_0350.csvn171128_0210.csvn171126_1150.csvn171128_1710.csvn171128_1220.csvn171127_0450.csvn171128_1110.csvn171127_1820.csvn171127_0410.csvn171126_1200.csvn171128_0900.csvn171125_2240.csvn171126_0130.csvn171126_1710.csvn171127_1320.csvn171128_0530.csvn171127_0340.csvn171127_0220.csvn171128_0920.csvn171127_1510.csvn171126_0800.csvn171128_0050.csvn171127_1930.csvn171126_0220.csvn171128_2020.csvn171126_1250.csvn171127_2340.csvn171127_2300.csvn171127_1140.csvn171127_1800.csvn171126_0650.csvn171127_1900.csvn171127_2020.csvn171127_1720.csvn171128_1530.csvn171127_0600.csvn171127_0910.csvn171128_1200.csvn171128_1630.csvn171127_0800.csvn171127_1230.csvn171126_0320.csvn171126_1210.csvn171127_2110.csvn171128_0300.csvn171126_0050.csvn171128_1720.csvn171128_0200.csvn171126_0440.csvn171126_1700.csvn171127_1310.csvn171128_0240.csvn171127_1950.csvn171127_1340.csvn171128_1800.csvn171127_0710.csvn171128_1950.csvn171128_1130.csvn171126_0930.csvn171125_2150.csvn171127_1240.csvn171126_0250.csvn171126_2030.csvn171128_1620.csvn171126_1940.csvn171127_0520.csvn171127_0020.csvn171127_1620.csvn171127_0350.csvn171128_2030.csvn171127_2010.csvn171127_1430.csvn171128_0410.csvn171126_0110.csvn171128_1030.csvn171126_1410.csvn171127_0100.csvn171127_2240.csvn171126_1020.csvn171128_1240.csvn171126_0740.csvn171127_0640.csvn171126_1920.csvn171127_1710.csvn171126_0450.csvn171127_0740.csvn171127_0200.csvn171128_1520.csvn171128_0110.csvn171126_0410.csvn171126_1820.csvn171127_1450.csvn171128_0600.csvn171126_1030.csvn171128_0230.csvn171126_1010.csvn171128_0850.csvn171127_2140.csvn171127_0610.csvn171128_0220.csvn171128_0310.csvn171127_2330.csvn171127_0150.csvn171128_1510.csvn171127_0130.csvn171126_2310.csvn171126_0640.csvn171128_1820.csvn171126_1420.csvn171126_0530.csvn171126_0520.csvn171126_0120.csvn171125_2200.csvn171126_0730.csvn171127_2050.csvn171126_1430.csvn171127_1810.csvn171127_0940.csvn171128_0340.csvn171126_1440.csvn171127_2130.csvn171126_0340.csvn171127_1150.csvn171127_0420.csvn171126_1910.csvn171126_2250.csvn171128_0830.csvn171128_1340.csvn171128_0330.csvn171127_2230.csvn171127_0900.csvn171128_0940.csvn171128_0550.csvn171128_0430.csvn171128_0650.csvn171126_2000.csvn171126_1850.csvn171126_0610.csvn

Strange thing is "n" letter between filenames.

Turbocapitalist · 11-28-2017, 01:45 PM

Quote:

Originally Posted by mackowiakp

Strange thing is "n" letter between filenames.

The \n is somehow not escaped and becoming n instead.

scasey · 11-28-2017, 02:01 PM

Quote:

Originally Posted by Turbocapitalist

Yes, it's correct process substitution

As I said on another post today: I love the learnin' that happens here.

More food for thought: Then maybe escape the < and/or the ( and ) in the substitution, so that rsync doesn't misread them?...as it appears to (may)be causing a syntax error -- and/or quote the clause to the --files-from ?
-- also maybe escape the \ of the \n in the printf portion? "%f\\n" to get the \n to happen in the output.