SSH -rsync

kzo81 · 03-19-2024, 05:32 AM

Hi Folks,

I encountered a strange problem. There is a video encoder that constantly records files, and I created this syncroniser script.
If I --remove-source-files parameter to rsync it senses that a file is beeing written by another process, and skips that file. I tested it on my laptop and it really skips it, but not always. On the server side some files are truncated to 828 bytes. Now I user it without that option, but I would really like to remove the files after sync:

Code:

#!/bin/sh
set -x

MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')
CONFIG="/usr/bin/syncroniser.cfg"
KEY="/usr/bin/dropbear_rsa_host_key"
SRC="/media/sdmmc/mmcblk0p1/RECORD/"
DST="$MAC"
FREQ=60
while true
    do
        if test -f "$CONFIG"; then
            . "$CONFIG"
        fi
        rsync -avme "ssh -i $KEY -p $REMOTE_PORT" $SRC $REMOTE_USER@$REMOTE_IP:$DST 
        sleep $FREQ
    done

I was trying to pipe to rsync with find and xargs, but it still did that file truncation on the file beeing written.

Code:

find $SRC -type f -exec sh -c 'if ! lsof `readlink -f {}` > /dev/null; then echo `basename {}`; fi' \; | tr '\n' '\0' | rsync -avme "ssh -i $KEY -p $REMOTE_PORT" $SRC $REMOTE_USER@$REMOTE_IP:$DST

Turbocapitalist · 03-19-2024, 05:52 AM

There's a lot which appears undefined in the script as presented. The variables $REMOTE_PORT, $REMOTE_USER, $REMOTE_IP, and $DST seem undefined. The variable $MAC seems unused but if you are going to fetch it using the formula which you have given, then it can all be done in AWK:

Code:

MAC=$(ip a | awk '/link\/ether/ { gsub(":", "-", $2); print $2; exit; };')

The ip utility can take the name of the specified interface as an option too.

pan64 · 03-19-2024, 06:20 AM

Do you mean you want to copy a file which is actually being written? That won't work, you cannot make it work. You need to skip that file. Next time you can copy/move it when it's saved and ready.

Instead of

Code:

echo `basename {}`

you can simply write

Code:

basename {}

Also would be nice to use shellcheck to validate your script.

kzo81 · 03-19-2024, 07:17 AM

This is shell checked :-)

Code:

set -x

MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')
CONFIG="/usr/bin/syncroniser.cfg"
KEY="/usr/bin/dropbear_rsa_host_key"
SRC="/media/sdmmc/mmcblk0p1/RECORD/"
DST="$MAC"
FREQ=60
while true
    do
        if test -f "$CONFIG"; then
            . "$CONFIG"
        fi
        find $SRC -type f -exec sh -c 'if ! lsof `readlink -f {}` > /dev/null; then basename {}; fi' \; | tr '\n' '\0' | rsync -avme "ssh -i $KEY -p $REMOTE_PORT" "$SRC" "$REMOTE_USER"@"$REMOTE_IP":"$DST" 
        sleep $FREQ
    done

kzo81 · 03-22-2024, 05:04 AM

Hi Folks,

Do you have idea why I cannot use this script with systemD anymore? It was running seamlessly before I added the red line.

Code:

#!/bin/sh
set -x

MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')
CONFIG="/usr/bin/syncroniser.cfg"
KEY="/usr/bin/dropbear_rsa_host_key"
SRC="/media/sdmmc/mmcblk0p1/RECORD/"
EXPORTS="/media/sdmmc/mmcblk0p1/RECORD/EXPORTS/"
DST="$MAC"
FREQ=60

while true
    do
        if test -f "$CONFIG"; then
            . "$CONFIG"
        fi

        mkdir -p "$EXPORTS";

        find $SRC -name *.mkv -mmin +1 -exec mv "{}" "$EXPORTS" \;
        rsync --remove-source-files -avme "ssh -i $KEY -p $REMOTE_PORT" $EXPORTS $REMOTE_USER@$REMOTE_IP:$DST        

        sleep $FREQ

    done

michaelk · 03-22-2024, 08:08 AM

Code:

 find $SRC -name *.mkv -mmin +1 -exec mv "{}" "$EXPORTS" \;

What exactly isn't working correctly? What sticks out is that by default find is recursive. I have not tried testing your code but try adding -maxdepth 0 and you need quotes around *.mkv

lvm_ · 03-22-2024, 12:19 PM

If your video encoder runs on the same linux machine, why don't you hook you script to it - via inotifywait, if there are no more straightforward means. And if it is a separate device which exposes its storage to linux as memory card, lsof trick won't work as it cannot see files opened there. As for that 'find $SRC -name *.mkv...' line, you have to enquote the file mask, otherwise it will be expanded prematurely.