LinuxQuestions.org - SSH -rsync

- Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)

- - SSH -rsync (https://www.linuxquestions.org/questions/linux-software-2/ssh-rsync-4175735062/)

Hi Folks,

I encountered a strange problem. There is a video encoder that constantly records files, and I created this syncroniser script.
If I --remove-source-files parameter to rsync it senses that a file is beeing written by another process, and skips that file. I tested it on my laptop and it really skips it, but not always. On the server side some files are truncated to 828 bytes. Now I user it without that option, but I would really like to remove the files after sync:

Code:

#!/bin/sh

set -x



MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')

CONFIG="/usr/bin/syncroniser.cfg"

KEY="/usr/bin/dropbear_rsa_host_key"

SRC="/media/sdmmc/mmcblk0p1/RECORD/"

DST="$MAC"

FREQ=60

while true

    do

        if test -f "$CONFIG"; then

            . "$CONFIG"

        fi

        rsync -avme "ssh -i $KEY -p $REMOTE_PORT" $SRC $REMOTE_USER@$REMOTE_IP:$DST 

        sleep $FREQ

    done

I was trying to pipe to rsync with find and xargs, but it still did that file truncation on the file beeing written.

Code:

find $SRC -type f -exec sh -c 'if ! lsof `readlink -f {}` > /dev/null; then echo `basename {}`; fi' \; | tr '\n' '\0' | rsync -avme "ssh -i $KEY -p $REMOTE_PORT" $SRC $REMOTE_USER@$REMOTE_IP:$DST

There's a lot which appears undefined in the script as presented. The variables $REMOTE_PORT, $REMOTE_USER, $REMOTE_IP, and $DST seem undefined. The variable $MAC seems unused but if you are going to fetch it using the formula which you have given, then it can all be done in AWK:

Code:

MAC=$(ip a | awk '/link\/ether/ { gsub(":", "-", $2); print $2; exit; };')

The ip utility can take the name of the specified interface as an option too.

Do you mean you want to copy a file which is actually being written? That won't work, you cannot make it work. You need to skip that file. Next time you can copy/move it when it's saved and ready.

Instead of

Code:

echo `basename {}`

you can simply write

Code:

basename {}

Also would be nice to use shellcheck to validate your script.

This is shell checked :-)

Code:

set -x



MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')

CONFIG="/usr/bin/syncroniser.cfg"

KEY="/usr/bin/dropbear_rsa_host_key"

SRC="/media/sdmmc/mmcblk0p1/RECORD/"

DST="$MAC"

FREQ=60

while true

    do

        if test -f "$CONFIG"; then

            . "$CONFIG"

        fi

        find $SRC -type f -exec sh -c 'if ! lsof `readlink -f {}` > /dev/null; then basename {}; fi' \; | tr '\n' '\0' | rsync -avme "ssh -i $KEY -p $REMOTE_PORT" "$SRC" "$REMOTE_USER"@"$REMOTE_IP":"$DST" 

        sleep $FREQ

    done

Hi Folks,

Do you have idea why I cannot use this script with systemD anymore? It was running seamlessly before I added the red line.

Code:

#!/bin/sh

set -x



MAC=$(ip a | grep ether | head -1 | awk '{print $2}' | sed 's/:/-/g')

CONFIG="/usr/bin/syncroniser.cfg"

KEY="/usr/bin/dropbear_rsa_host_key"

SRC="/media/sdmmc/mmcblk0p1/RECORD/"

EXPORTS="/media/sdmmc/mmcblk0p1/RECORD/EXPORTS/"

DST="$MAC"

FREQ=60



while true

    do

        if test -f "$CONFIG"; then

            . "$CONFIG"

        fi



        mkdir -p "$EXPORTS";



        find $SRC -name *.mkv -mmin +1 -exec mv "{}" "$EXPORTS" \;

        rsync --remove-source-files -avme "ssh -i $KEY -p $REMOTE_PORT" $EXPORTS $REMOTE_USER@$REMOTE_IP:$DST        



        sleep $FREQ



    done

Code:

find $SRC -name *.mkv -mmin +1 -exec mv "{}" "$EXPORTS" \;

What exactly isn't working correctly? What sticks out is that by default find is recursive. I have not tried testing your code but try adding -maxdepth 0 and you need quotes around *.mkv

If your video encoder runs on the same linux machine, why don't you hook you script to it - via inotifywait, if there are no more straightforward means. And if it is a separate device which exposes its storage to linux as memory card, lsof trick won't work as it cannot see files opened there. As for that 'find $SRC -name *.mkv...' line, you have to enquote the file mask, otherwise it will be expanded prematurely.