[SOLVED] Setting up an rsync server: advice needed

wpeckham · 12-26-2023, 12:38 AM

Quote:

Originally Posted by Turbocapitalist

The hard links would still point to the same inode as before. See the exercise in #24 above. Now symbolic links would be another matter, but Rsync produces hard links and not symbolic links so that potential concern would be moot.

Not QUITE exact description, but a good way to think of it.
SOFT links are a directory entry that points to a file or folder elsewhere. Basically a pointer to the other table entry. A HARD link is a total COPY of the table entry so that two table entries do NOT point to each other, each points directly to the data.
If you delete a soft link it is just gone. IF you delete the original file the soft links still exist, but are lost.

If you delete a HARD link you can still address the data as long as just ONE directory entry pointing to the data remains. Deleting any ONE, even the original, has NO effect on the others.

IsaacKuo · 12-26-2023, 07:48 AM

BTW, in case it's not obvious, rsync can happily replicate soft links, even if the soft link is to a file/directory path that does NOT exist on computer running rsync.

So, you do NOT have to worry about a backup being broken because of missing or messed up soft links.

hazel · 12-26-2023, 08:15 AM

Quote:

Originally Posted by IsaacKuo

BTW, in case it's not obvious, rsync can happily replicate soft links, even if the soft link is to a file/directory path that does NOT exist on computer running rsync.

That's because a soft link is just a file containing the name of some other file. Like any file, it has its own inode but it doesn't link directly to the inode of the file that it refers to.
https://www.linuxquestions.org/quest...d-links-39109/

hazel · 01-01-2024, 08:21 AM

So today I stepped through the script. What was good was that it finished in almost no time.

Code:

rsync: [sender] opendir "/home/data/lost+found" failed: Permission denied (13)
kate/
....
sent 319,261 bytes  received 417 bytes  91,336.57 bytes/sec
total size is 3,872,577,308  speedup is 12,113.99
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]

but I still got what looked like a full directory at the other end.

What was not so good was that the directory was given the wrong name, dumps-48 instead of dumps-01. I'll have to check how that happened.

hazel · 01-22-2024, 08:47 AM

I have been running this for a few weeks now and it seems to have shaken down OK but I have run into a problem. Today, it should have done the first remote directory delete:

Code:

[ -f  hazel@littleboy:/home/hazel/Dumps/data-$deleteweek ] && \
   rm -fR hazel@littleboy:/home/hazel/Dumps/data-$deleteweek

It looked (correctly) for data-52 but failed to find it. I did a few manual searches using the same basic code and it looks as if test/[ does not know how to find remote files. Is this the case? If so, how do I proceed?

PS: Looks like the correct code may be "ssh hazel@littleboy [ -f /home/hazel/Dumps/data-$deleteweek ]". I've got the lappy charging upstairs at the moment, but I'll try again later.

michaelk · 01-22-2024, 08:57 AM

If you want to delete files on littleboy from your script you can either run a command over ssh i.e.

Code:

ssh hazel@littleboy "[ -f /home/hazel/Dumps/data-$deleteweek ] && \
   rm -fR /home/hazel/Dumps/data-$deleteweek"

Or use sshfs instead of rsync which is a fuse filesystem to mount a remote share using sftp. And then use rsync as if it were a local directory. I have not tried it so not sure if all of the options work the same...

hazel · 01-22-2024, 09:00 AM

Yup! That was what I thought. Your double quotes, bracketing a compound command to make it a single argument for ssh, have been noted!

hazel · 01-23-2024, 05:32 AM

So I tried that code today and the remote delete command works (if given separately) but the test doesn't, so in practice the directory doesn't get deleted. Now why is that, I wonder? After all the pathname is exactly the same for both command parts.

Return code ($?) for the composite command is 1.

michaelk · 01-23-2024, 06:51 AM

I don't know why but the basic syntax should work. Here is my short script.

Code:

#!/bin/bash
tfile="test.txt"
ssh server "touch $tfile; ls $tfile; [ -f $tfile ] && rm $tfile; ls $tfile"

./test.sh
test.txt
ls: cannot access test.txt: No such file or directory

hazel · 01-23-2024, 10:39 AM

I notice that your script has "server" and not "name@server" like mine. Why and what is the difference?

michaelk · 01-23-2024, 10:51 AM

Sorry, I have a user ssh config file (~/.ssh/config) with the settings for that server with hostname, username, keys etc so I don't have to specify it on the command line. ssh will also default to your client username if not specified.

https://linuxize.com/post/using-the-ssh-config-file/

hazel · 01-24-2024, 11:46 AM

So here are the latest results:

Code:

$ ssh hazel@littleboy "cat /home/hazel/Dumps/test.txt"
XXX               //file listed
$ ssh hazel@littleboy "[ -f /home/hazel/Dumps/test.txt ] && cat /home/hazel/Dumps/test.txt"
XXX                // file found by condition and listed
$ ssh hazel@littleboy "[ -f /home/hazel/Dumps/test.txt ] && \
   rm -fR /home/hazel/Dumps/test.txt"    //file found by condition and deleted (?)

Checked on littleboy: file deleted successfully!

Code:

$ deleteweek=52
$ echo $deleteweek
52
$ ssh hazel@littleboy "[ -f /home/hazel/Dumps/data-$deleteweek ] && \
   rm -fR /home/hazel/Dumps/data-$deleteweek"

data-52 not deleted!

So what is going wrong here?

Turbocapitalist · 01-24-2024, 11:52 AM

Does /home/hazel/Dumps/data-$deleteweek point to a directory or a file? If it is a directory then the test should be -d instead of -f in the conditional statement.

hazel · 01-24-2024, 12:01 PM

Quote:

Originally Posted by Turbocapitalist

Does /home/hazel/Dumps/data-$deleteweek point to a directory or a file? If it is a directory then the test should be -d instead of -f in the conditional statement.

Bless you, I think you've nailed it! Yes, it's a directory and that's why the test fails. I was assuming that "-f" meant "found" but it actually means "[regular]file".

hazel · 01-25-2024, 05:55 AM

Finally got it. As well as using -d for the test, I also need to use ordinary local pathnames on the remote system (i.e. no duplication of "hazel@littleboy" there).

Now I think I am ready to proceed cautiously to Part 2 of the project: dumping my root partitions. I plan to do this while not logged in to the system in question (i.e. dump Slackware out of AntiX and vice versa). That way I won't have to make any special arrangements for excluding dynamic directories.

I understand that I have to be root on the local system (bigboy) because some files are only accessible to root. But can I just be me on the remote system? Or do I have to be root there too? Am I even allowed to be root when working remotely? I don't particularly want to post sshd_config files on a public forum but I can answer questions about what is in them.

And I need to know what ssl keys will be used because I'm rather confused about that. I made a pair of keys for myself at the outset and copied the public key over to littleboy, so that's what I've been using up to now. But do I have to make a different pair for root or can I just copy my personal .ssh directory over? What about the system keys in /etc/ssl? Would they be used automatically for a root ssh? Some pointers would be useful.