problem with tar over ssh

Guttorm · 01-22-2007, 04:36 AM

Hi

I have set up a little script to make a backup of the /etc directory - using tar and ssh to make the backup over ssh (with key based authentication).

The same command works on some other computers but not one running Debian Sarge.

It's basically this command:
tar -cf "username@computername:~/brannsafe/etc.tar" --preserve-permissions --same-order --rsh-command=/usr/bin/ssh etc/

The idea is to put a backup a directory called "brannsafe" - this is then copied to tape and put in a fireproof safe the next day.

When I run it, I get the error message:

bash: line 1: /usr/libexec/rmt: No such file or directory
tar: username@computername\:~/brannsafe/etc.tar: Cannot open: Input/output error
tar: Error is not recoverable: exiting now

I tried with rsync too - it gave a similar error message.

I managed to get a backup using scp
scp -r -q /etc username@computername:~/brannsafe/
This works fine, but the permissions are lost because username@computername is a regular user on the remote computer. Also all symlinks are lost.

I know I could tar locally and then scp the tar file over, but I was thinking of making a backup of a lot more than /etc - and it would be a lot better if tar command worked.

There is no file "/usr/libexec/rmt" - I don't know what it is. Am I maybe missing some package?

kees-jan · 01-22-2007, 06:21 AM

Hi,

For extra readability, you might want to put commands between "CODE" tags

Quote:

Originally Posted by Guttorm

It's basically this command:

Code:

tar -cf "username@computername:~/brannsafe/etc.tar" --preserve-permissions --same-order --rsh-command=/usr/bin/ssh etc/

I cannot see anything obviously wrong with this, but I've never tried it quite like this myself.

Quote:

Originally Posted by Guttorm

When I run it, I get the error message:

Code:

bash: line 1: /usr/libexec/rmt: No such file or directory

This suggests that on line 1 of your script. there is a call to /usr/libexec/rmt. I do not have that binary on either ubuntu (6.06) or debian (sarge), nor is it available in the repositories I have enabled.

Where did you get this binary? On which distribution do you have it available? What are you using it for?

Quote:

Originally Posted by Guttorm

Code:

tar: username@computername\:~/brannsafe/etc.tar: Cannot open: Input/output error
tar: Error is not recoverable: exiting now

Just out of curiousity: Why is there a backslash before the colon? As far as I know, the colon does not need escaping, and hence there will be a backslash at the end of your hostname, possibly confusing the hell out of things.

Again out of curiousity, what happens when you try this variation of the same theme?

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of=~/brannsafe/etc.tar

Groetjes,

Kees-Jan

Dutch Master · 01-22-2007, 06:37 AM

The rmt program is part of the tar package. According to the Debian-packages page it should be located in /etc/rmt and /sbin/rmt.

Mind you: if you execute the above as a script you should (read as: must) add

Code:

#! /bin/bash

to the first line of your script! Learn about scripting here

unSpawn · 01-22-2007, 06:40 AM

ssh username@computername dd of=~/brannsafe/etc.tar
If you don't mind me asking, why the dd? I use:

Code:

tar -cf - /some/dir | ssh remote_ssh_username@remote_host 'cat > /storage_dir/archive.tar'

.

For me rsync over SSH works fine like this:

Code:

rsync -ave "ssh -l remote_ssh_username" --rsh=ssh /home/dir remote_username@remote_host:/storage_dir/

Guttorm · 01-22-2007, 07:00 AM

Hi again.

First of all, sorry for not using code tags - I should have thought of that.

I tried your command:

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of=~/brannsafe/etc.tar
dd: opening `/root/brannsafe/etc.tar': No such file or directory

It looks like it is trying to put the tar file in /root/brannsafe/etc.tar on the local computer and not the remote one.

As for the error message:

Code:

bash: line 1: /usr/libexec/rmt: No such file or directory

I get it when I run the tar command from the shell - not using any scripts. I have no /usr/libexec/rmt file on the computer - that's why I thought I might be missing some package.

Just now I tried

Code:

ssh username@computername ls

It works like it should - listing the files in the home directory as well as showing me that the directory "brannsafe" exists and is ok.

And the error message:

Code:

tar: username@computername\:~/brannsafe/etc.tar: Cannot open: Input/output error
tar: Error is not recoverable: exiting now

I did not put the "\" character before the ":". That's why I tried to add quotes on that argument, I thought the shell could escape something, but quotes didn't help.

Now I also tried:

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computer dd of=~/etc.tar
dd: opening `/root/etc.tar': Permission denied

It seems it is trying to write to the home directory of the local computer, not the remote one. Really strange, I can't figure out why.

Guttorm · 01-22-2007, 07:16 AM

Hi again

Update:

I didn't see the other answes while I was writing my reply.

Code:

tar -cf - etc/ | ssh username@remotecomputer 'cat > ~/brannsafe/etc.tar'

Worked great! Thank you very much.

I have no idea why the other commands will not work on this computer - but my problem is solved and I am very happy.

unSpawn · 01-22-2007, 07:35 AM

I am very happy
Still a minor nit: "~/brannsafe/etc.tar" isn't scalable, maybe first set a string like:

Code:

BK_STR="`hostname -s`_`date '+%Y%m%d'`"
tar -cf - etc/ | ssh username@remotecomputer 'cat > ~/brannsafe/${BK_STR}_etc.tar'

so a listing should give you something like:
"~/brannsafe/frob_20061101_etc.tar"
"~/brannsafe/darkstar_20070121_etc.tar"
which is more efficient to look for if you plan on using this on more hosts.

Guttorm · 01-22-2007, 07:44 AM

Hi

Thanks for the tip.

However, I use a separate account for each remote computer on the destination computer with the tape drive. That way, if someone hacks some of the computers, all they can see is a backup of the computer they hacked - not the other ones. (Unless they manage to escalate privileges on the backup server) I've also set up firewall rules so you cannot ssh to the computer with the tape drive from anywhere except the clients needing backup.

As for dates, everything at /home/*/brannsafe/* is backed up to tape every day and rotated after a while, so I can figure out which date it is by looking at the tape.

If not for this, using filenames with hostname and date would be a good idea.

kees-jan · 01-22-2007, 08:07 AM

Quote:

Originally Posted by unSpawn

ssh username@computername dd of=~/brannsafe/etc.tar
If you don't mind me asking, why the dd? I use:

Code:

tar -cf - /some/dir | ssh remote_ssh_username@remote_host 'cat > /storage_dir/archive.tar'

.

There is no good reason. The two are equivalent. Just use whatever you prefer.

The only thing is that with the 'cat > /storage/dir/archive.tar' you have to be very careful with the quotes. If you forget them, the archive will be created on your local dir after having traversed the network twice.

Using dd of=/storage/dir/archive.tar, you are less sensitive to mistakes in quoting. dd will always create the file on the machine it runs on (i.e. remote)

Groetjes,

Kees-Jan

kees-jan · 01-22-2007, 08:18 AM

Quote:

Originally Posted by Guttorm

Hi again.

First of all, sorry for not using code tags - I should have thought of that.

I tried your command:

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of=~/brannsafe/etc.tar
dd: opening `/root/brannsafe/etc.tar': No such file or directory

It looks like it is trying to put the tar file in /root/brannsafe/etc.tar on the local computer and not the remote one.

My mistake. I did some digging in the bash man page (interesting reading, if you are sort of a masochist). Tildes are expanded on the local machine, instead of the remote one, unless they are quoted with either double or single quotes.

So an improved version may be

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of="~/brannsafe/etc.tar"

or

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of=/absolute/path/to/brannsafe/etc.tar

There goes my argument of dd being less sensitive to quoting errors

Groetjes,

Kees-Jan

unSpawn · 01-22-2007, 08:34 AM

Aw crap, and I just prepared my similar reply ;-p

unSpawn · 01-22-2007, 08:39 AM

interesting reading, if you are sort of a masochist
..and I think *that* is totally uncalled for!
Everybody knows masochists use Csh :-]

Guttorm · 01-22-2007, 09:05 AM

Code:

tar -cf - --preserve-permissions --same-order etc/ | ssh username@computername dd of=/absolute/path/to/brannsafe/etc.tar

This worked too - except dd gives some output to stderr:
7440+0 records in
7440+0 records out
3809280 bytes transferred in 0.539044 seconds (7066733 bytes/sec)

Since this will run under cron, I want error messages to be emailed, so I prefere not to pipe stderr to /dev/null, and I couldn't find a "quiet" option to the dd command. So I'll use the cat version.

Quote:

Tildes are expanded on the local machine, instead of the remote one, unless they are quoted with either double or single quotes.

Thank you, that was most likely my mistake. What I can't figure out still is why this command works:

Code:

scp somefile username@computername:~

kees-jan · 01-22-2007, 09:19 AM

Quote:

Originally Posted by Guttorm

Thank you, that was most likely my mistake. What I can't figure out still is why this command works:

Code:

scp somefile username@computername:~

Again quoting the bash manpage, tilde expansion is only done when the tilde appears at the beginning of a word. In this example, that's not the case.

Groetjes,

Kees-Jan

Guttorm · 01-22-2007, 01:00 PM

Ah. That explains it. Thanks again for the help.