jazbinschek

[Log in to get rid of this advertisement]

I have been asked to administer a school's computer. It is running Gentoo Linux. The computer gets a full backup everyday onto tape. The code used to backup the system is:

/usr/sbin/mt -f /dev/nst0 rewind
(cd /; nohup find . home/csci home/students var -print -depth -mount | cpio -ov 2> /root/backup/fullback.$(date +%j).cpio.log | gzip -c > /dev/nst0 &)

The documentation that I recieved said that I could retrieve backed up information by using:

cs mathieu # mt -f /dev/nst0 rewind
cs mathieu # gunzip -c < /dev/nst0 | cpio -ivdm /home/csci/mathieu/1test/* &

After running this, no data is retrieved. Am I doing something wrong or do I have a bad tape?

jlightner

The "-c" means stdout for gzip/gunzip. You're not sending to standard out when you read - you're trying to read from stdin but there is not flag for that in gunzip. Doing a test here I believe the following syntax will work for you:


gunzip -S "" < /dev/nst0 |cpio -ivdm /home/csci/mathieu1test/* &

gunzip expects the file to have a .gz or .z suffixe so the "-S" tells it not to have one. The double-double quotes tells it the suffix is null (i.e. there isn't one) so it will use /dev/nst0. The cpio command tells it to read in /home/csci/matheiu1test directory with its contents.

FYI: You don't need the "&". That sends it into the background - if you leave it out you can watch what it does. If you want to background it because you think it will take a while you need to nohup it like you did the original write. The nohup tells it not to stop the background process if your terminal dies or is logged out.

jazbinschek

I have tested out your suggestion two times and both times it did not retrieve any data. It takes about six hours for each test, that is why it has taken me so long to get back to you.

Is there any way I can test to see if the tape drive is malfunctioning?

jlightner

If the "mt -f /dev/nst0 rewind" is working without error its likely the drive is OK. If it weren't this command would likely hang or say "no such device". I didn't mention it but of course you'd want to do the rewind before you did my command. The rewind you do before starting the backup makes it put start the image at the beginning of the tape so you'd have to do one before the restore so it would start reading from the beginning of the tape.

The command works with a cpio file. Originally I was going to write that gunzip wouldn't know what to do with the tape device but decided not to since my test worked. Now I'm thinking my first thought was right.

All is not lost though. You can use dd to read the tape. If nothing else you can dd the tape into a file:

dd if=/dev/nst0 of=/pathto/filename.cpio (there is a bs= flag you can add often "bs=20b" for 20 byte block size - optimal depends on the kind of drive you're using - running without the bs= may work or prompt you for an optimal block size to use.)

You could then use the command I gave you against the file created:
gunzip -S "" < /pathto/filename.cpio |cpio -ivdm /home/csci/mathieu1test/* &
(all one line)

You could try using it as a pipe instead of creating the file:
dd if=/dev/nst0 | gunzip -S "" |cpio -ivdm /home/csci/mathie1test* &
(all one line).

jazbinschek

This is the result I got.

cs mathieu # mt -f /dev/nst0 rewind
cs mathieu # dd if=/dev/nst0 | gunzip -S "" | cpio -ivdm /home/csci/mathieu/test1/* &
[1] 12844
cs mathieu # dd: reading `/dev/nst0': Cannot allocate memory
0+0 records in
0+0 records out

gunzip: stdin: unexpected end of file
cpio: premature end of archive

[1]+ Exit 1 dd if=/dev/nst0 | gunzip -S "" | cpio -ivdm /home/csci/mathieu/test1/*
cs mathieu #

I also tried “dd if=/dev/nst0 of=/home/csci/mathieu/test1.cpio” and got memory errors too.

Is the problem that the original backup program does not give a filename and extension to the backup tape file? Would it be possible to rewrite the original backup program and put a filename and extension on it? Is there any other way that I could rewrite the original backup program to make this process easier?

At this point in time I am not trying to retrieve any lost data. I am trying to be prepared in case one of my users loses something and wants me to retrieve it. So I have been practicing deleting test files and then trying to retrieve them again.

jlightner

Maybe it's best to see some more detail:

1) What kind of tape drive do you have (not just brand but is it DAT, Ultrium, LTO, QIC)? It's possible the drive allows for hardware compression which would eliminate the need for the gzip/gunzip.

2) Have you verified the drive is being activated (flashing lights , whirring sound) when you issue the rewind? How about when you issue the original command?

3) How much memory do you have? Run "free" and see what it shows.

4) What happens if you do a simple cpio to the drive and cpio back from it without the gzip/gunzip and the rest? Try the following:

cd /tmp
find . -depth -print |cpio -ovBcumd >/dev/nst0

mkdir /tmp/testrestore

cd /tmp/testrestore
cpio -ivBcumd </dev/nst0

Do the contents of /tmp/testrestore now contain what was in /tmp when you started the backup?

5) Have you explored using tar instead of cpio? It has built in flags for gzip/gunzip or compress/uncompress. Haven't used these for tape writes but they may work.

jazbinschek

1. The computer with the tape drive is 10 miles away and locked up for the holidays. If this information is extremely important, I can go through security to allow me access to the machine. All I do know about it is the tapes used. The tape cartridges used are FujiFilm ½” Data Tape 40GB/80GB DLT 8000.

2. The tape drive does work. Over Christmas I made a tar ball of a directory and recovered it perfectly.

3. total = 1032332 used = 1018932 free = 13400 shared = 0 buffers =82384 cached = 127904

Swap: total = 2048276 used = 45520 free = 2002756


4. I had to modify it a little but the statement below worked.
cd /tmp
find . -depth -print | cpio -ovBc > /dev/nst0
mkdir /tmp/testrestore
mt –f /dev/nst0 rewind
cd /tmp/testrestore
cpio -ivBc < /dev/nst0

5. Yes, it looks like I will have to look into tar.

jlightner

Brand wasn't important. DLT was the information I sought.

Well cpio to and from the drive works. The command I originally gave you works to and from a file so there appears to be an issue with the way it interacts with the tape drive. Doing the dd was a way to try to get around this and you're likely running out of memory because the dd is putting all of the read into memory before piping it so that isn't the way to go.

DLT does have a hardware compression mechanism. (The 40GB is uncompressed and the 80GB is compressed.) Therefore you don't need to (and shouldn't) do compression at the software level. Software compression can actually increase the amount of data backed up.

According to the link at:
http://www.ofb.net/~jheiss/perftune_backups.shtml

"Drive compression
Linux defaults to enabling drive compression. Drive compression can be disabled by running mt -f /dev/tape-device compression 0. See the mt(1) man page for more info."

This means writing to /dev/nst0 IS doing it in compressed mode. (I had to look this up as on UNIX there is often a separate device for compressed and uncompressed modes. Apparently in Linux they use the same device and you need the mt command to change which way it does it.)

Also while looking I saw several comments that one can use "tar zcf ..." but then recommending AGAINST the z (or any other software compression) because if one bit gets hosed on the backup then the entire backup is useless. Specifically I saw recommendations saying not to use this on a DLT on Linux.

Given all the above I'd say you are best just piping your output into cpio as you did in your test. You'll have the best compression that way and since its at the hardware level shouldn't have the issues noted above.

P.S.
Final thought - Do "ls -l /dev/nst0" to verify it is in fact a device and not a regular file:
crw-rw---- 1 root tape 9, 128 Mar 14 2002 /dev/nst0

The c at the beginning indicates its a characer (raw) device. The "9, 128" before the date is the major and minor number.

Since cpio and tar are quite happy writing to files the fact that you're able to read what you wrote doesn't prove its actually going to tape. If for some reason the device didn't exist it would just create a file by this name - you'd know because it wouldn't have a c in the first position and the number in front of the date would be a byte count without a comma.

jazbinschek

Thank you very much for all your help. With your help I have decided not to continue using the apparently flawed backup program and I have started writing my own. I know now that compressing the files using gzip is unnecessary and may be the cause of my problems. I am planning on taking the next couple of days to run some tests using tar and maybe cpio without the gzip command.

Your assistance has been very valuable and appreciated.

Thank you very much.