Hi,

I am attempting to write some data to tape drive.

The data is written using the following comamand:

dd if=file bs=8000 cbs=80 of=/dev/nst0

According to the spec I need to follow the at the end of each file there must be a "IBM end of file mark" on the tape. Then the next file. After the last file on the tape there should be 2 "IBM end of file marks".

The files consist of 80byte records and the final record in the file is "EOF".

What I would like to know is how to write two consecutive "IBM end of file marks". I am assuming by running the dd command the tape drive will write an "IBM end of file mark" after it has finished writing the file.

I am a little confused by so called logical EOF (the ASCII record containig "EOF" chars) and the physical EOF on the tape media.

Are there other terms such as EOT EOD etc... I have seen that by cycling through the files I have written to tape I can get :

mt -f /dev/nst0 status

to report each file end and finally a "EOD" which I presume is End of Device.

Thanks.

Steve

I can give you a partial answer to your question.

"I am a little confused by so called logical EOF (the ASCII record containig "EOF" chars) and the physical EOF on the tape media."

The logical EOF is in ASCII and can be read by an ordinary read. The physical EOF is an electronic mark on the tape that does not correspond to any of the 256 possible byte values. When you read a tape mark no data is transferred to memory. Instead the device returns status that indicates you have hit a tape mark.

"What I would like to know is how to write two consecutive "IBM end of file marks". I am assuming by running the dd command the tape drive will write an "IBM end of file mark" after it has finished writing the file."

dd will not write a tape mark. If you are running on an IBM machine then close will write 2 tape marks at the end of the file.

---------------
Steve Stites

Hi,

Thanks very much for the reply.

I am running on Red Hat Enterprise Linux 4 update 4 x86-64.

The tape drives in question are IBM 3590 3592 types.

Could you please explain how to write the IBM end of file mark in linux?

I was assuming that after I finished writing a file in dd. The tape drive itself would write the IBM end of file mark. It seems I may have misunderstood the situation.

My original plan was to just get dd to write nothing which would result in an IBM EOF mark. Therefore I would get 2 in a row after the last file had been written.

Thanks,

Steve

"I was assuming that after I finished writing a file in dd. The tape drive itself would write the IBM end of file mark. It seems I may have misunderstood the situation."

I have had experience writing IBM tape labels and tape marks on non-IBM machines but never with any type of PC. I have also never done it under Linux. That said, I can tell you how I would approach the problem.

The hardware will not write a tape mark on its own initiative. Software must tell the tape drive to write a tape mark. I doubt that dd will write a tape mark. Writing a tape mark would definitely be an optional feature and I saw no such option in man dd.

I found some Linux software which will write IBM labels. See:
man tmlabel

http://techpubs.sgi.com/library/tpl/...cat8/tmlabel.z

tmlabel is part of some software called Tape Management Facility. I suggest that you look through the programs in TMF and see if you can find one that will write 2 IBM tape marks at the end of a file.

Another thought:
Since the tape drives are IBM drives close probably writes an IBM tape mark. You might be able to close the file after the dd write without rewinding the tape. Then open and close a null file to try to put another tape mark after the first tape mark.

------------------
Steve Stites

Hi,

thanks for the replies.

Here is the exact file description that I need to adhere to:

File Description:
A tape or disk may contain one or more files depending on the type of survey. Each file is started by a
number of "Reader Records" followed by data records and closed by an EOF statement in col 1-3 of the
final record.
A tape file is closed by one IBM file mark.
A tape must be closed by two IBM file marks.

I looked at your link but I am sorry to say I did not really understand the content. However I re-read your original post where you wrote "The logical EOF is in ASCII and can be read by an ordinary read. The physical EOF is an electronic mark on the tape that does not correspond to any of the 256 possible byte values. When you read a tape mark no data is transferred to memory. Instead the device returns status that indicates you have hit a tape mark."

I do not understand what the physical EOF mark is, if it does not correspond to any of hte possible byte values. Could you explain a little? Perhaps you ahve a link to some information on this kind of topic?

Based on your information I assume that for a tape drive to write a physical EOF mark I presume it does so by being told to do so, I.e. by being sent a SCSI command to do so.

I found this link: http://home.comcast.net/~SCSIguy/SCS...si-summary.doc

Which details a table with this sequential access command defined:

SCSI – Sequential Access
Command Opcode Scu Command

WRITE FILEMARKS 10h mt { eof | weof } [ count ]

And also this link: http://www.computerhope.com/unix/umt.htm

Which details the linux "mt" command a little more.

It seems that

==> mt -f /dev/nst0 eof

will write a eof mark

==> mt -f /dev/nst0 eof 2

will write 2 eof marks.

Their is also an weof option. I am not sure what the difference is, perhaps the are equivalent.


If I amend my software to write each file like this:

for file=1 to n

dd if=file_n bs=8000 cbs=80 of=/dev/nst0
mt -f /dev/nst0 eof

end
mt -f /dev/nst0 eof

I assume I should get the result I want. Does this sound ok?

The problem I have is that I have no standard compliant tape to reverse engineer or software to test my software against.

What do you think? Am I on the right track here? the mt command is written by Kai Makisara, I wonder if he might reply if I sent him such a "newbie" question on email?

Thanks,

Stvy

Hi,

A couple more questions.

Now I know what the difference between the logical EOF and the physical EOF.

Is there such as thing as a logical EOT (end of tape), EOD (end of device) etc? Do these have physical equivalents?

What does the command

==> mt -f /dev/nst0 eod,seod

actually do? Does it write or look for some physical mark on the tape? I.e. a physical EOD equiavelent to the physical IBM EOF mark?

Is this equivalent to 2 file end marks? or is it a separate entity?

Thanks,

Steve

"Based on your information I assume that for a tape drive to write a physical EOF mark I presume it does so by being told to do so, I.e. by being sent a SCSI command to do so."

Yes.

"If I amend my software to write each file like this:

for file=1 to n

dd if=file_n bs=8000 cbs=80 of=/dev/nst0
mt -f /dev/nst0 eof

end
mt -f /dev/nst0 eof

I assume I should get the result I want. Does this sound ok?"

This sounds like you are on the right track. One potential problem is if either dd or mt does a rewind you may end up writting tape marks and/or files over the first file. mt has a rewind option. You don't want to rewind until you are completely finished writting files and tapemarks.

-----------------------
Steve Stites

Hi,

Thanks so much for your replies.

However I have been thinking about this today (unfortunately I was not able to test on a host with a tape drive yet).

However from memory i will try and explain why I ended up misunderstanding what was happening with the way dd worked writing 1 file at a time to the tape device.

Say for example I write two files using the no rewind device driver to a completely empty tape that is ready at load point.

The two files consist of two lines each, 80 char (white space padded) records.

The first file is:

This is file 1.
EOF


The 2nd file is:

This is file 2.
EOF


I write the two files to tape as follows:


==> dd if=file_1 bs=8000 cbs=80 of=/dev/nst0

when that completes:

==> dd if=file_2 bs=8000 cbs=80 of=/dev/nst0

Then I do the eof trick with mt:

==> mt -f /dev/nst0 eof


Ok so what I should have is 2 files that are not separated by an EOF file mark, but that have an EOF file mark after the 2nd file.

Next I rewind the tape.

==> mt -f /dev/nst0 rewind

Now I will try and read back but unfortunately I no longer remember what I have written to the tape. So I decide to try and read back one "file" at a time.

==> dd if=/dev/nst0 bs=8000 cbs=80 of=/tmp/file_1

I wait for dd to complete and then I less the file_1. I get the following result:

This is file 1.
EOF

2 records.

If I do a

==> mt -f /dev/nst0 status

I find that the tape device is at file 1, at EOF.

Next I try and read another file:

==> dd if=/dev/nst0 bs=8000 cbs=80 of=/tmp/file_2

I again wait for dd to complete and then I less the file_2. I get the following result:

This is file 2.
EOF

2 records.

If I do a

==> mt -f /dev/nst0 status

I find that the tape device is at file 2, at EOF.


So my question is this; how does dd know where the end of the file is, if it (or the generic sg/st scsi tape device driver in linux) has not written an EOF mark on the tape. If dd did not write (or the tape drive sg/st driver automatically do it) an EOF mark surely I would have read back the first file and go the following result with a less:

This is file 1.
EOF
This is file 2.
EOF

Total 4 records.


Please if you could explain this I think that I understand the situation well enough to complete my task.


Tomorrow I will run this test again and log the terminal so that I can post the report from ==> mt -f /dev/nst0 status as I go along through the test.


Cheers,

Steve

"So my question is this; how does dd know where the end of the file is, if it (or the generic sg/st scsi tape device driver in linux) has not written an EOF mark on the tape."

When a program writes a file to tape and then closes the file the close routine writes a tape mark.

---------------------
Steve Stites

Hi,

Here is the log of me reading back a tape. I have written 2 files to the tape. I then issues a "mt -f /dev/nst0 eof" command:

/tmp% mt -f /dev/nst0 status We are at the beginning of the tape and have not read anything.

SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x2a (no translation).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN

/tmp% dd if=/dev/nst0 cbs=80 bs=8000 of=file_1 - We have just read back the first file

9+1 records in
9+1 records out

/tmp% mt -f /dev/nst0 status - The /dev/nst0 reports that it is positioned after file 1 physical EOF mark.

SCSI 2 tape drive:
File number=1, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x2a (no translation).
Soft error count since last status=0
General status bits on (81010000):
EOF ONLINE IM_REP_EN

/tmp% dd if=/dev/nst0 cbs=80 bs=8000 of=file_2 - We have just read back the second file

9+1 records in
9+1 records out

/tmp% mt -f /dev/nst0 status - The /dev/nst0 reports that it is positioned after file 2 physical EOF mark.

SCSI 2 tape drive:
File number=2, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x2a (no translation).
Soft error count since last status=0
General status bits on (81010000):
EOF ONLINE IM_REP_EN


/tmp% dd if=/dev/nst0 cbs=80 bs=8000 of=file_3 We have just read back the 3rd file

0+0 records in
0+0 records out


This file is of size zero bytes. This is I presume the IBM physical EOF mark only. Hence the zero byte size.


/tmp% mt -f /dev/nst0 status The /dev/nst0 reports that it is positioned after file 3 physical EOF mark.

SCSI 2 tape drive:
File number=3, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x2a (no translation).
Soft error count since last status=0
General status bits on (81010000):
EOF ONLINE IM_REP_EN

/tmp% dd if=/dev/nst0 cbs=80 bs=8000 of=file_4 We have just read back the fourth file

0+0 records in
0+0 records out

Again this file is of size zero bytes. This is therefore I presum another IBM physical EOF mark only. Is this a valid assumption? Is there any way I can check this?

/tmp% mt -f /dev/nst0 status The /dev/nst0 reports that it is positioned after file 4 physical EOF mark.

SCSI 2 tape drive:
File number=4, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x2a (no translation).
Soft error count since last status=0
General status bits on (9010000):
EOD ONLINE IM_REP_EN

/tmp% dd if=/dev/nst0 cbs=80 bs=8000 of=file_5 - This final read fails. This is because their is no file to read as we have tried to read past the EOD (end of device).

dd: reading `/dev/nst0': Input/output error
0+0 records in
0+0 records out

/tmp%



Now to my mind this is what is on the tape:

FILE 1: file1, with IBM EOF mark at the end.
FILE 2: file2, with IBM EOF mark at the end.
FILE 3: IBM EOF mark only
FILE 4: Assuming an IBM EOF mark only


Now I know that I have manually created files 1,2 & 3. However FILE 4 seems to have been automatically created by the SCSI driver.

Does this double IBM physical EOF mark cause the "mt -f /dev/nst0 status" command to return "EOD" status.


Do those 4 files adhere to this specification:

A tape file is closed by one IBM file mark.
A tape must be closed by two IBM file marks.


Or do I have one extra IBM file mark than I should have.

Thanks,

Stvy

"Do those 4 files adhere to this specification:

A tape file is closed by one IBM file mark.
A tape must be closed by two IBM file marks.


Or do I have one extra IBM file mark than I should have."


The files:

FILE 1: file1, with IBM EOF mark at the end.
FILE 2: file2, with IBM EOF mark at the end.
FILE 3: IBM EOF mark only

are two files in IBM format. The FILE 4 EOF mark is just garbage on the unused portion of the tape.

--------------------
Steve Stites