LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 06-29-2009, 07:18 PM   #1
mehorter
LQ Newbie
 
Registered: Mar 2008
Location: NJ
Distribution: Suse
Posts: 18

Rep: Reputation: 3
End-of-line Characters missing from last line of md5 file. Md5sum fails


I have a number of md5 files that were either generated on windows machines or transfered in binary mode (bit for bit). As a result the end-of-line characters are formated with CRLF.

Code:
cat -e windows.md5
d4c638e28b1990c55b79e5249cf33038 *testfile_01.ext^M$
9e15845bdae8c1d1699dade90b73c744 *testfile_02.ext
When I translate them with
Code:
tr -d '\r'  <windows.md5> linux.md5
the CR is removed
Code:
cat -e linux.md5
d4c638e28b1990c55b79e5249cf33038 *testfile_01.ext$ 
9e15845bdae8c1d1699dade90b73c744 *testfile_02.ext
and I can use
Code:
md5sum -c linux.md5
with success except for the last file/line to be checked.

My problem is that the last line in the file never had any end-of-line characters to begin with.

As a result, md5sum -c linux.md5 returns an error for the last line.
Code:
md5sum -c linux.md5
testfile_01.ext: OK
md5sum: testfile_02.ext : No such file or directory
testfile_02.ext : FAILED open or read
md5sum: WARNING: 1 of 2 listed files could not be read
If I open the linux.md5 file with EMACS, delete the last "blank" characters of the file then hit return, then save that I find the last line is fixed
Code:
9e15845bdae8c1d1699dade90b73c744 *testfile_02.ext$
and will be used by md5sum correctly.

I would like to be able to script the fix and not have to manually go through every md5 file by hand. It seems like an easy problem but it has got me vexed...

I have tried appending to the file:
Code:
cat >> linux.md5
Blah
^D
but that only returns
Code:
d4c638e28b1990c55b79e5249cf33038 *testfile_01.ext$ 
9e15845bdae8c1d1699dade90b73c744 *testfile_02.ext $
Blah$
Please notice the space between the "testfile_02.ext" and "$".
This returns the same error.

If I could only insert the correct end-of-line character for the last line I think I would be fixed.

Any suggestions?
 
Old 06-29-2009, 08:04 PM   #2
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
To get you started, there is a tool/script already available, and included with many *nix systems, called 'dos2unix' and it has a partner tool called 'unix2dos' and it does the end-of-line conversions on text files. Check Google for it if it is not on your machine.

As to the missing end-of-line on the very last line, I'm not sure if dos2unix would insert one (I haven't used the tool myself) but it wouldn't be too much of a big leap to have a final newline appended to a file that doesn't have one.

Sasha
 
Old 06-29-2009, 08:14 PM   #3
mehorter
LQ Newbie
 
Registered: Mar 2008
Location: NJ
Distribution: Suse
Posts: 18

Original Poster
Rep: Reputation: 3
Quote:
Originally Posted by GrapefruiTgirl View Post
To get you started, there is a tool/script already available, and included with many *nix systems, called 'dos2unix' and it has a partner tool called 'unix2dos' and it does the end-of-line conversions on text files. Check Google for it if it is not on your machine.

As to the missing end-of-line on the very last line, I'm not sure if dos2unix would insert one (I haven't used the tool myself) but it wouldn't be too much of a big leap to have a final newline appended to a file that doesn't have one.

Sasha
Thanks,

I do have dos2unix installed and it gives the same result as
Code:
tr -d '\n' <oldfile> newfile
Both methods fail to leave any kind of EOL character on the final line.

Please make the leap and show me how to have a final newline appended to a file that doesn't have one!

I might also mention that I get the same results if I use vi and use 'set fileform=unix'. Same thing again when i use
Code:
sed -e 's/\r$//' old.md5 > new.md5
Seems like a job for sed or awk but I'm vexed..
 
Old 06-29-2009, 08:25 PM   #4
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
sh# awk '{ printf "\n" >> $0 }' md5sum_file.txt

sh# echo >> md5sums_file.txt

There are a couple methods I just quickly grabbed off the 'net but haven't tested.
In the case of a file that already has a newline, a second one shouldn't hurt (I don't think it will )

Sasha
 
Old 06-29-2009, 08:52 PM   #5
mehorter
LQ Newbie
 
Registered: Mar 2008
Location: NJ
Distribution: Suse
Posts: 18

Original Poster
Rep: Reputation: 3
Thumbs up

Quote:
Originally Posted by GrapefruiTgirl View Post
sh# awk '{ printf "\n" >> $0 }' md5sum_file.txt

sh# echo >> md5sums_file.txt

There are a couple methods I just quickly grabbed off the 'net but haven't tested.
In the case of a file that already has a newline, a second one shouldn't hurt (I don't think it will )

Sasha
Allright!

Both methods work great!
Code:
tr -d '\r' <windows.md5> linux.md5; awk '{ printf "\n" >> $0 }' linux.md5
Code:
tr -d '\r' <windows.md5> linux.md5; echo >> linux.md5
Both methods remove CRLF from all lies and leave all lines with a LF (line feed).

The 2nd method using echo does insert an extra Character but that doesn't seem to effect the function of md5sum.

Thank-you!
 
Old 06-29-2009, 08:56 PM   #6
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
No problem; I learned something myself too: a simple AWK function. I have spent much time learning what I can of GAWK, but AWK has stuff too that can be handy-- if only there weren't SO MANY permutations of AWK/GAWK/SED they would be easier to get a handle on.

Cheers,
Sasha
 
  


Reply

Tags
md5, md5sum


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Attempting to append a line of text to the end of the previous line market_garden Linux - General 4 12-11-2008 11:37 AM
Removing new line characters on every line execpt first line bioinformatics_guy Linux - Newbie 4 10-21-2008 12:41 PM
C: fread to read a file line by line until the end Blue_muppet Programming 2 09-19-2008 09:42 AM
kwrite - show end of line characters? babag Linux - Software 2 06-09-2008 05:04 AM
Shell scripting: How to add characters at the end of the line Micro420 Programming 7 05-18-2007 01:56 AM


All times are GMT -5. The time now is 11:18 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration