LinuxQuestions.org - Shell Scripting: Strange Problem with Text Files

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Shell Scripting: Strange Problem with Text Files (https://www.linuxquestions.org/questions/programming-9/shell-scripting-strange-problem-with-text-files-425445/)

Shell Scripting: Strange Problem with Text Files

I've got a problem with the following (simplified) script:

while read line
do
echo $line
done < MYFILE

When I create MYFILE with Kate or KWrite like this

One
Two
Three

I get the following output from my script:

One
Two

-> One line is missing! However, when I create the same file in Vim, the output is OK:

One
Two
Three

When I open the file I created in Vim with KWrite or Kate, it shows an empty line at the end that should not be there:

1: One
2: Two
3: Three
4:

It seems that when I use Kate or Kwrite, I have to end every file with an empty line for my shell script to read it. Why is this so? Please enlighten me! :confused:
The problem is I have this huge stack of text files created with Kate that I need to process line by line with my script...

Thanks,
MOO-COW

The "extra line" should be there - it is not a line, it's just the way vi shows that a newline exists at the end of the previous line.

Your bash problem is that the last line does not end with a newline.

This can kill you when you edit files like /etc/fstab - every line HAS to have a newline in order to be seen. Lots of new unix users run afoul of this.

OK, then I have to check for each file whether it ends with a newline character and, if not, append one. Any idea how to do that in Bash?

You would first check the last character of the file to see if it is a newline:
tail -c1 file|grep "^$"
If it is not a newline then you can add newline at the end of a file like this:
echo >>file

So you could put it all together like this:

Code:

#!/bin/bash

for i in *;do

  tail -c1 $i|grep "^$"

  if [ "$?" = "1" ]; then #If $? = 1 then grep failed

    echo >>$i            #So we add a newline to the end of the file

  fi

done

The above checks all files in the current directory, you may need to modify that.

That's exactly what I needed... and so short & elegant!
Thanks very much :)

#!/bin/bash
for i in *;do
tail -c1 $i|grep "^$"
if [ "$?" = "1" ]; then #If $? = 1 then grep failed
echo >>$i #So we add a newline to the end of the file
fi
done

I ran the above program. Nothing happened.

What does it check?

What is the line '' tail -c1 $1 | grep "$" " is doing here ?
I can't understand it.
---------------------------------------------------------------------------

I created a file using the '' vi '' editor and named '' unwanted1 ''.

Some of the output is here.

[nissanka@c83-250-107-194 ~]$ ./unwanted1

tail: error reading `Applications': Is a directory
./unwanted1: line 10: Applications: Is a directory

tail: cannot open `The' for reading: No such file or directory
tail: cannot open `Holy' for reading: No such file or directory
tail: cannot open `Family\'s' for reading: No such file or directory
tail: cannot open `flight' for reading: No such file or directory
tail: cannot open `to' for reading: No such file or directory
tail: cannot open `Egypt' for reading: No such file or directory
./unwanted1: line 10: 1: ambiguous redirect
tail: cannot open `The' for reading: No such file or directory
tail: cannot open `Portugese' for reading: No such file or directory
tail: cannot open `advent' for reading: No such file or directory
tail: cannot open `in' for reading: No such file or directory
tail: cannot open `Sri' for reading: No such file or directory
tail: cannot open `Lanka' for reading: No such file or directory
./unwanted1: line 10: 1: ambiguous redirect

tail: error reading `tmp': Is a directory
./unwanted1: line 10: tmp: Is a directory

tail: error reading `ussh': Is a directory
./unwanted1: line 10: ussh: Is a directory

tail: error reading `Varansi_files': Is a directory
./unwanted1: line 10: Varansi_files: Is a directory

tail: error reading `Video': Is a directory
./unwanted1: line 10: Video: Is a directory

[nissanka@c83-250-107-194 ~]$

Quote:

What is the line '' tail -c1 $1 | grep "$" " is doing here ?
I can't understand it.

It should be:
tail -c1 $i|grep "^$"
tail -c1 returns the last character of a file ($i in this case.)
grep "^$" is looking for a newline character. Probably should have used something else to make it more readable:
grep "\n" would do the same thing.

As for the directory errors, that could be avoided by first checking that $i is a file.

Code:

#!/bin/bash

for i in *;do

  if [ -f "$i" ];then

    tail -c1 $i|grep "\n"

    if [ "$?" = "1" ]; then #If $? = 1 then grep failed

      echo >>$i            #So we add a newline to the end of the file

    fi

  fi

done

As posted earlier this just checks the last character of a file to see if it is a newline. If it isn't a newline then it adds one.

bin/bash suggested the following program. I ran it. I named it ' unwanted2 '.

#!/bin/bash
for i in *;do
if [ -f "$i" ];then
tail -c1 $i|grep "\n"
if [ "$?" = "1" ]; then #If $? = 1 then grep failed
echo >>$i #So we add a newline to the end of the file
fi
fi
done
---------------------------------------------------

Some of the output is as follows:

[nissanka@c83-250-107-194 ~]$ chmod 755 unwanted2
[nissanka@c83-250-107-194 ~]$ ./unwanted2
./unwanted2: line 15: Important_Command: Permission denied
./unwanted2: line 15: klamav-0.32-2mdk.x86_64.rpm: Permission denied
tail: cannot open `Nepal\'s' for reading: No such file or directory
tail: cannot open `Buddha' for reading: No such file or directory
tail: cannot open `boy' for reading: No such file or directory
tail: cannot open `goes' for reading: No such file or directory
tail: cannot open `missing' for reading: No such file or directory
./unwanted2: line 15: 1: ambiguous redirect
./unwanted2: line 15: NTI: Permission denied
tail: cannot open `Portugese' for reading: No such file or directory
tail: cannot open `in' for reading: No such file or directory
tail: cannot open `Sri' for reading: No such file or directory
tail: cannot open `Lanka' for reading: No such file or directory
./unwanted2: line 15: 1: ambiguous redirect
tail: cannot open `Scientists' for reading: No such file or directory
tail: cannot open `to' for reading: No such file or directory
tail: cannot open `check' for reading: No such file or directory
tail: cannot open `Nepal' for reading: No such file or directory
tail: cannot open `Buddha' for reading: No such file or directory
tail: cannot open `boy.stm' for reading: No such file or directory
./unwanted2: line 15: 1: ambiguous redirect
tail: cannot open `Sri' for reading: No such file or directory
tail: cannot open `Lanka' for reading: No such file or directory
tail: cannot open `ceasefire' for reading: No such file or directory
./unwanted2: line 15: 1: ambiguous redirect

----------------------------------------------------------------------
----------------------------------------------------------------------

[nissanka@c83-250-107-194 ~]$ ./unwanted2 teco8

./unwanted2: line 15: Important_Command: Permission denied
./unwanted2: line 15: klamav-0.32-2mdk.x86_64.rpm: Permission denied
tail: cannot open `Nepal\'s' for reading: No such file or directory
tail: cannot open `Buddha' for reading: No such file or directory
tail: cannot open `boy' for reading: No such file or directory
tail: cannot open `goes' for reading: No such file or directory
tail: cannot open `missing' for reading: No such file or directory
./unwanted2: line 15: 1: ambiguous redirect
./unwanted2: line 15: NTI: Permission denied
tail: cannot open `Portugese' for reading: No such file or directory
tail: cannot open `in' for reading: No such file or directory
tail: cannot open `Sri' for reading: No such file or director

It didn't do anything worthwhile. I have a file called 'teco8'. It is just a text file I wrote using the 'vi' editor

What is the problem?

I urge someone to look at this post. I badly need your help.

I forgot to quote $i. Try this.

Code:

#!/bin/bash

for i in *;do

  if [ -f "$i" ];then

    tail -c1 "$i"|grep "\n"

    if [ "$?" = "1" ]; then #If $? = 1 then grep failed

      echo >>"$i"          #So we add a newline to the end of the file

    fi

  fi

done

Thanks bin/bash for taking time again to help me.
By the way, I like the name you invented. You are bin/bash!

#!/bin/bash
for i in *;do
if [ -f "$i" ];then
tail -c1 "$i"|grep "\n"
if [ "$?" = "1" ]; then #If $? = 1 then grep failed
echo >>"$i" #So we add a newline to the end of the file
fi
fi
done

---------------------------------------------

Yes, I ran the above. I named it 'unwanted3'.

echo >>$i

tail -c1 $i|grep "\n"

Only the above two lines are different than the previous program
-----------------------------------------------------

The output is as follows:

[nissanka@c83-250-107-194 ~]$ chmod 755 unwanted3
[nissanka@c83-250-107-194 ~]$ ./unwanted3
./unwanted3: line 12: Important_Command: Permission denied
./unwanted3: line 12: klamav-0.32-2mdk.x86_64.rpm: Permission denied
./unwanted3: line 12: NTI: Permission denied
[nissanka@c83-250-107-194 ~]$

Is it fine? I am not satisfied

I hope /bin/bash or someone will read this and tell me something more.

Permission denied

You probably do not have write permission for those files.

/bin/bash

First of all I must thank you for taking time to reply me.

You say that I probably don't have permission to write those files.
In my experience, definetly I don't have permission to write those files unless I logged in as a root user.

For example, I tried to use the 'find' command to search files which are written using the Open office wordprocessor. The funny message of 'no permission' surfaced. Then I logged in as a root user. Everything went smoothly.

However, my question was a different one.

Has the program worked fine? I am not sure whether it yielded the right output.

As I said before I am a newbie to these scripting, sed awk. I am comfortable with 'grep commands', 'find commands' and other UNIX related commands. I am not an expert , however. There are more things to learn.