LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-05-2010, 03:22 AM   #1
CheckiSt
LQ Newbie
 
Registered: Apr 2010
Location: Moscow, Russia
Distribution: Ubuntu 9.10, RHEL, AIX 6.1
Posts: 6

Rep: Reputation: 0
Question head adds chars to end of each line (Red Hat Enterprise Linux)


Hello!

I wrote a bash-script that splits each of many .sql-files into two parts by some condition using head utlity.
After that I execute all the scripts in sqlplus, and in one or two of them I get an error:
SP2-0042: unknown command ")" - rest of line ignored.

If I open the file with vi, I can see that in the end of each line there's a "^M", which is treated as a single character. If I delete this character placed before the closing parenthesis, the scripts executes without any errors.

In the initial script opened by vi there's no such characters.

Is it a problem with the head utility or with something else? What could I do with such a problem?
Of course, I cannot grep these special chars.
 
Old 04-05-2010, 03:27 AM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,985
Blog Entries: 11

Rep: Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879
Hi, welcome to LQ!

I very much doubt that head would introduce those. What you see are DOS
CarriageReturns. What platform are you on, where are the files from?


Cheers,
Tink
 
Old 04-05-2010, 03:28 AM   #3
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Hello and Welcome to LinuxQuestions,

I'm not sure if head is the culprit. Can you post the relevant code of your bash script?

One way to remove those characters is by using the dos2unix command. Most of the time one encounters those characters in a file that originates from Windows.

Code:
dos2unix <filename>
Kind regards,

Eric
 
1 members found this post helpful.
Old 04-05-2010, 03:33 AM   #4
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
This is because the file you are looking at has CRLF line endings instead of LF. Have a look for fileformat in vi.

'fileformat' 'ff' string (MS-DOS, MS-Windows, OS/2 default: "dos",
Unix default: "unix",
Macintosh default: "mac")
local to buffer
{not in Vi}
This gives the <EOL> of the current buffer, which is used for
reading/writing the buffer from/to a file:
dos <CR> <NL>
unix <NL>
mac <CR>
When "dos" is used, CTRL-Z at the end of a file is ignored.
See |file-formats| and |file-read|.
For the character encoding of the file see 'fileencoding'.
When 'binary' is set, the value of 'fileformat' is ignored, file I/O
works like it was set to "unix'.
This option is set automatically when starting to edit a file and
'fileformats' is not empty and 'binary' is off.
When this option is set, after starting to edit a file, the 'modified'
option is set, because the file would be different when written.
This option can not be changed when 'modifiable' is off.
For backwards compatibility: When this option is set to "dos",
'textmode' is set, otherwise 'textmode' is reset.


http://vim.wikia.com/wiki/File_format
 
Old 04-05-2010, 04:36 AM   #5
CheckiSt
LQ Newbie
 
Registered: Apr 2010
Location: Moscow, Russia
Distribution: Ubuntu 9.10, RHEL, AIX 6.1
Posts: 6

Original Poster
Rep: Reputation: 0
Thumbs up

Quote:
Originally Posted by Tinkster View Post
I very much doubt that head would introduce those. What you see are DOS
CarriageReturns. What platform are you on, where are the files from?
These files were checked out in RedHat and then copied to another RedHat server using WinSCP plugin for FAR. They were not edited using any Windows editors.

Quote:
Originally Posted by EricTRA View Post
I'm not sure if head is the culprit. Can you post the relevant code of your bash script?

Code:
dos2unix <filename>
Thank you, this solution worked perfectly, but I'm also interested in finding the causes of this problem.

Relevant code of bash script:
Code:
while read fname
do
  newf=`basename $fname`
  lnum=`grep -inm 1 "ALTER" $fname | cut -d: -f1`
  if [ 0$lnum != 0 ]
  then
    head -$(echo $lnum"-1" | bc) $fname >$tab_dir/$newf
    tail -$(echo $(cat $fname | wc -l)"-"$lnum"+1" | bc) $fname >$const_dir/$newf
  else
    cp $fname $tab_dir/$newf
  fi
done
This bash script reads filenames from a pipeline and splits each SQL-script into two parts: before the first ALTER operator and after it.
 
Old 04-05-2010, 04:45 AM   #6
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Quote:
Originally Posted by CheckiSt View Post
These files were checked out in RedHat and then copied to another RedHat server using WinSCP plugin for FAR. They were not edited using any Windows editors.
Hello,

I'm almost sure that WinSCP is the culprit in this case. You copied from server1 to a Windows workstation and then from the Windows pc to server2 with the same WinSCP, right? You can verify by copying the file directly from server1 to server2 using scp if you can connect from one to another.

Kind regards,

Eric
 
Old 04-05-2010, 05:09 AM   #7
CheckiSt
LQ Newbie
 
Registered: Apr 2010
Location: Moscow, Russia
Distribution: Ubuntu 9.10, RHEL, AIX 6.1
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by EricTRA View Post
You copied from server1 to a Windows workstation and then from the Windows pc to server2 with the same WinSCP, right?
That's right. But in the starting post I said about the initial scripts where the were no DOS-like line endings. The rub is that the "initial" scripts are the scripts that had already been copied using WinSCP.

Quote:
Originally Posted by EricTRA View Post
You can verify by copying the file directly from server1 to server2 using scp if you can connect from one to another.
I've tried to, but didn't manage to do so. I'm not sure, what is the problem with my SSH-key. I created it using PuTTY (in MS Windows) and placed it to ~/.ssh/, it is loaded when I use the scp and asks for my passphrase, but the passphrase that I successfully use to create a session in WinSCP is said to be erroneous
 
Old 04-05-2010, 06:15 AM   #8
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
Is this happening consistently, i.e. on all line endings, or only on a few?
If it's only happening on a few, what is the common denominator connecting those?
 
Old 04-06-2010, 12:20 AM   #9
CheckiSt
LQ Newbie
 
Registered: Apr 2010
Location: Moscow, Russia
Distribution: Ubuntu 9.10, RHEL, AIX 6.1
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by bakdong View Post
Is this happening consistently, i.e. on all line endings, or only on a few?
If it's only happening on a few, what is the common denominator connecting those?
I'm confused. Previously it was a stable error: it was on all line-endings (in all files that I checked), and after the next execution of my scripts the problem disappeared...
 
Old 04-06-2010, 12:36 AM   #10
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Hi,

Just an idea: the first time you copied over your files they were copied using WinSCP which might have resulted in the error. Next when you ran the script, if it writes to the files then I'd assume it's doing it correctly in the 'unix/linux' way so the error will not reproduce.

Kind regards,

Eric
 
Old 04-06-2010, 02:54 AM   #11
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
By the way, the WinSCP settings have an option for EOL character(s). I've never used the plugin for FAR but I gather that is also configurable.
Attached Images
File Type: png winscp.PNG (21.2 KB, 6 views)
 
Old 04-08-2010, 07:33 AM   #12
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
Moved to Linux-General
 
Old 04-14-2010, 03:26 AM   #13
CheckiSt
LQ Newbie
 
Registered: Apr 2010
Location: Moscow, Russia
Distribution: Ubuntu 9.10, RHEL, AIX 6.1
Posts: 6

Original Poster
Rep: Reputation: 0
Question

I've found that the cause of this problem is in another script. That's why the problem was unstable - I didn't execute this script always.
But I'm still confused, I can't understand why the problem appears.
Here's the script:
Code:
while read dir
do
  pwd1=`pwd`
  if [ x$dir != x ]
  then
    cd $dir
    for file in `ls | grep -i '\.sql'`
    do
     echo $'\nexit;\n' >>$file
    done
  fi
  cd $pwd1
done
It just opens each .sql file in a directory and appends "newline exit; newline" to the end of the file. So, I can't understand why it appends windows-style line-endings to all other lines.
The script is executed from PuTTY terminal via "find * -type d | append_exit" command, the "append_exit" file is placed in ~/bin.
 
Old 04-14-2010, 03:52 AM   #14
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
What is the initial $ sign for on the echo command?

Should you have '-e' as an option for echo with backslash characters?

(btw, I suspect that echo command will add two newlines after the 'exit;' because you haven't suppressed the automatic one with -n)

I was going to suggest replacing all of the above with a find/xargs combination but then couldn't quite get it to work!

find . -type f -iname "*.sql" -print | xargs -n 1 'echo -ne "\nexit;\n" >> '

Edit:

Ok, got it now, this does the trick:

Code:
for f in $(find . -type f -iname "*.sql");do echo -ne "\nexit;\n" >> $f;done

Last edited by bakdong; 04-14-2010 at 10:16 AM.
 
  


Reply

Tags
bash, linux


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Red Hat Ready for Top End of Enterprise Servers LXer Syndicated Linux News 0 12-05-2007 05:10 AM
LXer: Red Hat to launch Red Hat Enterprise Linux 5 before March LXer Syndicated Linux News 0 12-30-2006 08:03 AM
Project made in Red Hat Enterprise 4 need to be able to run on Red Hat Enterprise 3 Elin Linux - Software 1 07-13-2006 12:22 PM
Red Hat does not plan to release another product in the red hat linux line... Whitehat General 5 11-03-2003 06:33 PM


All times are GMT -5. The time now is 03:30 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration