LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-02-2011, 08:32 PM   #1
greenpool
Member
 
Registered: Sep 2010
Posts: 41

Rep: Reputation: 0
beginner script errors


Hi guys,

So i've been playing around with some really basic scripts to get a good grip on bash scripting.

some help on the following problems would be helpful.

problem 1:
./exitStatusTest.sh: line 8: [: too many arguments

problem 2:
./checkUser.sh: line 6: if[ 0 -eq 0 ]: command not found
./checkUser.sh: line 7: syntax error near unexpected token `then'
./checkUser.sh: line 7: `then'

Last edited by greenpool; 09-02-2011 at 09:48 PM.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-02-2011, 08:34 PM   #2
greenpool
Member
 
Registered: Sep 2010
Posts: 41

Original Poster
Rep: Reputation: 0
exitStatusTest.sh

Code:
#!/bin/bash
cd ~/Documents/sfiles

if [ $? -ne 0 ]
then
echo "failed: file doesn't exist"
else
for i in *
do
 if [ `cat $i | grep read` ];
  then   
    echo  $i 
 fi
done
fi


the above script works except that it gives me an error as well:

./exitStatusTest.sh: line 8: [: too many arguments

can someone please tell me why this is happening?

Last edited by greenpool; 09-02-2011 at 09:10 PM. Reason: script updates. error has not changed
 
Old 09-02-2011, 09:49 PM   #3
greenpool
Member
 
Registered: Sep 2010
Posts: 41

Original Poster
Rep: Reputation: 0
checkUser.sh

Code:
#!/bin/bash
echo -n "Enter user name : "
read USR
cut -d: -f1 /etc/passwd | grep "$USR" > /dev/null
OUT=$?
if[ $OUT -eq 0 ];
then
echo "user account found!"
else
echo "user account does not exist in /etc/passwd file!"
fi

errors:Enter user name : test
./checkUser.sh: line 6: if[ 0 -eq 0 ]: command not found
./checkUser.sh: line 7: syntax error near unexpected token `then'
./checkUser.sh: line 7: `then'
 
Old 09-02-2011, 10:08 PM   #4
tbrand
Member
 
Registered: Jul 2006
Location: Toronto, Canada
Distribution: gentoo
Posts: 33

Rep: Reputation: 17
In your second script, on line 6, you have no space between ``if'' and ``[''. You may not realize that in bash ``['' is in fact and alias for ``test''. It is not a language construct.

The problem with the first script is more subtle. It seems that the directory ``~/Documents/sfiles'' contains enough files to exceed the bash buffer space.
However, if I understand your script correctly what you are trying to do can be done much more efficiently and simply by this single line command:

ls | xargs grep -l read

The only problem you may have with this is if your directory contains files whose names contain spaces or some other non-alphanumeric characters that are interpreted by bash. But such a problem you would have with your script as well.
 
Old 09-03-2011, 12:23 AM   #5
greenpool
Member
 
Registered: Sep 2010
Posts: 41

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by tbrand View Post
In your second script, on line 6, you have no space between ``if'' and ``[''. You may not realize that in bash ``['' is in fact and alias for ``test''. It is not a language construct.

The problem with the first script is more subtle. It seems that the directory ``~/Documents/sfiles'' contains enough files to exceed the bash buffer space.
However, if I understand your script correctly what you are trying to do can be done much more efficiently and simply by this single line command:

ls | xargs grep -l read

The only problem you may have with this is if your directory contains files whose names contain spaces or some other non-alphanumeric characters that are interpreted by bash. But such a problem you would have with your script as well.
hi tbrand,

thanks for that.

i took your advice and modified my script.

Code:
ls | xargs grep -l read

so from what i understood xargs will split list into sublists and calls grep once for every sublist.

although i got the following script to work, i'm having trouble wrapping my head around it.

here's the modified script:


Code:
#!/bin/bash
myscriptname=`basename $0`;
for i in `ls -A`
do
   if [ $i != $myscriptname ]
   then
   echo $i | xargs grep -l read
    
   fi
done


the part i'm having trouble understanding:

Code:
echo $i | xargs grep -l read
i must understand this wrong cos in my head i'm thinking echo $i pipes a single filename (as opposed to a list) so why is xargs necessary? but it doesn't work without xargs either..

can somebody please clarify this for me? thanks.
 
Old 09-03-2011, 01:10 AM   #6
tbrand
Member
 
Registered: Jul 2006
Location: Toronto, Canada
Distribution: gentoo
Posts: 33

Rep: Reputation: 17
Hi Greenpool,

my point was that, practically, the whole script can be replaced by that one-liner.

The way I would rewrite your script would be something like this:

Code:
#!/bin/bash
ls -A | xargs grep -l read
Let me explain:

The ``ls -A'' produces the list of files in the current directory and writes it to standard output (when standard output from ls goes to the console it is arranged in columns but when it gets redirected to a file or a pipe the output contains one file name per line).

Using ``xargs'' has two benefits for your purpose. First, it reads the list of file names from ls on its standard input and executes ``grep -l'' with the list of file names it got from its standard input as command line arguments.

So, if your directory contains three files: file1, file2 and file3, xargs will execute command

Code:
grep -l file1 file2 file3
But you may say why should I do ``ls -A | xargs grep -l'' rather then just ``grep -l *''? This is where you will appreciate the second benefit of xargs. If you use ``*'', bash will expand it to a list of all matching file names but that list may be bigger than bash can accommodate in its buffer and you get the same problem you described earlier.

As you say, xargs splits the list of files into sub-lists. How large those sub-lists are depends on the size of the shell's buffer. ``xargs'' makes sure that each of those sublists will fit into the shell's buffer. If it needs to produce more than one sub-list it will execute ``grep -l'' repeatedly until it runs out of sub-lists.

One last detail, which you probably already know if you read grep manpage, is that ``grep -l'' simply prints the name of the file in which it found the given pattern (``read'' in your case) at least once, rather than printing all the lines that contain the pattern which is grep's default action.

In your second version of your script, you exclude the file which contains your script. That's probably not good practice. It is customary to store your script with execute permissions in some standard directory (typically bin) specified in your PATH variable, so, that you can run it in any directory you wish.
 
1 members found this post helpful.
Old 09-03-2011, 05:37 AM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
If you haven't seen this guide yet, stop now and read through it first. It will cover all the basic concepts you need to know.

http://mywiki.wooledge.org/BashGuide

Also read these three links. It's vital in scripting to understand exactly how the shell handles arguments and whitespace:

http://mywiki.wooledge.org/Arguments
http://mywiki.wooledge.org/WordSplitting
http://mywiki.wooledge.org/Quotes

Now for some specific comments.

1) Let's start off with formatting. Clear and consistent formatting helps make your script easier to read and debug. Indent all of your loops, branches, and functions, and use blank lines to separate blocks of lines that naturally group together. Add some comments to explain your code. Putting the "do" on the same line as the for/while/until also looks a little nicer.

2)
Code:
if [ $? -ne 0 ]
Change this to...
Code:
if (( $? != 0 )); do
Numeric tests can better be performed with ((..)), and for string/complex tests the newer [[ test is recommended.

3)
Code:
if [ `cat $i | grep read` ];
a. $(..) is highly recommended over `..`

b. Always quote variable and other substitutions unless you want word splitting to occur; particularly inside [ tests. See the links I gave above.

c. grep can read files directly, so you don't need cat.

Code:
if [[ "$( grep "read" "$i" )" ]]; then
d. But this can be simplified even further. If a successful match is made, grep will exit true, and that's all we're interested in. Now grep -q will suppress output, meaning you can simply test the exit code only. This can even be done directly without the test brackets.
Code:
if grep -q "read" "$i" ; then
	echo "found"
fi
4)
Code:
cut -d: -f1 /etc/passwd | grep "$USR" > /dev/null
This is backwards and inefficient. first use grep to get the line you want, then cut it. Or use a tool like sed or awk instead that can do both the matching and the extracting (grep -o would even work here).

But actually, in this case, the cutting is superfluous; you already have the name, you just want to find out if it's in the file. So just grep for the line that contains the username, using grep -q as in the previous example. You might consider anchoring the match to the beginning of the line, however, to avoid possible false matches elsewhere in the file.
Code:
grep -q "^$USR" /etc/passwd
Many tools like grep have "quiet" or "silent" options, meaning you don't have to dump the output to /dev/null. Always try to learn the capabilities of the tools you're using and see if it can do what you want internally, before turning to shell options.

5)
Code:
myscriptname=`basename $0`;
bash has parameter expansion and other text extraction abilities, making many tools like basename, pathname, and even cut, unnecessary much of the time.
Code:
myscriptname="${0##*/}"
6)
Code:
for i in `ls -A`
Parsing ls is risky, difficult to do properly, and usually unnecessary. So don't use it. Most of the time you can use globbing or find instead (and use the null-separator option when you do, if possible).

In this case, the dotglob shell option will allow you to match hidden files.
Code:
shopt -s dotglob
for i in * ; do
7)
Code:
   if [ $i != $myscriptname ]
   then
   echo $i | xargs grep -l read
   fi
xargs is a bulk command processor. As you seem to have realized, you shouldn't need to use it on a single entry like this if you're processing individual inside a loop. But if you have a whole list of filenames from some input source, such as from a find or a file, it's a great tool to use.

Your buffering problem above is mostly caused by the use of the for loop, which expands the glob into a complete list of files to process before running. But instead of using xargs, you can probably use a while+read loop, which takes in only a single line at a time.

The only question then is how generate the list that it needs to read. I'm thinking that using printf to break up the globbing pattern by newlines may work, although it might encounter the same problem. We might have to switch to find.
Code:
while read i; do
	echo "$i"
done < <( shopt -s dotglob; printf "%s\n" * )

# or if that doesn't work:

while read i; do
	echo "$i"
done < <( find -maxdepth 1 -iname "*" -print )
Another option may be to load the entire filelist into an array first, then process them by index number instead. Using the index numbers in the for loop instead of names will probably bring the total line length down to less than the maximum.

Code:
shopt -s dotglob
files=( * )
for i in "${!files[@]}": do
	echo "{files[i]}"
done
Finally, have a look at the ulimit built-in command. You may be able to increase the size of the command buffer using it ("stack size", perhaps?).

Last edited by David the H.; 09-03-2011 at 05:47 AM. Reason: minor editing and additions
 
2 members found this post helpful.
Old 09-03-2011, 10:12 AM   #8
tbrand
Member
 
Registered: Jul 2006
Location: Toronto, Canada
Distribution: gentoo
Posts: 33

Rep: Reputation: 17
Unfortunately, ulimit does not affect the size of the argument buffer.

If you'd like to know what the size is you can do

Code:
getconf ARG_MAX
but increasing ARG_MAX requires a global system change which means that all processes will require more memory and, furthermore, it will simply set another limit albeit higher.

xargs is the cleanest and most efficient method of overcoming the problem.
 
Old 09-04-2011, 01:13 PM   #9
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Thanks, tbrand. I'm not really up on what all the shell limits are. I've rarely encountered them personally.

Yes, xargs is best if what you want to do is something that can be arranged in bulk, as it automatically takes limits like this into account.

But if you need to use a loop for whatever reason, then you some other kind of work-around will be needed, which is what I wanted to document.
 
Old 09-04-2011, 01:39 PM   #10
tbrand
Member
 
Registered: Jul 2006
Location: Toronto, Canada
Distribution: gentoo
Posts: 33

Rep: Reputation: 17
Thanks, David the H. I understand your point, xargs doesn't solve everything but you must admit that that particular case was a one-liner bait
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
reading a value from a textfile using bash script (beginner problem) AndrewJS Linux - Newbie 1 06-10-2011 08:16 PM
Beginner question, Bash-script - date kickarzt Programming 15 03-10-2010 01:18 PM
a beginner's PHP/MySQL script: MySQL in spreadsheet-like layout julianb LinuxQuestions.org Member Success Stories 2 01-01-2010 09:25 PM
Multiple Errors in C program (beginner) circuit_girl Programming 9 09-19-2006 04:47 AM
Beginner on Scilab, matlab script conversion munichtexan Linux - Software 6 12-08-2005 07:42 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 08:19 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration