Bash scripting problem: Can't get a list of all files, including hidden ones

archtoad6 · 02-23-2007, 10:53 AM

You can get ls to add quotes w/ the -Q option:

Code:

ls -1QA

This might further help eliminate pathological filenames.

Yes, although I can't remember ever doing it, I believe that it is possible to create a filename w/ a newline in it.

If that were a problem, then find -print0 may be the solution. I know this isn't applicable to this case (grep only), but it might be of help to a reader w/ a more general problem.

Finally, (re-)RT

M ls & find.

cfaj · 02-27-2007, 07:30 PM

Quote:

Originally Posted by Dark_Helmet

I certainly agree. I prefer the other because it's one fewer keystroke

and the fact that you press Enter/Return reinforces the idea that word-splitting occurs only on newlines.

I must admit that I usually use the literal newline, not because it is shorter, but because it is more portable.

Quote:

I'm afraid I don't understand what you're saying here. By having the shell split on newlines, the filename variable will contain the as-displayed-by-ls filename (because the shell already did the wildcard expansion for the '*' before executing the ls command).
...

You are right; I was forgetting that IFS had been redefined. However, it is still almost always the wrong way to do it, not to mention being inefficient. Wildcard expansion is always the safest and most efficient.

Quote:

And just to be thorough, I used a 1 (one) in the ls command--not an l (el). Makes a difference in the output

Since the output is not going to a terminal, using the -1 option makes absolutely no difference.

Dark_Helmet · 02-27-2007, 07:49 PM

Quote:

Originally Posted by cfaj

Since the output is not going to a terminal, using the -1 option makes absolutely no difference.

The output of the ls command may not be going to a terminal, it is being fed into a variable, and these two lines will yield completely different variable contents:

Code:

var1=$( ls -1 test_file )
var2=$( ls -l test_file )

EDIT:
Just a couple other things to mention.

The purpose of the -1 is to pair it with the change in IFS. The IFS causes word-splitting on newlines and the -1 option to ls causes a newline after each filename; Therefore, unless a filename has a newline in it (hopefully an exceedingly rare condition), the splitting is correct, and gives the iterative variable the desired contents. Using the -l option will cause the variable to contain the full line of file permissions, date, size, name, etc--most definitely not what is wanted.

And in regard to efficiency... I'll admit I'm clueless. The solution you provided may very well be more efficient. But to me, that's a low priority. My philosophy: shell scripts are quick-and-dirty solutions, and I don't see the OP's request as relying heavily on efficiency to be a satisfactory solution. In those cases, I just throw up solutions that have worked for me in the past.

cfaj · 02-27-2007, 09:49 PM

Quote:

Originally Posted by Dark_Helmet

The output of the ls command may not be going to a terminal, it is being fed into a variable, and these two lines will yield completely different variable contents:

Code:

var1=$( ls -1 test_file )
var2=$( ls -l test_file )

I said nothing about using -l (lowercase 'L'); of course that gives different results.

But these two are exactly the same (even with multiple file arguments):

Code:

var1=$( ls -1 * )
var1=$( ls * )

Quote:

EDIT:
Just a couple other things to mention.

The purpose of the -1 is to pair it with the change in IFS. The IFS causes word-splitting on newlines and the -1 option to ls causes a newline after each filename; Therefore, unless a filename has a newline in it (hopefully an exceedingly rare condition), the splitting is correct, and gives the iterative variable the desired contents. Using the -l option will cause the variable to contain the full line of file permissions, date, size, name, etc--most definitely not what is wanted.

Using -1 changes nothing if the output is not going to a terminal (e.g., when output is stored in a variable).

Quote:

And in regard to efficiency... I'll admit I'm clueless. The solution you provided may very well be more efficient. But to me, that's a low priority. My philosophy: shell scripts are quick-and-dirty solutions, and I don't see the OP's request as relying heavily on efficiency to be a satisfactory solution. In those cases, I just throw up solutions that have worked for me in the past.

If you use shell scripts for serious programming, then efficiency is important. Every quick-and-dirty script you write may later be incorporated into a larger script. A lot of inefficient Q'n'D scripts can make the larger script sluggish and unresponsive.

Dark_Helmet · 02-27-2007, 11:55 PM

The reason I focused on the -1 versus -l difference is because of a response made earlier:

Quote:

Originally Posted by cfaj

Yes, a variable containing a filename should always be quoted, but that is too late to prevent the word splitting that will already have occurred by using for filename in $( ls -lA ).

That was part of the response to my initial post in the discussion. So I assumed you were essentially trying to quote my for-loop beginning to make a comment about it. However, your quote uses an l (el) as opposed to my original 1 (one). Hence, my reply wanting to make the distinction. Which then led to the other replies. And, of course, ls -l (el) as opposed to ls -1 (one) will cause a significant difference in the values assigned to the variable filename as the loop proceeds (i.e. the output of the ls command)--which then explains my last post.

shailesh2000 · 03-10-2007, 05:28 PM

i want to make a script in linux for running ftp or taking backup in lto tape drive ????

I'm making a bash script to search for a word recursively using grep.

Problem is, I don't find a reliable way to get a full listing of files for the current dir.

Cases are:

Code:

for $file in `ls -a`
...

This will get me a list with all files, including hidden ones. Problem is, they are separated by spaces, and if a filename has spaces in it, I have no way to make a difference between the chunks of that filename and others.

Code:

for $file in `ls -am`
...

Nope, sorry but this one isn't good enough for me. This will get me a list of filenames separated by a comma and a space each. Then I'd make fixes to make it usable for "for". Problem is, you can create a filename called "blah, blah" this way: "$ touch blah\,\ blah". So I just can't use it. Now, if it used another character instead of a comma...

Code:

for $file in ./*
...

This gives me a list of the files, which is correctly assigned each loop for $file, that is, $file value is what it is supposed to be each loop, but I don't get the hidden files

Code:

for $file in ./.*
...

With this I get only the hidden files.

So, I was wondering if I could use any kind of regular expression or glob to get both hidden and unhidden filenames. Or, if there's way to make "./.*", then "./*" and the join them in one list. Or if I can make ls use other separator than a ", ".

I've been googling for a looooot of time and couldn't find a solution to my problem, that's why i'm asking here.

I really appreciate your help and thanks in advance pals.

Cheers

[/QUOTE]

shailesh2000 · 03-10-2007, 05:30 PM

can i make a script to run ftp in a single command if yes then how it possible////
i can use vi editor for it???

ghostdog74 · 03-10-2007, 07:42 PM

Quote:

Originally Posted by oxi

Hi,

I'm making a bash script to search for a word recursively using grep.

Problem is, I don't find a reliable way to get a full listing of files for the current dir.

Cases are:

Code:

for $file in `ls -a`
...

This will get me a list with all files, including hidden ones. Problem is, they are separated by spaces, and if a filename has spaces in it, I have no way to make a difference between the chunks of that filename and others.

Code:

for $file in `ls -am`
...

Nope, sorry but this one isn't good enough for me. This will get me a list of filenames separated by a comma and a space each. Then I'd make fixes to make it usable for "for". Problem is, you can create a filename called "blah, blah" this way: "$ touch blah\,\ blah". So I just can't use it. Now, if it used another character instead of a comma...

Code:

for $file in ./*
...

This gives me a list of the files, which is correctly assigned each loop for $file, that is, $file value is what it is supposed to be each loop, but I don't get the hidden files

Code:

for $file in ./.*
...

With this I get only the hidden files.

So, I was wondering if I could use any kind of regular expression or glob to get both hidden and unhidden filenames. Or, if there's way to make "./.*", then "./*" and the join them in one list. Or if I can make ls use other separator than a ", ".

I've been googling for a looooot of time and couldn't find a solution to my problem, that's why i'm asking here.

I really appreciate your help and thanks in advance pals.

Cheers

you can try the suggestion at 14. here.
If you have Python, here's an alternative:

Code:

##!/usr/bin/python
import os
dir = "/home"
word2find = "word to search"
os.chdir(dir)
for fi in os.listdir("."):
    if os.path.isfile(fi):
        for num,line in enumerate(open(fi)):            
            if word2find in line:
                 print "%s: '%s' found at line %d" %(fi,word2find,num)

cfaj · 03-11-2007, 05:17 PM

Quote:

Originally Posted by shailesh2000

i want to make a script in linux for running ftp or taking backup in lto tape drive ????

I'm making a bash script to search for a word recursively using grep.

Problem is, I don't find a reliable way to get a full listing of files for the current dir.

Cases are:

...

Code:

for $file in ./*
...

This gives me a list of the files, which is correctly assigned each loop for $file, that is, $file value is what it is supposed to be each loop, but I don't get the hidden files

Code:

for $file in ./.*
...

With this I get only the hidden files.

Neither of those will get you anything but a syntax error:

bash: `$file': not a valid identifier

Quote:

So, I was wondering if I could use any kind of regular expression or glob to get both hidden and unhidden filenames. Or, if there's way to make "./.*", then "./*" and the join them in one list. Or if I can make ls use other separator than a ", ".

You are using bash, so set the dotglob option:

Code:

shopt -s dotglob
for file in ./*

archtoad6 · 03-12-2007, 06:19 AM

1) There's nothing that prevents you from putting more than 1 pattern in a for statement:

Code:

for FILE in * ./.*

If each works separately, combining them will also work. Just make sure they don't overlap. (I believe they won't.)

2) Why are you having to deal w/ mal-formed file names containing spaces? Such are not "the Unix way" (tm

). See my post #11.