LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-25-2012, 05:14 AM   #1
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Rep: Reputation: Disabled
Smile Searching for files and populate in array


Hi Guys, I was wondering if someone can assist in the following? I want to search for a filename (with wildcard) and then use the name I found to do some stuff with it. Let me explain... Let say I want to search for a file, I would normally do the following: # find / -iname filename*.dmp. The problem is, there might be more than one filename returned. Therefore, I need to handle each filename as they are returned. How can I put this in a shell script to search for the filename and then one by one, enter it into an array or something to be dealt with later? I know how to pass the search string from the command line to the shell script, I just cannot figure out how to make each filename found equal to a variable, or and array. Can someone give me some pointers? Thanks!
 
Old 01-25-2012, 05:55 AM   #2
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,250

Rep: Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684
Try redirecting your find into a while loop and processing each file in turn, something along the lines of:
Code:
while read -r filename
do
    <do your stuff here to $filename>
done< <(find / -iname filename*.dmp)
Now an initial trap for young players here is that the find will actually return the entire path to the file. If this is not desired, have a look at the man page for find
regarding the -printf option.
 
Old 01-25-2012, 06:17 AM   #3
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Original Poster
Rep: Reputation: Disabled
Hi, thanks for the reply. I tried your suggestion, but no luck:

filename=TWT*.dmp
while read -r filename
do
echo "$filename"
done << (find /usr/online/ -iname filename)
 
Old 01-25-2012, 09:54 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
The while read -r filename replaces the value of filename set by filename=TWT*.dmp.

filename=TWT*.dmp will not work as intended if there are any files in the current directory matching pattern TWT*.dmp

Here's a more robust version of grail's suggestion (also works with file names including newlines etc.) that adds names to an array called files and solves the pattern matching issue:
Code:
pattern="TWT*.dmp"
while IFS= read -r -d '' file
do
   files+=("$file")
done < <(find / -type f -name "$pattern" -print0)
If you want to pass the pattern as an argument to the script, replace pattern="TWT*.dmp" with pattern="$1" and use single quotes when calling the script:
Code:
./my_script 'TWT*.dmp'
You could also use double quotes in that example but it would fail if the pattern includes expressions that would be substituted by the shell such as $TWT*.dmp

Last edited by catkin; 01-25-2012 at 09:56 AM. Reason: sub-editing tyranny
 
Old 01-25-2012, 09:57 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
First of all, please use [code][/code] tags around your code and data, to preserve formatting and to improve readability. Please do not use quote tags, colors, or other fancy formatting.

Second, it doesn't do much good to say something didn't work, unless you explain in what way it failed. What output or errors did you get? What have you tried doing to diagnose or solve the problem?

You probably don't want to use the filename variable to store the pattern, as the while loop just overwrites it. And you need to use $ when you want to expand it. Don't forget to double-quote it too, so that the shell doesn't expand it before find can use it.

Anyway, to set the output of find, or any command, into an array, you can use the following, a variation on grail's loop:

Code:
pattern="TWT*.dmp"

while IFS="" read -d "" -r filename; do
	array[i++]="$filename"
done < <( find /usr/online/ -iname "$pattern" -print0 )

printf "%s\n" "${array[@]}"
Using the -print0 option and IFS makes it safe for all filenames, even if they happen to contain newline characters. The find command itself is run inside bash's process substitution pattern.

See this link for more on reading data into scripts:
http://mywiki.wooledge.org/BashFAQ/001

Edit: Beaten by 3 minutes by catkin. And to an almost identical solution. Sigh...

Last edited by David the H.; 01-25-2012 at 09:59 AM.
 
Old 01-25-2012, 10:13 AM   #6
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Original Poster
Rep: Reputation: Disabled
Thanks for the help guys. I copy "David the H" code and paste the sample code but when I execute, I get:

test.sh: line 6: syntax error near unexpected token `<'
test.sh: line 6: `done < <( find /usr/online/ -iname "$pattern" -print0 )'

Sorry for the trouble... I dont have the foggiest clue what all the stuff means in the sample code, but I trust that it will work.

Cheers,
J
 
Old 01-25-2012, 10:20 AM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
As I mentioned, process substitution is a bash feature. Make sure your script is using #!/bin/bash as the shebang.

If you use #!/bin/sh then it's interpreted as a posix script, and many things like that won't work. Your system may even be set up to use a different shell to interpret such scripts, such as dash.
 
Old 01-25-2012, 10:25 AM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Which version of bash are you using and does your script have a shebang line as its first line, something like #!/bin/bash

Edit: Beaten by 5 minutes by David the H.. And with almost identical content. Sigh...

Last edited by catkin; 01-25-2012 at 10:27 AM.
 
Old 01-25-2012, 10:35 AM   #9
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Original Poster
Rep: Reputation: Disabled
Awesome, this is exactly what I needed. Thanks "Catkin" and "David the H" it is working now. I added the bash part, set permissions and its all working well. I still don't have a clue how it does it, but it's working. Thanks again for the help. Now I can assign each filename to do some magic... Cheers, J
 
Old 01-25-2012, 11:23 AM   #10
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
Glad you found a solution. Threads can be marked SOLVED via the Thread Tools menu.

If you would like an explanation of how the script works, please ask.
 
Old 01-25-2012, 12:05 PM   #11
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,250

Rep: Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684Reputation: 2684
Here's a couple of pages to help you work it out:

http://tldp.org/LDP/abs/html/
http://mywiki.wooledge.org/TitleIndex
 
Old 01-26-2012, 02:31 AM   #12
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Original Poster
Rep: Reputation: Disabled
Hi, Yes please. I will appreciate it if you can explain what each line does. It will definitly help for future dev work. Thanks!
 
Old 01-26-2012, 08:19 AM   #13
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
OK. Starting at the end and working from the inside out
Code:
done < <(find / -type f -name "$pattern" -print0)
  1. The -print0 makes find separate each path matching the pattern with a NUL character (ASCII 00) instead of the usual newline.
  2. The <( ... ) is Process Substitution. Bash substitutes the output of the find command for it.
  3. The done < is redirection. It makes bash redirect the output of the process substitution to the while-do-done loops standard input.
Now we know what the standard input is, here's how it is used
Code:
while IFS= read -r -d '' file
  1. while (and the following do and done lines) makes bash loop as long as what follows is true, that is as long as it sets a 0 return code.
  2. What follows is IFS= read -r -d '' file
  3. This begins with a variable assignment (IFS=) followed by a command. The variable assignment differs from one not followed by a command in that it is only in effect during the execution of the command, as explained here.
  4. The string contained in IFS are the characters used by bash when it expands an expression into separate words (word-splitting). By default it is space, tab and newline. Here it is set to an empty string so word-splitting does nothing when bash runs the following read.
  5. read -r -d '' file reads from standard input:
    1. ignoring any "backslash escapes" (-r, raw mode) such as \t for tab.
    2. using the NUL character as the delimiter to separate "lines" (-d ''). The '' is an empty string but bash silently appends the NUL character to empty strings (sorry -- I cannot find any reference for this behaviour).
    3. and reading the entire line into variable file.
  6. When read tries to read another "line" when there are none left from the find output, it fails and sets a non-zero return code so the while terminates.
EDIT: the purpose of setting IFS to the empty string is to ensure that leading and trailing characters in the input "lines" which match characters in the IFS value are not stripped when it is assigned to the file variable (thanks to David the H's post in another thread for explaining this).

Last edited by catkin; 01-26-2012 at 10:03 PM.
 
Old 01-26-2012, 08:33 AM   #14
Jakkie
LQ Newbie
 
Registered: Jan 2012
Posts: 26

Original Poster
Rep: Reputation: Disabled
Wow, it actually makes sense if you see it line for line and reading it a few times (with your explination). Thank you very much for the info. I clearly need more training on this stuff.
 
Old 01-26-2012, 09:38 AM   #15
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
The code does use some uncommon bash features so don't overly worry about needing more training
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Populate array with content of file - line by line madsovenielsen Programming 2 07-26-2011 05:32 PM
Bash - Searching strings for array elements... Phier Programming 18 05-09-2010 05:37 AM
[SOLVED] Bash: how to populate a list of arbitrarily named files? catkin Programming 10 03-14-2010 03:58 PM
looping an array to populate menu for "dialog" kushalkoolwal Programming 4 01-30-2009 03:20 AM
searching an array in c andystanfordjason Programming 4 05-19-2007 10:41 AM


All times are GMT -5. The time now is 04:47 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration