Bash script issues -- halp!

Hey all. I'm trying to come up with a script to do what is, I'm sure, a pretty straightforward operation, but sadly I don't have a great deal of bash scripting experience and I'm hung up on one piece of it. After googling, tinkering, and researching for a couple days, I'd like to throw it out to the experts.

What I am trying to make is a script that will find and replace some strings across a series of files. The files exist as a number of text files in a series of subdirectories. The values come from a columnated, tab-delimited input file, formatted thusly:

File_1 OldValue_1 NewValue_1
File_1 OldValue_2 NewValue_2
File_2 Stuff Fnord

...etc.

The way I've figured on doing this is by, working one line at a time, assigning each field to a variable: on line 1, 'File_1' would become the $workfile variable; 'OldValue_1' would become the $original variable, and 'NewValue_1' would become the $replace variable. Once the information in the input file is appropriately chunked up, I could then construct a statement to actually perform the work. I would run this command in a loop so it would proceed one line at a time to carry things out.

As a proof of concept, I have created the following:

#!/bin/bash
workfile=`cat InputFile.txt | sed -n 2p | cut -f 1`
echo $workfile

This of course will cat InputFile.txt, only print the second line (the first line of the input file is just headers), cut the first field in that line, and set the variable 'workfile' to this value (this process would be repeated twice more with the other sections of the InputFile, with the cut command modified appropriately). This works great -- for line 2. In order for this to work in a loop, the 2 in the 'sed -n 2p' part needs to be a variable -- a counter that would increment with each iteration. I haven't constructed the loop yet -- I don't understand loops very well at all, I'm sorry to admit -- but towards this end, I have:

#!/bin/bash
counter=2
workfile=`cat InputFile.txt | sed -n "$counter"p | cut -f 1`
echo $workfile

When $counter increments, the sed statement becomes 'sed -n 3p', 'sed -n 4p', etc, etc. The problem is that this doesn't work from within a bash script. When I run the script, I get the following output:

$ ./test_1.sh
'ed: -e expression #1, char 2: unknown command: `

If I simply copy the contents of the script and run it at the shell, this works perfectly:

(Note; this is Cygwin on a Windows 7 machine. I'd do it in an actual Linux machine, but this is actually something that my wife asked me to put together for her, and she doesn't even have access to one -- she can install Cygwin and run things there, though, so this is kind of what I'm stuck with, not that Cygwin isn't great and all!)

dano@lappy686 /cygdrive/c/stuff/test
$ counter=2

dano@lappy686 /cygdrive/c/stuff/test
$ echo $counter
2

dano@lappy686 /cygdrive/c/stuff/test
$ workfile=`cat InputFile.txt | sed -n "$counter"p | cut -f 1`

dano@lappy686 /cygdrive/c/stuff/test
$ echo $workfile
File_1

So -- this is where I'm stuck. I've tried every iteration of spacing, single-quotes, double-quotes, and every escape character around the $counter part in the sed statement that I can think of, but it's just not working out for me. =(

I think that once this part is sorted the rest of the script should be easy enough -- it'll just be putting this bit into a loop, making sure the variables are cleared out appropriately in the next iteration of it, and then sending this information to another bit that will actually handle the finding and replacing, which I think will be a bit easier. I'm sure the way I've decided to go about doing this whole activity is utterly primitive and stupid -- I feel like I'm banging rocks together here -- but I just can't really think of a better one. Whatever the case, if anybody has any advice, I would certainly appreciate it. Thanks very much, everyone!

I think there's probably a better way of doing what you want. Why not post an example of the input you have followed by the output you want.

Depending on the complexity of the output, you may be able to run a single sed command rather than worrying about loops, variable substitutions, and whatever else.

Originally Posted by Dark_Helmet I think there's probably a better way of doing what you want. Why not post an example of the input you have followed by the output you want. Depending on the complexity of the output, you may be able to run a single sed command rather than worrying about loops, variable substitutions, and whatever else.

So, the actual contents of the (example) input file I have to work with:

FileName	ReplaceFrom	ReplaceTo
PostDischarge	admission_date	AdmissionDate
PostDischarge	discharge_date	DischargeDate
PostCdm	Cdm_Code	Cdm
PostCdm	Cdm_Description	CdmDescription
PostAvoidableDays	Av_Days	AvoidableDays

What I need to do is, for example, examine the 'PostDischarge' file, find the string 'admission_date', and replace it with 'AdmissionDate'. Then examine the 'PostDischarge' file again, find 'discharge_date', and replace it with 'DischargeDate'. Then examine the 'PostCdm' file, replace 'Cdm_Code' with 'Cdm', etc, etc. All the values in the 'FileName' column are actually .sql files (but for this purpose are handled pretty much the same way as a text file, I would think). This is just an example set I have to work with -- there will very likely be a pretty large number (several dozen at least, and possibly up to a few hundred or so) of different files beneath a series of subdirectories, and all of them will have different ReplaceFrom and ReplaceTo values; some files may only have one thing to be replaced, some may have a whole bunch. That's why I wanted to just go through it all one line at a time, as I'm not sure how else to make sure the FileName, ReplaceFrom, and ReplaceTo values maintain a consistent relationship. Thanks!

I think the code below will do what you want. Or at least, demonstrate what you were getting at to begin with. Based on the way you phrased your original problem, I didn't make the connection that you would be modifying contents in another file.

Anyway:

#!/bin/bash

while read filename oldValue newValue ; do

  echo "Filename: ${filename}"
  echo "Old Value: ${oldValue}"
  echo "New Value: ${newValue}"

#done < <( awk '{ print $1, $2, $3 }' test_data.txt )
done

exit 0

EDIT:
I should clarify... the while loop will execute once for each input data line and each variable should be set appropriately assuming that none of your filenames or value names have spaces in them.

EDIT2:
Oh, and I used a file named "test_data.txt" for my testing. That file included the sample data you provided. You'll, of course, have to substitute the appropriate filename for your data in the command.

EDIT3:
last edit, I swear... The fancy redirection is not needed at all. With the "awk" command commented out, you can get the same results with this pipeline:

cat test_data.txt | ./your_script.bash

Wow, thanks! Definitely WAY nicer than what I had cobbled together.

So I made a new script with the code you posted; I updated the data file with the appropriate filename, of course, and just for consistency's sake updated the field names here to match the headers in the feeder file, then tried it out -- but it doesn't seem to like something:

#!/bin/bash

while read FileName ReplaceFrom ReplaceTo ; do

  echo "File Name: ${FileName}"
  echo "Replace From: ${ReplaceFrom}"
  echo "Replace To: ${ReplaceTo}"

done < <( awk '{ print $1, $2, $3 }' InputFile.txt )
#done

exit 0

$ ./test_1.sh
./test_1.sh: line 2: $'\r': command not found
./test_1.sh: line 9: syntax error near unexpected token `done'
'/test_1.sh: line 9: `done < <( awk '{ print $1, $2, $3 }' InputFile.txt )

I also tried switching out the comments on the 'done' lines:

#done < <( awk '{ print $1, $2, $3 }' InputFile.txt )
done

and ran it again with the cat command, as below:

$ cat InputFile.txt | ./test_1.sh
./test_1.sh: line 2: $'\r': command not found
./test_1.sh: line 13: syntax error: unexpected end of file

Originally Posted by Dark_Helmet I think the code below will do what you want. Or at least, demonstrate what you were getting at to begin with. Based on the way you phrased your original problem, I didn't make the connection that you would be modifying contents in another file. Anyway: !/bin/bash while read filename oldValue newValue ; do echo "Filename: ${filename}" echo "Old Value: ${oldValue}" echo "New Value: ${newValue}" #done < <( awk '{ print $1, $2, $3 }' test_data.txt ) done exit 0 EDIT: I should clarify... the while loop will execute once for each input data line and each variable should be set appropriately assuming that none of your filenames or value names have spaces in them. EDIT2: Oh, and I used a file named "test_data.txt" for my testing. That file included the sample data you provided. You'll, of course, have to substitute the appropriate filename for your data in the command. EDIT3: last edit, I swear... The fancy redirection is not needed at all. With the "awk" command commented out, you can get the same results with this pipeline: cat test_data.txt | ./your_script.bash

Did you write/save the script in a Windows text editor? If so, did you run dos2unix on it before trying to run it under Linux?

EDIT:
I just realized that I forgot the '#' at the beginning of the first line. Although, that doesn't seem like it would explain the '\r' message on line 2 in the errors you posted. The '\r' problems are usually caused by using a file with Windows/DOS line endings in Linux.

The missing '#' is probably not the culprit anyway--the script you posted had the full "#!/bin/bash"

For reference, the first line of the script should be:

#!/bin/bash

I have updated my original post with the script.

Ahha, that was the deal there. I was using Notepad++ as a script editor, and didn't know about dos2unix. I just dos2unix'd the script and it worked perfectly:

$ ./test_1.sh
File Name: FileName
Replace From: ReplaceFrom
Replace To: ReplaceTo
File Name: PostDischarge
Replace From: admission_date
Replace To: AdmissionDate
File Name: PostDischarge
Replace From: discharge_date
Replace To: DischargeDate
File Name: PostCdm
Replace From: Cdm_Code
Replace To: Cdm
File Name: PostCdm
Replace From: Cdm_Description
Replace To: CdmDescription
File Name: PostAvoidableDays
Replace From: Av_Days
Replace To: AvoidableDays

VERY cool, thank you very much!

Can you help me understand how this works, exactly? It looks like the first line:

while read FileName ReplaceFrom ReplaceTo ; do

is declaring a while loop, and setting the FileName, ReplaceFrom, and ReplaceTo variables... so, 'while there are FileName, ReplaceFrom, and ReplaceTo, go do this stuff.'
It's then echoing a description, and the contents of each of those variables... then there is some black magic, followed by the awk statement putting fields 1, 2, and 3 into those variables, somehow, followed by an exit code.

How does the done line work, exactly?

done < <( awk '{ print $1, $2, $3 }' InputFile.txt )

I get the awk statement -- print the first field ($1), second field ($2), etc, that you find in 'InputFile.txt', but I don't get how those fields make their way into the appropriate variable. Clearly they do, because you are awesome, but I don't get it. =\

I'm very sorry for the dumb questions here, and I really appreciate the help... thanks again!

Originally Posted by Dark_Helmet Did you write/save the script in a Windows text editor? If so, did you run dos2unix on it before trying to run it under Linux?

Originally Posted by danowar I'm very sorry for the dumb questions here, and I really appreciate the help... thanks again!

Nope... no dumb questions.

The while statement does create a loop. The "read" portion of that statement takes in one line at a time of input. I'll get to where the source of that input is later. The following three variables tell the read command how and where to separate the line of input.

Specifically, the read command places the text in the input line up to the first break (i.e. a space-type character) in the first variable listed. The next group of text, up to the next break, is placed in the second variable. Lastly, the read command places all of the remaining text of the input line in the last variable listed (regardless of any other breaks).

So that is what I meant when I said the script will work, assuming that none of your filenames, oldvalue, or newvalue names contain a space. If any of them do contain a space, it will cause the read command to split the input line incorrectly.

The body of the script just displays the variables to demonstrate what values the variables contain. You can, of course, eliminate the echo statements and do whatever processing you need based on the variables.

The "done" command ends the loop. The remaining portion of that line does some shell redirection. In brief, the result of the awk command is given as input to the while-read loop. I used awk on the chance that your files might contain a mixture of spaces and/or tabs to separate your columns. By doing so, the awk command guarantees that each column's value is separated from one another by a single space--more or less in line with what the read command expects.

The redirection is functionally equivalent to the second example I gave: "cat filename | ./yourscript.bash"

The only difference is the filtering/grooming that awk performs before the read command sees the input line.

Awesome -- very useful. As a continuation of my efforts here, I've put together some stuff to actually handle the text replacement; isn't working quite right yet, but this seems like a more surmountable issue. =)

#!/bin/bash
while read FileName ReplaceFrom ReplaceTo ; do
  mv $FileName $FileName.bak 
  sed -e 's/$ReplaceFrom/$ReplaceTo/g' $FileName.bak > $FileName
  
done < <( awk '{ print $1, $2, $3 }' InputFile.txt )
#done

exit 0

Seems like it's doing all it's supposed to except for actually updating the file -- when it spits anything out at the shell it seems to have the updated values in it, but doesn't appear to be actually writing to the output file.

Originally Posted by Dark_Helmet Nope... no dumb questions. The while statement does create a loop. The "read" portion of that statement takes in one line at a time of input. I'll get to where the source of that input is later. The following three variables tell the read command how and where to separate the line of input. Specifically, the read command places the text in the input line up to the first break (i.e. a space-type character) in the first variable listed. The next group of text, up to the next break, is placed in the second variable. Lastly, the read command places all of the remaining text of the input line in the last variable listed (regardless of any other breaks). So that is what I meant when I said the script will work, assuming that none of your filenames, oldvalue, or newvalue names contain a space. If any of them do contain a space, it will cause the read command to split the input line incorrectly. The body of the script just displays the variables to demonstrate what values the variables contain. You can, of course, eliminate the echo statements and do whatever processing you need based on the variables. The "done" command ends the loop. The remaining portion of that line does some shell redirection. In brief, the result of the awk command is given as input to the while-read loop. I used awk on the chance that your files might contain a mixture of spaces and/or tabs to separate your columns. By doing so, the awk command guarantees that each column's value is separated from one another by a single space--more or less in line with what the read command expects. The redirection is functionally equivalent to the second example I gave: "cat filename | ./yourscript.bash" The only difference is the filtering/grooming that awk performs before the read command sees the input line.

One problem and two suggestions:

sed -e 's/$ReplaceFrom/$ReplaceTo/g' $FileName.bak > $FileName

Problem:
Your use of single quotes for the expression is your downfall. Bash will not substitute/interpret anything within single quotes. The consequence is, your $ReplaceFrom and $ReplaceTo references will not have their values as variables substituted--they will be treated as literal text.

To fix this, use double quotes--which bash will scan for appropriate substitutions. For example:

sed -e "s/$ReplaceFrom/$ReplaceTo/g" $FileName.bak > $FileName

Now, for the suggestions.

1. It's a good idea for you to get in the habit of using curly braces around your variable references. For instance:

sed -e "s/${ReplaceFrom}/${ReplaceTo}/g" ${FileName}.bak > ${FileName}

Doing so prevents any ambiguity as to the variable you want to reference. Using variable references without the curly braces in commands (like sed) that are loaded with other text and/or special characters could cause unintentional misinterpretation by the shell when it comes to substitution.

2. You can eliminate your script's mv command entirely and retain the "backup" file. The sed command has an option for that type of approach:

sed -i.bak -e "s/${ReplaceFrom}/${ReplaceTo}/g" $FileName

The -i option tells sed to operate on the file directly. The '.bak' immediately after the '-i' (notice the lack of a space), tells sed to make a backup of the file before making changes, and to append '.bak' to the filename to save the unmodified version of the file. So, the ${FileName} file will be modified version and ${FileName}.bak will be a copy of the file prior to any modifications by sed.

This is insanely helpful -- thank you so much! I guess my last couple questions would be, can you think of a way to make it recursively search subdirectories for the files contained in $filename, and is there a way to handle case sensitivity issues? Also, while nothing in my example data set contains any spaces, I don't know if something else might later on; is there a handle that?

Thanks again!

Originally Posted by Dark_Helmet One problem and two suggestions: sed -e 's/$ReplaceFrom/$ReplaceTo/g' $FileName.bak > $FileName Problem: Your use of single quotes for the expression is your downfall. Bash will not substitute/interpret anything within single quotes. The consequence is, your $ReplaceFrom and $ReplaceTo references will not have their values as variables substituted--they will be treated as literal text. To fix this, use double quotes--which bash will scan for appropriate substitutions. For example: sed -e "s/$ReplaceFrom/$ReplaceTo/g" $FileName.bak > $FileName Now, for the suggestions. 1. It's a good idea for you to get in the habit of using curly braces around your variable references. For instance: sed -e "s/${ReplaceFrom}/${ReplaceTo}/g" ${FileName}.bak > ${FileName} Doing so prevents any ambiguity as to the variable you want to reference. Using variable references without the curly braces in commands (like sed) that are loaded with other text and/or special characters could cause unintentional misinterpretation by the shell when it comes to substitution. 2. You can eliminate your script's mv command entirely and retain the "backup" file. The sed command has an option for that type of approach: sed -i.bak -e "s/${ReplaceFrom}/${ReplaceTo}/g" $FileName The -i option tells sed to operate on the file directly. The '.bak' immediately after the '-i' (notice the lack of a space), tells sed to make a backup of the file before making changes, and to append '.bak' to the filename to save the unmodified version of the file. So, the ${FileName} file will be modified version and ${FileName}.bak will be a copy of the file prior to any modifications by sed.

Originally Posted by danowar I guess my last couple questions would be, can you think of a way to make it recursively search subdirectories for the files contained in $filename, and is there a way to handle case sensitivity issues?

Yes, but it may add some complexity that might appear overwhelming if you're not familiar with it.

Briefly, the find command will do recursive searches with case insensitivity. For example:

find . -iname "${FileName}"

The '-iname' is a case insensitive match on the filename pattern. The '.' specifies to start searching from the current directory. So the command's output will be a list of files that match the value of ${FileName}. In a script, you could do something like:

fileList=$( find . -iname "${FileName}" )

Then you would need to code up a way to work through the contents of ${fileList} to do your file-by-file processing.

Just as a warning, you might be able to create a nested while-read loop to handle the file list, but I don't know if doing so would interfere with the outer/containing while loop. I'm not at a Linux machine that can do any testing at the moment. So I can't say for certain. That's why I suggested the "var=$( ... )" format above.

Originally Posted by danowar Also, while nothing in my example data set contains any spaces, I don't know if something else might later on; is there a handle that?

You could adjust for it, but it would require some reworking. First, if there are spaces, then you would need to identify unambiguous points that define column boundaries. If you can do that, then it's possible to use sed, awk, or other tools to extract the pieces. The exact commands would depend on how those boundaries are defined.

Additionally, for any filename that your script would need to process... your script would need to enclose the filename variable references in double quotes--to prevent bash from thinking you're referencing two separate files. For instance:

sed -i.bak -e "s/${ReplaceFrom}/${ReplaceTo}/g" "$FileName"

But generally speaking, there is no single approach that would handle all cases where a space, a tab, or other "whitespace" might appear within the data. The approach would need to be tailored to the environment.

I would add that the awk is not required at all in the current loop scenario. The following will have the same outcome:

#!/bin/bash

while read FileName ReplaceFrom ReplaceTo ; do
  sed -i.bak "s/$ReplaceFrom/$ReplaceTo/g" "$FileName"
done<"$InputFile.txt"

If you do in fact have a header row this can be bypassed with a simple counter and continue:

(( count++ == 0 )) && continue

Regarding my mention of the find command earlier, perhaps this would work:

#!/bin/bash

while read FileName ReplaceFrom ReplaceTo ; do
  find . -iname "${FileName}" -exec sed -i.bak "s/${ReplaceFrom}/${ReplaceTo}/g" {} \;
done < <( awk '{ print $1, $2, $3 }' InputFile.txt )

Or, as grail mentioned, removing the awk and piping in the contents of InputFile.txt directly, or using cat to pipe in the contents of the input file through the command line (my third edit in the second reply), or any number of other possible approaches. Pick whichever one makes the most sense to you.

One thing to note, the find command above will modify files and it will not do any double-checking. It assumes that every file it finds that matches the filename pattern is a file to modify. I strongly advise you to be careful when using this type of command. It is worth the time and effort for you to run at least a minimal number of tests to verify what files the find command will match.

To do so, run the find command without the '-exec' or any text that follows it. For example:

find . -iname "${FileName}"

That command will print a list of matching files for you to review. Keep an eye out for any problems. When you are absolutely positive that everything is OK, then put the '-exec' portion of the command back.

halp!

rofl