using sed to insert lines with special characters

disorderly · 04-12-2006, 09:17 AM

thanks for the update dive. i gave up for the night last night after 13 hours and the script erased half my files and then itself. i'll try the new code now that my head is clear again

dive · 04-12-2006, 12:49 PM

I can't really see how files are being erased here. Did you try using mv with the -v flag to get a more verbose output? It will say something like test.html -> test.php for each mv. That way you can check back in terminal for errors.

If the cat and sed commands are to blame for deleting files (maybe cat'ing a file back into itself?) you could try using a temporary file:

echo "<?php include(\"/home/functions.php\");validateUser(\$_SERVER['PHP_SELF']);?>" > tempfile
cat $y >> tempfile
mv -v tempfile "${y%.*}.php"

disorderly · 04-13-2006, 03:20 PM

hi dive - excellent alternative solution! the script worked great after that. the problem seemed to be with sed; the script produced some very strange results:
1) most files were truncated (i think sed has a memory limit or something..)
2) files were being copied & dropped in random folders
3) files were having their contents erased so the byte size was 0

here is the finished script

Code:

#!/bin/bash

# this script recursively inserts a line into the head of all files 
# in the directory from where it was called
# it notes all files changed to output.txt
# it shows everything the script does in logs.txt

logs=/home/logs.txt
output=/home/output.txt
phead='<?php include("func.php");validateUser($_SERVER["PHP_SELF"]);?>'

for i in *
do
        if [ -d "$i" ] # if * is a directory
        then
                echo "1) $PWD" >> $logs
                cd "$i" # descend into the directory

                for y in *
                do
                        if echo "$y" | grep ".htm"
                        then
                                echo "2) modifying: $PWD/$y" >> $logs
                                echo $phead > temp
                                cat $y >> temp
                                mv -v temp "${y%.*}.php"
                                chmod 744 "${y%.*}.php"
                                rm -v $y
                                echo "$PWD/$y" >> $output #for testing
                        fi

                        if [ -d "$y" ]   # if a directory, call self
                        then
                                echo "3) $PWD" >> $logs
                                cd "$y"
                                sh /home/c.sh; # call self (recursively) 
                                cd ..
                                echo "4) going back up to: $PWD" >> $logs
                        fi
                done
                cd ..
        fi

        if echo "$i" | grep ".htm"
                then
                        echo "5) modifying: $PWD/$i" >> $logs
                        echo $phead > temp
                        cat $i >> temp
                        mv -v temp "${i%.*}.php"
                        chmod 744 "${i%.*}.php"
                        rm -v $i
                        echo "$PWD/$i" >> $output #for testing
        fi
done

it has one limitation - some of the files in these directories were made with Frontpage [puke] and thus have folder in the same directory named the same way but with the word "html" in them. for example the file "content.html" includes two files: ./_vti_cnf/content.html and ./_derived/content.html_sourcecontrol whatever the f those are, and they get changed by the script too.

for anyone that is intested i also found another way to check for the file extension:

Code:

for i in *;do
        if [ ! -d "$i" ];then
                ext=${i##*.}
                echo '$i' is $i and it\'s extension is $ext
                if [ $ext = htm ]; then
                       echo "$i ends with htm"
                if
        fi
done

dive · 04-13-2006, 04:01 PM

I'm glad that you got it working.

One thing: after test to see if $i is -d, it acts upon that and either changes the file, or descends into dir, and finally ascends out of it. Next it tests again for .htm and then changes the file, whether or not it is a dir. That is why folders are being changed too. Try this:

Code:

#!/bin/bash

# this script recursively inserts a line into the head of all files 
# in the directory from where it was called
# it notes all files changed to output.txt
# it shows everything the script does in logs.txt

logs=/home/logs.txt
output=/home/output.txt
phead='<?php include("func.php");validateUser($_SERVER["PHP_SELF"]);?>'

for i in *
do
        if [ -d "$i" ] # if * is a directory
        then
                echo "1) $PWD" >> $logs
                cd "$i" # descend into the directory

                for y in *
                do
                        if echo "$y" | grep ".htm"
                        then
                                echo "2) modifying: $PWD/$y" >> $logs
                                echo $phead > temp
                                cat $y >> temp
                                mv -v temp "${y%.*}.php"
                                chmod 744 "${y%.*}.php"
                                rm -v $y
                                echo "$PWD/$y" >> $output #for testing
                        fi

                        if [ -d "$y" ]   # if a directory, call self
                        then
                                echo "3) $PWD" >> $logs
                                cd "$y"
                                sh /home/c.sh; # call self (recursively) 
                                cd ..
                                echo "4) going back up to: $PWD" >> $logs
                        fi
                done
                cd ..

        elif echo "$i" | grep ".htm" # else if not a dir -d
                then
                        echo "5) modifying: $PWD/$i" >> $logs
                        echo $phead > temp
                        cat $i >> temp
                        mv -v temp "${i%.*}.php"
                        chmod 744 "${i%.*}.php"
                        rm -v $i
                        echo "$PWD/$i" >> $output #for testing
        fi
done

disorderly · 04-13-2006, 04:46 PM

aHA - nice catch dive i'll try that out tomorrow morning!
again, i appreciate all the help

-disorderly

dive · 04-13-2006, 07:38 PM

Sort of going on the basis that if you call a script recursively you shouldn't need the same file writing code twice, I came up with this:

Code:

#!/bin/bash

# this script recursively inserts a line into the head of all files
# in the directory from where it was called
# it notes all files changed to output.txt
# it shows everything the script does in logs.txt

logs=/home/logs.txt
output=/home/output.txt
phead='<?php include("func.php");validateUser($_SERVER["PHP_SELF"]);?>'

for i in *
do
        if [ -d "$i" ] # if * is a directory
        then
                echo "1) $PWD" >> $logs
                cd "$i" # descend into the directory
                        sh /home/c.sh # call self (recursively)
                        echo "4) going back up to: $PWD" >> $logs
                cd ..
        elif echo "$i" | grep ".htm"
        then
                        echo "5) modifying: $PWD/$i" >> $logs
                        echo $phead > temp
                        cat $i >> temp
                        mv -v temp "${i%.*}.php"
                        chmod 744 "${i%.*}.php"
                        rm -v $i
                        echo "$PWD/$i" >> $output #for testing
        fi
done

It worked for me with a small test with dirs going 2 deep, but let me know how it works. No problem for the help btw - I love scripting and it's helped me with one of my own scripts too.

disorderly · 04-13-2006, 08:13 PM

short and sweet - that's much tighter coding! it worked perfectly - the end result was the same as the former longer script. thanks for all the help - i appreciate it. this stuff really intrigues me- i've now gone out and picked up the books, "learning the bash shell, unix shell programming" & "wicked cool shell scripts." hopefully i can learn to automate some of my my job so i can have more time to read slashdot at work

-disorderly

dive · 04-13-2006, 08:56 PM

I found this quite helpful: http://www.tldp.org/LDP/abs/html/ plus doing any sort of search for bash scripting or specific commands usually comes up with useful titbits

disorderly · 04-13-2006, 09:19 PM

oh cool - that site is a goldmine! would you happen to know why sed couldn't work on the files i was trying to edit? the text files were between 1k and 139k

dive · 04-13-2006, 09:23 PM

It may be a filesize limitation of sed I guess. But I've had problems in the past escaping characters in sed and iirc that also involved disapearing files.

jschiwal · 04-14-2006, 07:09 AM

There are two things that you kept doing with the sed examples.

cat $file | sed '<sed program' >$file

First you could write this as sed '<sed program>' $file. You don't need the cat command.
Second if you redirect the output to the same file you use as the input, you will zero out the file before it starts. Either produce a different file, or use the '-i' option (in situ editing).

sed '<sed program>' "${file}" >"${file%.html}.php"
rm "$file"

or

sed -i '<sed program>' "${file}"
mv "$file" "${file%.html}.php"

Also, there is an insert command in sed.
sed '1i\
<? your php line ?>' $file >${file%.html}.php}

The newline is needed after the 'i\'

disorderly · 04-20-2006, 06:30 PM

thanks jschiwal for the explaination! i feel much better understanding WHY something didn't work. i put

Code:

sed '<sed program>' "${file}" >"${file%.html}.php"

into the program and it worked like a charm

i think i had given up on the use of the -i switch because my version of sed doesn't seem to accept it - just gives me an error. might be too old