Trying to make a script that will search and replace text in a file

Jongi · 06-15-2007, 04:52 AM

Quote:

route add -net 164.88.0.0/16 $LOCAL
route add -net 164.146.0.0/15 $LOCAL
route add -net 164.147.34.0/24 $LOCAL
route add -net 164.147.241.195/32 $LOCAL
route add -net 164.148.0.0/14 $LOCAL

I have a file with roughly 800 such lines. I want to run it as a script. I need to do two things to manipulate this file automatically:

1. Change all $LOCAL to ppp1.
2. Where the IP addresses end in /32 I need to change -net to -host.

With respect to 1, I could open up the file in OOo and do a search and replace I suppose.

But essentially I will de downloading the file from here and I will want these changes to be made automatically in the download script.

Code:

# sed 's/$LOCAL/ppp1/g' localroutes4.txt > newroutes.txt

I am thinking that newroutes.txt will now have ppp1 instead of $LOCAL for all 800+ lines.

It then strikes me that I can probably use sed again in an if statement with loop, to pick up all the lines with /32 and change net to host. This part I am not sure about. So the script would look something like this:

Code:

!#/bin/sh
wget http://alm.za.net/ip/localroutes4.txt
sed 's/$LOCAL/ppp1/g' localroutes4.txt > newroutes.txt
cat newroutes.txt | while read routes do
if [ $routes =~ "/32" ]
then
routes=`echo $routes | sed s/net/host/ > hosts.txt`
fi
done

Here I believe there should be a file called hosts.txt with only the lines with /32 and where the net has been changed to host. The one thing that strikes me as possibly going wrong is that in the end I will have a file (hosts.txt) with only one line rather than the /32 lines with net replaced by host. Would the use of >> instead of > help?

What I would then like to do is to create a second file (call it net.txt), which will be newroutes.txt but without any of the lines with /32. Once I have that file I would then do a cat net.txt hosts.txt > routes.sh and then chmod +x routes.sh. I'm sure I'll easily find a way to have !#/bin/sh added as the first line of routes.sh.

acid_kewpie · 06-15-2007, 05:05 AM

Code:

curl http://domain.com/file.txt | while read line
do 
  [ "$( echo $line | grep \/32 )" ] && line=$( echo $line | sed s/net/host/ ) 
  echo $line | sed s/\$LOCAL/ppp1/
done > /tmp/newroutes.txt

should abuot do it. it's a little long winded, and totally untested, but hopefully readable enough to understand. note that $ in sed is the end of the line, not a dollar sign, so you need to escape it to treat it as a normal character.

Jongi · 06-15-2007, 05:11 AM

Thanks. I'm a little confused by what you say in the last line. I'll just copy and paste it as is and hopefully that will work.

I do like you saying yours is long winded and then I compare yours to mine

acid_kewpie · 06-15-2007, 05:14 AM

if you mean the $ stuff, sed s/string$/replacement/ would replace "string" with "replacement" as long as it was the very last thing on the line. so sed s/$string/replacement/ would replace "string" with "replacement" if it was *AFTER* the end of the line... which is clearly nonsense.

well it was about 3 lines longer originally... you could do most if that with direct bash commands, no seds or anything, but it's pretty confusing, and i always forget it...

ta0kira · 06-15-2007, 09:35 AM

Maybe this?

Code:

sed "s* \$LOCAL* ppp1*" localroutes4.txt | sed -r "s* -net ([^ ]+)/32 * -host \1/32 *" > newroutes.txt

ta0kira

edit: fixed first sed expression.

Jongi · 06-15-2007, 11:55 AM

Quote:

Originally Posted by acid_kewpie

Code:

curl http://domain.com/file.txt | while read line
do 
  [ "$( echo $line | grep \/32 )" ] && line=$( echo $line | sed s/net/host/ ) 
  echo $line | sed s/\$LOCAL/ppp1/
done > /tmp/newroutes.txt

should abuot do it. it's a little long winded, and totally untested, but hopefully readable enough to understand. note that $ in sed is the end of the line, not a dollar sign, so you need to escape it to treat it as a normal character.

Does the job wonderfully :thup:

ghostdog74 · 06-15-2007, 09:52 PM

Code:

awk '{
        $NF="ppp1";
        if ( $0 ~ /\/32/ ) {
           gsub("net","host",$0)
        }       
        print $0
     }

' "file"

Jongi · 06-20-2007, 05:22 AM

If I can impose again, it seems the linux on my router (BusyBox v0.61.pre (2006.11.03-09:13+0000) Built-in shell (ash)) does not support the notation xxx.xxx.xxx.xxx/yy (while my Debian box does). So what I then need to do is to replace /yy with " netmask zzz.zzz.zzz.zzz" (note the space before netmask) . Now I would prefer to do this to the output of acid's script, ie /tmp/newroutes.txt. So would the following work:

Code:

cat /tmp/newroutes.txt | while read line
do 
  [ "$( echo $line | grep \/32 )" ] && line=$( echo $line | sed s//32/ netmask 255.255.255.255/ )
  echo $line
  [ "$( echo $line | grep \/14 )" ] && line=$( echo $line | sed s//14/ netmask 255.252.000.000/ ) 
  echo $line
done > /tmp/netmaskroutes.txt

I've only shown for two subnets but I'd look to put all 32 in the script. So route add -host 164.147.241.195/32 ppp1 should now look like route add -host 164.147.241.195 netmask 255.255.255.255 ppp1 (with the space between 195 and netmask).

acid_kewpie · 06-20-2007, 06:38 AM

similar to the above, you can see that the use of the / character within text clearly makes the sed command illogical... how do you seperate the /'s of sed with the /'s of the data? try "s/\/14/\ netmask\ 255.252.0.0/" instead, again it's the \ which makes the / and also the space literal characters to act upon in the data.

Jongi · 06-20-2007, 08:04 AM

Thanks. Yeah I still have problems with using sed. But you picked up the issue I had.

So the script now looks like this:

Code:

#!/bin/sh
cd ~
curl http://alm.za.net/ip/localroutes4.txt | while read line
do
  [ "$( echo $line | grep \/32 )" ] && line=$( echo $line | sed s/net/host/ )
      echo $line | sed s/\$LOCAL/ppp1/
          done > ~/newroutes.txt
cat ~/newroutes.txt | while read route
do
  [ "$( echo $route | grep \/1 )" ] && route=$( echo $route | sed "s/\/1/\ netmask\ 128.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/2 )" ] && route=$( echo $route | sed "s/\/2/\ netmask\ 192.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/3 )" ] && route=$( echo $route | sed "s/\/3/\ netmask\ 224.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/4 )" ] && route=$( echo $route | sed "s/\/4/\ netmask\ 240.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/5 )" ] && route=$( echo $route | sed "s/\/5/\ netmask\ 248.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/6 )" ] && route=$( echo $route | sed "s/\/6/\ netmask\ 252.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/7 )" ] && route=$( echo $route | sed "s/\/7/\ netmask\ 254.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/8 )" ] && route=$( echo $route | sed "s/\/8/\ netmask\ 255.0.0.0/" )
    echo $route
  [ "$( echo $route | grep \/9 )" ] && route=$( echo $route | sed "s/\/9/\ netmask\ 255.128.0.0/" )
    echo $route
  [ "$( echo $route | grep \/10 )" ] && route=$( echo $route | sed "s/\/10/\ netmask\ 255.192.0.0/" )
    echo $route
  [ "$( echo $route | grep \/11 )" ] && route=$( echo $route | sed "s/\/11/\ netmask\ 255.224.0.0/" )
    echo $route
  [ "$( echo $route | grep \/12 )" ] && route=$( echo $route | sed "s/\/12/\ netmask\ 255.240.0.0/" )
    echo $route
  [ "$( echo $route | grep \/13 )" ] && route=$( echo $route | sed "s/\/13/\ netmask\ 255.248.0.0/" )
    echo $route
  [ "$( echo $route | grep \/14 )" ] && route=$( echo $route | sed "s/\/14/\ netmask\ 255.252.0.0/" )
    echo $route
  [ "$( echo $route | grep \/15 )" ] && route=$( echo $route | sed "s/\/15/\ netmask\ 255.254.0.0/" )
    echo $route
  [ "$( echo $route | grep \/16 )" ] && route=$( echo $route | sed "s/\/16/\ netmask\ 255.255.0.0/" )
    echo $route
  [ "$( echo $route | grep \/17 )" ] && route=$( echo $route | sed "s/\/17/\ netmask\ 255.255.128.0/" )
    echo $route
  [ "$( echo $route | grep \/18 )" ] && route=$( echo $route | sed "s/\/18/\ netmask\ 255.255.192.0/" )
    echo $route
  [ "$( echo $route | grep \/19 )" ] && route=$( echo $route | sed "s/\/19/\ netmask\ 255.255.224.0/" )
    echo $route
  [ "$( echo $route | grep \/20 )" ] && route=$( echo $route | sed "s/\/20/\ netmask\ 255.255.240.0/" )
    echo $route
  [ "$( echo $route | grep \/21 )" ] && route=$( echo $route | sed "s/\/21/\ netmask\ 255.255.248.0/" )
    echo $route
  [ "$( echo $route | grep \/22 )" ] && route=$( echo $route | sed "s/\/22/\ netmask\ 255.255.252.0/" )
    echo $route
  [ "$( echo $route | grep \/23 )" ] && route=$( echo $route | sed "s/\/23/\ netmask\ 255.255.254.0/" )
    echo $route
  [ "$( echo $route | grep \/24 )" ] && route=$( echo $route | sed "s/\/24/\ netmask\ 255.255.255.0/" )
    echo $route
  [ "$( echo $route | grep \/25 )" ] && route=$( echo $route | sed "s/\/25/\ netmask\ 255.255.255.128/" )
    echo $route
  [ "$( echo $route | grep \/26 )" ] && route=$( echo $route | sed "s/\/26/\ netmask\ 255.255.255.192/" )
    echo $route
  [ "$( echo $route | grep \/27 )" ] && route=$( echo $route | sed "s/\/27/\ netmask\ 255.255.255.224/" )
    echo $route
  [ "$( echo $route | grep \/28 )" ] && route=$( echo $route | sed "s/\/28/\ netmask\ 255.255.255.240/" )
    echo $route
  [ "$( echo $route | grep \/29 )" ] && route=$( echo $route | sed "s/\/29/\ netmask\ 255.255.255.248/" )
    echo $route
  [ "$( echo $route | grep \/30 )" ] && route=$( echo $route | sed "s/\/30/\ netmask\ 255.255.255.252/" )
    echo $route
  [ "$( echo $route | grep \/32 )" ] && route=$( echo $route | sed "s/\/32/\ netmask\ 255.255.255.255/" )
    echo $route
done > ~/netmaskroutes.txt

Here's the interesting bit, not all the lines are being processed. For instance some of the /32 lines are not having the /32 replaced by netmask 255.255.255.255. Yet if I delete all the preceding entries related to adding netmask and leave the one for /32, the netmask is added. I don't touch anything with respect to that line.

It seems it is something to do with the number of entries being processed.

Jongi · 06-20-2007, 09:39 AM

I've tried breaking it down into groups to process. As below:

Code:

echo "Subnet 21-25 (of 32)"
cat ~/newroutes.txt | while read route
do
  [ "$( echo $route | grep \/21 )" ] && route=$( echo $route | sed "s/\/21/\ netmask\ 255.255.248.0/" )
    echo $route
  [ "$( echo $route | grep \/22 )" ] && route=$( echo $route | sed "s/\/22/\ netmask\ 255.255.252.0/" )
    echo $route
  [ "$( echo $route | grep \/23 )" ] && route=$( echo $route | sed "s/\/23/\ netmask\ 255.255.254.0/" )
    echo $route
  [ "$( echo $route | grep \/24 )" ] && route=$( echo $route | sed "s/\/24/\ netmask\ 255.255.255.0/" )
    echo $route
  [ "$( echo $route | grep \/25 )" ] && route=$( echo $route | sed "s/\/25/\ netmask\ 255.255.255.128/" )
    echo $route
done > net5.txt
cat net5.txt | grep netmask > netmask5.txt
#rm -f net5.txt

So check this out

Code:

[jongi:~#] cat newroutes.txt | grep 216.104.192.0
route add -net 216.104.192.0/21 ppp1
[jongi:~#] cat netmask5.txt | grep 216.104.192.0
route add -net 216.104.192.0 netmask 255.255.248.0 ppp1
route add -net 216.104.192.0 netmask 255.255.248.0 ppp1
route add -net 216.104.192.0 netmask 255.255.248.0 ppp1
route add -net 216.104.192.0 netmask 255.255.248.0 ppp1
route add -net 216.104.192.0 netmask 255.255.248.0 ppp1

It is clearly processing the line everytime rather than only when it should be looking for /21.

Jongi · 06-21-2007, 01:56 AM

I guess I'll try the long way round the problem and do each subnet seperately instead of in blocks of 5.

Code:

echo "Subnet 21"
cat ~/newroutes.txt | while read route
do
  [ "$( echo $route | grep \/21 )" ] && route=$( echo $route | sed "s/\/21/\ netmask\ 255.255.248.0/" )
    echo $route
done > net21.txt

echo "Subnet 22"
cat ~/newroutes.txt | while read route
do
  [ "$( echo $route | grep \/22 )" ] && route=$( echo $route | sed "s/\/22/\ netmask\ 255.255.252.0/" )
    echo $route
done > net22.txt

acid_kewpie · 06-21-2007, 02:17 AM

how's about this...

Code:

cat newroutes.txt | while read route
do 
  export $(ipcalc -m $(echo $route | sed -e 's/^.*net\ \([0-9\.\/]*\).*$/\1/'))
  echo $route | sed -e "s/\/[0-9]*/ netmask $NETMASK/"
done

ipcalc takes an input like 192l168.0.1/24 and with the -m optino returns "NETMASK=255.255.255.0" in this example. then you just need to replace the CIDR notation with the conventional netmask. it's a little obscure as you only see reference to the $NETMASK variable once, but it's being set implicitly in the previous command.

Jongi · 06-21-2007, 02:36 AM

Chris,

Thanks for your help. I'll only be able to try it probably tomorrow or over the weekend.

If I get what you saying, the ipcalc command (with the -m option), is able to read the /x part in the IP address and churn out the corresponding subnet. Is the part you say I need to replace the bold part below?

Quote:

export $(ipcalc -m $(echo $route | sed -e 's/^.*net\ $[0-9\.\/]*$.*$/\1/'))

Thanks again.

acid_kewpie · 06-21-2007, 03:10 AM

not sure what you mean.. *YOU* don't need to replace any part. the "\1" means that the first field in the first half of the sed command (i.e. the contents of [0-9\.\/]* as delimited by the escaped brackets $ and $ ) is put there, whatever it is. that sed basically finds a subnet, e.g. 1.2.3.4/5 and removes everything either side of it.