[SOLVED] loop sed to remove all strings in file

pedropt · 09-09-2019, 03:59 PM

Hi , i want to use sed in a loop to remove directly all the lines containing the search variables , but in the end the file stays the same and i have no idea why .

Heres the code

Code:

# Ips already blocked in firewall , get how many they are
cntfr=$(wc -l "$path/fireips" | awk '{print$1}')	

# Sequence of code : Read firewall ip line by line , if ip
# Exists in logfile then remove that line from logfile
# if not exist then do nothing

for i in $(seq "$cntfr")
do
# read line i from file and get the ip

rdip=$(sed -n ${i}p < "$path/fireips")

# it may exist the subnet blocked in firewall , so filter the ip
# to be ex : 192.168.1.1 to 192.168.1.0/24

ip2="${rdip%.*}.0/24"

# Do a quick search with grep if the ip exists on logfile
ckip=$(grep "$rdip" < $cmlog)

#  Do a quick search with grep if the subnet of that ip exists on #logfile
cksb=$(grep "$ip2" < $cmlog)

# Case subnet output was not empty
if [[ ! -z "$cksb" ]]
then

# Remove all lines containg that ip from main log file
sed -i -e '/$rdip/d' $cmlog
fi

# Case ip exists in logfile
if [[ ! -z "$ckip" ]]
then
# remove that ip from logfile
sed -i -e '/$ckip/d' $cmlog
fi
done

The thing here is that code runs without any error , but in the end the logfile was not changed , this means that sed did not made the changes to current file .

Any Idea why ?

Firerat · 09-09-2019, 04:37 PM

is that the whole script?

if it is cmlog has not been declared

but you should see errors

Code:

foo=$(grep "foo" <$foobar ) 
echo $foo
[[ -z $foo ]] && echo empty || echo not empty

bash: $foobar: ambiguous redirect
empty

syg00 · 09-09-2019, 04:42 PM

Quote:

Originally Posted by pedropt

Any Idea why ?

Single quotes inhibit interpolation of the bash variable.

Firerat · 09-09-2019, 04:49 PM

Ahh, I see it
if that is just part of the script

the problem is your sed

Code:

sed -i -e '/$rdip/d' $cmlog

'' will stop the shell expanding

Code:

rdip=foo
echo sed -i -e '/$rdip/d' $cmlog
echo sed -i -e '/'$rdip'/d' $cmlog

when debugging a script, use

Code:

set -x # shows the expansion
code to be debugged
more 
sed -i -e '/$rdip/d' $cmlog
more code
set +x # stops shoing teh expansion

or you can

Code:

bash -x ./script.sh

and see the lot

if you want some fun

on your shell

Code:

set -x
<tab><tab>

where <tab> is the tab key

set +x to turn off

remember

pedropt · 09-09-2019, 05:13 PM

That is just 5% of the script , and cmlog variable was already declared before .
cmlog is a file declared as a variable to be more easier .

Firerat · 09-09-2019, 05:34 PM

Quote:

Originally Posted by pedropt

That is just 5% of the script , and cmlog variable was already declared before .
cmlog is a file declared as a variable to be more easier .

yeap

your log probably doesn't have spaces in it, but you should get in the habit of accounting for that

Code:

somecommand "${cmlog}"

pedropt · 09-09-2019, 06:16 PM

Quote:

yeap

your log probably doesn't have spaces in it, but you should get in the habit of accounting for that

Look , i dont know why this is happening , but the file still stays the same .
If makes you all be conscious relieve then here is is a sample of the log that is called "server.log"

Quote:

Fri Sep 6 09:36:54 2019; TCP; eth0; 46 bytes; from 77.247.110.80:55376 to 192.168.1.20:128; first packet (SYN)
Fri Sep 6 09:36:55 2019; ICMP; eth0; 84 bytes; from 13.234.221.180 to 192.168.1.20; echo req
Fri Sep 6 09:36:56 2019; ICMP; eth0; 84 bytes; from 3.112.14.218 to 192.168.1.20; echo req
Fri Sep 6 09:36:57 2019; TCP; eth0; 46 bytes; from 206.189.181.12:34377 to 192.168.1.20:23; first packet (SYN)
Fri Sep 6 09:37:03 2019; TCP; eth0; 46 bytes; from 206.189.58.99:54794 to 192.168.1.20:8160; first packet (SYN)
Fri Sep 6 09:37:05 2019; TCP; eth0; 46 bytes; from 206.189.181.12:34377 to 192.168.1.20:2323; first packet (SYN)

My question is pretty simple , why does not sed remove the lines containing those ips , or the ips that is reading in the firewall ? a good example is the ip 77.247.110.80 that already it is in the firewall .

Basically the whole script will get rid the ips already in firewall so it can proceed to the next stage , witch is counting how many times each ip not blocked in firewall connected to server and witch requests it made and witch ports and witch data .
But this last part his working perfectly .

Lets get one thing here clear , i have this script already running in server and running perfectly except with this new part of code i am adding , the reason i am adding these sed instructions is to get the script running more faster because when you get more that 40000 entries in the log file for 1 day then it will take at least 15 minutes to do all the job , but if i clear from the mainlog file the ips that dont need to be checked by the script because there is no need by the fact that server does not reply to any request from that ip , i clear from 40000 entries at least 10000 or much more from port scans or ddos attacks from noobies and get the script running more faster .

grail · 09-09-2019, 06:19 PM

Try re-reading the answers as you have been told why already in posts #3 and #4

scasey · 09-09-2019, 06:29 PM

See #3 and #4.

pedropt · 09-09-2019, 06:43 PM

Post 3 : Changing from :

Quote:

sed -i -e '/$rdip/d' $cmlog

to :

Quote:

sed -i -e "/$rdip/d" $cmlog

Gives me this error :

sed: -e expression #1, char 112: unterminated address regex

scasey · 09-09-2019, 06:47 PM

Quote:

Originally Posted by pedropt

Post 3 : Changing from :
to :
Gives me this error :

sed: -e expression #1, char 112: unterminated address regex

What’s in the variable when that happens?

pedropt · 09-09-2019, 06:56 PM

i am not pretty sure because sed is deleting a lot of lines with that ip , but i believe that the problem is a front slash /

All special characters used in the log as far as i can see :

:
,
;
(
* - rarely appears
)
/ - when server gets : Connection reset; 1 packets, 40 bytes, avg flow rate 0.00 kbits/s

So , if i use [ in sed will it work ? [] are not used in log file, so probably the sed sentence could be made with this symbol , but i am not sure .

NOTE :
Misunderstood your question .

The variable have an ip address , show it should be 111.111.111.111 , probably the . (dot) is the problem according to what you are asking , if you asked in the log objective then are all those symbols i wrote before .

Sed is looking for a specific ip address and if it finds it then it should delete that line and all others with same result in the log file

pedropt · 09-09-2019, 07:06 PM

Found The issue .
My Mistake , sorry all .

The issue was that i was sending to sed the output search from grep , instead the ip address it self , so it is pretty normal for sed to crash the sentence .

The correct code :

Code:

cntfr=$(wc -l "$path/fireips" | awk '{print$1}')	
for i in $(seq "$cntfr")
do
rdip=$(sed -n ${i}p < "$path/fireips")
ip2="${rdip%.*}.0/24"
ckip=$(grep "$rdip" < $cmlog)
cksb=$(grep "$ip2" < $cmlog)
if [[ ! -z "$cksb" ]]
then
sed -i -e "/$rdip/d" $cmlog
fi
if [[ ! -z "$ckip" ]]
then
sed -i -e "/$rdip/d" $cmlog
fi
done

I was able to figure it out because scasey pointed the question about what was sed searching , and in my 1st code i mistaken in the variable to be sent to sed , i used "$ckip" , witch is a lot of output from log because it is a grep search to see if exists .
Now that i have look at 1st code , i notice that sed crashed in the 2nd if sentence :

Quote:

sed -i -e '/$ckip/d' $cmlog

the 1st if sentence was right , 1st if sentece activates sed if the subnet of some ip already exists in the logfile , and the 2nd sentence activates the 2nd sed instruction in case if only an ip exists .

To give you guys an idea why this code to be added then look at this :

I have here a sample log for half a day witch have 2.9M of text .
After clearing the dns requests , upnp requests and network arps i got 1.1M of text to be checked , now after this next clean up i only have 964Kbit witch i really need to check , because it is what matters and the server is responding to it .

Firerat · 09-09-2019, 07:14 PM

debug it like this

Code:

bash -xe /path/to/your/script.sh 2>&1 | tee debug.log

-x will show what bash sees
-e will stop the script at the first error

if your script stop "too soon" use bash -x

tee will output to both stdout *and* debug.log

( use tee -a to append to the file )

pedropt · 09-09-2019, 07:37 PM

This Script have 1315 lines of code and it is not yet completed , only the most important parts , this script interacts with everything in server , since restarting services , cleaning logs , editing firewall rules , checking fail2ban jails , check the country of specific ip , scan that ip , etc .... , i have been adding functions to it as i look it was necessary , now it is working in manual mode , this means i connect to server and run it and then i give the instruction in the script to do what was programmed for , but in future i want to make it as service in automatic mode .
This means that script will be looking at firewall logs all the time , and if by example detects a 404 error in https service then it will automatically knows that client made a request not available in server , this could be a web crawler script running on client side , from this point depending on the hits from that ip it can automatically add that ip to the blocked ips in firewall and reload the rules . This also can be done with some patterns that i can add to some file to script check before doing anything .
This technique can be used also for the mail server and all other services may be running
in the server . Right now is pretty simple i did it in a rush , took 6 or 7 hours to made it because have many logic routines to activate "in case of" , it is a bit messy and i have to make a new one more clear and more faster than the current one , but i dont know when i will get some time to do it .