LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 09-09-2019, 03:59 PM   #1
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Rep: Reputation: Disabled
loop sed to remove all strings in file


Hi , i want to use sed in a loop to remove directly all the lines containing the search variables , but in the end the file stays the same and i have no idea why .

Heres the code

Code:
# Ips already blocked in firewall , get how many they are
cntfr=$(wc -l "$path/fireips" | awk '{print$1}')	

# Sequence of code : Read firewall ip line by line , if ip
# Exists in logfile then remove that line from logfile
# if not exist then do nothing

for i in $(seq "$cntfr")
do
# read line i from file and get the ip

rdip=$(sed -n ${i}p < "$path/fireips")

# it may exist the subnet blocked in firewall , so filter the ip
# to be ex : 192.168.1.1 to 192.168.1.0/24

ip2="${rdip%.*}.0/24"

# Do a quick search with grep if the ip exists on logfile
ckip=$(grep "$rdip" < $cmlog)

#  Do a quick search with grep if the subnet of that ip exists on #logfile
cksb=$(grep "$ip2" < $cmlog)

# Case subnet output was not empty
if [[ ! -z "$cksb" ]]
then

# Remove all lines containg that ip from main log file
sed -i -e '/$rdip/d' $cmlog
fi

# Case ip exists in logfile
if [[ ! -z "$ckip" ]]
then
# remove that ip from logfile
sed -i -e '/$ckip/d' $cmlog
fi
done
The thing here is that code runs without any error , but in the end the logfile was not changed , this means that sed did not made the changes to current file .

Any Idea why ?

Last edited by pedropt; 09-09-2019 at 04:01 PM.
 
Old 09-09-2019, 04:37 PM   #2
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,992

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
is that the whole script?

if it is cmlog has not been declared


but you should see errors

Code:
foo=$(grep "foo" <$foobar ) 
echo $foo
[[ -z $foo ]] && echo empty || echo not empty
bash: $foobar: ambiguous redirect
empty
 
Old 09-09-2019, 04:42 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,078

Rep: Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900
Quote:
Originally Posted by pedropt View Post
Any Idea why ?
Single quotes inhibit interpolation of the bash variable.
 
Old 09-09-2019, 04:49 PM   #4
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,992

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
Ahh, I see it
if that is just part of the script

the problem is your sed


Code:
sed -i -e '/$rdip/d' $cmlog
'' will stop the shell expanding

Code:
rdip=foo
echo sed -i -e '/$rdip/d' $cmlog
echo sed -i -e '/'$rdip'/d' $cmlog
when debugging a script, use


Code:
set -x # shows the expansion
code to be debugged
more 
sed -i -e '/$rdip/d' $cmlog
more code
set +x # stops shoing teh expansion

or you can

Code:
bash -x ./script.sh
and see the lot

if you want some fun

on your shell
Code:
set -x
<tab><tab>
where <tab> is the tab key

set +x to turn off

remember
 
Old 09-09-2019, 05:13 PM   #5
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
That is just 5% of the script , and cmlog variable was already declared before .
cmlog is a file declared as a variable to be more easier .
 
Old 09-09-2019, 05:34 PM   #6
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,992

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
Quote:
Originally Posted by pedropt View Post
That is just 5% of the script , and cmlog variable was already declared before .
cmlog is a file declared as a variable to be more easier .
yeap

your log probably doesn't have spaces in it, but you should get in the habit of accounting for that

Code:
somecommand "${cmlog}"
 
Old 09-09-2019, 06:16 PM   #7
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
Quote:
yeap

your log probably doesn't have spaces in it, but you should get in the habit of accounting for that
Look , i dont know why this is happening , but the file still stays the same .
If makes you all be conscious relieve then here is is a sample of the log that is called "server.log"
Quote:
Fri Sep 6 09:36:54 2019; TCP; eth0; 46 bytes; from 77.247.110.80:55376 to 192.168.1.20:128; first packet (SYN)
Fri Sep 6 09:36:55 2019; ICMP; eth0; 84 bytes; from 13.234.221.180 to 192.168.1.20; echo req
Fri Sep 6 09:36:56 2019; ICMP; eth0; 84 bytes; from 3.112.14.218 to 192.168.1.20; echo req
Fri Sep 6 09:36:57 2019; TCP; eth0; 46 bytes; from 206.189.181.12:34377 to 192.168.1.20:23; first packet (SYN)
Fri Sep 6 09:37:03 2019; TCP; eth0; 46 bytes; from 206.189.58.99:54794 to 192.168.1.20:8160; first packet (SYN)
Fri Sep 6 09:37:05 2019; TCP; eth0; 46 bytes; from 206.189.181.12:34377 to 192.168.1.20:2323; first packet (SYN)
My question is pretty simple , why does not sed remove the lines containing those ips , or the ips that is reading in the firewall ? a good example is the ip 77.247.110.80 that already it is in the firewall .

Basically the whole script will get rid the ips already in firewall so it can proceed to the next stage , witch is counting how many times each ip not blocked in firewall connected to server and witch requests it made and witch ports and witch data .
But this last part his working perfectly .

Lets get one thing here clear , i have this script already running in server and running perfectly except with this new part of code i am adding , the reason i am adding these sed instructions is to get the script running more faster because when you get more that 40000 entries in the log file for 1 day then it will take at least 15 minutes to do all the job , but if i clear from the mainlog file the ips that dont need to be checked by the script because there is no need by the fact that server does not reply to any request from that ip , i clear from 40000 entries at least 10000 or much more from port scans or ddos attacks from noobies and get the script running more faster .
 
Old 09-09-2019, 06:19 PM   #8
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,719

Rep: Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034Reputation: 3034
Try re-reading the answers as you have been told why already in posts #3 and #4
 
Old 09-09-2019, 06:29 PM   #9
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.6
Posts: 3,635

Rep: Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205
See #3 and #4.
 
Old 09-09-2019, 06:43 PM   #10
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
Post 3 : Changing from :
Quote:
sed -i -e '/$rdip/d' $cmlog
to :

Quote:
sed -i -e "/$rdip/d" $cmlog
Gives me this error :

sed: -e expression #1, char 112: unterminated address regex
 
Old 09-09-2019, 06:47 PM   #11
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.6
Posts: 3,635

Rep: Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205Reputation: 1205
Quote:
Originally Posted by pedropt View Post
Post 3 : Changing from :
to :
Gives me this error :

sed: -e expression #1, char 112: unterminated address regex
What’s in the variable when that happens?
 
Old 09-09-2019, 06:56 PM   #12
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
i am not pretty sure because sed is deleting a lot of lines with that ip , but i believe that the problem is a front slash /

All special characters used in the log as far as i can see :

:
,
;
(
* - rarely appears
)
/ - when server gets : Connection reset; 1 packets, 40 bytes, avg flow rate 0.00 kbits/s



So , if i use [ in sed will it work ? [] are not used in log file, so probably the sed sentence could be made with this symbol , but i am not sure .


NOTE :
Misunderstood your question .

The variable have an ip address , show it should be 111.111.111.111 , probably the . (dot) is the problem according to what you are asking , if you asked in the log objective then are all those symbols i wrote before .

Sed is looking for a specific ip address and if it finds it then it should delete that line and all others with same result in the log file

Last edited by pedropt; 09-09-2019 at 07:01 PM.
 
Old 09-09-2019, 07:06 PM   #13
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
Found The issue .
My Mistake , sorry all .

The issue was that i was sending to sed the output search from grep , instead the ip address it self , so it is pretty normal for sed to crash the sentence .

The correct code :

Code:
cntfr=$(wc -l "$path/fireips" | awk '{print$1}')	
for i in $(seq "$cntfr")
do
rdip=$(sed -n ${i}p < "$path/fireips")
ip2="${rdip%.*}.0/24"
ckip=$(grep "$rdip" < $cmlog)
cksb=$(grep "$ip2" < $cmlog)
if [[ ! -z "$cksb" ]]
then
sed -i -e "/$rdip/d" $cmlog
fi
if [[ ! -z "$ckip" ]]
then
sed -i -e "/$rdip/d" $cmlog
fi
done
I was able to figure it out because scasey pointed the question about what was sed searching , and in my 1st code i mistaken in the variable to be sent to sed , i used "$ckip" , witch is a lot of output from log because it is a grep search to see if exists .
Now that i have look at 1st code , i notice that sed crashed in the 2nd if sentence :

Quote:
sed -i -e '/$ckip/d' $cmlog
the 1st if sentence was right , 1st if sentece activates sed if the subnet of some ip already exists in the logfile , and the 2nd sentence activates the 2nd sed instruction in case if only an ip exists .

To give you guys an idea why this code to be added then look at this :

I have here a sample log for half a day witch have 2.9M of text .
After clearing the dns requests , upnp requests and network arps i got 1.1M of text to be checked , now after this next clean up i only have 964Kbit witch i really need to check , because it is what matters and the server is responding to it .

Last edited by pedropt; 09-09-2019 at 07:21 PM.
 
Old 09-09-2019, 07:14 PM   #14
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,992

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
debug it like this

Code:
bash -xe /path/to/your/script.sh 2>&1 | tee debug.log
-x will show what bash sees
-e will stop the script at the first error

if your script stop "too soon" use bash -x


tee will output to both stdout *and* debug.log

( use tee -a to append to the file )
 
Old 09-09-2019, 07:37 PM   #15
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 217

Original Poster
Rep: Reputation: Disabled
This Script have 1315 lines of code and it is not yet completed , only the most important parts , this script interacts with everything in server , since restarting services , cleaning logs , editing firewall rules , checking fail2ban jails , check the country of specific ip , scan that ip , etc .... , i have been adding functions to it as i look it was necessary , now it is working in manual mode , this means i connect to server and run it and then i give the instruction in the script to do what was programmed for , but in future i want to make it as service in automatic mode .
This means that script will be looking at firewall logs all the time , and if by example detects a 404 error in https service then it will automatically knows that client made a request not available in server , this could be a web crawler script running on client side , from this point depending on the hits from that ip it can automatically add that ip to the blocked ips in firewall and reload the rules . This also can be done with some patterns that i can add to some file to script check before doing anything .
This technique can be used also for the mail server and all other services may be running
in the server . Right now is pretty simple i did it in a rush , took 6 or 7 hours to made it because have many logic routines to activate "in case of" , it is a bit messy and i have to make a new one more clear and more faster than the current one , but i dont know when i will get some time to do it .
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
replace strings in a file using strings from another file xpto09 Linux - Newbie 3 01-28-2016 06:11 PM
[Cygwin, sed] Using filenames as both files and search strings within sed lingh Linux - Newbie 5 10-20-2012 10:38 AM
BASH: replace strings in on file by the strings in another one cristalp Programming 5 10-28-2011 09:47 AM
suing sed to remove certain list of strings from a file ??? HuMan-BiEnG Linux - Distributions 11 10-13-2010 09:04 AM
[SOLVED] Searching and replacing strings in a file with strings in other files xndd Linux - Newbie 16 07-29-2010 02:40 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration