LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-14-2019, 11:08 AM   #1
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Rep: Reputation: Disabled
Loop , use sed to remove and update loop final count


Looks complicated but it is not .
Here it is the thing , i want to search and remove directly to the file using sed , but the loop sequence will be updated after sed deleted some lines .

Imagining that loop starts inicialy a 1 and supose to end at 100 , but then some pattern was found and sed have to delete 10 lines , then loop will end at new variable witch is 100 -10 = 90 , by this the initial variable in loop is updated , but this happens consequently in this loop because sed is removing if found what i am searching .

Now the problem is that i stay in a endless loop without change because sed is not replacing the original file with the new updates .

I already had created before a similar thread , but this one a very different .

The Code

Code:
# count the lines of file to loop
cntfr=$(wc -l "$cmlog" | awk '{print$1}')

# start sequence until variable cntfr

for i in $(seq "$cntfr")
do

# read the line and grab the ip 
# ip will be xxx.xxx.xxx.xxx:port , so this next code
# will remove everything after the : and will get only 
# the cleaned ip

var1=$(sed -n ${i}p < $cmlog | awk '{print$11}' | cut -f1 -d":")

# this one i learned here , witch is put something like
# ex: 111.111.111.111 to 111.111.111.0/24
# this will be to see if subnet exists on a file ahead

ip2="${var1%.*}.0/24"

# fireips is a file containing all the blocked ips and 
# subnets in the firewall
# search for the subnet in fireips file
cksb=$(grep "$ip2" < "$path/fireips")

# case something was found
if [[ ! -z "$cksb" ]]
then

# remove with sed lines containing that IP from cmlog
sed -i -e '/"$var1"/d' $cmlog 

# inform user that ip subnet was found and was deleted

echo " - Existent Subnet : $ip2  for IP : $var1 - Cleaned"

fi

# objective of this next line is to check if the file #changed still have the initial number of lines , in case
#does not then update its value in the loop above

cntfr=$(wc -l "$cmlog" | awk '{print$1}')
done
An example from cmlog can be found here
https://pastebin.com/apxg0DrB

What happens in this code is that grep finds a value but then sed ahead do not remove the lines directly in file .

There is no error in code that could give any clew .
 
Old 10-14-2019, 11:25 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,842

Rep: Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308
sed will not evaluate $var1 inside ' ' (single quotes). You have to use " " for that.
 
1 members found this post helpful.
Old 10-14-2019, 11:44 AM   #3
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
yup , thank you .
Was that .
However i have another issue .
The problem is the value in the loop sequence .
If after sed finishes its job and after the count to get in the loop i get a number inferior than the $i then i get an error :

sed: -e expression #1, char 0: no previous regular expression

the problem here is not sed , the problem is that sed have nothing to find .
I think i have to set an if statement in the beginning of loop to check if "cntfr" is inferior to "i" then to set i at "cntfr" .
 
Old 10-14-2019, 11:51 AM   #4
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
Fixed .

Code:
cntfr=$(wc -l "$cmlog" | awk '{print$1}')
for i in $(seq "$cntfr")
do

if [[ "$cntfr" -le "$i" ]]
then
cntrf="$i"
else
var1=$(sed -n ${i}p < $cmlog | awk '{print$11}' | cut -f1 -d":")
ip2="${var1%.*}.0/24"
cksb=$(grep "$ip2" < "$path/fireips")
if [[ ! -z "$cksb" ]]
then
sed -i -e "/$var1/d" $cmlog 
echo " - Existent Subnet : $ip2  for IP : $var1 - Cleaned"
fi
cntfr=$(wc -l "$cmlog" | awk '{print$1}')
fi
done
 
Old 10-14-2019, 12:00 PM   #5
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.9.2009
Posts: 5,727

Rep: Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211
This is mostly a style comment.
When doing a task like this, I will usually not use the -i option to sed, but rather direct the output to a new file. This would eliminate the problems with the line count changing while the loop is running.

The last line of the script would then mv the new file to the original file, replacing it. Just food for thought.
 
1 members found this post helpful.
Old 10-14-2019, 12:36 PM   #6
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,842

Rep: Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308
additionally you can avoid pipe chains (like sed|awk|cut), usually they can be replaced by a single command, but that is probably not really important.
 
Old 10-15-2019, 04:41 PM   #7
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
yup , but i prefer this way because i am dealing with logs with 70M for a single day , and if i dont check 1 day and i take 2 or 3 days to check the logs then they will be huge .
Also because when you have a single string to remove then your way would be simple , but when you have more than 1000 ips to remove from a log because they are already blocked and then there is no need to see what they were doing , then the best approach and faster way for me is this one .
 
Old 10-15-2019, 11:45 PM   #8
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 2,683

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by pedropt View Post
yup , but i prefer this way because i am dealing with logs with 70M for a single day , and if i dont check 1 day and i take 2 or 3 days to check the logs then they will be huge .
Also because when you have a single string to remove then your way would be simple , but when you have more than 1000 ips to remove from a log because they are already blocked and then there is no need to see what they were doing , then the best approach and faster way for me is this one .
but your script is slow

sed | awk | cut ... is just silly

replace the
wc -l, for loop , sed

with while read loop

replace the awk/cut with bash

replace testing the grabbed output of grep with testing the exit code of grep
( a match is exit 0 )

Code:
#!/bin/bash
# blind as I have no idea what the cmlog looks like,
# but I can guess from your awk ;)
while read -a cmlogline
do
    grep -q "${cmlogline[10]/\.+([0-9]):*/.0\/24}" "$path/fireips" \
        && printf "Existent Subnet : %s for IP : %s - Cleaned\n" \
            ${cmlogline[10]/\.+([0-9]):*/.0\/24} \
            ${cmlogline[10]%:*} \
        || cat - <<<"${cmlogline[@]}" >> "${newcmlog}"
done < $cmlog
mv "${newcmlog}" "$cmlog"
ok, probably not easy for you to read

this is a bit more steppy
Code:
#!/bin/bash
while read -a cmlogline
do
    foo2="${cmlogline[10]%:*}"
    foo1="${foo2%.*}.0/24"
    grep -q "$foo1" "$path/fireips" \
        && printf "Existent Subnet : %s for IP : %s - Cleaned\n" \
            ${foo1} \
            ${foo2} \
        || cat - <<<"${cmlogline[@]}" >> "${newcmlog}"
done < $cmlog
mv "${newcmlog}" "$cmlog"
binary conditionals can be messy, a traditional if then else is safer

Code:
#!/bin/bash
while read -a cmlogline
do
    foo2="${cmlogline[10]%:*}"
    foo1="${foo2%.*}.0/24"
    if ( grep -q "$foo1" "$path/fireips" )
    then
        printf "Existent Subnet : %s for IP : %s - Cleaned\n" ${foo1} ${foo2}
    else
        cat - <<<"${cmlogline[@]}" >> "${newcmlog}"
    fi
done < $cmlog
mv "${newcmlog}" "$cmlog"
now, if the majority of the lines 'Stay' it might be quicker to run a ( sed -i s/$var/d ) than catting each line to a new file

you could "save up" your removes and run a single sed at the end

Code:
#!/bin/bash
CheckLogs () {
removethese="" # zero length
while read -a cmlogline
do
    foo2="${cmlogline[10]%:*}"
    foo1="${foo2%.*}.0/24"
    if ( grep -q "$foo1" "$path/fireips" )
    then
        removethese+="${foo2}\|"
        # if you want to slow it down print stuff
    fi
done < $cmlog
return ${#removethese}
}
CleanLog () {
sed -i "/${removethese%\|}/d" "$cmlog"
# since removethese is a var should be ok with ARG_MAX
}

CheckLogs || CleanLog
# the count of removethese is used as return value of CheckLogs(),
# if it is none zero run CleanLog()
my moto, if you are going to use bash, use bash!

as a bonus

Code:
while read ip
do
    echo "${ip}"
    echo "${ip%.*}.0/24"
done < <(awk '{sub(/:.*/,"",$11);print $11}' "$cmlog")
this is probably not going to work
Code:
while read ip
do
    echo "${ip}"
    echo "${ip%.*}.0/24"
done < <(sed -E 's/.* ([0-9]{1,3}(\.[0-9]{1,3}){3}):[0-9]+.*/\1/' "$cmlog" )
Code:
while read ip
do
    echo "${ip%:*}"
    echo "${ip%.*}.0/24"
done < <(grep -Eo "([0-9]{1,3}(\.[0-9]{1,3}){3}):[0-9]+" "$cmlog" )

awk is probably the most useful, you could write the script in awk and do away with bash
but unless you are doing some serious number crunching, bash is going to be much easier.
 
1 members found this post helpful.
Old 10-16-2019, 09:04 AM   #9
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,842

Rep: Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308
Quote:
Originally Posted by pedropt View Post
yup , but i prefer this way because
What I would like to say is: try to simplify your script. I did not suggest you to modify the logic, just it looks a bit complicated for me. Firerat already suggested a few ideas you can apply. And we can give others if you are interested.
Anyway, it is your script and you do not need to modify it for me.
 
Old 10-16-2019, 12:52 PM   #10
pedropt
Member
 
Registered: Aug 2014
Distribution: Devuan
Posts: 345

Original Poster
Rep: Reputation: Disabled
Quote:
What I would like to say is: try to simplify your script. I did not suggest you to modify the logic, just it looks a bit complicated for me. Firerat already suggested a few ideas you can apply. And we can give others if you are interested.
Anyway, it is your script and you do not need to modify it for me.
I have to admit , you are right pan64 , it is great code indeed by firerat .

Interesting way to deal with the problem , and as i see it , it is more faster .
I will implement it in my main script .

My baby , a tool that facilitates every network administrator .
I build this tool to check things remotely on server .
https://i.postimg.cc/zDQbrkXx/natm.jpg

1277 lines of code until now , and 2 or 3 options are not yet finished as i want them.
I write when i got some free time , and when i get doubts or hard stuff i come here .

Last edited by pedropt; 10-16-2019 at 12:56 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Final Fantasy XIII and Final Fantasy XIII-2 Might Get Linux Releases LXer Syndicated Linux News 1 12-18-2014 01:01 PM
[SOLVED] MySql order by COUNT and list the total count countrydj Programming 3 07-19-2012 12:21 PM
sed remove string until final match Guyverix Linux - Newbie 6 03-20-2010 06:32 AM
Need a way to count sub-directories and get a total count Mo-regard Linux - Newbie 1 08-14-2009 09:10 AM
LXer: MS vs. EC: Final Q& A and final pleadings LXer Syndicated Linux News 0 04-30-2006 03:54 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 05:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration