LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-29-2015, 10:33 AM   #1
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Rep: Reputation: 16
Question How can i remove duplicated lines on a file?


Hi

I am using this command to get some info about bots/spiders from my Centos server access.log file:

Code:
grep 'spider\|bot' access.log | sort -u -f >> bots.txt
Result is like this (i know pingdom is not bad):

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.102 - - [28/Mar/2015:02:19:56 +0200] "GET / HTTP/1.1" 200 24277 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:13:21 +0200] "GET /user/74595-tery1/?tab=idm HTTP/1.1" 200 3905 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Is there any command that can remove duplicate lines if the ip and the user-agent is the same on each line?

To get something like:

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Or if there is no way for this then to get only one line (even if different ip's exist for each user agent) like:

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"

Last edited by ASTRAPI; 03-29-2015 at 07:45 PM.
 
Old 03-29-2015, 10:38 AM   #2
veerain
Senior Member
 
Registered: Mar 2005
Location: Earth bound to Helios
Distribution: Custom
Posts: 2,524

Rep: Reputation: 319Reputation: 319Reputation: 319Reputation: 319
You can write a perl script which stores previous state and produces only single line.
 
Old 03-29-2015, 10:46 AM   #3
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Thanks for your reply veerain but i don't know how i can code a script

I will wait if any other user can help me....

Last edited by ASTRAPI; 03-29-2015 at 10:53 AM.
 
Old 03-29-2015, 10:56 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,353
Blog Entries: 55

Rep: Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541
If you only need IP addresses and bot names how about:
Code:
grep 'spider\|bot' access.log | perl -lane 'print "@F[0,11..$#F]"' | sort -u -f >> bots.txt
 
1 members found this post helpful.
Old 03-29-2015, 11:15 AM   #5
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,886

Rep: Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258
Quote:
Originally Posted by ASTRAPI View Post
Thanks for your reply veerain but i don't know how i can code a script I will wait if any other user can help me....
We will be happy to HELP you, but we aren't going to do it for you. You will have to show some effort of your own...there are THOUSANDS of easily-found scripting tutorials, and many examples you can find with a brief Google search.

Also, you've been here since 2007...after EIGHT YEARS, you should be able to write a very simple script, and know about the grep and sort commands, or at least how to read the man pages. Especially since you've asked things like this before:
http://www.linuxquestions.org/questi...os-4175537803/
http://www.linuxquestions.org/questi...os-4175534130/
http://www.linuxquestions.org/questi...il-4175521129/
http://www.linuxquestions.org/questi...es-4175438314/
http://www.linuxquestions.org/questi...rogram-944485/

You were directed to bash scripting tutorials at least three years ago...unless you actually START reading those tutorials, and putting some effort into writing your own scripts, you will NEVER be able to do it on your own. Again, we are happy to ASSIST you, but asking us to continue to write scripts for you isn't too nice.
 
Old 03-29-2015, 11:38 AM   #6
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Thanks unSpawn i will test it and i will let you know

@TB0ne

Quote:
We will be happy to HELP you, but we aren't going to do it for you.
I didn't ask for someone to write a perl script for me...I just ask if a command exist and anyone can share it with me....

Quote:
Also, you've been here since 2007...after EIGHT YEARS
2007 is the registration date when i start using Linux at least i try to use and then you can see a huge gap more than 5 years as i was use back Windows....

So i do not try to use Linux for 8 years and do such stupid questions.

Quote:
START reading those tutorials, and putting some effort into writing your own scripts, you will NEVER be able to do it on your own.
I agree 100 percent with you and i try to do it when i have free time....You know wife, kids, job damn

Thanks

Last edited by ASTRAPI; 03-29-2015 at 11:43 AM.
 
Old 03-29-2015, 11:58 AM   #7
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,886

Rep: Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258
Quote:
Originally Posted by ASTRAPI View Post
Thanks unSpawn i will test it and i will let you know
Yes, that's a good command...and it is very much like others you were given previously, wasn't it? You were given a command similar to this in one of the posts I quoted, back from 2012. At some point, you need to take what you've been told, and apply it, rather than asking others to modify things to fit each situation for you.
Quote:
I didn't ask for someone to write a perl script for me...I just ask if a command exist and anyone can share it with me....
Right...which again, is very much like things you asked before, isn't it?
Quote:
2007 is the registration date when i start using Linux at least i try to use and then you can see a huge gap more than 5 years as i was use back Windows....So i do not try to use Linux for 8 years and do such stupid questions.
We have no idea what you do, or when you do it. And looking, there is no 'huge gap more than five years' in your posting history. You begin posting in 2007, and seem to only skip 2008 and 2011...there are posts from every other year in between.
Quote:
I agree 100 percent with you and i try to do it when i have free time....You know wife, kids, job damn
Just like everyone else here, then??? Again, we're happy to HELP you, but you have to show some effort...we have the same things in life you do, but want us to take our time to figure something out for you, because you're busy?? So are we.
 
Old 03-29-2015, 12:04 PM   #8
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
You are very aggressive with me and i don't know why

I got your advice and i will try to follow as i agree but when i can't follow it i will post here and IF anyone can/want to help then great if not no problem

So please stop giving me more advices
 
Old 03-29-2015, 12:08 PM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,696

Rep: Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261
"sort -u" with field specifications comes to mind.
 
1 members found this post helpful.
Old 03-29-2015, 12:08 PM   #10
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,886

Rep: Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258
Quote:
Originally Posted by ASTRAPI View Post
You are very aggressive with me and i don't know why
Not aggressive at all; pointing out you've been given similar answers many times, and don't seem to want to apply the knowledge, or try to learn on your own.
Quote:
I got your advice and i will try to follow as i agree but when i can't follow it i will post here and IF anyone can/want to help then great if not no problem
The issue is, you will never be able to follow it, if YOU don't try things on your own. You post questions that are VERY similar to each other, and show no effort to solve the issue, and ask us how to do it. EFFORT is what we're asking you to show...if you post "I've tried xxx, yyy, zzz, similar to what I did here <link>, but I'm getting <answer>, and I'm confused", that's one thing. Saying "I need this..help" is different.
Quote:
So please stop giving me more advices
Please stop asking people to do things like this for you, without showing effort of your own first.

---------- Post added 03-29-15 at 12:09 PM ----------

Quote:
Originally Posted by jpollard View Post
"sort -u" with field specifications comes to mind.
Indeed...which goes back to what the OP was told three years ago.
 
Old 03-29-2015, 12:29 PM   #11
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
You may not reply to my topics if you don't agree with the way i will try to do it as i explain above....

I will not reply anymore to yours....
 
Old 03-29-2015, 01:06 PM   #12
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,886

Rep: Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258
Quote:
Originally Posted by ASTRAPI View Post
You may not reply to my topics if you don't agree with the way i will try to do it as i explain above.... I will not reply anymore to yours....
Read the "Question Guidelines" link in my posting signature. It covers showing no effort of your own, and asking for handouts.

No matter what your excuse is, you've been here for years, have asked several similar questions, and didn't show effort of your own, and lied about the 'huge gap' in your posting.

Last edited by TB0ne; 03-29-2015 at 01:07 PM.
 
Old 03-29-2015, 04:46 PM   #13
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Quote:
2007 8 topics
2008 0 topics
2009 6 topics
2010 3 topics
2011 0 topics
2012 9 topics
2013 0 topics
If you don't see a gap here then i don't have anything to say....Or maybe you think with those numbers that i should be a Linux expert?

This is my last reply to you and yes i say a lie but only one at post #11 guess which one

But i will follow your quote style of replies just to be sure that you got it

Quote:
I will not reply anymore to yours....
 
Old 03-29-2015, 05:06 PM   #14
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 18,886

Rep: Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258Reputation: 4258
Quote:
Originally Posted by ASTRAPI View Post
If you don't see a gap here then i don't have anything to say....Or maybe you think with those numbers that i should be a Linux expert?
In post #4, you said: "you can see a huge gap more than 5 years"...then you post this, which very clearly shows you did NOT have a five year gap, does it? And as noted, you did skip 2008 and 2011. Expert? No..but you should have learned SOMETHING in that time, especially when you've been handed answers that could easily solve your issue now, if you put even a small amount of effort into reading a man page.
Quote:
This is my last reply to you and yes i say a lie but only one at post #11 guess which one
You didn't have a five year gap, sorry.
Quote:
But i will follow your quote style of replies just to be sure that you got it
Doesn't matter to me, but try to stay on topic, and learn. Again, asking people to feed you answers when you show ZERO effort of your own is rude. Especially when you're asking questions that only differ slightly from what you've asked before.

Show effort, and learn from what you've been told. Until you start doing that, you will never be an 'expert' at ANYTHING. Saying that YOU are too busy with your family to learn, then asking us to provide you answers (essentially, to save YOUR time, at the expense of ours), certainly won't get you much help.

Again, the "Question Guidelines" link in my posting signature was written by the LQ Site Owner...I suggest you read it. One relevant line:
Quote:
Originally Posted by Question Guidelines
(although we are happy to assist you with specifics, if you show some effort of your own!).

Last edited by TB0ne; 03-29-2015 at 05:07 PM.
 
Old 03-29-2015, 05:25 PM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,353
Blog Entries: 55

Rep: Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541
Quote:
Originally Posted by ASTRAPI View Post
Thanks unSpawn i will test it and i will let you know
So, let's get this thread back on track and let me know if it works.

@TB0ne: As you can clearly see now this thread is filled half with a relevant but meta discussion. I know you're trying hard to make things better but this isn't working. So please stop.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Need to remove empty lines from a file ashishkumar1000 Linux - Newbie 10 08-19-2012 02:13 PM
How to remove first 2 lines of a file in a script nazs Programming 16 02-19-2007 07:08 AM
How to remove lines from a file doza Linux - General 2 04-27-2005 11:59 AM
How do i remove blank lines from a file? kakho Programming 1 04-15-2004 03:57 AM
[bash] remove lines from a file Drimo Programming 3 03-20-2004 11:16 AM


All times are GMT -5. The time now is 03:04 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration