LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 11-18-2005, 09:42 PM   #1
essdeeay
Member
 
Registered: Aug 2004
Location: United Kingdom
Distribution: Gentoo, FC5
Posts: 87

Rep: Reputation: 15
Question Deleting duplicate messages


In the processing of recovery from a mail server crash on a Windows OS, I have a few thousand duplicate messages. I'm thinking along the lines of reading all the messages and checking for the "Message-ID" string, storing it if it's the 1st time I've seen it, then deleting all subsequent messages with the same Message-ID.

The messages may not be exactly the same because the duplicates have been through the mail router at least once more than the original message.

Is this a flawed method, and is there a better way?

As I'm fairly new to scripting (but understand quite well), what kind of techniques should I look to be using?

Many thanks,
Steve
 
Old 11-20-2005, 08:58 AM   #2
bigrigdriver
LQ Addict
 
Registered: Jul 2002
Location: East Centra Illinois, USA
Distribution: Debian Squeeze
Posts: 5,805

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
I should think you could do a three part operation on the mail.

Sort by Message_ID so that duplicates are adjacent in the sorted file.

Pipe the output through uniq, which requires duplicate lines in a file to be adjacent in order to filter out duplicates.

Write the first occurrance of a Message_ID to a new file, and drop the duplicates into null space, or another file "just in case".
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Bash - Deleting duplicate records Wire323 Programming 5 12-04-2005 09:51 AM
Redirecting the kernel messages to file other than /var/log/messages jyotika_b83 Linux - General 3 04-28-2005 07:39 PM
Pan deleting old messages? Bob-M1211 Mandriva 0 12-09-2004 09:28 AM
/var/log/messages full of these messages. Should I be concerned? mdavis Linux - Security 5 04-16-2004 11:08 AM
Deleting messages in evolution usercsr Linux - Software 2 03-31-2004 06:07 AM


All times are GMT -5. The time now is 10:23 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration