Welcome to the most active Linux Forum on the web.
Go Back > Forums > Linux Forums > Linux - General
User Name
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.


  Search this Thread
Old 11-18-2005, 09:42 PM   #1
Registered: Aug 2004
Location: United Kingdom
Distribution: Gentoo, FC5
Posts: 87

Rep: Reputation: 15
Question Deleting duplicate messages

In the processing of recovery from a mail server crash on a Windows OS, I have a few thousand duplicate messages. I'm thinking along the lines of reading all the messages and checking for the "Message-ID" string, storing it if it's the 1st time I've seen it, then deleting all subsequent messages with the same Message-ID.

The messages may not be exactly the same because the duplicates have been through the mail router at least once more than the original message.

Is this a flawed method, and is there a better way?

As I'm fairly new to scripting (but understand quite well), what kind of techniques should I look to be using?

Many thanks,
Old 11-20-2005, 08:58 AM   #2
LQ Addict
Registered: Jul 2002
Location: East Centra Illinois, USA
Distribution: Debian Squeeze
Posts: 5,846

Rep: Reputation: 342Reputation: 342Reputation: 342Reputation: 342
I should think you could do a three part operation on the mail.

Sort by Message_ID so that duplicates are adjacent in the sorted file.

Pipe the output through uniq, which requires duplicate lines in a file to be adjacent in order to filter out duplicates.

Write the first occurrance of a Message_ID to a new file, and drop the duplicates into null space, or another file "just in case".


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bash - Deleting duplicate records Wire323 Programming 5 12-04-2005 09:51 AM
Redirecting the kernel messages to file other than /var/log/messages jyotika_b83 Linux - General 3 04-28-2005 07:39 PM
Pan deleting old messages? Bob-M1211 Mandriva 0 12-09-2004 09:28 AM
/var/log/messages full of these messages. Should I be concerned? mdavis Linux - Security 5 04-16-2004 11:08 AM
Deleting messages in evolution usercsr Linux - Software 2 03-31-2004 06:07 AM

All times are GMT -5. The time now is 09:39 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration