My hosts file is big, I'm talking 1.1 MB. I want to try to reduce the size of the file by editing it to remove duplicate entries and all comment fields (i.e #). However, to try this by hand would take a long time. I tried to edit the file by hand and in about an hour, I was maybe 1/4 of the way done. I don't have the time to edit the file by hand, so I tried searching for Linux program or script that would do the work for me. I have been unable to find anything so I turned to the boards. When I ran Windows, I would edit the file using a program called Hostess. However, Hostess doesn't function properly when running it on Linux with Wine.

If anyone knows of a program similar to Hostess for Linux or a script, please let me know.

My first thought would be to
sed -i '/^#.*/d' /etc/hosts
which will get rid of all comments ...

Then a
sort -u /etc/hosts > /etc/hosts.tmp
which should sort the file and (if the lines are
EXACTLY the same) get rid of entires that are
dupes ...

If you have slight differences in the file you'd be
wanting to use another approach.
sort -k 1,1 -u /etc/hosts > /etc/hosts.tmp

If your sed doesn't support in-place editing try
sed '/^#.*/d' /etc/hosts | sort -k 1,1 -u /etc/hosts > /etc/hosts.tmp

I've created a dupe of my hosts, entered a few duplicate
lines and with those two commands got the file from 1355
to 352 bytes in under a second :)


Cheers,
Tink

Why do you have such a large HOSTS file?

You do not have to worry about 99% of the spy/adware that effects Windows.


I have casalemedia.com, paypopup.com, fastclick.net, doubleclick.net in my HOST file; just to block popups.

Like you when I was running windows my HOSTS file was huge.

I tried...

sudo sort -u /etc/hosts > /etc/hosts.tmp

and received a permission denied error. So, I then ran:

sudo sort -u /etc/hosts

and it appeared to scan my file, although nothing appears to have been removed.

I ran:

sort -k 1,1 -u /etc/hosts

which appeared to scan my file, however very few entries were removed.

I think the problem is that because I appended additional hosts files I found on-line to my file, there are so many duplicate entries and formatting differences that the commands were unable to correctly edit and sort my file. I don't know how accurate this information is as I've been using Linux for only one year so I'm not that experienced with it yet.

I want to block as many advertisements, web bugs, data brokers, and other malicious Internet entities as possible, even though they may or may not affect Linux. Simply put, I'm paranoid.

Sometimes I think it would be easier for me just to disconnect my PC from the Internet. I spend more time trying to harden my PC against intruders and other Internet nasties than I do using it for "normal use".

Your last problems sounds like you didn't su to root first. Files in /etc can't be edited by normal users.

Perhaps you would rather be running a proxy server to screen out these sites. There are blacklists containing over 100,000 sites. I'd hate to maintain that many entries in the hosts file. You could run this on a computer that accesses the internet, and protect the other computers on the network.
http://www.screaming-penguin.com/main.php?storyid=4767

This post contains a link that may be more comprehensive.
http://www.linuxquestions.org/questi...ight=blacklist

Handling such a large number of website aliases, perhaps you want to use bind instead.