Postfix: How to speed up postmap when using it on a huge database?

robel · 10-23-2008, 04:06 PM

I'm downloading this: http://wget-mirrors.uceprotect.net/uce-pfsm-1/access.gz

Uncompress and running postmap access

But it takes very long time to execute.

Any hints on how to improve performance, or is this just what it is; a very slow process?

Edit: The file is 1 481 000 lines long, 207 574 710 bytes. The .db-file is 340 500 480 bytes.

billymayday · 10-25-2008, 02:16 AM

Have you considered accessing it online, or are you querying it too often?

billymayday · 10-25-2008, 03:24 AM

Code:

]# time postmap access

real    40m52.828s
user    0m22.361s
sys     0m19.542s

Gosh that's slow. I assume the massive difference between real time and the sum of the others is really inefficient io of postmap? Have you tried running in a ramdisk?

billymayday · 10-25-2008, 04:08 AM

Is there some reason you don't use their rsync method?

robel · 10-25-2008, 05:05 PM

Quote:

Originally Posted by billymayday

Have you considered accessing it online, or are you querying it too often?

Well, on the 25th (yesterday, local time) we got 29747 connections to our mail server. And that's only one. We have four servers all in all.

One server would get over 20 connections a minute. And that would probably lead to 20 queries each minute....?

Yes, rsync is an option. But do they have the .db available (hash database)?

robel · 10-25-2008, 05:10 PM

Quote:

Originally Posted by billymayday

Code:

]# time postmap access

real    40m52.828s
user    0m22.361s
sys     0m19.542s

Gosh that's slow. I assume the massive difference between real time and the sum of the others is really inefficient io of postmap? Have you tried running in a ramdisk?

No I haven't. Good idea. Wonder how much I need.

But doesn't such hash routine use /tmp by default? Or am I mixing this with sort?

The numbers indicate there is a lot of HD activity, so being able to run all in RAM would speed up things. That was sort of what I was hoping for. Never thought if ramdisk though...

billymayday · 10-26-2008, 12:13 AM

Probably does use /tmp, but that's on disk in my case.

I think the rsync version doesn't bother postmapping either. I only had a quick read of their docs though

Mr. C. · 10-26-2008, 02:29 AM

Try using the CDB database type, or making it DNS-based instead using rblnsd: http://www.corpit.ru/mjt/rbldnsd.html

robel · 10-26-2008, 04:41 AM

Quote:

Originally Posted by billymayday

Probably does use /tmp, but that's on disk in my case.

According to lsof it doesn't use /tmp which is a good thing as it's on disk in my case. But I need a pretty big ramdisk (around 400 MB). Does a ramdisk swap to disk if there is not enough physical ram available? That would be a disaster in this case.

I'm testing postmapping to a ramdisk right now. Running Slackware-current on top of VMware Workstation (Windows XP). I didn't bother installing Postfix, just compiled the latest version and copied the postmap executable after make. Running this command:

Code:

#time cat access |./postmap -i -c /root /mnt/ramdisk/access

The computer (as seen from Slackware) is a dual Intel Core2 Q6600 at 2.4 GHz, which is actually a quad core by the way. VMware didn't let me use all four cores, but postmap would probably not run any faster on a quad core CPU.

Here is the results of two tests, one on HD, the other one on ramdisk:

First I'll measure the time to read the file:

Code:

# dd if=/root/access of=/dev/null bs=16384
12293+1 records in
12293+1 records out
201410974 bytes (201 MB) copied, 0.0786823 s, 2.6 GB/s

Good enough...

(Yes, the entire file is in cache by now)

Next it's time to run the test on disk:
(my HD-LED is constantly lit)

Code:

# time cat access |./postmap -i -c /root /root/access

real    23m56.618s
user    0m1.410s
sys     1m10.852s

Next, it's time to run the same test with ramdisk:

Code:

# time cat access |./postmap -i -c /root /mnt/ramdisk/access

real    0m41.475s
user    0m3.993s
sys     0m36.288s

Yeah! It's really cooking with gas! (Although the majority of norwegian stoves are electrical and so is my computer...)

It's really an I/O issue when it comes to postmap. Perhaps postmap wasn't intended to work on such scale?

robel · 10-26-2008, 05:03 AM

Quote:

Originally Posted by robel

Code:

# time cat access |./postmap -i -c /root /mnt/ramdisk/access

real    0m41.475s
user    0m3.993s
sys     0m36.288s

Yeah! It's really cooking with gas!

Code:

# ls -ld /root/access.db /mnt/ramdisk/access.db
-rw-r--r-- 1 root root 374329344 2008-10-26 10:51 /mnt/ramdisk/access.db
-rw-r--r-- 1 root root 339329024 2008-10-26 10:32 /root/access.db

The files are not identical! No errors by postmap, but it's probably not a good idea to use ramdisk. My guess is that the ramdisk was too small (400MB) and that postmap was trying to make the finishing touch and didn't succeed. It should give an error or warning though...

billymayday · 10-26-2008, 05:23 AM

I'd take Houston out the equation and go rblnsd, which is the recommendation of the guys at uceprotect - see http://www.uceprotect.net/en/index.php?m=6&s=10

robel · 10-26-2008, 05:50 AM

Quote:

Originally Posted by billymayday

I'd take Houston out the equation and go rblnsd, which is the recommendation of the guys at uceprotect - see http://www.uceprotect.net/en/index.php?m=6&s=10

That's it! Thanks!

Edit: Thanks to Mr. C.
You came up with the same solution, but I didn't see it.