hi,

I've got a webserver running for a while, perfectly for DNS and HTTP(s). Now we want to start mailing as well and it seems that the smtp server (postfix) accepts around 10 to 15 connections which accept mail and deliver them correctly and then stops givingen an HELO after an new connection to port 25 is made.

When this problem arises restarting postfix does not help. Only an reboot of the server does.

One thing i did stumble upon myself is in the output of 'netstat -natop', I see an connected client, but the connection doesn't seem to be routed to postfix. There are also an lot of warnings in mail.log, but i dont know what to make of them, because I have no experience with postfix.

If any one has an idea or an suggestion of where to look i'd be very thankfull.

gr,

Henry

(relevant) output of netstat -natop:
Active Internet connections (servers and established)
tcp 0 0 127.0.0.1:10025 0.0.0.0:* LISTEN 2984/master off (0.00/0/0)
tcp 0 0 XX.XXX.XXX.61:25 0.0.0.0:* LISTEN 2984/master off (0.00/0/0)
tcp 4 0 XX.XXX.XXX.61:25 XXX.XX.XXX.117:1876 ESTABLISHED - off (0.00/0/0)

warnings in mail.log:
Aug 6 13:37:44 www postfix/master[11986]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/smtp pid 12153 killed by signal 6
Aug 6 13:37:44 www postfix/qmgr[11991]: warning: private/retry socket: malformed response
Aug 6 13:37:44 www postfix/qmgr[11991]: warning: transport retry failure -- see a previous warning/fatal/panic logfile record for the problem description
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/error pid 12155 killed by signal 6
Aug 6 13:37:44 www postfix/master[11986]: warning: /usr/lib/postfix/error: bad command startup -- throttling
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/cleanup pid 12157 killed by signal 6
Aug 6 13:37:44 www postfix/master[11986]: warning: /usr/lib/postfix/cleanup: bad command startup -- throttling
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/bounce pid 12159 killed by signal 6
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/bounce pid 12161 killed by signal 6
Aug 6 13:37:44 www postfix/master[11986]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/error pid 12163 killed by signal 6
Aug 6 13:37:44 www postfix/master[11986]: warning: process /usr/lib/postfix/bounce pid 12165 killed by signal 6

Hi Henry,

Have you checked the maxproc in the master.cf file?

Normally the default for smtp in Postfix is 100, but you may have set this lower. Alternatively, have you a spam / virus checker which is listed in master.cf which has a lower limit? e.g.

127.0.0.1:10026 inet n - n - 15 smtpd

This would give a maxproc of 15 and limit the number of connections accordingly for this service.

I the mail coming from a webpage? perhaps the injection program is not disconnecting cleanly?

Hope that's of use.

Carl.

Hi again,

The problem isn't that postfix won't accept more then 15 connections at the same time, but not more then 15 connections ever. So i assume postfix creates 15 'workerthreads' wich do there work, but (like you suggest) don't disconnect properly so that the workerthread can't be recycled for an 16th connection. (The line in master.cf is not touched, so there are still 100procs allowed)

This first 15 connections are made through telnet by myself and not from an webpage and close cleanly. I assume there is some action postfix performs after accepting the message and closing the connection that gets an workerthread to 'hang' and that this was caused by the warnings in my mail.log, maybe you can give me a hand there?

gr,

Henry

PSi forgot to post "postconf -n" so here it is:
www:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases, hash:/var/lib/dtc/etc/postfix_aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
disable_vrfy_command = yes
inet_interfaces = 83.137.144.61
inet_protocols = ipv4
mailbox_size_limit = 0
mailbox_transport = cyrus
mydestination = posit.nl
myhostname = www.posit.nl
mynetworks = 127.0.0.0/8
myorigin = /etc/mailname
parent_domain_matches_subdomains =
recipient_delimiter = +
relay_domains = /var/lib/dtc/etc/postfix_relay_domains
relay_recipient_maps = hash:/var/lib/dtc/etc/postfix_relay_recipients
relayhost =
smtp_sasl_auth_enable = no
smtp_tls_session_cache_database = btree:${queue_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks, p ermit_sasl_authenticated, permit_sasl_authenticated, r eject_invalid_hostname, reject_non_fqdn_sender, r eject_non_fqdn_recipient, reject_unknown_sender_domain, r eject_unknown_recipient_domain, reject_rbl_client sbl-xbl.spamh aus.org, reject_rbl_client list.dsbl.org, reject_unauth_de stination, permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = /etc/mailname
smtpd_sasl_security_options = noanonymous
smtpd_sender_restrictions = permit
smtpd_tls_auth_only = no
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtpd_use_tls = yes
transport_maps = hash:/etc/postfix/transport
virtual_alias_maps = hash:/var/lib/dtc/etc/postfix_virtual
virtual_mailbox_domains = hash:/var/lib/dtc/etc/postfix_virtual_mailbox_domains
virtual_transport = maildrop

Henry,

Not sure about this... but I think the line:

Aug 6 13:37:44 www postfix/master[11986]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling

may have a lot to do with it.

The only time I've come across this is when a server is suffering from a huge number of bounced mails. This can be checked with:

deliver_lock_attempts (default: 20)

The maximal number of attempts to acquire an exclusive lock on a mailbox file or bounce(8) logfile.

deliver_lock_delay (default: 1s)

The time between attempts to acquire an exclusive lock on a mailbox file or bounce(8) logfile.

Time units: s (seconds), m (minutes), h (hours), d (days), w (weeks). The default time unit is s (seconds).

Try changing this to see if this is the issue.

Carl.

Hey Carl,

Actually I walk into some SPAM on my server, but it isn't so much it should be causing problems i think. It is no more then a few mails from time to time

I kept the deliver_lock_delay at 1s, but lowered the amount of retrys to 2. I'm affraid did didn't work

For now i'm hoping that you have any other idea's?

Henry

Hi !

Edit the following line in your master.cf file, changing the default "10" instances to something higher like 30, your mileage may vary....

PS: Please, note this is not the same line carlmarshall suggested changes.
This one controls how many smtpd process will run at same time. In a "busy" or "slow" server, this is the right parameter to change....

Hey,

Thanks for your reaction. I changed the option you suggested and my mail.log starts scrolling faster when i tail it But unfortunatly the warnings keep comming up. I've saved an complety log on http://www.posit.nl/postfix

I hope you have some more idea's?

Henry

You need to focus on and fix this:

Please show your entire master.cf.

Hello,

The complete master.cf:
#
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - - - 40 smtpd
#submission inet n - - - - smtpd
# -o smtpd_enforce_tls=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#smtps inet n - - - - smtpd
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#628 inet n - - - - qmqpd
pickup fifo n - - 60 1 pickup
cleanup unix n - - - 0 cleanup
qmgr fifo n - n 300 1 qmgr
#qmgr fifo n - - 300 1 oqmgr
tlsmgr unix - - - 1000? 1 tlsmgr
rewrite unix - - - - - trivial-rewrite
bounce unix - - - - 0 bounce
defer unix - - - - 0 bounce
trace unix - - - - 0 bounce
verify unix - - - - 1 verify
flush unix n - - 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - - - - smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay unix - - - - - smtp
-o fallback_relay=
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - - - - showq
error unix - - - - - error
retry unix - - - - - error
discard unix - - - - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - - - - lmtp
anvil unix - - - - 1 anvil
scache unix - - - - 1 scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent. See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# See the Postfix UUCP_README file for configuration details.
#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}

# Configured by DTC v0.17 : Please don't touch this line !
# Adds support for the sa-learn script
sa-spam unix - n n - - pipe
-o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
user=amavis:amavis argv=/usr/share/dtc/admin/sa-wrapper spam ${sender}

sa-ham unix - n n - - pipe
-o smtpd_client_restrictions=permit_sasl_authenticated,permit_mynetworks,reject
user=amavis:amavis argv=/usr/share/dtc/admin/sa-wrapper ham ${sender}


maildrop unix - n n - - pipe
flags=DRhu user=dtc argv=/usr/bin/maildrop -w 90 -d ${user}@${nexthop} ${extension} ${recipient} ${user} ${nexthop}


cyrus unix - n n - - pipe
flags=R user=cyrus argv=/usr/sbin/cyrdeliver -e -m ${extension} ${recipient}

# amavisd-new
smtp-amavis unix - - - - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20

127.0.0.1:10025 inet n - - - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks


# End of DTC configuration v0.17 : please don't touch this line !

Yeh, Mr. C is right. You really need to take care of that "throttling" error messages....
I notice almost none of subservices on master.cf (rewrite, bouce, defer, etc) is setup to run as root. Mine they run as postfix user.

You setup this in master.cf, 5th parameter. To not run as root, put a 'n' on 5th parameter. Take a look at how it is setup on mine:
note the 'n' at 'chroot' column...may be is worth to change it just to see what happens. Make a backup copy of yours just to make easy to go back.

cheers,

Hey hey,

Okidoi, i changed the recommended parameters and things started working almost. Now my Amavis was giving errors and holding up mail all of a sudden, but without amavis everything works fine.

Thanks for your help I'll get on my amavis

greetz,

Henry

Hello again,

Unfortunatly i was wrong After a while the throtling errors came back. They still are preceded with an kill of the smtp process.

Aug 10 13:30:32 www postfix/master[31670]: warning: process /usr/lib/postfix/smtp pid 31749 killed by signal 6
Aug 10 13:30:32 www postfix/master[31670]: warning: /usr/lib/postfix/smtp: bad command startup -- throttling

Does somebody have any other ideas?

greetz

Please, check /var/log/mail.err and mail.info too.

Stop postfix. On other two terminals, start "tail -f" on mail.err and mail.info. Hit a couple of CR just to make some blank lines and you can easily see any new message.
Start postfix and look for any output on mail.err and any messages on mail.info.

I just mess with a file on my installation for this test (in my case, sender_checks.pcre) and I got the throttling message on /var/log/mail, with the explicit reason on mail.err.

So, I hope a secondary file has a syntax error or something like that, causing throttling...


PS: BTW, go back with master.cf. The changes just didn't solve the problem but add another one with amavis, so is better to stick with the previous version....

Possibly the errors took longer to show up because root has higher resource limits than the user Postfix was running as previously?

Have you tried checking all the mail queues for stuck messages? Have you tried running with the normal configuration, without Amavisd? A few hits on Google mention similar problems to this caused by Spam Assassin. Have you made sure you're on the most up to date version of Amavisd and Spam Assassin?