Any postfix gurus around here?

[CIYAM]

For some reason sending SMTP from my server (i.e. locally sending an email using SMTP) is extraordinarily slow (nearly a minute).

From (edited for brevity and security) logging the SMTP session within my application server I see the following:

Code:

[2013-06-07 07:10:48] [000160] [mail_op] (connected now reading greeting)
[2013-06-07 07:10:53] [000160] [mail_op] 220 server1.ciyam.org ESMTP
[2013-06-07 07:10:53] [000160] [mail_op] EHLO localhost
[2013-06-07 07:10:58] [000160] [mail_op] 250-server1.ciyam.org
[2013-06-07 07:11:03] [000160] [mail_op] 334 VXNlcm5hbWU6
[2013-06-07 07:11:08] [000160] [mail_op] 334 UGFzc3dvcmQ6
[2013-06-07 07:11:13] [000160] [mail_op] 235 2.7.0 Authentication successful
[2013-06-07 07:11:18] [000160] [mail_op] 250 2.1.0 Ok
[2013-06-07 07:11:18] [000160] [mail_op] RCPT TO:<ian@who.net>^M
[2013-06-07 07:11:23] [000160] [mail_op] 250 2.1.5 Ok
[2013-06-07 07:11:23] [000160] [mail_op] DATA^M
[2013-06-07 07:11:28] [000160] [mail_op] 354 End data with <CR><LF>.<CR><LF>
[2013-06-07 07:11:28] [000160] [mail_op] From: "CIYAM Open" <notifier@ciyam.org>^M
[2013-06-07 07:11:33] [000160] [mail_op] 250 2.0.0 Ok: queued as 977A4FB8396
[2013-06-07 07:11:33] [000160] [mail_op] QUIT^M
[2013-06-07 07:11:33] [000160] [mail_op] 221 2.0.0 Bye

The OS is OpenSUSE 12.2 and postfix version is 2.8.11.

The relevant thing to notice is that there is a 5 second delay before every server response (*exactly* 5 seconds) - so is there some setting that would be causing this?

[CIYAM]

Lucky I had paid attention to the 5 seconds - then after carefully checked my SMTP source code:

https://github.com/ciyam/ciyam/commit/bfa8f2ffd54be314f78c73138aae63e5cc92010c

Doh!

[OnkelPaul]

Lucky I had paid attention to the 5 seconds - then after carefully checked my SMTP source code:

https://github.com/ciyam/ciyam/commit/bfa8f2ffd54be314f78c73138aae63e5cc92010c

Doh!

Well now you have unnecessary 1.5 seconds per SMTP command instead of unnecessary 5 seconds...
A timeout value does not mean that you need to wait that long before looking at the server response, but that you signal an error if you don't receive a server response within that time. If you get a response before, just continue.
So the problem is most likely in your SMTP client's timeout handling.

Onkel Paul

[CIYAM]

Well now you have unnecessary 1.5 seconds per SMTP command instead of unnecessary 5 seconds...

Yes - I did consider whether I could just remove the extra reads (and will re-read up on SMTP again when I have the spare time).

I think the problem is knowing exactly when to expect multiple line responses from the SMTP server (am sure I could probably do a better job of that than I currently am).

It is rather low priority for me though as the benefit of putting in the delays (even though most are probably unnecessary) does help make sure that the application server can't be abused for sending spam (nearly 1 minute per email was too slow but now that it takes around 15 seconds I am not unhappy).

[OnkelPaul]

One other thing I noticed: although you're talking to a local server that most likely will not try to exploit your SMTP client, your code is just asking for buffer overflows. Never read a line of input into a fixed-size buffer without giving a buffer size limit!
You can of course interpret the delays as a kind of "greylisting" to limit the usefulness of the interface to spammers, however, this looks a bit like defensive post-hoc reasoning.
Handling multi-line responses in SMTP is really simple: every SMTP response starts with a 3-digit code. After that code you always have a space (which indicates that the response is finished) or a hyphen (which indicates that this is a line of a multi-line response, and more lines should be read).

Onkel Paul

[CIYAM]

I am not sure which code you are referring to that is reading into a fixed length buffer - can you point it out for me (all the socket reads use a std::string)?

I like the "defensive post-hoc reasoning" bit and so yes I guess I should be less *lazy* and do the SMTP *properly*.

BTW - thanks for the info about the space or hyphen after the 3 digit code (I clearly missed that).

Sent you a 0.1 BTC tip for that (is it similar with POP3 btw?).

[CIYAM]

Per your advice:

https://github.com/ciyam/ciyam/commit/915db2388ef3b1d76ae30bbd572133f5e71f9fdd

Nice how open source can grow this way.

[OnkelPaul]

Sent you a 0.1 BTC tip for that (is it similar with POP3 btw?).

Thanks, that's generous - forum advice is normally free :-)

POP3 sadly is a totally different beast - unlike some other line-oriented protocols such as SMTP, NNTP or FTP, they have a different command/response structure. In particular, it depends on the command that was sent whether the client can expect a single-line or multi-line response. However, parsing the responses is pretty easy in POP3: The first response line always starts with +OK or -ERR (additional text can follow, a client should not interpret that but it can give useful additional hints during debugging).
If a command expects a multi-line response, the server sends a line containing just a single period as the last line. The client should check the first character of each received line, if it is a period but there's more stuff in the line it should drop the initial period (the server can use this to escape lines containing a single period). This is basically the reverse of what SMTP does in the DATA command.

I found the RFCs for these standard protocols to be pretty readable. POP3 is described in http://www.ietf.org/rfc/rfc1939.txt.

Onkel Paul

[CIYAM]

Thanks, that's generous - forum advice is normally free :-)

IMO this is the Bitcoin way (you saved me the time and effort and so I am happy to reward that).

This is one of things I really *love* about Bitcoin.

If a command expects a multi-line response, the server sends a line containing just a single period as the last line. The client should check the first character of each received line, if it is a period but there's more stuff in the line it should drop the initial period (the server can use this to escape lines containing a single period). This is basically the reverse of what SMTP does in the DATA command.

Aha - yes I do already have some stuff to do with the period in my POP3 code (will have to take another look and yes I have the RFCs it just takes a bit of time to *focus* which I rarely have unfortunately).

BTW - if you have any interest in other contributions to the CIYAM project then let me know (there are many things that need improvement in any such large project as this).

[OnkelPaul]

BTW - if you have any interest in other contributions to the CIYAM project then let me know (there are many things that need improvement in any such large project as this).

Sorry, this probably won't happen - I'm not a C++ programmer, and my rather limited spare time is already devoted to some projects... :-(
But good luck in getting things done!

Onkel Paul