{"id":243,"date":"2019-03-30T16:16:22","date_gmt":"2019-03-30T16:16:22","guid":{"rendered":"https:\/\/hostry.com\/blog\/?p=243"},"modified":"2019-04-12T06:27:01","modified_gmt":"2019-04-12T06:27:01","slug":"spam-filtering-smtp-header","status":"publish","type":"post","link":"https:\/\/hostry.com\/blog\/spam-filtering-smtp-header\/","title":{"rendered":"Spam Filtering Based on SMTP Header"},"content":{"rendered":"\n

Spam. Spam is incessant; therefore, each responsible mail administrator should keep up with spam filtering methods based on their headers\u2019 SMTP <\/strong>analysis. In this article we use the Postfix <\/strong>server for MTA <\/strong>server configuration.<\/p>\n\n\n\n

\"\"

MTA (Mail Transfer Agent) is the main component of the Internet mail transfer system<\/figcaption><\/figure>\n\n\n\n

All in all, this article\u2019s information is targeted at those administrators who seek to establish a highly efficient spam filter. It also contains essential information for those who use e-mail a lot and want to understand how the electronic correspondence mechanisms operate.<\/p>\n\n\n\n

In this article we assume that the Postfix mail server is used as MTA<\/strong>, but generally speaking it considers more of a theoretical approach. The Postfix options have to be specified in the relevant * _restriction parameters of the configuration file, for more information, refer to the Postfix <\/strong>configuration guide.<\/p>\n\n\n\n

At a glance: SMTP protocol <\/h3>\n\n\n\n

The main task of the SMTP (Simple Mail Transfer Protocol)<\/a> protocol is to ensure the electronic mail (e-mail) transfer. To operate via SMTP protocol<\/strong>, the client establishes a TCP connection<\/strong> with the server through port 25. After that the client and SMTP <\/strong>server exchange information until the connection is closed or terminated. The major procedure in SMTP <\/strong>is Mail Procedure. It is followed by Mail Forwarding procedures, mailbox names checking and mail groups listing. The very first procedure opens the transmit channel, while the last one is to closes it.<\/p>\n\n\n\n

SMTP commands communicate to the server what operation the client requires. These commands consist of keywords followed by one or more parameters. The keyword contains 4 characters and is separated from the argument by one or more spacebars. Each command line ends with CRLF symbols. Here is the syntax for all the SMTP protocol commands (SP – spacebar):<\/p>\n\n\n\n

HELO <SP> <domain> <CRLF>\nMAIL <SP> FROM:<reverse-path> <CRLF>\nRCPT <SP> TO:<forward-path> <CRLF>\nDATA <CRLF>\nRSET <CRLF>\nSEND <SP> FROM:<reverse-path> <CRLF>\nSOML <SP> FROM:<reverse-path> <CRLF>\nSAML <SP> FROM:<reverse-path> <CRLF>\nVRFY <SP> <string> <CRLF>\nEXPN <SP> <string> <CRLF>\nHELP <SP> <string> <CRLF>\nNOOP <CRLF>\nQUIT <CRLF><\/code><\/pre>\n\n\n\n
A typical SMTP <\/strong>server response contains a response number followed by subsidiary text separated by a spacebar. The response number serves as a server status indicator.<\/p>\n\n\n\n
 E-mail Sending<\/h3>\n\n\n\n
Let us, for instance, consider connecting to an SMTP server <\/strong>through port 25<\/strong>. Now you have to send a HELLO command and our IP address to the server: <\/p>\n\n\n\n
C: HELLO 235.193.31.127\nS: 250 smtp.example.com is ready<\/code><\/pre>\n\n\n\n
When sending mail, we transfer some required data (sender, receiver and the message itself):<\/p>\n\n\n\n
C: MAIL FROM: <user1> specify the sender\nS: 250 OK\nC: RCPT TO: <user2@example.com> specify the receiver\nS: 250 OK<\/code><\/pre>\n\n\n\n
Indicate to the server that we are going to transmit the messge contents (header and message body)<\/p>\n\n\n\n
C: DATA\nS: 354 Start mail input; end with <CRLF>. <CRLF><\/code><\/pre>\n\n\n\n
message transfer has to be completed with CRLF.CRLF symbols<\/p>\n\n\n\n
S: 250 OK\nC: From: User1 <user1@example.com>\nC: To: User2 <user2@example.com>\nC: Subject: Hello My Friend<\/code><\/pre>\n\n\n\n
between the message header and its text there is not one CRLF pair but two. <\/p>\n\n\n\n
C: Hello User2.\nC: How do you do!<\/code><\/pre>\n\n\n\n
complete transmission with CRLF.CRLF<\/p>\n\n\n\n
S: 250 OK<\/code><\/pre>\n\n\n\n
Now to complete the operation we send the QUIT command:<\/p>\n\n\n\n
S: QUIT\nC: 221 smtp.example.com is closing transmit channel<\/code><\/pre>\n\n\n\n
And now let us consider the procedure of a client connecting to your mail server. The mail server capabilities contain a mechanism for e-mail check based only on the SMTP<\/strong> message headers taken from the address information.<\/p>\n\n\n\n
As you already know, e-mail is transferred between mail servers via SMTP<\/strong>. SMTP communication begins with three requisite headers: HELO, MAIL FROM and RCPT TO. The implication is that before beginning data transfer, the server first introduces itself with HELO, then it reports the return sender address MAIL FROM and then the recipient address RCPT TO. These three headers form the signature on an electronic envelope, and most of the spam can be eliminated via their analysis. Survey of the  MAIL FROM header rejects most of the attempts to transfer something to my server, that is, the letters are refused even before actual receiving, which considerably eases things off.<\/p>\n\n\n\n
Let’s check the order of steps you need to take to reject emails that are obviously spam.<\/p>\n\n\n\n
 DNS Service <\/h3>\n\n\n\n
Initially, at the onset of the Internet, the message was delivered directly to the Nodes <\/strong>indicated in the mail address. In order to deliver a letter to user1@example.com, the mail server determined the IP address exemple.com and attempted to send an email to the determined IP. The appearance of MX records<\/strong> has solved many complications of such mailing process organization. Some programs still work with A records, but mail server organization requires at least one MX record<\/strong> for one domain.<\/p>\n\n\n\n
<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
MX records contain server addresses to which e-mails for the specified domain should be delivered. To eliminate spam, SPF <\/strong>(Sender Policy Framework) technology is also taken to; the following entry indicates the DNS addresses<\/strong> of possible sender servers from the specified domain.<\/p>\n\n\n\n
For a more detailed introduction to the SPF technology, refer to RFC 4408<\/strong> https:\/\/tools.ietf.org\/html\/rfc4408#section-3<\/a> . The next SPF record tells that emails can only be recieved from the servers listed in the MX records for your domain.<\/p>\n\n\n\n
v=spf1 +mx ~all<\/strong><\/p>\n\n\n\n
It looks like \u2018fairy dust\u2019<\/em> that ensures that the spam problem is eliminated once and for all. All that is lacking is to enable strict filtering based on SPF <\/strong>and fix the spat of the detestable spam forever. It should be noted though that letters can get through retractors and thus SPF-based<\/strong> hard filtering is extremely undesirable. Be sure to add an SPF record<\/strong> to your domain, and enable SPF checking on your mail servers. As for the information on setting up SPF for Postfix<\/strong>, the Google search system has everything you may require.<\/p>\n\n\n\n
A few more words about the DNS records<\/a>, the DNS master records are “A” records that transform domain name in the IP address. There are also CNAME<\/strong> records that contain an aliasing to an already existing domain name. These are the major records of the domain name system, but not all the users are aware of the so-called PTR reverse record<\/strong>, which translates the IP address into a domain name. If you follow these two rules, messages from your mail server are less likely to be seen as spam.<\/p>\n\n\n\n
As for your domain name, there must be an “A” record<\/strong>, and a PTR record that allow to recognize by the host name IP address, and by IP again the host name. For a MX record, you must always specify the host name for which an A record is specified. It is forbidden to use IP records or the aliasing CNAME in MX.<\/p>\n\n\n\n
Ignoring any of these rules causes some of the emails coming from your domain being recognized as spam.<\/p>\n\n\n\n
Keep an Eye for the Client.<\/h3>\n\n\n\n
At the initialization of the SMTP session, you get the client\u2019s IP address and therefore you can deduce some conclusions on its basis. The next option includes checking the PTR record of the client.<\/p>\n\n\n\n
reject_unknown_client_hostname<\/span><\/p>\n\n\n\n
This option determines that if the IP address that initiates connection is resolved to the name via PTR, therefore this name must be resolved back to the client IP of the person who established the connection.<\/p>\n\n\n\n
The following option provides a less tight constraint.<\/p>\n\n\n\n
reject_unknown_reverse_client_hostname<\/span><\/p>\n\n\n\n
This option only checks for a PTR<\/strong> record, without looking for the existence of a corresponding“A”<\/strong>  record.<\/p>\n\n\n\n
The next trick allows you to eliminate a great part of incoming spam. For this you have to make sure that your server does not responds to the client immediately, but with a short delay. Spammers do not have queue time, and a common SMTP server never breaks the connection without waiting for at least a couple of seconds. Upon receiving a request for a connection from a client, a little delay will be very helpful, since the connection is already established and there is communication between a client and a server, and delays cannot force the client to disconnect.<\/p>\n\n\n\n
The timeout during the connection is activated by the command<\/p>\n\n\n\n
sleep seconds<\/span><\/p>\n\n\n\n
The optimal time is around 20 seconds, as after the 30th second of waiting Postfix considers by default that the server to be unavailable.<\/p>\n\n\n\n
 Check Introduction<\/h3>\n\n\n\n
Before trasnitting a letter to your mail server, the transfer begins with a HELO introduction. If the FQDN of a sender is not specified in HELO, we can beyond a doubt refuse its acceptance. It requires the following options<\/p>\n\n\n\n
reject_invalid_helo_hostname\nreject_non_fqdn_helo_hostname<\/code><\/pre>\n\n\n\n
The first one prohibits the reception of letters from hosts introducing themselves with incorrect syntax, the second \u2013 from those hosts that send non-FQDN in a HELO request.<\/p>\n\n\n\n
However, the non-FQDN is only transmitted by the most unwitting spammers, and it is not so difficult to introduce as gmail.com. Here is the next option for such an occasion.<\/p>\n\n\n\n
reject_unknown_helo_hostname<\/code><\/pre>\n\n\n\n
This option prohibits receiving letters from those servers that are introduced with an address that had not been added to“A”<\/strong>  or MX record<\/strong>. SMTP clients have to to strictly adhere to RFC 5321<\/strong> and begin a session with the EHLO (HELO) command and submit their full domain name (FQDN)<\/strong> or full IP address in case of absence of a domain name. Since mail servers on the Internet cannot go without a domain name, all of the above options can be established without hesitation.<\/p>\n\n\n\n
Sender Verification <\/h3>\n\n\n\n
After the server has introduced itself, the program turns to to the sender’s address, which also contains useful information. There is a misconception that both sender’s address and the server itself always belong to the same domain. It is not true; a single mail server can serve several domains.<\/p>\n\n\n\n
If the sender\u2019s address contains a non-existent domain, or an address that is nothing but a random charset or even an empty space, the letter should obviously be banned. The following options are used to reject such messages.<\/p>\n\n\n\n
reject_non_fqdn_sender\nreject_unknown_sender_domain<\/code><\/pre>\n\n\n\n
The first option will reject the request if the domain in the MAIL FROM address line does not comply with the FQDN, as it is required by the RFC. The second option will reject the message if there are no “A” and MX DNS<\/strong> records in the MAIL FROM<\/strong> address line, or if the address contains an incorrect MX record, for example, a record with a MX<\/strong> name of zero length.<\/p>\n\n\n\n
Next we are going to request the server of the specified sender address for the existence of a user with such an address. This allows us to verify that the return address actually exists.<\/p>\n\n\n\n
Verification is performed as follows, our server initiates a reciprocal SMTP session, attempting to send an email to the sender’s address. If it is sending RCPT TO<\/strong> is successful with this address, i.e. if the receiving server does not suddenly declare that the specified mailbox is nonexistent, it is considered a valid one. The data (that is, the message),of course, is not to be transmitted during verification, the session is terminated after RCPT TO.<\/p>\n\n\n\n
reject_unverified_sender<\/code><\/pre>\n\n\n\n
It follows from the above mentioned, that for each address you send an email from your domain, there must be a mailbox on your server. Otherwise, these messages will fail the return address verification on the receiving side and, accordingly, will not be delivered to the receiver.<\/p>\n\n\n\n
If you plan to issue a newsletter, always create mailboxes for all addresses you send your message from. If somehow you do not want to receive letters to a particular address, send them to it to \/ dev \/ null, but you are obliged to receive messages.<\/p>\n\n\n\n
How to Verify a Recipient! <\/h3>\n\n\n\n
The following directive serves to verify the header of the recipient envelope for information actually being an email address.<\/p>\n\n\n\n
reject_non_fqdn_recipient<\/code><\/pre>\n\n\n\n
The next step is to prohibit receive email from nonexistent addresses. To achieve this, you have to first create a list of serviced boxes, then introduce Postfix to it through one of the * _recipient_maps parameters of the configuration file, and then you can use the configuration file parameter<\/p>\n\n\n\n
smtpd_reject_unlisted_recipient = yes<\/code><\/pre>\n\n\n\n
 or a disabling option that has the same effect:<\/p>\n\n\n\n
reject_unlisted_recipient<\/code><\/pre>\n\n\n\n
Thus, Postfix stops recieving emails for serviced domains where there is no mailbox for the recipient. <\/p>\n\n\n\n
The open transmit of letters through your Postfix we prohibit, leaving only the opportunity to receive letters to known addresses. For these purpose serves the option<\/p>\n\n\n\n
reject_unauth_destination<\/code><\/pre>\n\n\n\n
Thus, we prohibit sending of letters to all unregistered users, except for those that have passed authorization or have been added to permit_mynetworks. It is strongly recommended to use this option, otherwise you risk to get in the DNSBL (RBL)<\/strong> – Domain Name Service-based Block List.<\/p>\n\n\n\n
Based on the analysis of the three envelope headers, you can eliminate a huge amount of spam. Next, consider the following mechanism to combat spam.<\/p>\n\n\n\n
Postgrey <\/strong>– implementation of gray lists for spam filtering.<\/h3>\n\n\n\n
Greylist (gray list) is an effective instrument in combating spam. The method is based on the fact that when a mail server receives a message, the server does not accept it, gives a warning of temporary unavailability, and takes the sender\u2019s coordinates in a special list. If the sender’s mail server is configured correctly, soon there will be a second attempt to send the same message, which this time will be processed by our mail server. Information about whether to accept or reject the letter is taken from the database of temporary lists. The first time we reject, if the same mail comes the second time, then we accept it.<\/p>\n\n\n\n
As practice shows, 50-70% of spam software usually makes only 1 delivery sample, so this method can safely halve junk mailing.<\/p>\n\n\n\n
Cons of the greylist technique: <\/strong><\/p>\n\n\n\n