Additional Anti-UCE settings

Additional anti-UCE settings for our Debian spamfilter (version 2).

This HOWTO is a guide to substantially enhancing the spam fighting abilities of our Debian spamfilter (built using one of the documents found at http://verchick.com/mecham/public_html/spam/). The disclaimer at that address also applies to this document. Let me start by saying that the reason these instructions are not included in the original documents, and instead are included in this supplement, is for two reasons. One reason is these techniques will only work on a machine that is on the public Internet, and the other reason is there is additional risk involved. These techniques cannot be used on a spamfilter that sits behind another gateway server because they depend directly on the conversation with the mail server attempting to deliver mail to your network (the client conversation). There is increased risk because mail may get rejected by what I call "a jury of one", or some legitimate mail servers may possibly be unable to send mail through your server. That said, you can use these settings to substantially reduce the amount of spammy mail that is accepted by your server, and therefore substantially reduce the load on your server (and also reduce network traffic).

I am going describe setting up two different "policy servers" in Postfix similar to what is described in the SMTPD_POLICY_README. I am also going to illustrate the use of the reject_rbl_client restriction as described in Postfix_Configuration_-_UCE_Controls. The two policy servers I use are 'policyd-weight' and 'postgrey', and I use one RBLs (real-time block/blackhole list): sbl-xbl.spamhaus.org. Of the hundreds of RBLs, this one is often regarded as being effective, and is considered to be effective with few false positives.

Of these three types of settings, I have found that policyd-weight (by design) is the least likely to reject legitimate mail and therefore I use it on my primary MX Debian spamfilter. If you only have one Debian gateway server, I suggest you use policyd-weight alone as opposed to the two other methods. If you have problems with policyd-weight, or simply prefer not to use it, then my second choice would be to use a single RBL: sbl-xbl.spamhaus.org. Now, if you have built a second Debian spamfilter and it is currently in use on the Internet as your secondary MX (not round robin), then you can get more aggressive and take more risks with the mail that is sent to that machine. For that machine, I suggest using both postgrey and the two RBLs. I would not use postgrey on my primary MX server because there are legitimate servers on the Internet that do not have the ability to "try back later" when postgrey tells them that the server is temporarily unavailable. The simple idea behind a Greylist server is that when a new client connects for the first time, the postgrey service tells them the server is temporarily unavailable, then it waits a predetermined amount of time to see if the same client tries back later, and if it does, it lets that client send mail. Most spammer's servers will not try back later, so it is quite effective (for now). There are some Trend Micro servers (among others) that will not try back later, so they will not be able to send you mail unless you whitelist them in the Greylist server's whitelist table or a check_client_access table that we will create. To summarize: all your domains will use the server running policyd-weight as their primary MX server (priority 10 for example) and all your domains will use the server running postgrey and the RBL as their secondary server (priority 20 for example).

Using any of these settings requires that you consider what happens when a legitimate message is not able to get past these settings. We will need to add a mechanism that will bypass these checks for mail addressed to postmaster@ and abuse@, and we may need to allow other particular clients to bypass these checks, so we will set up a new check_recipient_access hash table, and a new check_client_access hash table.

A cautionary note on policyd-weight. Policyd-weight is beta software (and is no longer maintained). Once you have the program set up and it is working without issues, don't be overly hasty to install a newer version. A little patience may reveal others who have problems with a newer version. Take the "if it's not broke, don't fix it approach" and keep your eye on the mailing list. Newer versions often have new features that make the upgrade very worthwhile, but as I said, have a little patience.

To see how effective these techniques are going to be for you, I suggest saving yesterday's pflogsumm report (or recreating it) so you can compare reports created over the next couple days to it. I think you will be amazed at how much more mail is rejected.

Let's download and install the current policyd-weight. You will want to begin by visiting http://policyd-weight.org/ and write down the version number we are getting today. The last time I updated this, the version was "0.1.15-beta-1", so we are going to make a directory that reflects the version we download, and place the files there. The program changes often, and sometimes there are issues with a new version, so we always want to keep old versions in case we need to use them. If you are upgrading policyd-weight from a version older than 0.1.15-beta-1, it is now required that a user and group 'polw' is created; policyd-weight will no longer run as user 'nobody' and policyd-weight.conf and master.cf must be updated to reflect this change. Also, we will change from using a Unix socket to a TCP socket. Modify the commands below as needed to reflect the current version:

mkdir /usr/local/src/policyd-weight-0.1.15-beta-1

cd /usr/local/src/policyd-weight-0.1.15-beta-1

wget http://policyd-weight.org/policyd-weight

chmod 0755 policyd-weight

wget http://verchick.com/mecham/public_html/spam/policyd-weight.conf

Edit policyd-weight.conf to see if you would like to change or add other settings (read policyd-weight or policyd-weight.conf.sample to get an idea of the settings you can change) then move it into place. I suggest using the settings I provide until you get more familiar with the program. Note that I'm using the vi editor here, but of course you can use a different editor if you desire:

vi policyd-weight.conf

Save and exit the file. Now we can move our program into place:


cp policyd-weight /usr/sbin/

cp policyd-weight.conf /etc/

If this is the first time installing policyd-weight, create a user and group 'polw':

groupadd polw

useradd -d /var/polw -s /bin/false -g polw polw

Then make our chrooted Postfix aware of the new user:
cp /etc/passwd /var/spool/postfix/etc/passwd

Make sure we have Net::DNS installed:

apt-get update

apt-get install libnet-dns-perl

If using Etch, you may want to install the 'killall' utility:

apt-get install psmisc

I personally like to cut in half the number of hard errors allowed before we cut a client off, this cuts off dictionary attackers a little quicker.


postconf -e "smtpd_hard_error_limit = 10"

postconf -e "smtpd_soft_error_limit = 8"

Download an initscript from me.
This part may also be new if you are familiar with older versions:


cd /etc/init.d

wget http://verchick.com/mecham/public_html/spam/policydweight

chmod +x policydweight

update-rc.d policydweight defaults

If you are upgrading from a previous version of policyd-weight, we will change from using a Unix socket to a TCP socket. Edit master.cf and remove (or comment out as I show here) these two lines:
vi /etc/postfix/master.cf

# policy	unix	-	n	n	-	-	spawn
#                user=polw argv=/usr/bin/perl /usr/lib/postfix/policyd-weight

Save and exit the file. Now we may need to create a couple new hash files for Postfix to use:

vi /etc/postfix/roleaccount_exceptions

And insert the following (insure you have a carriage return at the end of the last line):


# mail addressed to these recipients are allowed to bypass RBL checks

postmaster@ OK

abuse@ OK

Save and exit the file, postmap it, and create and postmap the next file:

postmap hash:/etc/postfix/roleaccount_exceptions

vi /etc/postfix/rbl_client_exceptions

And insert the following:

# these client IP addresses are allowed to bypass RBL checks

66.35.250.225 OK

(that IP address is a 'picked at random' sourceforge server) Then postmap that file:

postmap hash:/etc/postfix/rbl_client_exceptions

Now we need to vi /etc/postfix/main.cf and add a few things to the smtpd_recipient_restrictions restriction stage. Let's make an assumption your smtpd_recipient_restrictions looks similar to this at this time:

smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_unauth_destination

It is VERY important these lines are added AFTER "reject_unauth_destination". We want to add three lines below (after) reject_unauth_destination, so the result looks something like this:

smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_unauth_destination,
    reject_unauth_pipelining,
    check_recipient_access hash:/etc/postfix/roleaccount_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions,
    check_policy_service inet:127.0.0.1:12525

The TCP socket at port 12525 is policyd-weight. This has changed from previous versions, so please make this change if you have used policyd-weight before. The white space in front of the settings above is required. This is true whenever you extend a Postfix setting to more than one line. Note that the commas are actually optional, Postfix can use either a comma, or a space to separate settings. Policyd-weight requires the use of reject_invalid_hostname. You can place this in smtpd_recipient_restrictions prior to check_policy_service or as is shown in this example in smtpd_helo_restrictions:

smtpd_helo_restrictions = reject_invalid_hostname

Save and exit your main.cf and stop Postfix:


postfix stop

If you have an older version of policyd-weight installed, you will need to manually kill those processes ('ps aux | grep pol' and 'kill xxxx' where xxxx is (are) the pid(s)) and delete the disk based cache in the /tmp directory (probably named /tmp/polw.sock or /tmp/policyd-weight or /tmp/.poliyd-weight/polw.sock).

Start up policyd-weight:


/etc/init.d/policydweight start

Verify it starts up:


ps aux | grep policyd-weight

This should show policyd-weight is listening on port 12525:

lsof | grep LISTEN | grep pol

Ok, policyd-weight is installed and running, and all that remains is to start Postfix, AND WATCH OUR LOG FOR ERRORS:


postfix start

tail -f /var/log/mail.log

Note that because we have "permit_mynetworks" prior to "check_policy_service", you cannot test policyd-weight by sending messages from a computer listed in $mynetworks, and you cannot test with messages sent to postmaster@ or abuse@. Keep an eye on the log as mail goes by. Watch it for at least 10 messages or so. Use [Ctrl]+c to return to the shell prompt. If there are errors that relate to policyd-weight, remove the "check_policy_service inet:127.0.0.1:12525" line in main.cf, and reload Postfix to get your mail going again. It is normal to get a "could not connect to cache" message the first time Postfix loads policyd-weight, but you should not get this error repeatedly. Hopefully all went well. Note that Postfix controls the startup and shutdown of policyd-weight, we do not start the daemon outside of Postfix. Now, assuming you have logcheck installed, ( apt-get install logcheck logcheck-database if you don't) every single line in our log that is created by policyd-weight will trigger a line in a logcheck report so we need to tell logcheck to ignore those lines:
vi /etc/logcheck/ignore.d.server/postfix

and at the top of the file, insert on a line by itself, the text:
policydweight

I suggest you subscribe to the (low volume) policyd-weight mailing list. I also suggest you read more about the program to gain an understanding of how it works. I think of it as a 'miniature SpamAssassin'. Also read the somewhat incomplete man page: http://www.policyd-weight.org/policyd-weight.conf.5.html. Here is a useful command:


 egrep -o rate:.* /var/log/mail.log

###########################################################################################################

Now we move on to installing postgrey. Remember that normally I do not recommend Greylisting on the primary MX. I believe it is quite appropriate for a secondary MX however. Since most mail sent to a secondary server is spam, it is also quite effective there. We start by installing postgrey:

apt-get update

apt-get install postgrey

By default postgrey will Greylist for 300 seconds. I noticed that Yahoo retries every 60 seconds, so I think it's a good idea to change the delay from 300 to something like 29 seconds so a valid server like that does not have to retry more than once. Shortening the time does not seem to make the program any less effective.


vi /etc/default/postgrey

and change:
POSTGREY_OPTS="--inet=127.0.0.1:60000"
to:
POSTGREY_OPTS="--inet=127.0.0.1:60000 --delay=29"

Since we made this change, we need to stop and start (not reload) postgrey:

/etc/init.d/postgrey stop

/etc/init.d/postgrey start

smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_unauth_destination

It is VERY important these lines are added AFTER "reject_unauth_destination". We want to add four lines below (after) reject_unauth_destination, so the result looks something like this:

smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_unauth_destination,
    reject_unauth_pipelining,
    check_recipient_access hash:/etc/postfix/roleaccount_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions,
    check_policy_service inet:127.0.0.1:60000

Postgrey is the server that listens on port 60000.
By adding postgrey, the /etc/passwd file has changed, so we will give Postfix an updated copy of it:
cp /etc/passwd /var/spool/postfix/etc/passwd

As we did on our other server:


postconf -e "smtpd_hard_error_limit = 10"

postconf -e "smtpd_soft_error_limit = 8"

OK, postgrey is installed, and all that remains is to reload Postfix, AND WATCH OUR LOG FOR ERRORS:

postfix stop

postfix start

tail -f /var/log/mail.log

Keep an eye on the log as mail goes by. Watch it for at least 10 messages or so. If there are errors that relate to postgrey, remove (do NOT comment out) the "check_policy_service unix:private/policy" line in main.cf, and reload Postfix to get your mail going again. If someone complains they are unable to send you mail, you can whitelist their server in /etc/postgrey/whitelist_clients or our rbl_client_exceptions map. There is also a /etc/postgrey/whitelist_recipients file that does the same thing as our roleaccount_exceptions map.

Now, adding the RBL client check is simply a matter of placing it in front of postgrey. It can be used by itself if you prefer.

smtpd_recipient_restrictions =
    permit_mynetworks,
    reject_unauth_destination,
    reject_unauth_pipelining,
    check_recipient_access hash:/etc/postfix/roleaccount_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions, 
    reject_rbl_client sbl-xbl.spamhaus.org,
    check_policy_service inet:127.0.0.1:60000

You should also consider adding a check_sender_access map (or two) after the check_client_access map that could be used to bypass the RBL and greylist checks for particular senders or sender domains. Also, if you have a valid populated relay_recipient_maps file (or other list of valid recipients appropriate for your domain class) and wish to reject mail to invalid recipients prior to greylisting, you should add 'reject_unlisted_recipient' prior to check_policy_service. The result may look something like:

smtpd_recipient_restrictions =    
    permit_mynetworks,
    reject_unauth_destination,
    reject_unlisted_recipient,
    reject_unauth_pipelining,
    check_recipient_access hash:/etc/postfix/roleaccount_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions,
    check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
    reject_rbl_client sbl-xbl.spamhaus.org,
    check_sender_access hash:/etc/postfix/greylist_sender_exceptions,
    check_policy_service inet:127.0.0.1:60000

OK, the RBL checks are installed, and all that remains is to reload Postfix, AND WATCH OUR LOG FOR ERRORS:

postfix stop

postfix start

tail -f /var/log/mail.log

If you are interested in making postgrey safer (but less effective), take a look at: http://lists.ee.ethz.ch/postgrey/msg01214.html or http://www.arschkrebs.de/postfix/postfix_greylisting.shtml

http://policyd-weight.org/
http://isg.ee.ethz.ch/tools/postgrey/
http://www.spamhaus.org/xbl/index.lasso

mr88talent at yahoo dot com
13 APR 2006