Index  Page 1  Page 2
This document is considered obsolete and is no longer maintained. See http://verchick.com/mecham/public_html/spam/

Amavisd-new lookups
Install DCC (dccifd) - Optional
Install Postgrey - Optional
Install policyd-weight - Optional
Install SaneSecurity and MSRBL ClamAV signatures - Optional
Install Botnet SA plugin - Optional
Install p0f - Optional
Compile SpamAssassin Body Rules - Optional
Install AIDE - Optional
Install Altermime - Optional
Odds and ends

Amavisd-new lookups

Index Top Bottom
The amavisd-new README.lookups page may be somewhat difficult to understand. However, for our needs the two major concepts are reasonably simple. First let's talk about SQL and static lookups. A lookup is performed using an email address as the key (usually the envelope recipient). SQL lookups are performed before static hash lookups, hash lookups are performed before defaulting to constants. To be more precise, from the README: "default sequence of lookups: SQL, LDAP, hash, ACL, regexp, constant." (you can find examples of hash, ACL, regexp and constant settings in amavisd.conf-sample). For example: if a matching recipient email address (or part of an address, or catchall) exists in the SQL database (in the users table), and has a policy (in the policy table) with the field 'spam_tag2_level' set at 6.0 and there is also a static constant setting '$sa_tag2_level_deflt' (in the 50-user file) set to 6.31; then 6.0 is used (because SQL is checked first and first match wins). The search (for that particular piece of data) stops at the first definitive answer (not NULL or undef). If the SQL field is NULL however (which evaluates to undef), the search continues on. In this example there is no hash table, no ACL list and no regexp list, so the search ends at the constant (which does exist - and matches any recipient); the net result: 6.31 is used. That is the first important concept.

The second important concept is priority within the SQL and static hash lookups. Full email addresses are broken down into pieces and there is an attempt to match each piece with the data in the SQL, hash or other table/map. In static hash lookups the most specific comparisons are made first (full email address). The most general comparisons are made last (catchall "."). Hash lookups (e.g. for user+foo@example.com) are performed in the following order:
user+foo@example.com
user@example.com
user+foo@
user@
example.com
.example.com
.com
.
Note that the format is a little different for SQL:
user+foo@example.com
user@example.com
user+foo
user
@example.com
@.example.com
@.com
@.
Because of the order of searches and the fact that the search stops at the first match, in hash (and other) tables it only makes sense to place more specific patterns before the more general ones. For example, in this hash postmaster@example.net is placed before .example.net:
$sa_tag2_level_deflt = 6.31;

@spam_tag2_level_maps = (
  { 'postmaster@example.net' => 99.0,
    '.example.net' => 7.0 },
    \$sa_tag2_level_deflt,   # catchall default
);
In SQL however, the order in which the lookups are performed is determined by the SQL SELECT statement and the users.priority field. The supplied SQL SELECT statement orders by the priority field in DESCending order. This means you have to create the same 'most specific to most general' order by setting the proper priority. In other words, you could assign a priority of 7 to all users with a full email address in the email field (which we do), and you could add a domain wide user "@example.net" that would have a lower priority. If user@example.net had spam_kill_level set to 6.0 and @example.net had spam_kill_level set to 7.0, then 6.0 would be used because user@example.net has a higher priority. If user@example.net had spam_kill_level set to NULL, then the lookup would fall through to the @example.net user (and it would find 7.0). If @example.net had a higher priority than user@example.net, then 7.0 would be used. For a given recipient this is what the SQL SELECT statement that amavisd-new creates might look like:

SELECT *,users.id FROM users LEFT JOIN policy ON users.policy_id=policy.id
 WHERE users.email IN ("user@example.net", "user", "@example.net", "@.example.net", "@.net", "@.")
 ORDER BY users.priority DESC;


Here is an example from a level 5 log entry where user@example.net has a policy with spam_tag2_level NULL (which is translated to undef) but a static hash exists (@spam_tag2_level_maps as shown above). The lookup falls through to the static hash, where a match is found:
lookup_sql_field(spam_tag2_level) "user@example.net" result=undef
query_keys: user@example.net, user@, example.net, .example.net, .net, .
lookup_hash(user@example.net) matches key ".example.net", result=7
lookup (spam_tag2_level) => true,  "user@example.net" matches, result="7", matching_key=".example.net"
In this next example the same user still has spam_tag2_level set to NULL, but there is a lower priority (domain wide) user '@example.net' that has spam_tag2_level set to 9999:
lookup_sql_field(spam_tag2_level) "user@example.net" result=undef
lookup_sql_field(spam_tag2_level) "user@example.net" result=9999
lookup (spam_tag2_level) => true,  "user@example.net" matches, result="9999", matching_key="/cached/"
"/cached/" simply means it is using results found during the initial query. Here is an example where the user does not exist in SQL and is not found in the hash, therefore the lookup falls through to the scalar variable:
lookup_sql_field(spam_tag2_level), "user@example.org" no matching records
query_keys: user@example.org, user@, example.org, .example.org, .org, .
lookup_hash(user@example.org), no matches
lookup: (scalar) matches, result="6.31"
lookup (spam_tag2_level) => true,  "user@example.org" matches, result="6.31",
        matching_key="(constant:6.31)"
Now hopefully it will make more sense when you read README.lookups. When I initially created the policies, I left all the data fields NULL with the exception of spam_tag2_level, spam_kill_level and occasionally spam_quarantine_cutoff_level. This way all the remaining settings are controlled by entries in the 50-user file. When new policies are created in the SquirrelMail interface, only those two or three fields have non-NULL values.


Install DCC (dccifd) - Optional

Index Top Bottom
DCC detects messages that are bulk mailed. The way I understand the license for DCC is if you are not a reseller and you receive less than somewhere around 100,000 messages per day, you can use the DCC client by itself. If you run a larger site you need to run a DCC server and flood your data to public DCC servers. If you are a reseller you need to pay for a license. I only outline installing a client. Be patient when compiling the program - it may appear to hang at one point:
cd /usr/local/src
wget http://www.rhyolite.com/anti-spam/dcc/source/old/dcc-1.3.120.tar.Z
tar xzf dcc-1.3.120.tar.Z
cd dcc-1.3.120
./configure --with-uid=amavis && make && make install

ln -s /var/dcc/libexec/cron-dccd /usr/bin/cron-dccd
chown -R amavis:amavis /var/dcc
cdcc info


You should see at least some entries with 'requests ok' (127.0.0.1 is expected to fail). If not, then you might be having an issue with a proxy server or firewall. http://www.dcc-servers.net/dcc/firewall.html gives an example of a Cisco router access list entry. Keep in mind that you will likely not run a DCC server, only the DCC client. If the tests fail you won't be able to test again until some period of time has passed. The program does this to prevent broken clients from trying too often.
crontab -e

and insert (at the bottom of existing entries):
43 11 * * * /usr/bin/cron-dccd

cd
echo "dcc_home /var/dcc" >> /etc/spamassassin/local.cf
sed -i 's/DCCIFD_ENABLE=off/DCCIFD_ENABLE=on/' /var/dcc/dcc_conf
sed -i 's/DBCLEAN_LOGDAYS=14/DBCLEAN_LOGDAYS=1/' /var/dcc/dcc_conf
sed -i 's/DCCIFD_LOGDIR/#DCCIFD_LOGDIR/' /var/dcc/dcc_conf
cp /var/dcc/libexec/rcDCC /etc/init.d/adcc
update-rc.d adcc defaults
/etc/init.d/adcc start
sed -i 's/#loadplugin Mail::SpamAssassin::Plugin::DCC/loadplugin Mail::SpamAssassin::Plugin::DCC/' /etc/spamassassin/v310.pre
/etc/init.d/amavis restart


You will get errors because I chose to turn of logging. I wouldn't worry about them:
Jun 15 19:33:12 sfa dccifd[7459]: log thresholds set with -t but no -l directory
Jun 15 19:33:12 sfa dccifd[7459]: no -l directory prevents per-user logging with -U

Test with spamassassin:
su amavis -c 'spamassassin -D dcc <sample-spam.txt'

You should get something like this in the output:
[7615] dbg: dcc: dccifd is available: /var/dcc/dccifd
[7615] dbg: dcc: dccifd got response: X-DCC--Metrics: sfa 1049; Body=many Fuz1=many Fuz2=many


We need to suppress a few logcheck messages:
echo "stat\(log directory /var/dcc/log\): No such file or directory" >> /etc/logcheck/ignore.d.server/dcc
echo ": missing message body" >> /etc/logcheck/ignore.d.server/dcc
echo "detected 0 spam, ignored for 0, rejected for 0," >> /etc/logcheck/violations.ignore.d/dcc
echo "detected 0 spam, ignored for 0, rejected for 0," >> /etc/logcheck/ignore.d.server/dcc


Read the LICENSE:
cat /usr/local/src/dcc-1.3.120/LICENSE


Install Postgrey - Optional

Index Top Bottom
Of course Greylisting will reject a ton of spam - but at a cost. Some mail will be delayed. To help out I cut the retry time from 6 minutes to 59 seconds. We will use 'selective greylisting' with the idea being we will attempt to only target clients that appear to come from dial-up and dynamic IP addresses. If you decide to implement this I also recommend you whitelist clients you trust. The whitelists are located in the /etc/postgrey/ directory. We can also whitelist senders, clients or networks in Postfix.
apt-get install postgrey

sed -i 's/--inet=127.0.0.1:60000/--inet=127.0.0.1:60000 --delay=59/' /etc/default/postgrey
/etc/init.d/postgrey restart
ps aux | grep postgrey | grep -v grep

cd /etc/postfix
wget http://verchick.com/mecham/public_html/spam/check_client_fqdn
cp -ip main.cf main.cf-before-grey

postconf -e "smtpd_restriction_classes = check_greylist"
postconf -e "check_greylist = check_policy_service inet:127.0.0.1:60000"
cp access greylist_sender_exceptions
postmap greylist_sender_exceptions
touch cidr_greylist_network_exceptions


In main.cf, our smtpd_recipient_restrictions should already look pretty like the sample below. Read the beginning (half dozen lines or so) of http://www.postfix.org/postconf.5.html. Now we need to add greylisting and a sender whitelist for greylisting.
Either vi /etc/postfix/main.cf or use the WinSCP editor and edit smtpd_recipient_restrictions, adding the items in red as usual:
smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination,
    reject_unlisted_recipient,
    check_recipient_access hash:/etc/postfix/reject_over_quota,
    check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions,
    check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
    reject_rbl_client sbl-xbl.spamhaus.org,
    check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
    check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
    check_client_access pcre:/etc/postfix/check_client_fqdn
Once the file is saved:
postfix reload
tail -f /var/log/mail.log


Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if your client gets greylisted like mine did. If I wait 30 seconds I should be able to send the message. Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed):
Jun 15 21:12:19 sfa postfix/smtpd[10095]: NOQUEUE: reject: RCPT from
 unknown[192.168.1.41]: 450 4.7.1 <unknown[192.168.1.41]>: Client host rejected:
  Greylisted, see http://isg.ee.ethz.ch/tools/postgrey/help/example.com.html;
   from=<garyv@example.org> to=<garyv@example.com> proto=ESMTP helo=<nobody>


I am going to whitelist 192.168.1.0 so clients on my internal network are not subjected to greylisting. Note that typically you would simply add your internal network to the mynetworks setting in main.cf. Cidr type tables do not need to be postmapped. http://www.postfix.org/cidr_table.5.html. I will place the whitelist entry in
vi /etc/postfix/cidr_greylist_network_exceptions
192.168.0.0/16 OK

postfix reload

I am going to wipe out the entire Postgrey database in order to prove to myself that my change did in fact whitelist my network. You don't want to do this once the system is up and running (unless the database becomes corrupt). Then I am going to try to send another message.

ls -l /var/lib/postgrey/
/etc/init.d/postgrey stop
rm /var/lib/postgrey/*
/etc/init.d/postgrey start

ls -l /var/lib/postgrey/


tail -f /var/log/mail.log

And it worked:
Jun 15 21:36:51 sfa postfix/smtpd[10167]: connect from unknown[192.168.1.41]
Jun 15 21:36:52 sfa postfix/smtpd[10167]: 1EDD624218: client=unknown[192.168.1.41]



Install policyd-weight - Optional

Index Top Bottom
Policyd-weight is no longer maintained. One thing to consider when using this is that RBLs at www.spamhaus.org are queried. If spamhaus is blocking you, then this program looses some value http://www.spamhaus.org/organization/dnsblusage.html. Remember that the Bind9 forwarding configuration (forwarding - or no forwarding) may affect whether spamhaus allows you to use their free service or not. Policyd-weight will block a client if it hits a few RBLs or appears to be poorly configured (or not configured) as a mail server. The $REJECTLEVEL setting is what will make the biggest difference as far as how much mail is rejected. I set it to a conservative 4.8 in /etc/policyd-weight.conf in order to give idiots running real (but poorly configured) mail servers a fighting chance. The RBL checks in policyd-weight may cause a slight delay when a client connects.
apt-get install policyd-weight

LINUX2
cd /etc
wget http://verchick.com/mecham/public_html/policyd-weight.conf
sed -i "s/'IX_MANITU',/'IX_MANITU'/" /usr/sbin/policyd-weight
sed -i "s/ 'rbl.ipv6-world.net'/#'rbl.ipv6-world.net'/" /usr/sbin/policyd-weight
/etc/init.d/policyd-weight stop
/etc/init.d/policyd-weight start
cd /etc/postfix
cp main.cf main.cf-before-policyd-weight
postconf -e "smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_invalid_hostname"
test -e greylist_sender_exceptions || cp -i access greylist_sender_exceptions
postmap greylist_sender_exceptions
touch cidr_greylist_network_exceptions


Now vi /etc/postfix/main.cf or use the WinSCP editor to edit main.cf. If you installed Postgrey, insert policyd-weight ahead of check_client_fqdn:
smtpd_recipient_restrictions =
    permit_mynetworks,
    permit_sasl_authenticated,
    reject_unauth_destination,
    reject_unlisted_recipient,
    check_recipient_access hash:/etc/postfix/reject_over_quota,	 
    check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
    check_client_access hash:/etc/postfix/rbl_client_exceptions,
    check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
    reject_rbl_client sbl-xbl.spamhaus.org,
    check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
    check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
    check_policy_service inet:127.0.0.1:12525,
    check_client_access pcre:/etc/postfix/check_client_fqdn
otherwise, add a few lines to the end:
     [...]
    reject_rbl_client sbl-xbl.spamhaus.org,
    check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
    check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
    check_policy_service inet:127.0.0.1:12525
Then:
postfix reload
tail -f /var/log/mail.log


Make sure reject_unlisted_recipient, is set as shown above. Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if you get a hit like mine did (you may have to temporarily remove your network from /etc/postfix/cidr_greylist_network_exceptions and reload postfix). Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed). If you installed postgrey and entered your local network in cidr_greylist_network_exceptions you will have to remove it and reload postfix in order to test this. After doing so, I once got somthing similar to:
Apr  5 13:48:17 msa postfix/policyd-weight[19123]: decided action=550 
 Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX
 settings or to get removed from DNSBLs; MTA helo: myclient, MTA hostname: unknown[192.168.1.41]
 (helo/hostname mismatch); in dsn.rfc-ignorant.org; in postmaster.rfc-ignorant.org;
 in abuse.rfc-ignorant.org; <client=192.168.1.41> <helo=myclient>
 <from=test@example.com> <to=garyv@example.com>; delay: 3s

Apr  5 13:48:17 msa postfix/smtpd[19117]: NOQUEUE: reject: RCPT from unknown[192.168.1.41]:
 550 5.7.1 <garyv@example.com>: Recipient address rejected: Mail appeared to be SPAM or
 forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings or to get removed
 from DNSBLs; MTA helo: myclient, MTA hostname: unknown[192.168.1.41] (helo/hostname mismatch);
 in dsn.rfc-ignorant.org; in postmaster.rfc-ignorant.org; in abuse.rfc-ignorant.org;
 from=<test@example.com> to=<garyv@example.com> proto=ESMTP helo=<myclient>
In order to prevent internal clients from getting scanned (and possibly rejected) by policyd-weight (and postgrey),
vi /etc/postfix/main.cf and place your internal network in mynetworks (and reload postfix) or whitelist the network by adding it to /etc/postfix/cidr_greylist_network_exceptions as mentioned above in the postgrey instructions.

Make a couple changes to logcheck (this is four lines)
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: decided action=" >>/etc/logcheck/ignore.d.server/postfix
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: weighted check:" >>/etc/logcheck/ignore.d.server/postfix
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: child: spawned" >>/etc/logcheck/ignore.d.server/postfix
tail -2 /etc/logcheck/ignore.d.server/postfix



Install SaneSecurity and MSRBL ClamAV signatures - Optional

Index Top Bottom
Optionally use additional anti-phishing and scam signatures. This setup uses Bill Landry's clamav-unofficial-sigs.sh script, version 3.7.1:
apt-get install curl rsync

cd /usr/share/man/man8/
wget http://verchick.com/mecham/public_html/spam/clamav-unofficial-sigs.8.gz.b
mv clamav-unofficial-sigs.8.gz.b clamav-unofficial-sigs.8.gz
cd /etc
wget http://verchick.com/mecham/public_html/spam/clamav-unofficial-sigs.conf.txt
mv clamav-unofficial-sigs.conf.txt clamav-unofficial-sigs.conf
cd /usr/sbin
wget http://verchick.com/mecham/public_html/spam/clamd-status.sh.txt
mv clamd-status.sh.txt clamd-status.sh
chmod +x clamd-status.sh
wget http://verchick.com/mecham/public_html/spam/clamav-unofficial-sigs.sh.txt
mv clamav-unofficial-sigs.sh.txt clamav-unofficial-sigs.sh
chmod +x clamav-unofficial-sigs.sh
cd /etc/logrotate.d
wget http://verchick.com/mecham/public_html/spam/clamav-unofficial-sigs-logrotate.txt
mv clamav-unofficial-sigs-logrotate.txt clamav-unofficial-sigs
clamav-unofficial-sigs.sh


The script will execute. When it's finished:

ls -l /var/lib/clamav

You will notice the data has been downloaded:
-rw-r--r-- 1 clamav clamav    11478 2010-06-06 19:52 bytecode.cvd
-rw-r--r-- 1 clamav clamav  2196135 2010-06-06 19:52 daily.cvd
-rw-r--r-- 1 clamav clamav    50400 2010-05-27 08:12 honeynet.hdb
-rw-r--r-- 1 clamav clamav  3950238 2010-06-05 08:49 junk.ndb
-rw-r--r-- 1 clamav clamav   559557 2010-06-06 19:49 jurlbl.ndb
-rw-r--r-- 1 clamav clamav 22906487 2010-06-06 19:52 main.cvd
-rw-r--r-- 1 clamav clamav       52 2010-06-06 19:52 mirrors.dat
-rw-r--r-- 1 clamav clamav  2299236 2010-06-06 09:50 phish.ndb
-rw-r--r-- 1 clamav clamav    60727 2010-06-05 08:49 rogue.hdb
-rw-r--r-- 1 clamav clamav     1116 2010-05-13 04:49 sanesecurity.ftm
-rw-r--r-- 1 clamav clamav  1712518 2010-06-04 12:54 scam.ndb
-rw-r--r-- 1 clamav clamav    75068 2010-05-27 08:12 securiteinfobat.hdb
-rw-r--r-- 1 clamav clamav   193234 2010-05-27 08:12 securiteinfodos.hdb
-rw-r--r-- 1 clamav clamav    53913 2010-05-27 08:12 securiteinfoelf.hdb
-rw-r--r-- 1 clamav clamav   225919 2010-06-02 02:53 securiteinfo.hdb
-rw-r--r-- 1 clamav clamav  1329399 2010-06-03 02:26 securiteinfohtml.hdb
-rw-r--r-- 1 clamav clamav   244083 2010-05-27 08:13 securiteinfooffice.hdb
-rw-r--r-- 1 clamav clamav   238856 2010-05-27 08:13 securiteinfopdf.hdb
-rw-r--r-- 1 clamav clamav    21106 2010-05-27 08:13 securiteinfosh.hdb
-rw-r--r-- 1 clamav clamav    52755 2010-06-06 09:50 spamimg.hdb
-rw-r--r-- 1 clamav clamav   866409 2010-06-06 19:45 winnow_malware.hdb
-rw-r--r-- 1 clamav clamav   785121 2010-06-06 19:45 winnow_malware_links.ndb
Now we add a crontab entry with download attempts performed every 6th hour:
crontab -e

Insert these two entries. Replace MM (minutes) below with a number between 1 and 59:
MM */6 * * * /usr/sbin/clamav-unofficial-sigs.sh
*/6 * * * * /usr/sbin/clamd-status.sh


Save and exit the file. The first cron job should run every 6 hours, and the second, every 6 minutes. The clamav-status.sh script will restart clamd after a crash. There is a log file at /var/log/clamav-unofficial-sigs.log and you can read the man page at 'man clamav-unofficial-sigs'.


Install Botnet SA plugin - Optional

Index Top Bottom
This plugin is designed to detect botnet (zombie) clients. I trim the score down in case of false positives (which do happen). Running this check will increase scan times.
cd /usr/local/src
mkdir Botnet-0.8
cd Botnet-0.8
wget http://verchick.com/mecham/public_html/spam/Botnet-0.8.tar
tar -xf Botnet-0.8.tar
wget http://verchick.com/mecham/public_html/spam/botnet8patch2.txt
cp Botnet.pm Botnet.pm~
patch -p0 <botnet8patch2.txt
cp Botnet.pm /etc/spamassassin/
cp Botnet.cf /etc/spamassassin/
cd /etc/spamassassin/
sed -i 's/5.0/1.2/' Botnet.cf
/etc/init.d/amavis restart
spamassassin --lint


You will have to wait for some mail from distant servers to test this. Look for a hit to the BOTNET rule in the X-Spam-Status: header. This will only happen occasionally as much of this stuff may be blocked before SpamAssassin sees it.

Install p0f - Optional

Index Top Bottom
Passive OS Fingerprinting attempts to discover what type of system a message is sent from. If mail comes directly from a Windows XP machine (for example) one might assume the machine is a zombie. If this machine is behind a NAT or proxy server that proxies the TCP sessions, p0f may not work. Trusted clients are not checked (due to settings in our MYNETS and TRUSTED policy banks):
apt-get install p0f

cd /etc/init.d
wget http://verchick.com/mecham/public_html/spam/p0f
chmod +x p0f
update-rc.d p0f defaults
cd /etc/spamassassin
wget http://verchick.com/mecham/public_html/spam/p0f.cf
/etc/init.d/p0f start


vi /etc/amavis/conf.d/50-user and add the lines shown in red in the location shown:
# nice to have $log_level (1-5) available:
$log_level = 0;


# Use passive OS fingerprinting
$os_fingerprint_method = 'p0f:127.0.0.1:2345';


Once the edits are finished:
/etc/init.d/amavis restart

You will have to wait for some mail from distant servers to test this. Look for hits to P0F rules in the X-Spam-Status: header. Look for an X-Amavis-OS-Fingerprint: header. If you have both p0f and Botnet installed, you may find this post relevant http://marc.info/?l=spamassassin-users&m=118272807918926

Compile SpamAssassin Body Rules - Optional

Index Top Bottom
With SpamAssassin version 3.2 or newer, we can optionally compile some body rules so they execute faster. Start by running sa-compile for the first time and check for errors:
sa-compile

If it looks like it didn't crash:
vi /etc/spamassassin/v320.pre

To enable the ability to use compiled rules, uncomment the line:
# loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody

Save and exit the file, then run --lint:
su amavis -c 'spamassassin --lint'

Now grab a script that will run sa-compile once a week:
cd /etc/cron.weekly
wget http://verchick.com/mecham/public_html/spam/sa-compile.sh.txt
mv sa-compile.sh.txt sa-compile-weekly
chmod +x sa-compile-weekly


Run the script. If it is working correctly, it should take a while, there should be no output and amavisd-new should get reloaded. The script does require that sa-compile has ran at least once before. If you tail the mail.log: tail -50 /var/log/mail.log you should see where amavisd-new was reloaded:
./sa-compile-weekly

tail -50 /var/log/mail.log



Install AIDE - Optional

Index Top Bottom
The concept with AIDE is to take a snapshot of some of the characteristics of critical files on our system, and then store that information in a database on read-only media; a write protected floppy for example. Then run a program that compares the current state of those files on our system with the database and report any differences. We should be able to tell if a hacker has compromised our system if the characteristics of the files on our system differ from the files in the database.

Place a good writable floppy in the floppy drive, then:
mkdir /floppy
test -e /dev/fd0u1722 || mknod /dev/fd0u1722 b 2 60
chmod 660 /dev/fd0u1722
chown root:floppy /dev/fd0u1722
fdformat /dev/fd0u1722


If the format fails, try again. If it fails again, try a different floppy. If it continues to fail, try a different brand of floppy or a different floppy drive. If the format goes well:
mke2fs /dev/fd0u1722

cd /etc/init.d
wget http://verchick.com/mecham/public_html/spam/startflop.txt
mv startflop.txt startflop
chmod +x startflop
update-rc.d startflop defaults 80
ln -s /etc/init.d/startflop /usr/bin/floppy


Now you can mount the floppy with floppy start and unmount the floppy with floppy stop (as long as the floppy is formatted as /dev/fd0u1722). The floppy will be left in the machine at all times and will automatically get mounted each time the system starts. You should reboot the computer and see if it starts back up. This is not a bootable floppy but some systems may hang if the BIOS is set to boot from the floppy drive. You may have to change the boot order in the BIOS so your server boots from the (first) hard drive before it boots from the floppy. Now install aide (choose the default responses to questions):
cd /usr/local/src
wget http://verchick.com/mecham/public_html/spam/aide_0.10-4_i386.deb
dpkg -i aide_0.10-4_i386.deb


Daily reports are mailed to root by default. [Ok]
Initialize aide database? [No]
Before AIDE can be used, you will have to initialize a database. [Ok]


We are going to obscure aide somewhat, use my configuration file and report and then initialize the database (which will take some time):
floppy start
mkdir /root/aide
cd /root/aide
cp /usr/bin/aide /root/aide
cp /usr/share/man/man1/aide.1.gz /root/aide
cp /usr/share/man/man5/aide.conf.5.gz /root/aide
cp /usr/share/man/man8/aideinit.8.gz /root/aide
cp /etc/cron.daily/aide /root/aide/report
cp /etc/cron.daily/aide /root/aide/report~
cp /etc/aide/aide.conf /root/aide/aide.conf
cp /etc/aide/aide.conf /root/aide/aide.conf~
wget http://verchick.com/mecham/public_html/spam/aide.conf.patch.txt
patch -p0 <aide.conf.patch.txt
wget http://verchick.com/mecham/public_html/spam/report.patch.txt
patch -p0 <report.patch.txt
chmod +x report

apt-get remove aide

mkdir /var/log/aide
cp /root/aide/aide.1.gz /usr/share/man/man1/
cp /root/aide/aide.conf.5.gz /usr/share/man/man5/
cp /root/aide/aideinit.8.gz /usr/share/man/man8/
cp /root/aide/aide /floppy
cp /root/aide/aide.conf /floppy
cp /root/aide/report /floppy
/floppy/aide --config=/floppy/aide.conf --init


Wait for the initial database to be created (/root/aide/aide.db.new) then copy it to the floppy:
cp /root/aide/aide.db.new /floppy/aide.db

Once the file copies over (wait about 30 seconds), run the report for the first time (this will take a while). Once the report completes, check the mailbox for the user that gets root's mail:
/floppy/report

Each time you run the report, it reads the database /floppy/aide.db but it also creates a new aide.db.new database. So why do we care about the aide.db.new? It's a matter of convenience. If the report is getting kind of long, and there appears to be nothing in it you would not expect; the aide.db.new database reflects the current state of the system and could be used to replace aide.db. You replace your aide.db on the floppy with aide.db.new, just as you did when you first initialized the system. If you run the report again, it should be clean, unless files have changed since it was created. You need to understand the risks of relying on the quality of the aide.db.new database if you use the copy that was created during the cron job. The cron job could have been hacked. A hacker could manipulate the file. Only rely on the aide.db.new database if you run   /floppy/report   from the command line and you inspect the report just prior to copying it over to the floppy. If you left the write protection off the floppy drive for any length of time, it's possible you can no longer rely on the database. Be paranoid.

So here's the gist of maintenance:
  • Run the report.
  • Read the report.
  • If it looks normal, and it has not grown to the point it is annoying, you're done for today.
  • If the report lists any critical system files you know you did not change or update, or you notice other suspicious activity,
    I hope you have a cloned hard drive ready to swap out because you may have been hacked. Study the report carefully.
  • If everything looks normal and you want it cleaned up, unmount the floppy, flip the write protect tab off, mount the floppy,
    copy /root/aide/aide.db.new to /floppy/aide.db, unmount the floppy, flip the write protect tab on, mount the floppy, done.
I have a script that can help with this last item. It's a lot more convenient if you are at the console when you run this (but it's not required). Please download it:
cd
wget http://verchick.com/mecham/public_html/spam/go.txt
mv -i go.txt go
chmod +x go


Then run it to see how it works:
./go

You should now have the program and data on the write protected floppy. Set up a cron job to run the report:
crontab -e

and insert something like this:
25 7 * * * /floppy/report

It is typical that the file '/dev/.udev/uevent_seqnum' will be on every report.


Install altermime - Optional

Index Top Bottom
If you feel the need to add a disclaimer to outgoing mail you can use altermime to accomplish this. We are going to use the latest devel release.
cd /etc
wget http://verchick.com/mecham/public_html/spam/disclaimer.txt
cd /usr/local/src
wget http://pldaniels.com/altermime/altermime-0.3-dev.tar.gz
tar xzf altermime-0.3-dev.tar.gz
cd altermime-0.3-dev
make && make install


Now vi /etc/amavis/conf.d/50-user and add this:
# altermime disclaimers
$altermime = '/usr/local/bin/altermime'; # a path to the program
@altermime_args_disclaimer = qw(--disclaimer=/etc/disclaimer.txt);
$defang_maps_by_ccat{+CC_CLEAN} = [ 'disclaimer' ];
$defang_maps_by_ccat{+CC_BADH} = [ 'disclaimer' ];


While still in /etc/amavis/conf.d50-user, we want to enable disclaimers for outgoing mail. This is done through the MYNETS and/or TRUSTED policy banks. You need to insure that the IP address of clients sending mail from the internal network are included in @mynetworks - or your clients are authenticated and therefore using the TRUSTED policy bank:
$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
  originating => 1,

  allow_disclaimers => 1,
  os_fingerprint_method => undef,


$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
  originating => 1,
  allow_disclaimers => 1,
  os_fingerprint_method => undef,


Then of course:
/etc/init.d/amavis restart

Log into SquirrelMail and send a message. If all is well your disclaimer will be added. Of course you have to customize it:
vi /etc/disclaimer.txt

For the moment there is one limitation: there can only be one manger in effect at a time, it is not currently possible to both defang and to append a disclaimer, e.g. for internal-to-internal mail inserting a disclaimer takes precedence. Also, if you need different disclaimers for different domains, search for @disclaimer_options_bysender_maps in http://www.ijs.si/software/amavisd/release-notes.txt.


Odds and ends

Index Top Bottom
If you wish to give clients within your network the ability to relay mail to domains other than yours then you should place your network (or individual IP addresses) in mynetworks in main.cf. You can also exclude certain hosts in your network by preceding the IP address with an exclamation point. Excluded addresses need to be listed before included addresses. You should also place the network in @mynetworks in 50-user in order for the clients to take advantage of the MYNETS policy bank.

You should build another server dedicated to holding a nightly backup of this server. The minimal Debian installation should be fine. It should have a very large hard drive, but I imagine it need not be a powerful machine. I used http://www.howtoforge.com/linux_rdiff_backup and can say it works well. Make sure iptables allows SSH from the backup server. I like the way it explains how to connect via SSH without using a password. Also how to limit what the connected party can run. I did have a problem with the "ssh-copy-id" command - read the comments at the bottom of the first page for a solution.

I suggest reading:
http://verchick.com/mecham/public_html/spam/amavisd-settings.html

You should read this:
http://verchick.com/mecham/public_html/spam/bypassing.html

I suggest reading:
http://jaqque.sbih.org/kplug/apt-pinning.html

This can help determine how long SpamAssassin spends on certain items:
http://marc.info/?l=amavis-user&m=117874388132138

I archive copies of messages. Here are some hints on how I do it:
http://www.freespamfilter.org/forum/viewtopic.php?t=526

You can redefine %banned_rules (make sure you include 'DEFAULT') in order to set per-recipient banned rules sets (in SQL).
http://marc.info/?l=amavis-user&m=111995411713191

I think you should consider building a secondary MX server that can spool mail if this machine goes down. It also reduces load on this server because spammers often target the secondary MX. You will need to populate relay_recipients on that machine. FYI you can gather all the local recipients into a file with commands such as these:
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(username, ' OK') from mailbox" >file1
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(address, ' OK') from alias" >file2
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(goto, ' OK') from alias" >file3
cat file1 file2 file3 >file4
cat file4 | egrep -v ^root@ | sort | uniq >file5


Here are links to most of the stuff you get from me (in the order downloaded):
my-medium.cnf.patch.txt
virtual.index.html
edit-mailbox.patch.v2.txt
create-mailbox.v2.patch.txt
mail_profile.sh.txt
quotaedit.sh.txt
postfixadmin.menu.patch.txt
convert.postfixadmin.tables.sql.v2211
mysql_virtual_alias_maps.cf
mysql_virtual_domains_maps.cf
mysql_virtual_mailbox_maps.cf
global.sieve.txt
deliver.logrotate.txt
dovecot.conf.txt
dovecot-sql.conf.txt
dovecot-crammd5.conf.txt
maildircheck.txt
rmmailboxspam.txt
amavis-init-20030616
amavisd.conf-262.txt
amavisd.conf-modified
amavisd.conf-sample
15-av_scanners.262.txt
amavisd.2.6.2.patch1.txt
amavisd.2.6.2.patch2.txt
amavisd.2.6.2.patch3.txt
amavisd.2.6.2.patch4.txt
amavisd.2.6.2.patch6.txt
amavis-sanesecurity_v2.cf
charset
template-dsn.txt
template-spam-admin.txt
template-spam-sender.txt
template-virus-admin.txt
template-virus-recipient.txt
template-virus-sender.txt
sa-update1.sh.txt
amavis-260-sqmail.sql.txt
domain.patch.v2.txt
trim-amavis-msgs.txt
trim-amavis.sql.txt
trim-amavis.txt
rmvirusquar.txt
gv-bayes-awl.sql.txt
local.cf-bayes-awl.txt
sample-spam.txt
trim-awl.sql.txt
trim-awl.txt
trim-sql-awl-weekly.txt
amavisnewsql.patch1.txt
pflogsumm.sh
vacation.2211.patch.txt
postfixadmin.wording.patch.txt
mzcpatch.txt
AmavisdEngine.patch1.txt
mailzu.logrotate.txt
quota-warning.sh.txt
mysql_relay_domains_maps.cf
mysql_virtual_domains_maps.cf.v2
check_client_fqdn
policyd-weight.conf
clamav-unofficial-sigs.conf.txt
clamd-status.sh.txt
clamav-unofficial-sigs.sh.txt
clamav-unofficial-sigs-logrotate.txt
botnet8patch2.txt
p0f
p0f.cf
sa-compile.sh.txt
startflop.txt
aide.conf.patch.txt
report.patch.txt
go.txt
disclaimer.txt

Here are general links:
amavisd-new
Apache 2.2
Bind
DCC
Dovecot
FuzzyOCR
Maildirmake
MSRBL
MySQL
mysql-zrm
MailGraph
MailZu
pflogsumm
phpMyAdmin
p0f
Postfix
PostfixAdmin
PostGrey
policyd-weight
Pyzor
sanesecurity
SARE
SARE Plugins
SpamAssassin
SquirrelMail docs
Vupil's Razor

Apache and Dovecot are not tuned to your particular level of traffic. I leave it up to you to discover how many processes (and other tweaks) you might need.

You need to subscribe to:
http://lists.debian.org/debian-security-announce/
http://lists.clamav.net/mailman/listinfo/clamav-announce

You may like this:
http://logreporters.sourceforge.net/

You may also want to install webalizer:
apt-get install webalizer
Run webalizer and then browse to your domain https://msa.example.com/webalizer
Add a crontab entry that runs after the daily cron run, e.g.:
1 8 * * * /usr/bin/webalizer

Some people (like me) do not like SpamAssassin AWL (auto whitelisting - which is actually a score averaging system). To turn it off vi /etc/spamassassin/local.cf and insert use_auto_whitelist 0 (don't disable the plugin in v310.pre) then:
spamassassin --lint
/etc/init.d/amavis restart


Every six months (or so), it would be a good idea to optimize your MySQL tables. I think it's a good idea (possibly even necessary) to stop MySQL activity during this period. This means you should shut down postfix and amavisd-new and possibly even disconnect the ethernet cable. There should be no MySQL maintenance scripts running (like /etc/cron.daily/ztrim-amavis which should start running at 06:25). Log in to mysql (as root) and optimize the tables:
USE amavis;
OPTIMIZE TABLE maddr, mailaddr, msg, msgowner, msgrcpt, msgs, policy, quarantine, users, wblist;
USE postfix;
OPTIMIZE TABLE admin, alias, domain, domain_admins, log, mailbox, vacation;
USE sa_bayes;
OPTIMIZE TABLE awl, bayes_expire, bayes_global_vars, bayes_seen, bayes_token, bayes_vars;


If you change your NIC card, udev will call your new NIC something other than eth0. In this case you need to
vi /etc/udev/rules.d/70-persistent-net.rules and correct the situation. I reboot afterwards.

You may want to edit some of the banned rules in the $banned_filename_re section of /etc/amavis/conf.d/20-debian_defaults . To start with, I usually comment out the "# banned extension - basic" line and uncomment the four lines below it that comprise the "# banned ext - long" section.

In /etc/postfix/master.cf I add " -o smtpd_client_connection_count_limit=21" to the smtpd daemon to limit the number of simultaneous connections a single client can make (within one minute's time). The default is 50 (half of the default maxproc). Lowering this setting from 50 to 21 is not appropriate if you get mail from another relay server - you would not want to place this lowered limit on a server you trust, but otherwise it may be a good idea to help with dictionary attacks:
smtp      inet  n       -       -       -       -       smtpd
    -o smtpd_client_connection_count_limit=21
Ok, start customizing. BTW, It would not surprise me if there are vulnerabilities in some of the pages of the various web inferfaces.


Index Top

Gary V
mr88talent at yahoo dot com
JUN 6 2010