|
The amavisd-new README.lookups
page may be somewhat difficult to understand. However, for our needs the two major concepts
are reasonably simple. First let's talk about SQL and static lookups. A lookup is performed using an
email address as the key (usually the envelope recipient).
SQL lookups are performed before static hash lookups, hash lookups are performed before defaulting to constants.
To be more precise, from the README: "default sequence of lookups: SQL, LDAP, hash, ACL, regexp, constant."
(you can find examples of hash, ACL, regexp and constant settings in amavisd.conf-sample).
For example: if a matching recipient email address (or part of an address, or catchall) exists
in the SQL database (in the users table), and has a policy (in the policy table) with the field 'spam_tag2_level' set at 6.0 and there
is also a static constant setting '$sa_tag2_level_deflt' (in the 50-user file) set to 6.31; then 6.0
is used (because SQL is checked first and first match wins). The search (for that particular piece of data)
stops at the first definitive answer (not NULL or undef).
If the SQL field is NULL however (which evaluates to undef), the search continues on.
In this example there is no hash table, no ACL list and no regexp list, so the search ends at the constant
(which does exist - and matches any recipient); the net result: 6.31 is used.
That is the first important concept. The second important concept is priority within the SQL and static hash lookups. Full email addresses are broken down into pieces and there is an attempt to match each piece with the data in the SQL, hash or other table/map. In static hash lookups the most specific comparisons are made first (full email address). The most general comparisons are made last (catchall "."). Hash lookups (e.g. for user+foo@example.com) are performed in the following order: user+foo@example.com user@example.com user+foo@ user@ example.com .example.com .com .Note that the format is a little different for SQL: user+foo@example.com user@example.com user+foo user @example.com @.example.com @.com @.Because of the order of searches and the fact that the search stops at the first match, in hash (and other) tables it only makes sense to place more specific patterns before the more general ones. For example, in this hash postmaster@example.net is placed before .example.net: $sa_tag2_level_deflt = 6.31;
@spam_tag2_level_maps = (
{ 'postmaster@example.net' => 99.0,
'.example.net' => 7.0 },
\$sa_tag2_level_deflt, # catchall default
);In SQL however, the order in which the lookups are performed is determined by the SQL SELECT
statement and the users.priority field. The supplied SQL SELECT statement orders by the
priority field in DESCending order. This means you have to create the same 'most specific to
most general' order by setting the proper priority. In other words, you could assign a priority
of 7 to all users with a full email address in the email field (which we do), and you could add a domain wide
user "@example.net" that would have a lower priority. If user@example.net had spam_kill_level
set to 6.0 and @example.net had spam_kill_level set to 7.0, then 6.0 would be used because
user@example.net has a higher priority. If user@example.net had spam_kill_level set to NULL,
then the lookup would fall through to the @example.net user (and it would find 7.0).
If @example.net had a higher priority than user@example.net, then 7.0 would be used. For a given
recipient this is what the SQL SELECT statement that amavisd-new creates might look like:SELECT *,users.id FROM users LEFT JOIN policy ON users.policy_id=policy.id
Here is an example from a level 5 log entry where user@example.net has a policy with spam_tag2_level NULL (which is translated to undef) but a static hash exists (@spam_tag2_level_maps as shown above). The lookup falls through to the static hash, where a match is found: lookup_sql_field(spam_tag2_level) "user@example.net" result=undef query_keys: user@example.net, user@, example.net, .example.net, .net, . lookup_hash(user@example.net) matches key ".example.net", result=7 lookup (spam_tag2_level) => true, "user@example.net" matches, result="7", matching_key=".example.net"In this next example the same user still has spam_tag2_level set to NULL, but there is a lower priority (domain wide) user '@example.net' that has spam_tag2_level set to 9999: lookup_sql_field(spam_tag2_level) "user@example.net" result=undef lookup_sql_field(spam_tag2_level) "user@example.net" result=9999 lookup (spam_tag2_level) => true, "user@example.net" matches, result="9999", matching_key="/cached/""/cached/" simply means it is using results found during the initial query. Here is an example where the user does not exist in SQL and is not found in the hash, therefore the lookup falls through to the scalar variable: lookup_sql_field(spam_tag2_level), "user@example.org" no matching records
query_keys: user@example.org, user@, example.org, .example.org, .org, .
lookup_hash(user@example.org), no matches
lookup: (scalar) matches, result="6.31"
lookup (spam_tag2_level) => true, "user@example.org" matches, result="6.31",
matching_key="(constant:6.31)"Now
hopefully it will make more sense when you read
README.lookups.
When I initially created the policies, I left all the data fields NULL with the exception of
spam_tag2_level, spam_kill_level and occasionally spam_quarantine_cutoff_level. This way all
the remaining settings are controlled by entries in the 50-user file. When new policies are created
in the SquirrelMail interface, only those two or three fields have non-NULL values.
|
SARE (SpamAssassin Rules Emporium) offers additional SpamAssassin rule sets.
OpenProtect gathers together the safest sets of SARE rules.
During this process we will replace the current sa-update.sh script with a script
that includes the OpenProtect SARE rules:
cd /etc/spamassassin
Personally I vi /usr/sbin/sa-update.sh and have it delete a few things that it downloads I don't want. If you install a new version of SpamAssassin you would have to remember to update this:
[...]
|
DCC detects messages that are bulk mailed. A hit results in 1.4 or 2.17 SpamAssassin points.
Personally, I set this to 1.75 points ('score DCC_CHECK 1.75' in local.cf).
The way I understand the license for DCC is if you are not a reseller and you receive
less than somewhere around 100,000 messages per day you can use the DCC client by itself.
If you run a larger site then you need to run a DCC server and flood your data to public
DCC servers. If you are a reseller you need to pay for a license. I only outline installing
a client. Be patient when compiling the program - it may appear to hang at one point:
cd /usr/local/src
You should see entries with 'requests ok'. If not, then you might be having an issue with a proxy server or firewall. http://www.dcc-servers.net/dcc/firewall.html gives an example of a Cisco router access list entry. Keep in mind that you will likely not run a DCC server, only the DCC client. If the tests fail you won't be able to test again until some period of time has passed. The program does this to prevent broken clients from trying too often.
crontab -e
and insert:
43 11 * * * /usr/bin/cron-dccd
cd
You will get errors because I chose to turn of logging. I wouldn't worry about them: find: /etc/rc.d: No such file or directory Jun 15 19:33:12 sfa dccifd[7459]: log thresholds set with -t but no -l directory Jun 15 19:33:12 sfa dccifd[7459]: no -l directory prevents per-user logging with -U Test with spamassassin:
su amavis -c 'spamassassin -D dcc <sample-spam.txt'
You should get something like this in the output:
[7615] dbg: dcc: dccifd is available: /var/dcc/dccifd
We need to suppress a few logcheck messages:
echo "stat\(log directory /var/dcc/log\): No such file or directory" >> /etc/logcheck/ignore.d.server/dcc
Read the LICENSE:
cat /usr/local/src/dcc-1.3.92/LICENSE
|
Of course Greylisting will reject a ton of spam - but at a cost. Some mail will be
delayed. To help out I cut the retry time from 6 minutes to 29 seconds. We will use
'selective greylisting' with the idea being we will attempt to only target clients
that appear to come from dial-up and dynamic IP addresses. If you
decide to implement this I also recommend you whitelist clients you trust. The whitelists
are located in the /etc/postgrey/ directory. We can also whitelist senders, clients or networks in Postfix.
apt-get install postgrey
In main.cf, our smtpd_recipient_restrictions should already look pretty like the sample below. Read the beginning (half dozen lines or so) of http://www.postfix.org/postconf.5.html. Now we need to add greylisting and a sender whitelist for greylisting. Either vi /etc/postfix/main.cf or use the WinSCP editor and edit smtpd_recipient_restrictions, adding the items in red as usual: smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
reject_unlisted_recipient,
check_recipient_access hash:/etc/postfix/reject_over_quota,
check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
check_client_access hash:/etc/postfix/rbl_client_exceptions,
check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
reject_rbl_client sbl-xbl.spamhaus.org,
check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
check_client_access pcre:/etc/postfix/check_client_fqdn
Once the file is saved:
postfix reload
Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if your client gets greylisted like mine did. If I wait 30 seconds I should be able to send the message. Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed): Jun 15 21:12:19 sfa postfix/smtpd[10095]: NOQUEUE: reject: RCPT from unknown[192.168.1.41]: 450 4.7.1 <unknown[192.168.1.41]>: Client host rejected: Greylisted, see http://isg.ee.ethz.ch/tools/postgrey/help/example.com.html; from=<garyv@example.org> to=<garyv@example.com> proto=ESMTP helo=<nobody> I am going to whitelist 192.168.1.0 so clients on my internal network are not subjected to greylisting. Note that typically you would simply add your internal network to the mynetworks setting in main.cf. Cidr type tables do not need to be postmapped. http://www.postfix.org/cidr_table.5.html. I will place the whitelist entry in vi /etc/postfix/cidr_greylist_network_exceptions
192.168.0.0/16 OK
postfix reload
I am going to wipe out the entire Postgrey database in order to prove to myself that my change did in fact whitelist my network. You don't want to do this once the system is up and running (unless the database becomes corrupt). Then I am going to try to send another message.
ls -l /var/lib/postgrey/
tail -f /var/log/mail.log
And it worked:
Jun 15 21:36:51 sfa postfix/smtpd[10167]: connect from unknown[192.168.1.41]
Note however that typically local networks are placed in $mynetworks, which means Postgrey would typically be bypassed for clients in $mynetworks. |
Note that Policyd-weight is no longer maintained. As with any program that uses RBLs,
certain RBLs may not work in the future. Depending on your volume, they may cost you money
to subscribe to them (zen.spamhaus.org for example). So with this in mind, be aware that
stuff that works today may not work in the future.
Policyd-weight will block a client if it hits a few RBLs or appears to be poorly
configured (or not configured) as a mail server. The $REJECTLEVEL setting is what
will make the biggest difference as far as how much mail is rejected. I set it
to a conservative 4.8 in /etc/policyd-weight.conf in order to give idiots running
real (but poorly configured) mail servers a fighting chance. The RBL checks in
policyd-weight may cause a slight delay when a client connects.
apt-get install policyd-weight
Now vi /etc/postfix/main.cf or use the WinSCP editor to edit main.cf. If you installed Postgrey, insert policyd-weight ahead of check_client_fqdn: smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
reject_unlisted_recipient,
check_recipient_access hash:/etc/postfix/reject_over_quota,
check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
check_client_access hash:/etc/postfix/rbl_client_exceptions,
check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
reject_rbl_client sbl-xbl.spamhaus.org,
check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
check_policy_service inet:127.0.0.1:12525,
check_client_access pcre:/etc/postfix/check_client_fqdn
otherwise, add a few lines to the end:
[...]
reject_rbl_client sbl-xbl.spamhaus.org,
check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
check_policy_service inet:127.0.0.1:12525
Then:
postfix reload
Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if you get a hit like mine did (you may have to temporarily remove your network from /etc/postfix/cidr_greylist_network_exceptions and reload postfix). Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed). If you installed postgrey and entered your local network in cidr_greylist_network_exceptions you will have to remove it and reload postfix in order to test this. After doing so, I got: Jul 20 20:39:42 sfa postfix/policyd-weight[4575]: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_NE_HELO=1.5 RESOLVED_IP_IS_NOT_HELO=1.5 (check from: .example. - helo: .myclient. - helo-domain: .myclient.) FROM_NOT_FAILED_HELO(DOMAIN)=3 IN_AHBL=2 IN_DSN_RFCI=3 IN_PM_RFCI=0.1 IN_ABUSE_RFCI=0.1; <client=192.168.1.41> <helo=myclient> <from=test@example.com> <to=test@example.com>; rate: 6.7 Jul 20 20:39:42 sfa postfix/policyd-weight[4575]: decided action=550 Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings or to get removed from DNSBLs; MTA helo: myclient, MTA hostname: unknown[192.168.1.41] (helo/hostname mismatch); in rhsbl.ahbl.org; in dsn.rfc-ignorant.org; in postmaster.rfc-ignorant.org; in abuse.rfc-ignorant.org; <client=192.168.1.41> <helo=myclient> <from=test@example.com> <to=test@example.com>; delay: 2sIn order to prevent internal clients from getting scanned (and possibly rejected) by policyd-weight (and postgrey), vi /etc/postfix/main.cf and place your internal network in mynetworks or whitelist the network by adding it to /etc/postfix/cidr_greylist_network_exceptions as mentioned above in the postgrey instructions. Make a couple changes to logcheck (this is four lines)
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: decided action=" >>/etc/logcheck/ignore.d.server/postfix
|
sanesecurity.com and msrbl.com author additional ClamAV signatures for scam, phishing,
image spam and spam.
apt-get install curl rsync
You will notice the data has been downloaded: drwxr-xr-x 2 clamav clamav 4096 2007-06-16 19:48 daily.inc -rw-r--r-- 1 clamav clamav 9351789 2007-06-10 21:16 main.cvd -rw------- 1 clamav clamav 260 2007-06-16 19:14 mirrors.dat -rw-r--r-- 1 clamav clamav 347982 2007-06-16 19:25 MSRBL-Images.hdb -rw-r--r-- 1 clamav clamav 228232 2007-06-08 04:33 MSRBL-SPAM.ndb -rw-r--r-- 1 clamav clamav 1033688 2007-06-16 19:48 phish.ndb -rw-r--r-- 1 clamav clamav 174338 2007-06-15 02:55 phish.ndb.gz -rw-r--r-- 1 clamav clamav 516182 2007-06-16 19:48 scam.ndb -rw-r--r-- 1 clamav clamav 102738 2007-06-15 02:55 scam.ndb.gzNow we add a crontab entry with download attempts performed every 4th hour:
crontab -e
Insert this entry. Replace MM (minutes) below with a number between 1 and 59:
MM */4 * * * /usr/sbin/UpdateSaneSecurity.sh
Add an entry to suppress logcheck messages:
echo "UpdateSaneSecurity" >>/etc/logcheck/ignore.d.server/amavisd-new
I have heard of an occasional false positive. Amavisd-new 2.5.2 or newer can treat these as spam instead of viruses. Unfortunately the latest Debian release of amavisd-new is 2.4.2. Logs of the last download are located in /var/tmp/clamdb/ |
This plugin is designed to detect botnet (zombie) clients. I trim the score down in
case of false positives (which do happen). Running this check will increase scan times.
cd /usr/local/src
You will have to wait for some mail from distant servers to test this. Look for a hit to the BOTNET rule in the X-Spam-Status: header. This will only happen occasionally as much of this stuff may be blocked before SpamAssassin sees it. |
Passive OS Fingerprinting attempts to discover what type of system a message is
sent from. If mail comes from a Windows XP machine (for example) one might assume the machine
is a zombie.
apt-get install p0f
vi /etc/amavis/conf.d/50-user and make these changes. Notice where $ is used, and where it is not used. You will have to jump around a little to find where to insert these three items (shown in red):
$penpals_threshold_low = 1;
Once the edits are finished:
amavisd-new reload
You will have to wait for some mail from distant servers to test this. Look for hits to P0F rules in the X-Spam-Status: header. If this machine is behind a NAT or proxy server that proxies the TCP sessions, P0F may not work. If you have both p0f and Botnet installed, you may find this post relevant http://marc.info/?l=spamassassin-users&m=118272807918926 |
The Debian etch package of amavisd-new is somewhat outdated. Personally I find it
much easier to stay current with amavisd-new if I migrate away from the Debian package.
With the instructions provided we will still continue to use the Debian style configuration
files. Once this is set up, we no longer have to rely on the Debian package maintainers
to produce updated versions of amavisd-new. Amavisd-new is actively developed by Mark
Martinec and new versions come out often.
Log in as root. Before we remove amavisd-new
we make backup copies of our important files. You should only perform this operation once (to
avoid overwriting the original backup). When the amavisd-new package is removed, the
/var/lib/amavis and /etc/amavis directories should not be removed
(and I do not make a backup copy of /var/lib/amavis). If you have items in /var/lib/amavis
that are important to you, you may want to consider making a copy of that directory
(and subdirectories):
mkdir /usr/ambackup
Of the files above, JpegTester.pm, /usr/sbin/amavisd-new and /usr/share/amavis/conf.d/
should get deleted when amavisd-new is removed. We will need to restore those from the backups.
There may be other files elsewhere that get deleted but they do not
affect how amavisd-new functions so their absence will not hurt us. The documentation
that we may loose will be provided with the original source code.
Test that when you remove amavisd-new, only amavisd-new will be removed:
apt-get -s remove amavisd-new
If it is the only thing that will be removed, continue on:
/etc/init.d/amavis stop
At this point we are temporarily going to put the same version
of amavisd-new back together again so we can continue to process mail. The first command below
should all fit on one line:
if ! grep -q amavis /etc/passwd; then adduser --group --system --home /var/lib/amavis --shell /bin/sh amavis; fi
The Debian amavisd-new package has been removed but you are still
using most of the original components of the Debian package. The main difference is
the /etc/init.d/amavis init script. Now it should be easy to upgrade to amavisd-new 2.5.4.
We will install an amavisd.conf
that simply loads in all the Debian conf files:
cd /etc/amavis
The amavisd.conf-modified file is an example of the amavisd.conf supplied with the source code but it has been modified for use with Debian. I suggest you read /etc/amavis/amavisd.conf. This version of amavisd-new can treat sanesecurity and MSRBL 'virusus' as spam. If you are using the sanesecurity ClamAV signatures, download a rule set to score these as spam:
cd /etc/spamassassin
If you are currently using external notification templates (you probably are since they are enabled by default in /etc/amavis/conf.d/30-template_localization), you should get the templates that match amavisd-new 2.5.4 and if necessary modify them in the same manner you have modified the currently installed 2.4.2 templates. The only templates available are English templates. If you don't normally customize your templates you should comment out: read_l10n_templates('en_US', '/etc/amavis'); in the 30-template_localization file.
cp -r /etc/amavis/en_US /etc/amavis/en_US-backup
If for some crazy reason you should want to reinstall amavisd-new from the Debian package:
apt-get update
If you added configuration items that are only valid for newer versions of amavisd-new then amavisd-new should complain and you will have to remove them before it will start. Now that you are running a newer version of amavisd-new we can use a function that was not available in older versions. Temporarily set $log_level to 1 and reload amavisd-new:
vi /etc/amavis/conf.d/50-user
and change:
$log_level = 0;
to:
$log_level = 1;
Then reload amavisd-new:
amavisd-new reload
Now send a message through (it needs to pass through SpamAssassin) and look in the log for a line similar to this: Jun 23 10:00:05 msa amavis[5359]: (05359-01) extra modules loaded: /etc/spamassassin/Botnet.pm We want to vi /etc/amavis/amavisd.conf again and take each module(s) listed and place it (them) in @additional_perl_modules (via a list of quoted words). You need to remove any commas that may separate items (and comments are not allowed within the qw() function). You could place this near the bottom of amavisd.conf in the area I have set aside for placing configuration items. The result for the log output above would be something like: @additional_perl_modules = qw( /etc/spamassassin/Botnet.pm );There is a new setting 'spam_dsn_cutoff_level_bysender_maps'. If our users send spam we want to bounce it back to them. We now need to set this in addition to 'spam_dsn_cutoff_level_maps' in order to continue to do so. There is also a new 'originating' policy bank key available. Setting this to a '1' (true) in a policy bank says to amavisd-new that the client is a local client of ours. This is similar to a client that is included in @mynetworks and you are using the MYNETS policy bank. We will make use of this in the TRUSTED policy bank (and explicitly set it in MYNETS too), so vi /etc/amavis/conf.d/50-user and insert these red items in our two policy banks:
$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
Then reload amavisd-new:
amavisd-new reload
If you were to send another test message through (at $log_level = 1; or greater), the "extra modules loaded" log entry should be suppressed. After testing it you may want to set the $log_level to 0 (or leave it alone until after we are finished with the installation. |
It is highly recommended that you are using amavisd-new 2.5.4 or newer before you upgrade SpamAssassin
using these instructions. I have only tested this with 2.5.4.
At this time, the safest way to upgrade SpamAssassin is to install it using the
backports.org sources.
First check that SA 3.2.3 lints clean. If it does not, you should fix it before continuing:
spamassassin --lint
Make a backup copy of your spamassassin config:
test -e /usr/ambackup || mkdir /usr/ambackup
While SA version 3.2.3 supplied with etch works well, there will be some who want a newer version of SpamAssassin. The 3.2.x versions allows you to compile some rules using a new 'sa-compile' command (you are not required to do so, but it may help with the speed issue a little bit). This command requires the re2c program (from regular expression to c). Once you have compiled these rules, you use them by enabling the Rule2XSBody plugin in v320.pre. If you were using SpamAssassin 3.1.x, you may or may not have installed the ImageInfo plugin, but if you have, you need to remove it because it is now included in SA 3.2.x. Also, we must remove the loopback interface from internal_networks and trusted_networks (if they are there) because it is now automatically included. We begin by adding the backports source to /etc/apt/sources.list and configure apt pinning:
vi /etc/apt/sources.list
and insert the backports source, similar to what I have done here:
deb http://mirrors.kernel.org/debian/ etch main non-free contrib
Since we now have a few different sources it's probably a good idea to increase the default apt-cache limit. Assuming you have not already done this at some point in the past:
echo 'APT::Cache-Limit "25165824";' >> /etc/apt/apt.conf
I also suggest adding gnupg keys for a few of the apt sources to apt:
gpg --keyserver subkeys.pgp.net --recv-key BBE55AB3
Since we are running etch, etch currently has a default pin-priority of 500. We need to make certain that the backports source has a lower priority:
vi /etc/apt/preferences
and insert (if you don't already have something similar):
Package: *
We must install re2c if we want to use the new sa-compile command:
apt-get update
Now we install a re2c and few other programs that will support Mail::SPF, Mail::DKIM and Image::Info:
apt-get install re2c libversion-perl liberror-perl libimage-info-perl
Version 3.2.x prefers we use Mail::SPF instead of Mail::SPF::Query. We will install this from backports. We will also install :
apt-get -t etch-backports install libmodule-build-perl
If you are running SA version 3.1.x (use "apt-cache policy spamassassin" to see if you are) we need to remove the manually installed 3rd party ImageInfo plugin if installed:
test -e /usr/share/perl5/Mail/SpamAssassin/Plugin/ImageInfo.pm && rm /usr/share/perl5/Mail/SpamAssassin/Plugin/ImageInfo.pm
If you are running SA version 3.1.x, that grep statement should show that the ImageInfo plugin is not set to load. If you are running 3.2.x, it will show that it is set to load (which is fine). Now we need to make another change:
vi /etc/spamassassin/local.cf
and remove the loopback interface from internal_networks and trusted_networks:
# explicitly set our internal_networks (might be the same or similar to mynetworks)
Now see what version you are going to get:
apt-cache policy spamassassin
Today my machine shows: spamassassin:
Installed: 3.2.3-0.volatile1
Candidate: 3.2.3-0.volatile1
Version table:
3.2.5-1 0
200 http://www.backports.org etch-backports/main Packages
*** 3.2.3-0.volatile1 0
500 http://volatile.debian.net etch/volatile/main Packages
100 /var/lib/dpkg/status
3.1.7-2 0
500 http://mirrors.kernel.org etch/main PackagesLooks fine,
so I will remove 3.2.3 and install 3.2.5. You may be prompted
whether to overwrite v310.pre or not. We can keep our modified file,
so you can answer with the default 'N':
amavisd-new stop
I assume it will only remove spamassassin. If this is true, we can proceed to remove it and then install the new version (not that your configuration files in /etc/spamassassin should not be removed as a result of removing spamassassin):
apt-get remove spamassassin
Reimport the openprotect GPG key:
cd /etc/spamassassin
If there were show stopping problems with the --lint, you need to fix them. If you reinstall version 3.1.7, please delete v320.pre when you do. If things get real bad you might have to '--purge remove' and then reinstall. You might have to reimport the OpenProtect gpg key. You need soon need to get your mail flowing again:
amavisd-new start
You should vi /usr/sbin/sa-update.sh and see if you entered in any custom commands that may need to be updated to the new directory /var/lib/spamassassin/3.002005. Then we should update the openprotect rules by manually running the script:
sa-update.sh
Continue to tail the mail.log in the current PuTTY session. Open a second PuTTY session and work from there. If you are up and running on 3.2.x, you should consider running sa-compile:
sa-compile
If that looks good, you can enable the compiled body rules (this command is on one line):
sed -i 's/# loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody/loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody/' /etc/spamassassin/v320.pre
Make sure that loadplugin setting is no longer commented out and make sure lint is clean.
grep Rule2XSBody /etc/spamassassin/v320.pre
Now grab a script that will run sa-compile once a week:
cd /etc/cron.weekly
Run the script. If it is working correctly, it should take a while; there should be no output and amavisd-new should get reloaded. The script does require that sa-compile has ran at least once before. If you are still tailing the log in the other session, you should eventually see amavisd-new reload:
./sa-compile-weekly
You can close the second PuTTY session if you like. You should be good to go. Send a test message through to make sure you can receive mail. Don't be surprised if messages take longer to get scanned and SA puts more load on you CPU(s). If you upgraded amavisd-new, you set a $log_level at 1 or above and you enabled the Rule2XSBody plugin; if you were to grep modules /var/log/mail.log you will notice:
Dec 6 10:42:46 msa amavis[7290]: extra modules loaded after daemonizing:
Don't worry about these (do not add Mail/SpamAssassin/CompiledRegexps/body_500.pm to @additional_perl_modules), but other modules may be found (remember, you must ge at $log_level = 1; to see these), for example:
Dec 6 10:43:52 msa amavis[7297]: (07297-01) extra modules loaded:
So, vi /etc/amavis/amavisd.conf and update @additional_perl_modules as needed (this is only an example):
@additional_perl_modules = qw(
You can also vi /etc/amavis/amavisd.conf and optionally include these settings. However, this is SA version specific and will need to be edited if you upgrade SpamAssassin:
my($sa_instdir) = '/var/lib/spamassassin/compiled/3.002005';
Then, of course:
amavisd-new reload
and send a test message through. Since we should have all the pieces in place, we can optionally enable the DKIM plugin:
sed -i 's/#loadplugin Mail::SpamAssassin::Plugin::DKIM/loadplugin Mail::SpamAssassin::Plugin::DKIM/' /etc/spamassassin/v312.pre
Then:
spamassassin --lint
If all is well:
amavisd-new reload
and watch for errors once again. Remember to set $log_level back to 0. |
FuzzyOCR attempts to detect image spam. It is of somewhat limited value at this time.
It can add to scan times. It works best
if you spend time understanding how to use debug output to help with tweaking scan sets
and key words. Spammers are especially wise to FuzzyOCR and intentionally obscure image spam to
work around it. FuzzyOCR can at times be CPU intensive.
apt-get install lsb-base libice6 libx11-6 xlibs-data libsm6 x11-common gifsicle ocrad
If you are using SpamAssassin 3.2.x, perform these next green colored statements:
cd /usr/local/src
If you are using SpamAssassin 3.1.7, perform these next purple colored statements:
cd /usr/local/src
The rest can be used with either version:
wget http://verchick.com/mecham/public_html/spam/FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch
Make sure this comes up clean:
spamassassin --lint
Run a test (you should get FUZZY_OCR rules hitting): If running SA 3.2.x:
cd /usr/local/src/FuzzyOCR-svn-131/samples
If running SA 3.1.7:
cd /usr/local/src/FuzzyOCR-3.5.1/samples
Other tests are also available:
su amavis -c 'spamassassin -tD < ocr-gif.eml'
su amavis -c 'spamassassin -tD < ocr-jpg.eml'
su amavis -c 'spamassassin -tD < ocr-multi.eml'
su amavis -c 'spamassassin -tD < ocr-obfuscated.eml'
su amavis -c 'spamassassin -tD < ocr-png.eml'
su amavis -c 'spamassassin -tD < ocr-wrongext.eml'
If you want to see details, increase focr_verbose in /etc/spamassassin/FuzzyOcr.cf and look in /var/log/FuzzyOcr.log (but set it back to 0 when finished testing). If everything looks OK so far:
amavisd-new stop
Browse to http://verchick.com/mecham/public_html/spam/host.gif and save it to your computer. Send another test message into the mailserver with this gif attached and watch the debug output for signs of errors. In my output I found where FuzzyOcr added 5.188 points: [23165] info: FuzzyOcr: Message is spam, score = 5.188 Also read the headers of the message that was received. If everything looks Ok, stop debugging with [Ctrl]+c, then:
amavisd-new start
vi /etc/amavis/conf.d/50-user and set $log_level = 1; and then amavisd-new reload . tail -f /var/log/mail.log and send a test message through and once again look for: Jul 26 16:07:08 msa amavis[24971]: (24971-01) extra modules loaded: /etc/spamassassin/FuzzyOcr.pm, /usr/lib/perl/5.8/auto/Storable/autosplit.ix, /usr/share/perl5/auto/Log/Agent/Priorities/autosplit.ix, /usr/share/perl5/auto/Log/Agent/autosplit.ix, FuzzyOcr/Config.pm, FuzzyOcr/Deanimate.pm, FuzzyOcr/Hashing.pm, FuzzyOcr/Logging.pm, FuzzyOcr/Misc.pm, FuzzyOcr/Preprocessor.pm, FuzzyOcr/Scanset.pm, FuzzyOcr/Scoring.pm, Log/Agent.pm, Log/Agent/Formatting.pm, Log/Agent/Message.pm, Log/Agent/Priorities.pm, MLDBM.pm, MLDBM/Sync.pm, MLDBM/Sync/SDBM_File.pm, SDBM_File.pm, Storable.pm, String/Approx.pm, Tie/Cache.pmEach of these modules (less body_500.pm and body_0.pm) should be added to @extra_perl_modules (without the commas). The net result may look like: @additional_perl_modules = qw( /etc/spamassassin/Botnet.pm /etc/spamassassin/FuzzyOcr.pm /usr/lib/perl/5.8/auto/Storable/autosplit.ix /usr/share/perl5/auto/Log/Agent/Priorities/autosplit.ix /usr/share/perl5/auto/Log/Agent/autosplit.ix FuzzyOcr/Config.pm FuzzyOcr/Deanimate.pm FuzzyOcr/Hashing.pm FuzzyOcr/Logging.pm FuzzyOcr/Misc.pm FuzzyOcr/Preprocessor.pm FuzzyOcr/Scanset.pm FuzzyOcr/Scoring.pm Log/Agent.pm Log/Agent/Formatting.pm Log/Agent/Message.pm Log/Agent/Priorities.pm MLDBM.pm MLDBM/Sync.pm MLDBM/Sync/SDBM_File.pm SDBM_File.pm Storable.pm String/Approx.pm Tie/Cache.pm );So, vi /etc/amavis/amavisd.conf and do your stuff, then of course:
amavisd-new reload
Send one more test message through - if all is well you can set the $log_level back to 0 and reload amavisd-new again. |
|
The concept with AIDE is to take a snapshot of some of the characteristics of critical files on
our system, and then store that information in a database on read-only media; a write protected
floppy for example.
Then run a program that compares the current state of those files on our system with the database
and report any differences. We should be able to tell if a hacker has compromised our system if
the characteristics of the files on our system differ from the files in the database. Place a good writable floppy in the floppy drive, then:
mkdir /floppy
If the format fails, try again. If it fails again, try a different floppy. If it continues to fail, try a different brand of floppy or a different floppy drive. If the format goes well:
mke2fs /dev/fd0u1722
Now you can mount the floppy with floppy start and unmount the floppy with floppy stop (as long as the floppy is formatted as /dev/fd0u1722). The floppy will be left in the machine at all times and will automatically get mounted each time the system starts. You should reboot the computer and see if it starts back up. This is not a bootable floppy but some systems may hang if the BIOS is set to boot from the floppy drive. You may have to change the boot order in the BIOS so your server boots from the (first) hard drive before it boots from the floppy. Now install aide (choose the default responses to questions):
cd /usr/local/src
Daily reports are mailed to root by default. [Ok] Initialize aide database? [No] Before AIDE can be used, you will have to initialize a database. [Ok] We are going to obscure aide somewhat, use my configuration file and report and then initialize the database (which will take some time):
floppy start
Wait for the initial database to be created (/root/aide/aide.db.new) then copy it to the floppy:
cp /root/aide/aide.db.new /floppy/aide.db
Once the file copies over (wait about 30 seconds), run the report for the first time (this will take a while). Once the report completes, check the mailbox for the user that gets root's mail:
/floppy/report
Each time you run the report, it reads the database /floppy/aide.db but it also creates a new aide.db.new database. So why do we care about the aide.db.new? It's a matter of convenience. If the report is getting kind of long, and there appears to be nothing in it you would not expect; the aide.db.new database reflects the current state of the system and could be used to replace aide.db. You replace your aide.db on the floppy with aide.db.new, just as you did when you first initialized the system. If you run the report again, it should be clean, unless files have changed since it was created. You need to understand the risks of relying on the quality of the aide.db.new database if you use the copy that was created during the cron job. The cron job could have been hacked. A hacker could manipulate the file. Only rely on the aide.db.new database if you run /floppy/report from the command line and
you inspect the report just prior to copying it over to the floppy. If you
left the write protection off the floppy drive for any length of time, it's possible you
can no longer rely on the database. Be paranoid.
So here's the gist of maintenance:
cd
Then run it to see how it works:
./go
You should now have the program and data on the write protected floppy. Set up a cron job to run the report:
crontab -e
and insert something like this:
25 7 * * * /floppy/report
It is typical that the file '/dev/.udev/uevent_seqnum' will be on every report. |
If you feel the need to add a disclaimer to outgoing mail you can use altermime to accomplish
this. I am going to assume you have installed amavisd-new 2.5.2 or newer. Version 2.4.2 does not
have the necessary hook to call altermime. If you are still using 2.4.2 then you can use some of
my preliminary notes
if you like, but I abandoned that project when altermime could be called from amavisd-new. We
are going to use the latest
devel release.
cd /etc
Now vi /etc/amavis/amavisd.conf and add this:
# altermime disclaimers
Now we want to enable disclaimers for outgoing mail. This is done through the MYNETS and/or TRUSTED policy banks. You need to insure that the IP address of clients sending mail from the internal network are included in @mynetworks - or your clients are authenticated and therefore using the TRUSTED policy bank. Now vi /etc/amavis/conf.d/50-user and add the red items as shown:
$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
Then of course:
amavisd-new reload
Log into SquirrelMail and send a message. If all is well your disclaimer will be added. Of course you have to customize it:
vi /etc/disclaimer.txt
For the moment there is one limitation: there can only be one manger in effect at a time, it is not currently possible to both defang and to append a disclaimer, e.g. for internal-to-internal mail inserting a disclaimer takes precedence. Also, if you need different disclaimers for different domains, search for @disclaimer_options_bysender_maps in http://www.ijs.si/software/amavisd/release-notes.txt. |
|
If you wish to give clients within your network the ability to relay mail to domains
other than yours then you should place your network (or individual IP addresses) in
mynetworks in main.cf. You can also
exclude certain hosts
in your network by preceding the IP address with an exclamation point. Excluded addresses
need to be listed before included addresses. You should also place the network in
@mynetworks in 50-user in order for the clients to take advantage of the MYNETS policy bank.
You should build another server dedicated to holding a nightly backup of this server. The minimal Debian installation should be fine. It should have a very large hard drive, but I imagine it need not be a powerful machine. I used http://www.howtoforge.com/linux_rdiff_backup and can say it works well. Make sure iptables allows SSH from the backup server. I like the way it explains how to connect via SSH without using a password. Also how to limit what the connected party can run. I did have a problem with the "ssh-copy-id" command - read the comments at the bottom of the first page for a solution. I suggest reading: http://verchick.com/mecham/public_html/spam/amavisd-settings.html You should read this: http://verchick.com/mecham/public_html/spam/bypassing.html I suggest reading: http://jaqque.sbih.org/kplug/apt-pinning.html This can help determine how long SpamAssassin spends on certain items: http://marc.info/?l=amavis-user&m=117874388132138 I archive copies of messages. Here are some hints on how I do it: http://www.freespamfilter.org/forum/viewtopic.php?t=526 You can redefine %banned_rules (make sure you include 'DEFAULT') in order to set per-recipient banned rules sets (in SQL). http://marc.info/?l=amavis-user&m=111995411713191 I think you should consider building a secondary MX server that can spool mail if this machine goes down. It also reduces load on this server because spammers often target the secondary MX. You will need to populate relay_recipients on that machine. FYI you can gather all the local recipients into a file with commands such as these:
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(username, ' OK') from mailbox" >file1
Here are links to most of the stuff you get from me (in the order downloaded): my-medium.cnf.patch.txt virtual.index.html database_mysql.patch.txt postfixadmin.autocreate.patch.txt postfixadmin.lowercase.patch.txt postfixadmin.multipatch.txt admin-edit-mailbox.patch.txt edit-mailbox.patch.txt maildirmake.sh.txt quotachange.sh.txt mysql_virtual_alias_maps.cf mysql_virtual_domains_maps.cf mysql_virtual_mailbox_maps.cf maildroprc.txt maildrop.logrotate.txt authmysqlrc.txt maildircheck.txt rmmailboxspam.txt sa-update1.sh.txt amavisd-new-trim-whitespace.patch.txt amavisd-new-trim-whitespace.patch2.txt amavis-sqmail.sql.txt domain.patch.txt trim-amavis.sql.txt trim-amavis.txt rmvirusquar.txt gv-bayes-awl.sql.txt local.cf-bayes-awl.txt sample-spam.txt trim-amavis-msgs.txt trim-awl.sql.txt trim-awl.txt trim-sql-awl-weekly.txt amavisnewsql.patch1.txt pflogsumm.sh gv-vac.txt postfixadmin_forward.patch.txt mzcpatch.txt mailzu.logrotate.txt mysql_relay_domains_maps.cf mysql_virtual_domains_maps.cf.v2 sa-update.sh.txt check_client_fqdn policyd-weight.conf UpdateSaneSecurity.sh.txt p0f-analyzer.txt p0f p0f.cf amavis-init-20030616 amavisd.conf-etch2.txt amavisd.conf-modified amavisd.conf-sample charset template-dsn.txt template-spam-admin.txt template-spam-sender.txt template-virus-admin.txt template-virus-recipient.txt template-virus-sender.txt ImageInfo.pm.txt imageinfo.cf.txt sa-compile.sh.txt FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch2 FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.p3 capFuzzy.txt gary.3.5.0-rc1.old.netpbm.patch1 gary.3.5.0-rc1.old.netpbm.patch2 gary.3.5.0-rc1.old.netpbm.patch3 FuzzyRotate.txt startflop.txt aide.conf.patch.txt report.patch.txt go.txt Here are general links: Postfix MySQL Apache http 1.3 Courier IMAP docs phpMyAdmin amavisd-new SpamAssassin PostfixAdmin SquirrelMail docs Courier Maildrop Vupil's Razor Pyzor pflogsumm Bind MailZu MailGraph mysql-zrm OpenProtect SA updates SARE SARE Plugins DCC PostGrey policyd-weight sanesecurity MSRBL BotNet p0f FuzzyOCR Apache and Courier are not tuned to your particular level of traffic. I leave it up to you to discover how many processes (and other tweaks) you might need. Something to consider. When a user creates an alias that forwards to another address, depending on spam settings, there is a potential for that address to be rewritten to user+spam@example.com and the final destination server may not accept mail in that format. In this case another alias would need to be created to change the address back to user@example.com You need to subscribe to: http://lists.debian.org/debian-security-announce/ http://lists.clamav.net/mailman/listinfo/clamav-announce You may like this: http://www.mikecappella.com/logwatch/ You may also want to install webalizer:
apt-get install webalizer
Then vi /etc/webalizer/webalizer.conf and change the LogFile so it points to /var/log/apache-ssl/access.log. Run webalizer and then browse to https://your_home_page/webalizer Some people (like me) do not like SpamAssassin AWL (auto whitelisting - which is actually a score averaging system). To turn it off vi /etc/spamassassin/local.cf and insert use_auto_whitelist 0 (don't disable the plugin in v310.pre) then:
spamassassin --lint
Every six months (or so), it would be a good idea to optimize your MySQL tables. I think it's a good idea (possibly even necessary) to stop MySQL activity during this period. This means you should shut down postfix and amavisd-new and possibly even disconnect the ethernet cable. There should be no MySQL maintenance scripts running (like /etc/cron.daily/ztrim-amavis which should start running at 06:25). Log in to mysql (as root) and optimize the tables:
USE amavis;
If you change your NIC card, udev will call your new NIC something other than eth0. In this case you need to vi /etc/udev/rules.d/z25_persistent-net.rules and correct the situation. I reboot afterwards. You may want to edit some of the banned rules in the $banned_filename_re section of /etc/amavis/conf.d/20-debian_defaults. To start with, I usually comment out the "# banned extension - basic" line and uncomment the four lines below it that comprise the "# banned ext - long" section. In /etc/postfix/master.cf I add " -o smtpd_client_connection_count_limit=21" to the smtpd daemon to limit the number of simultaneous connections a single client can make (within one minute's time). The default is 50 (half of the default maxproc). This lowered limit is not appropriate if you get mail from another relay server - you would not want to lower the limit for a server you trust, but otherwise it may be a good idea. It may help with curtailing dictionary attacks: smtp inet n - - - - smtpd
-o smtpd_client_connection_count_limit=21
Ok, start customizing. BTW, It would not surprise me if there are vulnerabilities in some of the
pages of the various web interfaces.
|