Fighting spam

From Kolab Wiki

Jump to: navigation, search


Spam can be cut down significantly with the use of spamassassin, bundled with kolab openpkg. It is already working, but to make it work better, it needs training with the sa-learn command.

There are scripts which may be useful for this in the Goodies section

Contents

An Example Antispam Scenario

This scenario uses single shared folders for spam/ham - if all clients use the same MUA, then you could adjust according to the location of the default spam folder (ie for thunderbird, you could use /kolab/var/imapd/spool/domain/*/user/*/Junk/ as the common source). I prefer to use a single folder, because it can me "moderated" - it is frustrating when stupid users mark mail from people they dont like, or managers etc as spam, and this can mess things up.

Step 1 - Create shared folders

Log in to the admin web gui as a maintainer and create 2 shared folders:

  • spam: anyone:all
  • ham : anyone:all

You could also change the permissions so that anyone can read/post, but only designated people can delete.

Step 2 - Set a spam trap

I found it very useful to set a spamtrap address. Create a new account, then go to google and search for "free porn email" or "free email offers" and fill in this email address into as many as you can. Wait for the spam to start, it should take a few hours up to a few days. Almost ALL email received on this address can be put into the spam folder.

Step 3 - getting spam into the spam folder

It is important that mail is moved or redirected to the spam folder, so that the headers remain intact. If mail is just forwarded to the folder, it will break things, because the headers will show your internal users.

  • Kmail - drag message > move here
  • Thunderbird - drag message
  • Outlook - drag message

Step 4 - teaching spamassassin

Next step is to teach spamassassin that mail in this folder is spam. I prefer to do it with a cron job that also updates my clamav bases . All that is really needed is the following command:

 #Remember that you have to run sa-learn command with kolab-r user and with home directory /kolab/var/amavisd
 su kolab-r
 export HOME=/kolab/var/amavisd
 # or use sa-learn with --dbpath argument
 
 /kolab/bin/sa-learn --spam /kolab/var/imapd/spool/domain/*/shared^spam/[1-9]*

This will look for all messages (in kolab the "raw" messages always start with a number) in the spam folder and train spamassassin to look for these patterns and mark them as spam. After a while, you will notice good spam detection improvements. You may want to try Spamandclam.sh to update your antivirus bases, and do spam training at the same time. This can be put in as a cron job for each evening.

bayes requirements

You need to learn at least 200 spam and 200 ham mails before bayesfilters will be activated. Take a look at

/kolab/bin/sa-learn -D --dump magic 

for information of what spamassassin has learned.

Step 5 - teaching spamassassin about ham

Some users tend to think that spam is any mail they do not want, and may start to put internal mail into the spam folder (a favourite is virus warnings). You can delete them, but if they make it through a spam training session, or are marked as spam when they are not spam, use the ham folder. This should command should be used to teach spamassassin that the mail in this folder is safe and not spam:

 /kolab/bin/sa-learn --ham /kolab/var/imapd/spool/domain/*/shared^ham/[1-9]*

Step 6 - automating

Automating this process saves a lot of time and effort, and since kolab2 now has dcron installed, it is quite simple. First we need the correct environment variables set. Get Opa and put it into root's .bashrc - this is very useful for other things as well! Once you have installed Opa, and run

 opa /kolab

Just put it into root's crontab (You could use your own script here as well):

 #crontab -e
 # Add this line:
 30      00      *       *       *       /data2/Spamandclam.sh

(You can also just edit the file /kolab/var/dcron/crontabs/root)

Spamandclam.sh Will update both clamav bases and spam learning, and send a mail, but you might want your own script here. If you have a useful script you use, please put it on the wiki!

Be carefull the dbpath is /kolab/var/amavisd/.spamassassin , so you add to salearn the option --dbpath /kolab/var/amavisd/.spamassassin in Spamandclam.sh


Maybe you also want to autodelete the mails in your shared folders from time to time. An easy way to proceed this task, is to use the ipurge tool, which is delivered with the cyrus imapd. ipurge intended use is to delete messages, which met specific characteristics like age, size and so on. Because of the invert option, it is possible to simply use ipurge as an "mailbox-emptier". For example, to empty the shared^spam folder, the following code is needed:

 /kolab/bin/ipurge -d1 shared.spam@kolab.domain
 /kolab/bin/ipurge -i -d1 shared.spam@kolab.domain

Make sure, that you don't run ipurge without giving an mailbox as argument. Otherwise it will be used against all mailboxes! To automate the deletion process, you'll have to put this code into a local crontab. You should leave some time between the start of the sa-lern-process and the commands to empty the corresponding mailbox.

Step 7 - checking progress

You can also check how many messages have been learned as spam and ham using the sa-learn command:

sa-learn --dump magic
...
0.000          0      60912          0  non-token data: nspam
0.000          0      32425          0  non-token data: nham
...

The lines abouve indicate that this spam system learned about 60.000 spam messages and 30.000 ham messages. As a rough estimate: Your Bayes DB should have learned from about 1.000 messages of each type to be really useful for spam recognition.

Using SBL and XBL lists from Spamhaus and ORDB

Using these lists can dramatically reduce the amount of spam you receive.

More Info:

Please, pay attention on Jim's note A Note About DNS BlackLists in the document above.


Be careful if you do business with Eastern Europe and your clients use servers there, they may be sharing with spammers unknowingly and thus have their (shared) MX on the SBL list.

Using it in kolab is easy, just edit /kolab/etc/kolab/templates/main.cf.template and add one line to it, look for:

 ## Kolab Policy Server
 smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,
       reject_unauth_destination, reject_unlisted_recipient, check_policy_service unix:private/kolabpolicy

and add the line to make it look like this:

 ## Kolab Policy Server
 smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,
       reject_unauth_destination, reject_unlisted_recipient, check_policy_service unix:private/kolabpolicy
       reject_rbl_client sbl-xbl.spamhaus.org

You may want to add the open relay database, but be careful with this as false positives may pop up. (Add on to the end of smtpd_recipient_restrictions)

       reject_rbl_client relays.ordb.org

After making the change, run /kolab/sbin/kolabconf

Update Jan 20, 2007:the openrelays database has been closed.

Spam Quarantine Shared Folder

Once spamassassin is trained, you can lower your discard limits for mail, but it is wise to make a quarantine "just in case". By default mail will be quarantined in /kolab/var/amavisd or /kolab/var/amavisd/virusmails - it can be easier to make it quarantine to a shared email folder. To do this, just make a new shared folder, and then change your amavisd.conf template file at /kolab/etc/kolab/templates/amavisd.conf.template. Edit the section for spam quarantine (around line 610)

 $spam_quarantine_to = 'kolab+shared.spambox@example.com';

This line will send all spam email over the kill level to a shared folder called spambox on the example.com domain. Make sure you have set permissions for "All" to at least "Post" or you will get errors. Now adjust your $sa_kill_level_deflt to a sane kill level, for spam to not be sent to users, and be quarantined in your shared folder.

A similar approach can be used for virus emails as well, check the amavisd.conf file for details (the virus and spam quarantine sections are close together)

Using non local test

An important side node on non local spamassassin tests (which is not yet relevant for the current Kolab server):

The default amavisd-new configuration in Kolab2 does not activate spam tests that require internet access. This can be changed by setting

$sa_local_tests_only = 0;

in the amavisd.conf.template file in your Kolab template directory. This is necessary for tools such as Razor, Pyzor or DCC. In addition these non local tests will usually check for dynamic IP addresses. This can lead to problems and you might see these spam tags in the headers of your mail headers:

RCVD_IN_DSBL
RCVD_IN_NJABL_DUL
RCVD_IN_NJABL_PROXY
RCVD_IN_SORBS_DUL

This basically means that your users did send the message using a dynamic uplink (like DSL). This is actually perfectly ok since your Kolab2 server will authenticate the users so the mail is certainly not a spam message (also see this link). But postfix =< 2.3 fails to add the necessary headers to add the information about successful authentication into the mail headers. This can be achieved using postfix >= 2.3 by adding

# Add a sasl authenticated header so that spamassassin knows the user was checked
smtpd_sasl_authenticated_header = yes

to your main.cf.template.

Other resources

Personal tools