Experiences - Antonio Larrosa
From Kolab wiki
I'll explain here how I migrated from using fetchmail, procmail and kontact in a workstation at home to use kolab.
My previous system configuration
I've used procmail since day 0, I've always used it to filter my mails when I used Netscape many years ago, and so continued using it when I updated to kmail and then to kontact. I used fetchmail to get the mail from my pop account at my ISP, which injected the mails into the local mail server (SuSE's postfix standard package) which then delivered the mail through a $HOME/.forward file to procmail, which would process it, filter through spamassassin and then deliver it to different mbox files depending on many factors. These mbox files are stored on a directory outside of KMail's main Mail directory.
In KMail/Kontact I just did a different Receive account for each of these mboxes which would put the mails into a different folder (a one-to-one mapping).
As you may guess, I wasn't totally happy with it (since it involved too much work to create one new folder and filter), but it was neccesary since procmail delivers mails to mbox files while I prefer to store mails into maildir folders with KMail/Kontact.
My requirements for a new system
Point one, I must be sure that I don't lose any mail in the transition (that is, from the previous mail system or mails received while migrating) Point two, I must be sure that I don't lose any mail (not even a single spam mail) Point three, it must be better than the current system Point four, it must make easier to read/answer mails from my laptop/work/when travelling/etc.
Note that I'm not an expert concerning mail handling, mail servers or whatever, in fact, I didn't know anything at all about postfix/imapd/kolab/etc before starting this installation so I cannot assure you that those requirements are met by the steps I took, they just worked for me, and I think they were met. If by following these steps you lose a mail (or all of them), your contacts, your appointments, your task list, your shop list or your dog, then it's all your responsability and I cannot be held liable.
Installing Kolab
My first concern was that installing kolab would overwrite some system configuration of current postfix installation and everything would stop working. Nothing farther from reality. Kolab installs completely on its own directory at /kolab so all the system applications and configuration files should be safe.
Once I learned that, I continued the installation by following the instructions on Kolab2 Installation - Source
Note that on the bootstrap step, I mainly answered the default answers (except for the passwords :) ), and I (of course) wanted a certificate, and then signed it.
Once kolab was installed, I went with konqueror to https://localhost/admin/ , to the Users tab, and created a user for me. As primary email address I used antonio@mydomain.org .
There's something you definitely want to do here if you plan to use fetchmail like I do, and it's filling in the email aliases box. Postfix can only relate your inbox to the email address you're creating at your domain, but you also want it to accept and deliver to you the mails received at your isp mail server and which have a different destination! If you don't do so, you'll end up not being able to receive any mails!
Thanks to Mario Teijeiro, who helped me to search for the cause of the mails not being delivered and proposed the (working) solution of setting those emails at postfix's virtual config file and making postfix use it by modifying the value of virtual_maps by adding
virtual_maps=ldap:ldapvirtual,hash:/kolab/etc/postfix/virtual
instead of
virtual_maps = ldap:ldapvirtual
Anyway, making that change is not neccesary. Later Matt Douhan told me on IRC that "email aliases" on the admin page for the user is also another name for virtual maps, but done in kolab's way (that is, changing the value of "email aliases" on the page, modifies the contents of the LDAP directory), so I've used that solution finally, and just left
virtual_maps = ldap:ldapvirtual
in /kolab/etc/postfix/main.cf . Btw, maybe you also want to read Mario's HOWTO which explains how to configure Postfix+Cyrus Imap+Sasl+tls (http://cernicalo.escomposlinux.org/~emeteo/imap/ imap+postfix/) (in spanish).
Note that if my email is antonio@mydomain.mytld then I also had to add here antonio@localhost.mydomain.mytld since using fetchmail (and/or procmail, as we'll see later) somehow requires this.
When I went to the "Services" tab, I was told I had to choose a user that would receive mails for postmaster, etc. so I choosed the just created one.
Kolab is now mostly installed and configured
Configuring Kontact to use kolab
The first step here is to close kontact and kmail if any of them is running. Now I run kolabwizard (which comes together with kontact, so you should have it installed on your system). I introduced the full host name with domain of my server name, my email address (that is, username@domain.tld , without hostname), name, password, and selected "Save password" and "Kolab 2".
Now I opened kontact, went to the "Settings" menu, "Configure KMail...", opened the "Misc" section and the "Groupware" tab inside it. IMAP resource functionality should be enabled. The Format used for groupware folders should be "Kolab (XML)" and you can select the account where you'll store the resource folders (but this is usually good by default, since most people only have one option :) ).
I let the "Hide groupware folders" unset while I was making the tests for the system and later set it, you can choose to do what you prefer.
Now I checked mail on that account, the first time you do this, Kontact creates the neccesary groupware folders into your kolab server.
I tested then that I could in fact store a mail on kolab's imap server. I copied a mail from a local folder to my Inbox folder in kolab's account, and then checked new mail. At that point, Kontact synchronizes both folders (local and remote) thus copying the mail there. I then checked to be sure that the mail was really stored in the server by looking at /kolab/var/imapd/spool/domain/mydomain.es/user/antonio/ where the mail was stored in a file called 1. (or 2. , 3. , etc.). Note that you should have in that same directory folders for your resources (Calendar, Contacts, Journal, Tasks, Notes), if you don't have them, then go back and confirm that you've followed all the steps.
Btw, you shouldn't manipulate the imap server contents manually, note that it's not just a maildir folder.
Migrating the resources to Kolab
Thanks to Martin Konold who suggested this idea in order to migrate easily.
The steps taken in order to copy the contact list are easy. First I exported all my contacts to vCard 2.1 files (one contact in each separate file), and then I imported them all. When importing them, Kontact (Kaddressbook) asks if they should be added to the local resource or to the kolab server. I selected Kolab server et voilá. They're there.
Well, just a few notes in this step: First, I tried several different file formats when exporting (vCard 3.0, etc.) and some didn't import correctly the unicode characters and some others didn't import correctly the photo I have for each contact. But basically, vCard 2.1 worked quite well.
For the calendar and journals, the process is the same: Export, Import into kolab.
Fetching mails and inserting them into Kolab
To download mails from my ISP's POP server and insert them into Kolab, the recommended way is to use fetchmail. Normally, fetchmail with a configuration similar to the next one, downloads the emails and insert them into the mail server on the local machine:
poll pop.myisp.es protocol pop3 user john password secretword is johnny here;
That should work as expected, so you can now skip this section if that's good for you, but since I'm a bit paranoid I wanted to have a backup of every fetched mail.
For that, I added an intermediate step. This involves fetchmail handing mails to procmail, which I use to store a backup copy in a mbox file and then forward each mail to my local user. In order to do so, I added a mda setting to fetchmailrc :
poll pop.myisp.es protocol pop3 user johnny password secretword is antonio here mda "/usr/bin/procmail -f-" ;
This tells fetchmail that instead of delivering fetched mails to the local mail server, it should forward it to procmail. So then I configured procmail by writing this into ~/.procmailrc :
PATH=/bin:/usr/bin LOGFILE=$HOME/procmail.log SENDMAIL=/kolab/sbin/sendmail :0 c /home/antonio/backup-kolab :0 !antonio@mydomain.mytld
This makes a copy of every mail processed by procmail into an mbox file at /home/antonio/backup-kolab and then forwards the mail to the address antonio@mydomain.mytld, which curiously, happens to be a user of kolab, and since it uses kolab's sendmail (which is in fact a wrapper over postfix ), the mail is delivered to my user.
Now I tried to run this as my own user, but kolab's sendmail tries to read some configuration files for which my user has no read permissions, so I just copied ~/.fetchmail and ~/.procmail to the root's HOME directory, and I run fetchmail as root. This time it worked as expected, and mails where delivered to my inbox folder into kolab.
Something to note is that by default SuSE runs its own postfix server, and when trying to run kolab's postfix, it will fail (since they both use the same ports). I just told SuSE not to run postfix at system startup by running:
insserv -r postfix
Since kolab is already started automatically, kolab's postfix will be started instead.
Strange things
One of the times I downloaded mails I noticed that fetchmail said it downloaded n mails but when I went to kontact to read them, the number of new mails was just n-1 or n-2 . This isn't nice at all, so I rejoiced about having set the backup mechanism and started looking at the logs to see what happened.
The first thing I noticed is that the lost mail was a spam mail (at least it was nothing to worry in this case), then I saw this on the postfix log files:
Apr 16 16:10:53 tazend <info> postfix/smtpd[17815]: D520A6574F:client=localhost[127.0.0.1] Apr 16 16:10:53 tazend <info> postfix/cleanup[18333]: D520A6574F:message-id=<20050416044013.XFTF1107.smtp01.retemail.es@sbmosgmb> Apr 16 16:10:54 tazend <info> postfix/qmgr[6663]: D520A6574F: from=<root@thisuniverse.org>, size=14364, nrcpt=1 (queue active) Apr 16 16:10:54 tazend <info> postfix/lmtp[17839]: D520A6574F: to=<antonio@thisuniverse.org>, relay=/kolab/var/kolab/lmtp[/kolab/var/kolab/lmtp], delay=1, status=bounced (host /kolab/var/kolab/lmtp[/kolab/var/kolab/lmtp] said: 554 5.6.0 Message contains NUL characters (in reply to end of DATA command)) Apr 16 16:10:54 tazend <info> postfix/qmgr[6663]: D520A6574F: removed
I checked in the backup file and the logged message is right, that mail contained a NUL character. So the cyrus imap server that kolab uses is simply rejecting to store mails containing NUL characters. Up to now, all the mails I've received containing a NUL character are spam mails. Also, I found this searching on several mailing lists (http://www.exim.org/pipermail/exim-users/Week-of-Mon-20031201/063358.html):
> Cyrus IMAP is simply protecting its mail readers from the dangers of > allowing things like embedded NUL characters. The fact that the RFCs > have the same rules is no mere coincidence, of course. They are also strict about the RFC requirement that all octets in the header block be in the range 1..127. > It's all very well and good for the MTA to be able to transport > arbitrary binary content, but it's very dangerous to assume that MUAs > will deal sanely with such content. Or mail filters. Or archive indexing programs. Or even downstream MTAs...
Also the RFC (2821) says:
The mail data may contain any of the 128 ASCII character codes, although experience has indicated that use of control characters other than SP, HT, CR, and LF may cause problems and SHOULD be avoided when possible.
And still, it's right that a NUL character may cause security problems in
mail filters or any application that manages mails, so I suppose ignoring those mails is not a problem. Anyway, nobody told me about this, and so I choosed to write this document to explain it for other users.
Filtering mails into Folders
Ok, so everything was working now, but all mails still go to the same INBOX folder. And (of course) I wanted to have them grouped by mailing lists, so I searched for a way to configure that.
I could configure kmail filters and just move new mails to their correct folders in the mail client, but that would mean that I would have to configure each mail client I use to do that, also, it would involve moving mails between server<->client too often, so I continued looking and found the standard way to move mails in an imap server is using the sieve filtering language.
Sieve is language similar to procmailrc config file, but specially written to allow fast and secure filtering of mails either in the mail client or in the mail server.
All you have to do is write (with kate, for example, which supports syntax highlighting for sieve scripts) a file where you define how to filter mails. The format is quite simple. Basically what I did was to insert first a Require line where I specified the features I'd use and then specified a long if/then/elsif/then/elsif/then/.../else chain. Here is the beginning of my sieve script:
require ["fileinto", "reject", "vacation", "regex", "relational", "comparator-i;ascii-numeric"];
fileinto "INBOX/backup/received";
if not exists ["From","Date"] {
fileinto "INBOX/zz/spam";
} elsif header :is ["X-Spam-Flag", "X-Spam-Status"] ["YES","Yes"] {
fileinto "INBOX/zz/spam";
} elsif header :contains "List-Id" "<kde-core-devel.kde.org>" {
fileinto "INBOX/desarrollo/kde-core-devel";
} elsif header :contains "List-Id" "<kde-devel.kde.org>" {
fileinto "INBOX/desarrollo/kde-devel";
} elsif header :contains "List-Id" "<kde-cvs.kde.org>" {
if header :contains "To" "kde-svn" {
fileinto "INBOX/cvs/kde/svn";
} elsif header :contains "Subject" "www/areas/events/info/conference2005" {
fileinto "INBOX/cvs/kde/www-akademy";
} elsif header :contains "Subject" "kde-i18n" {
fileinto "INBOX/cvs/kde/i18n";
} elsif header :regex "Subject" "/docs?(/|$)" {
fileinto "INBOX/cvs/kde/i18n";
} elsif header :contains "Subject" "kdebase" {
fileinto "INBOX/cvs/kde/kdebase";
} elsif header :contains "Subject" "kdelibs" {
fileinto "INBOX/cvs/kde/kdelibs";
} elsif header :contains "Subject" "kdepim" {
fileinto "INBOX/cvs/kde/kdepim";
} elsif header :contains "Subject" "qt-copy" {
fileinto "INBOX/cvs/kde/qt-copy";
} elsif header :contains "Subject" ["www/","developer.kde.org"] {
fileinto "INBOX/cvs/kde/www";
} else {
fileinto "INBOX/cvs/kde";
}
} elsif header :contains "List-Id" "<kde-devel-es.kde.org>" {
if header :contains "X-List-Administrivia" "yes" {
fileinto "INBOX/admin/kde-devel-es";
} else {
fileinto "INBOX/desarrollo/kde-devel-es";
}
} elsif exists "X-Bugzilla-URL" {
fileinto "INBOX/desarrollo/bugs/kde-bugs";
} elsif header :regex "Subject" "Local.*Security for tazend" {
fileinto "INBOX/tazend-security-logs";
} else {
fileinto "INBOX";
}
My file is much longer, but with those lines you can get an idea on how to write one, since it's quite readable.
First thing to note is that all mail folders hang from the INBOX folder, this is because kolab is using cyrus imapd, it seems it's possible to configure cyrus to use an alternate namespace but if kolab doesn't allow it. I think I can live with that, in fact, it's ok to have it that way.
I would also note that the first thing I do in my script is making a copy of every mail to the INBOX/backup/received folder . I plan to eventually remove that, since I've not had any problem with the filtering of mails.
My concern in this section was what would happen if I wrote a typo and specified a fileinto command to a folder that doesn't exist. In that case, the sieve filtering application just files the mail into the nearest parent directory that exists. In the worst case, the mail is filed into the INBOX folder, so there's no problem.
Once you have your own sieve script, save it into a file, open konqueror and go to the address: sieve://user%40mydomain.mytld@localhost (I assume you're using konqueror on the same computer where you have installed kolab). Once there, open another konqueror window and copy there your sieve script. Another option is to use sieveshell -u user -a user host and then use the commands:
put file.sieve activate file.sieve ls (to check the file has been correctly copied) quit (to exit).
If you want to read more about sieve and the sieve format, there are links to the relevant RFCs at http://www.cyrusoft.com/sieve/documents.html, also doing an internet search for "sieve fileinto INBOX" gives some other examples.
Reading the logs, and some other curiosities
Well, everything was already configured and working, I just had a look to the logs to see if they said anything special. I noted a line that said: duplicate_prune: purged 495 out of 2842 entries As you may guess by now, I never delete any mail (just move them to backup media), so before worrying about what this message meant, I asked and rcsu (from #cyrus) told me that it was a database cleanup of message IDs. It seems cyrus stores the ID of messages so if a message arrives duplicated, it makes a hard link in the filesystem instead of storing the mail twice.
A "problem" I had (to call it somehow) was that I couldn't read my mails from work. When I first installed kolab and I just had an empty INBOX folder (no fetchmail, no sieve), my first test was to copy a mail with my kontact client at home (let's call it kontactH) and then check mails with kontact running at work (let's call it kontactW) to see that the same mail appeared there too. Unfortunately it didn't seem to work since the mail simply didn't appear in the INBOX folder of kontactW and more important, the mail was deleted from the kolab server! . As funny as it seems, doing the same test in the other direction worked flawlessly. I'm afraid only after Till Adam and me lost much time, we found the problem was in the configuration of kontactW.
In kontactW, I had filters (kmail filters) for incoming mails so that mails related to kde were moved to a kde folder. The problem was that kmail was applying those filters to new mails into the INBOX folder of kolab, so the mail I was testing with (a regular mail from one kde mailing list), was being silently moved (since the mail was already in the read state) to my local folder at kontactW, and so, it was being deleted from the kolab server.
What I made then was to add a filter as the first mail filter at kontactW with the criteria that the "Received" field "contains" "myhost.mydomain.mytld", applied "to incoming messages" and "on manual filtering", with no action, and then with "If this filter matches, stop processing here" selected.
Some days ago, I got a strange message when checking mails, my INBOX/Contacts folder was over the quota and it couldn't write a contact (contacts from my addressbook are stored into the INBOX/Contacts folder, which is usually hidden, so nobody usually cares about that :) ) . I found it quite strange, since I didn't set any quotas. I rechecked that I don't use quotas and started looking for a solution to this message that appeared every time I checked mails.
The solution was provided by http://www.oreilly.com/catalog/mimap/chapter/ch09.html#41575
That page says "If quotas become corrupted, all that is required to fix the problem is to run the quota command with the -f switch". Since the quota command is renamed to cyrquota, I run:
su - kolab-r cyrquota -f
and it fixed the problem.
To-do
Kolab is a bit slower processing mails than just procmail (of course, since it involves quite many other applications and provides much more functionality), so maybe it could be configured to be a bit faster (you can check the mails queued, being processed, with the mailq command). For example, I don't care about viruses since I only run Linux on all my computers, I should look some day if it's possible to remove the virus scanning keeping the spam detection done by spamassassin (any help is welcomed :) ).
Talking about spamassassin, I haven't tought it yet to recognize my spam mails. I had a very well trained spamassassin running with my previous mail configuration, but since mail headers in my incoming mails have changed (kolab adds a different set of Received fields for example). I prefer to teach spamassassin with a new set of spam/ham mails. Right now (after 2 weeks of using this system) spamassassin has detected 287 spam mails, has had 371 false negatives, and 0 false positives. I would say that detecting 44% of the spam without being trained is a good number. Let's hope I can find some time to configure it, since previously I had between 98% and 99% of spams detected by spamassassin. Anyway, I prefer to wait some other weeks to do this, since that way I'll have a larger number of mails to take a more representative sample.
I'm also thinking on changing the permissions of some files so that normal users can run sendmail without problems. If someone can think of a security implication of that (or a better alternative than changing the permission of some config files), I'll be happy to read about it.
Something I still have to solve is that in some computers my calendar events are shifted one hour earlier than they appear in my kontact client at home. I guess that's something related to the timezone I'm in, but I still have to find the reason for that and the solution.
Conclusion
Despite some small problems I've had to get it configured as I wanted, mostly occurring by the lack of knowledge I had about kolab/cyrus imap/postfix/sieve/kolab/mail servers in general and my concrete needs, kolab has always worked rock solid and stable and my objetive of not losing any mail has been fulfilled. For the nearly two weeks that have passed since I finished installing/configuring kolab until the time of this writing, I've been much more productive in reading and administering my mails, so I'm quite happy of choosing to install kolab, and I hope this document serves to help other people like me to get started even faster.
Also, I'd like to thank everyone who helped me in the installation and configuration processes (some of whom I've mentioned in the respective section).
Antonio Larrosa <larrosa_at_kde_dot_org>
Categories: Installation | Howto | Usage
