Openexchange

From Kolab Wiki

Some tips based on our migration from a SuSE Linux Openexchange (SLOX) server to a Kolab 2 server.

NOTE: This is not a complete migration guide!

Contents 1 General Notes 2 Migrating User Accounts and Mail 2.1 transfer_accounts Script 2.1.1 What does it do? 2.1.2 Any requirements or configuration? 2.1.3 How does it work? 2.1.4 IMPORTANT NOTES 2.1.5 The script 3 Migrating Calendars 4 Migrating Addressbooks

General Notes

SLOX offers a lot more functionality than Kolab - among other things, you have a simple document storage structure, a web forum, bookmark storage and more. One should be aware of the fact that these cannot be migrated to Kolab.

SLOX - like Kolab - uses an LDAP database for user account data and IMAP for mail. Migrating these two elements is therefore rather straightforward and can even be partially automated (see below). Calendar/agenda data, however, is not as easy, as SLOX uses an SQL database for this. At our site, the SLOX agenda was barely used. Therefore, our efforts concentrated on migrating user accounts, user data and mail.

Migrating User Accounts and Mail

Kolab allows user accounts to be created directly via LDAP (see the page about ldap). Most of the data that Kolab has stored in LDAP is already present in SLOX' LDAP database and can therefore be copied with some minor preparation. To do so, we created a Perl script called transfer_accounts.

transfer_accounts Script

What does it do?

transfer_accounts first gets all information from the SLOX server for a given user, then creates the same user account on the Kolab server and finally transfers all mail for that user using imapsync.

NOTE: transfer_accounts does not transfer shared folders - only ordinary users are dealt with!

Any requirements or configuration?

transfer_accounts is meant to be run on the Kolab server, as the Perl that comes with Kolab has all necessary modules already installed.

There are several parameters that need to be configured (e.g. the servers). All of those parameters are placed at the beginning of the script - make sure to edit them according to your needs before using the script. Make sure to read the IMPORTANT NOTES below!

How does it work?

transfer_accounts works as user cyrus on the SLOX side and as user manager on the Kolab side. Therefore, it will need the passwords for both these accounts. The script will log in as cyrus on the SLOX server, get the encrypted password of the given user to be transferred and store it. Also, all user data is read from the SLOX LDAP and prepared for the Kolab LDAP format. Then, a new user with the given UID is created on the Kolab server with the data read from the SLOX server. The user's password gets changed to a standard password on both servers, then imapsync is run to transfer the user's mail using the standard password. We discovered that it is necessary to run imapsync iteratively, as not all folders and subfolders seem to be copied successfully during the first run. The script takes care of this. Finally, the user's password is changed back to the original encrypted password again on both servers, thus giving the user the same password on the Kolab server as he had on the SLOX server.

IMPORTANT NOTES

• The script has to contain the passwords for the cyrus account (SLOX) and the manager account (Kolab). This can pose a security risc! Therefore, handle the script with care and do not make it readable for everybody!
• As the user's password is changed temporarily during the transfer, the user cannot log in during that time!
• The script does have an option to create a default, world-readable calendar folder as well (option '-c') - this is only marginally tested (as we abandoned that requirement later) and does not work 100%!
• The script also has an option to bulk-migrate all users present on the SLOX server in one go - this has not been tested!

The script

'NOTE: THIS SCRIPT COMES WITHOUT ANY WARRANTIES WHATSOEVER - USE AT YOUR OWN RISK!!!'

#!/kolab/bin/perl -w
###########################################################################
# transfer_accounts                                                       #
#                                                                         #
# A script to migrate users from a SLOX server to a Kolab 2 server        #
#                                                                         #
# USE AT YOUR OWN RISK!                                                   #
#                                                                         #
# Thomas Ribbrock <argathin@gmx.net>, March 2006                          #
#                                                                         #
###########################################################################

use strict;
use Net::LDAP;
use Net::LDAP::LDIF;
use Digest::SHA1;
use MIME::Base64;
use Cyrus::IMAP::Admin;
use POSIX qw(strftime);
use Getopt::Std;

###########################################################################
# USER OPTIONS - edit as appropriate!                                     #
###########################################################################
# SLOX settings (source):
# hostname
my $SLOX_host = "SLOX.HOST.DOMAIN";

# LDAP Base
my $SLOX_LDAP_base = "dc=YOUR,dc=SLOX_LDAP_BASE";

# Password for "cyrus" on SLOX
my $SLOX_cyrus_pwd = "CYRUS_PASSWORD";

# Kolab settings (target):
# hostname
my $Kolab_host = "KOLAB.HOST.DOMAIN";

# LDAP Base
my $Kolab_LDAP_base = "dc=YOUR,dc=KOLAB_LDAP_BASE";

# Password for "manager" on Kolab
my $Kolab_manager_pwd = "MANAGER_PASSWORD";

# Mail Domain
my $MailDomain = "MY.DOMAIN";

# don't migrate the users in this list
my %excluded_users = ('testuser1' => 1,
		      'testuser2' => 1);

# default password to be used during migration
my $defaultpwd = 'system';

# imapsync command
my $imapsync = "imapsync";

# flags to be used with imapsync - edit with care!
my $imapsyncflags = "--include '^INBOX*' --subscribe";

# logdir
my $logdir = "/kolab/var/kolab/log";

# quota for mailbox on Kolab server in MB
my $cyrus_userquota = 500;

###########################################################################
# END OF USER OPTIONS!                                                    #
###########################################################################

# global variables
my %option = ();
getopts("chr", \%option);

my $defaultpwd_crypt;
my $ldap_SLOX;
my $ldap_Kolab;
my $mesg_SLOX;
my $mesg_Kolab;
my $SLOX_entry;
my $Kolab_entry;
my $SLOX_dn;
my $Kolab_dn;
my $SLOX_cn;
my $Kolab_cn;
my $SLOX_pwd;
my $Kolab_pwd;
my $filter;
my $result;
my $tries;
my $ctx;
my $new_entry = 0;
my $logfile;
my $date = strftime "%Y%m%d%H%M", localtime;
my $cmd;

# generate encrypted version of default password
$ctx = Digest::SHA1->new;
$ctx->add($defaultpwd);
$defaultpwd_crypt = '{SHA}' . encode_base64($ctx->digest,);

# open LDAP connections
$ldap_SLOX = Net::LDAP->new( $SLOX_host ) or die "$@";
$ldap_Kolab = Net::LDAP->new( $Kolab_host ) or die "$@";

$mesg_SLOX = $ldap_SLOX->bind( "uid=cyrus,$SLOX_LDAP_base",
			       password => $SLOX_cyrus_pwd) or die "$@";
$mesg_Kolab=$ldap_Kolab->bind("cn=manager,cn=internal,$Kolab_LDAP_base",
			      password => $Kolab_manager_pwd) or die "$@";

if ($option{h} || $#ARGV != 0)
  {
    print ("\nUsage: $0 [options] ALL - to migrate all accounts (UNTESTED!)\n");
    print ("       $0 [options] UID = to migrate user with user ID UID\n\n");
    print ("       Options:\n");
    print ("       -h - display this message\n");
    print ("       -r - reset password for ALL|UID to SLOX password\n");
    print ("       -c - force calendar folder creation (UNTESTED!)\n\n");
    exit;
}

if ($ARGV[0] eq "ALL")
  {
    # migrate all => search for all accounts on SLOX server
    $filter = "(objectClass=person)";
  }
else
  {
    # migrate a single user => set filter for that uid
    $filter = "(uid=$ARGV[0])";
  }

# get account info, but only attributes that have a Kolab equivalent
$mesg_SLOX = $ldap_SLOX->search(base => $SLOX_LDAP_base,
		      filter => $filter,
		      attrs => [ 'dn',
				 'ufn',
				 'kolabHomeServer',
				 'userPassword',
				 'objectClass',
				 'sn',
				 'cn',
				 'givenName',
				 'mail',
				 'uid',
				 'title',
				 'o',
				 'ou',
				 'roomNumber',
				 'street',
				 'postOfficeBox',
				 'postalCode',
				 'l',
				 'c',
				 'telephoneNumber',
				 'facsimileTelephoneNumber',
				 'kolabFreeBusyFuture',
				 'kolabInvitationPolicy',
				 'kolabDelegate',
				 'alias',
				 'cyrus-userquota']
		     );

if ($mesg_SLOX->count == 0)
  {
    printf ("\n\nNo entries found on $SLOX_host for filter '$filter'!\n\n");
    exit 1;
  }
  
# The following attributes are different in or unique to kolab:
# dn (change)
# objectClass (change)
# kolabHomeServer (add - doesn't get set properly otherwise)
# kolabFreeBusyFuture (uses default - don't set)
# kolabInvitationPolicy (add - doesn't get set properly otherwise)
# kolabDelegate (uses default - don't set)
# cyrus-userquota (set explicitly)
# roomNumber (must be set manually...)
# postOfficeBox (optional)

foreach $SLOX_entry ($mesg_SLOX->all_entries)
  {
    # get uid of current user
    my $uid = $SLOX_entry->get_value('uid');

    $logfile = "$logdir/$uid-migration.log";
    open (LOG, ">>$logfile") || die "Can't open $logfile!";

    LogMsg("--------------------- $date -----------------------", \*LOG);
    LogMsg("Migrating data for user with uid=$uid...\n", \*LOG);

    # skip excluded users
    if ($excluded_users{$uid})
      {
        LogMsg("uid=$uid is excluded - skipping!\n", \*LOG);
	next;
      }

    LogMsg("Default password (crypt): $defaultpwd_crypt", \*LOG);

    # get SLOX dn,cn,password
    $SLOX_dn  = $SLOX_entry->dn();
    $SLOX_cn  = $SLOX_entry->get_value('cn');
    $SLOX_pwd = $SLOX_entry->get_value('userPassword');
    LogMsg("SLOX_pwd=$SLOX_pwd\n", \*LOG);

    # test whether user already exists on Kolab server
    my $mesg_Kolab = $ldap_Kolab->search(base => $Kolab_LDAP_base,
					   filter => "(uid=$uid)");
    my $count = $mesg_Kolab->count;

    if ($mesg_Kolab->count == 0)
      {
	# user does not exist on Kolab server yet => transfer account
	LogMsg("uid=$uid does not exist on Kolab server - creating account!",
	        \*LOG);
        if ($option{r})
	  {
	    LogMsg("Option -r ignored - new accounts will always get the SLOX password!", \*LOG);
	  }

	$new_entry = 1;
	
	# Kolab_entry is a copy of the SLOX entry
	$Kolab_entry = $SLOX_entry;
	$Kolab_pwd = $SLOX_pwd;
	LogMsg("Kolab_pwd(create)=$Kolab_pwd", \*LOG);

	# now modify what is necessary
	# first, fix the dn
	LogMsg("SLOX_cn=$SLOX_cn", \*LOG);
	$Kolab_dn = "cn=$SLOX_cn,$Kolab_LDAP_base";
	$Kolab_entry->dn($Kolab_dn);
	
	# we've used "NL" so far - "The Netherlands" is nicer
	my $country = $SLOX_entry->get_value('c');
	$country =~ s/^NL$/The Netherlands/;
	$Kolab_entry->replace('c' => $country);
	
	# convert objectClass
	$Kolab_entry->replace('objectClass' => ['top', 'inetOrgPerson',
						 'kolabInetOrgPerson']);
	# add quota
	$Kolab_entry->add('cyrus-userquota' => "$cyrus_userquota");
	
	# add kolabHomeServer
	$Kolab_entry->add('kolabHomeServer' => "$Kolab_host");
	
	# add kolabInvitationPolicy: ACT_MANUAL
	$Kolab_entry->add('kolabInvitationPolicy' => 'ACT_MANUAL');
	
	# set password to default, so we can use imapsync
	$Kolab_entry->replace('userPassword' => "$defaultpwd_crypt");
	
	# dump new entry
	LogMsg("New LDAP entry:", \*LOG);
	$Kolab_entry->dump;
	$Kolab_entry->dump(\*LOG);
	
	# add new entry on KOLAB
	$result = $ldap_Kolab->add($Kolab_entry);
	if ($result->code)
	  {
	    LogMsg("ERROR: Failed to add Kolab entry:" . $result->error,
	           \*LOG);
            Exit(1,\*LOG);
          }
      }
    elsif ($mesg_Kolab->count == 1)
      {
	LogMsg("uid=$uid already exists on Kolab server.", \*LOG);
	LogMsg("Synching IMAP folders!", \*LOG);
	$Kolab_entry = $mesg_Kolab->entry(0);

	# get Kolab dn,cn,password
	$Kolab_dn  = $Kolab_entry->dn();
	$Kolab_cn  = $Kolab_entry->get_value('cn');
	$Kolab_pwd = $Kolab_entry->get_value('userPassword');
	LogMsg("Kolab_pwd($Kolab_host)=$Kolab_pwd", \*LOG);

        if ($option{r})
	  {
	    LogMsg("-r given - Kolab password will be reset to SLOX password!",
	           \*LOG);
            $Kolab_pwd = $SLOX_pwd;

	    LogMsg("Kolab_pwd(reset)=$Kolab_pwd", \*LOG);
	  }

	# set password on Kolab to default for imapsync
	$result = $ldap_Kolab->modify($Kolab_dn,
				      replace => {'userPassword' =>
						  "$defaultpwd_crypt"});
        if ($result->code)
          {
            LogMsg("ERROR: Failed to modify Kolab password:" . $result->error,
                   \*LOG);
            Exit(1,\*LOG);
          }
      }
    else
      {
	# there's more than one entry with this uid on the Kolab server
	# => fatal error
	LogMsg("FATAL ERROR: Several entries with uid=$uid exist on Kolab server!", \*LOG);
	Exit(1, \*LOG);
      }

    # now it's time to run imapsync for this account

    # set password on SLOX also to default for imapsync
    LogMsg("\nChanging SLOX password...", \*LOG);
    $result = $ldap_SLOX->modify($SLOX_dn, replace => {'userPassword' => "$defaultpwd_crypt"});
    if ($result->code)
      {
        LogMsg("ERROR: Failed to modify SLOX password:" . $result->error,
	       \*LOG);
        Exit(1,\*LOG);
      }

    # for some odd reason, the imapsync has a nasty tendency to interfere
    # with our LDAP sessions - no idea why
    # we choose to err on the side of caution and close the LDAP sessions
    # before we run imapsync and open them again afterwards.
    LogMsg("Closing LDAP conneciton with $SLOX_host and $Kolab_host.", \*LOG);
    $mesg_SLOX = $ldap_SLOX->unbind;   # take down session
    $mesg_Kolab = $ldap_Kolab->unbind;   # take down session

    $cmd = "$imapsync $imapsyncflags --host1 $SLOX_host --user1 $uid --password1 $defaultpwd --host2 $Kolab_host --user2 $uid --password2 $defaultpwd";

    LogMsg("Command=$cmd\n", \*LOG);

    $tries = 0;
    my $sys_result = 1;
    do {
      $tries++;

      # NOTE: Can't use 'tee' here, as we need the status of the
      #       imapsync command to be returned by "system"!!!
      $sys_result = system("$cmd>>$logfile 2>&1");
      LogMsg("\n$tries. try, system returned $sys_result", \*LOG);
    } until ($sys_result == 0||$tries == 10);

    # if this was a new entry, we also need to create a new Calendar
    # folder with the correct permissions.
    # For some odd reasons, this works better when done after imapsync
    # NOTE: FOR NOW IT IS NOT 100% CERTAIN THAT THIS WORKS ->
    #       ONLY DO IT IF -c WAS GIVEN!
    #if ($new_entry || $option{c})
    if ($option{c})
      {
        LogMsg("Creating world-readable calendar folder for $uid...", \*LOG);
        my $client = Cyrus::IMAP::Admin->new('localhost');

        $client->authenticate(-user => "manager",
                              -password=> $Kolab_manager_pwd);

        # find calendar folder for uid
        my @mailboxes = $client->list("*user/$uid/Calendar*");
        my $entries = @mailboxes;

        LogMsg("Found $entries mailboxes for uid=$uid.", \*LOG);

        if ($entries == 0)
          {
            LogMsg("No calendar folder found for $uid - creating...\n", \*LOG);

            my $calendar="user/$uid/Calendar\@$MailDomain";
            LogMsg("mailboxname(new) = $calendar", \*LOG);
            $client->create($calendar);
            if ($client->error)
              {
                LogMsg("WARNING: Couldn't create calendar folder:".$client->error, \*LOG);
              }
            else
              {
                $client->setacl($calendar, 'anyone' => 'read');
                if ($client->error)
                  {
                     LogMsg("WARNING: Couldn't set permissions of new calendar folder: ".$client->error, \*LOG);
                  }
              }
            @mailboxes = $client->list("*user/$uid/Calendar*");
	  }

        foreach my $mailbox (@mailboxes)
          {
            my $calendar = @$mailbox[0];
            LogMsg("mailboxname = $calendar", \*LOG);
            $client->setacl($calendar, 'anyone' => 'read');
            if ($client->error)
              {
                 LogMsg("Error when changing permissions of existing calendar folder: ".$client->error, \*LOG);
              }

            my %ACL=$client->listacl($calendar);

            foreach my $key (keys(%ACL))
              {
                LogMsg("ACL{$key}=$ACL{$key}", \*LOG);
              }
          }
      }

    # finally, reset the password on both servers
    # of course, we have to reopen the LDAP sessions to do so
    LogMsg("\nResetting passwords on both servers...", \*LOG);

    LogMsg("Re-opening LDAP connection with $SLOX_host", \*LOG);
    $ldap_SLOX = Net::LDAP->new( $SLOX_host ) or die "$@";
    $mesg_SLOX = $ldap_SLOX->bind( "uid=cyrus,$SLOX_LDAP_base",
                                    password => $SLOX_cyrus_pwd) or die "$@";
    LogMsg("SLOX_pwd(reset)=$SLOX_pwd", \*LOG);
    $result = $ldap_SLOX->modify($SLOX_dn, replace => {'userPassword' => "$SLOX_pwd"});
    if ($result->code)
      {
        LogMsg("ERROR: Failed to reset password on SLOX: " . $result->error,
	       \*LOG);
        Exit(1,\*LOG);
      }

    LogMsg("\nRe-opening LDAP connection with $Kolab_host", \*LOG);
    $ldap_Kolab = Net::LDAP->new( $Kolab_host ) or die "$@";
    $mesg_Kolab=$ldap_Kolab->bind("cn=manager,cn=internal,$Kolab_LDAP_base",
                                   password => $Kolab_manager_pwd) or die "$@";
    LogMsg("Kolab_pwd(reset)=$Kolab_pwd", \*LOG);  
    $result = $ldap_Kolab->modify($Kolab_dn, replace => {'userPassword' => "$Kolab_pwd"});
    if ($result->code)
      {
	LogMsg("ERROR: Failed to reset password on Kolab: ".$result->error,
	       \*LOG);
	Exit(1,\*LOG);
      }

    close(LOG);
  }

$mesg_SLOX = $ldap_SLOX->unbind;   # take down session
$mesg_Kolab = $ldap_Kolab->unbind;   # take down session

sub LogMsg
  {
    my ($Message,$fileref) = @_;

    print $fileref $Message, "\n";
    print $Message, "\n";
  }

sub Exit
  {
    my ($status,$fileref) = @_;

    print $fileref ("\n----------------------------------------------------------\n");
    close ($fileref);
    exit ($status);
  }

Migrating Calendars

As mentioned previously, there does not seem to be a straightforward way to migrate the calendar data from SLOX' SQL database to the Kolab IMAP setup. Cursory experiments indicated that it seems possible to do so for the users themselves using a suitable client like e.g. kontact by subscribing to both the SLOX and the Kolab server and subsequently copying their data. However, this has not been tested so far.

Migrating Addressbooks

At this point in time, we have not yet devised a way to do so.