Migration of IMAP accounts of Support and Billing questions to dedicated distributed servers

 

Emin Gabrielyan

2009-10-04

 

 

Migration of IMAP accounts of Support and Billing questions to dedicated distributed servers. 1

Steps remaining for completing the migration of the Billing account 2

Comparison of mailbox counters on source and destination IMAP servers as well as on the local computer 2

Importing IMAP cache files as local mailboxes. 7

To do for the Support classes. 8

References: 9

 

 

IMAP servers and the folder hierarchy of Switzernet’s Billing and Support accounts are being migrated to dedicated servers. The load will be distributed across several machines. The previous hosted solution raised serious issues with simultaneous connections and will be soon abandoned.

 

Two copy/backup methods of source IMAP folders are compared. Though server based imap transfer techniques exist, both our examined methods are based on the end user Thunderbird application. In both methods all concerned IMAP folders on the source server are marked for offline use and we completely sync thunderbird with the server. Creation of local copies is finalized only after the folders are synced completely and without error (done as of 2009-10-02).

 

With the 1st method we copy the synced IMAP folders into local folders from within Thunderbird while staying in off-line mode (which is important). With the 2nd method, we shut down the Thunderbird application, and we copy the hierarchy of synced IMAP folders from the file-system into the space corresponding to the Thunderbird’s local folders (files must be copied only when you are sure that no Thunderbird process is running in background).

 

Several misplaced folders (not respecting the structure of the how-to doc) are corrected manually. All misused dashes (‘-‘) in folder names are replaced by underlines (‘_’) according to the convention (the html bookmarks to chapters describing particular IMAP folders in our how-to docs support only alphanumerical symbols and underlines, but not dashes).

 

The table below shows the message counts and mailbox sizes of IMAP folders (a) on the original IMAP server (see the first column), (b) in the local Thunderbird directory copied from within Thunderbird (see the second column), (c) in the local Thunderbird directory imported from IMAP cache files (see the third column), and (d) on the destination IMAP server after a successful upload (see the fourth column).

 

For the final transfer path (source IMAP server – local Thunderbird – destination IMAP server), we have chosen the 1st copy method based on the copy of folders from within Thunderbird (the application being kept in the off-line mode during the copy). The numbers of messages in mailboxes imported from the file-system (according to the 2nd method) are given for comparison only and they show frequent discrepancies. Interpretation of IMAP cache folders as local thunderbird folders works, but frequent mismatches of message numbers take place because certain labels (e.g. for deleted messages) in imported cache files are not interpreted properly. Moreover, the manual tags (colors) and flags associated with messages are also lost when importing the IMAP cache files as local mailboxes.

 

When copying from within Thunderbird the tags and flags associated with messages were preserved (though tags were lost when mailboxes, but not the individual messages, are further re-copied in the local space, but this shall be a bug of Thunderbird).

 

There are few counter discrepancies also with the copy carried out according to the 1st method (see the first and second columns). The discrepancies between the source and the local copy should be explained only by fresh messages (according to a few occasional checks) and hopefully do not indicate on any error. New messages arrived because, thunderbird went on-line in between when restarted for importing the IMAP cache based version.

 

Steps remaining for completing the migration of the Billing account

 

Establish the automatic forwarding from the source server to the destination server.

Move the team members to the new server but keep temporarily the access to the old server.

When the last or most team members are migrated to the new server, block the access to the old server.

Transfer all new messages (since 2009-10-02) from all IMAP folders of the source server to the corresponding counterparts of the destination server. Transfer also the unsorted emails of inbox.

 

Folder names in the destination IMAP server are changed so as to solve the location-path problem of answered messages in Thunderbird search tools. The convention is easy to follow by looking at the names. Note that numbering order of all folders must be refreshed (manually!) whenever a new IMAP folder is added. If several IMAP folders are to be added in a short period of time, use temporarily in new names ‘x’ for sequential numbers. Renumber all folders of the concerned class at once, when no more types are considered for addition in a week, or so.

 

Comparison of mailbox counters on source and destination IMAP servers as well as on the local computer

 

On the source IMAP server

Copy of source IMAP folders carried out from within Thunderbird in off-line mode after a complete successful sync (1st copy method)

Rebuilding local Thunderbird folders from the files of IMAP sync/cache folders on the disk (2nd method)

On the destination IMAP server (uploaded from local folders of the 1st copy method)

Billing’s 01 Explain on IMAP

Billing’s 01 Explain copied in off-line mode

Billing’s 01 Explain rebuild from IMAP sync files

Billing’s 1 Explain on the destination (zimbra) IMAP server

‘blocked_free_calls’ is empty [solved]

Number mismatches (with 1st column):

‘account_balance’ (ok after refresh)

‘blocked_free_calls’

‘english_invoice’

‘offers’

‘payment_methods’

‘payment_note’

‘pdf_format’

Not user for the transfer to the destination server

No mismatches with the 2nd column

Billing’s 02 Cash on IMAP

Billing’s 02 Cash copied in off-line mode

Billing’s 02 Cash rebuild from IMAP sync files

Billing’s 2 Cash on the destination (zimbra) IMAP server

Mismatches with the 1st column:

‘lost_payment’

Not used for the transfer to destination server

No mismatches with the 2nd column

Billing’s 03 Account on IMAP

Billing’s 03 Account copied in off-line mode

Billing’s 03 Account rebuild from IMAP sync files

Billing’s 3 Account on the destination (zimbra) IMAP server

Mismatches with the 1st column:

‘contract_cancel’

‘email_bill_again’

‘free_billed’

‘refuse_to_pay’

Not user for the transfer to the destination server

Mismatches with the 2nd column:

3,10,re (ok after refresh)

3,11,re (ok after refresh)

Support’s 01 Reception on IMAP

Support’s 01 Reception copied in off-line mode

Support’s 01 Reception rebuild from IMAP sync files

Support’s 4 Reception on Zimbra

The hardware for the IMAP server is pending

Support’s 02 Phone on IMAP

Support’s 02 Phone copied in off-line mode

Support’s 02 Phone rebuild from IMAP sync files

Support’s 5 Phone on Zimbra

The hardware for the IMAP server is pending

Support’s 03 Service on IMAP

Support’s 03 Service copied in off-line mode

Support’s 03 Service rebuild from IMAP sync files

Support’s 6 Service on Zimbra

The hardware for the IMAP server is pending

 

Importing IMAP cache files as local mailboxes

 

Before copying the file-system folders (according to the 2nd method), the empty zombie folders left in the file-system by deleted or renamed folders in the past, are removed manually as well.

 

The following log shows the details of procedures carried out before importing the IMAP cache files as local Thunderbird folders. The MSF index files are deleted (so the indexes can be rebuilt from the scratch) and empty mailbox files are created for empty folders (no such files exist for empty mailboxes in case of IMAP caching). Recall that these folders however are not used for uploading.

 

Emin.Gabrielyan@pc8 /cygdrive/f/people/emin.gabrielyan/thunderbird/Local Folders/archive.sbd/091002 billing@ support@ sync.sbd

$ find . -name "*.msf"

./billing.sbd/01 Explain.sbd/account_balance.msf

./billing.sbd/01 Explain.sbd/account_balance.sbd/answered.msf

./billing.sbd/01 Explain.sbd/blocked_free_calls.msf

./billing.sbd/01 Explain.sbd/blocked_free_calls.sbd/answered.msf

./billing.sbd/01 Explain.sbd/contact_me.msf

./billing.sbd/01 Explain.sbd/contact_me.sbd/answered.msf

./billing.sbd/01 Explain.sbd/credit_warning.msf

./billing.sbd/01 Explain.sbd/credit_warning.sbd/answered.msf

./billing.sbd/01 Explain.sbd/english_invoice.msf

./billing.sbd/01 Explain.sbd/english_invoice.sbd/answered.msf

./billing.sbd/01 Explain.sbd/explain_invoice.msf

./billing.sbd/01 Explain.sbd/explain_invoice.sbd/answered.msf

./billing.sbd/01 Explain.sbd/final.msf

./billing.sbd/01 Explain.sbd/final.sbd/answered.msf

./billing.sbd/01 Explain.sbd/howto_refer.msf

./billing.sbd/01 Explain.sbd/howto_refer.sbd/answered.msf

./billing.sbd/01 Explain.sbd/offers.msf

./billing.sbd/01 Explain.sbd/offers.sbd/answered.msf

./billing.sbd/01 Explain.sbd/online_access.msf

./billing.sbd/01 Explain.sbd/online_access.sbd/answered.msf

 

 

$ find . -name "*.msf" | perl -ne 's/(\.sbd\/answered)?\.msf$//; print' | uniq -c

      2 ./billing.sbd/01 Explain.sbd/account_balance

      2 ./billing.sbd/01 Explain.sbd/blocked_free_calls

      2 ./billing.sbd/01 Explain.sbd/contact_me

      2 ./billing.sbd/01 Explain.sbd/credit_warning

      2 ./billing.sbd/01 Explain.sbd/english_invoice

      2 ./billing.sbd/01 Explain.sbd/explain_invoice

      2 ./billing.sbd/01 Explain.sbd/final

      2 ./billing.sbd/01 Explain.sbd/howto_refer

      2 ./billing.sbd/01 Explain.sbd/offers

      2 ./billing.sbd/01 Explain.sbd/online_access

      2 ./billing.sbd/01 Explain.sbd/payment_methods

      2 ./billing.sbd/01 Explain.sbd/payment_note

      2 ./billing.sbd/01 Explain.sbd/pay_yearly

      2 ./billing.sbd/01 Explain.sbd/pdf_format

 

$ find . -name "*.msf" | while read s; do rm "$s"; done

 

$ find . -name "*.sbd" | while read s; do f=`echo $s | perl -ne 's/\.sbd$//; print'`; if [ ! -f "$f" ]; then echo $f; fi; done

./billing.sbd/02 Cash.sbd/canceled_payment

./billing.sbd/03 Account.sbd/contract_resend

./billing.sbd/03 Account.sbd/private_to_commercial

 

$ find . -name "*-*"

./billing.sbd/03 Account.sbd/address-change

./billing.sbd/03 Account.sbd/address-change.msf

./billing.sbd/03 Account.sbd/address-change.sbd

 

To do for the Support classes

 

The same scenario shall be followed for the migration of Support accounts, but the three class indexes “01”, “02”, and “03” shall be replaced by “4”, “5”, and “6” in case we will be interested in a central backup account sharing Support and Billing (maybe with imapsync tool).

Type names in Support classes must be renamed according to same conventions (space separated suffixes, such as “4,1” in type names and unique names, such as “4,1,re” for formerly “answered” folders)

The class folders (such as “4 Reception”) must contain control messages showing the total number of type sub-folders, their names, and links to the bookmarks of the how-to docs.

References:

 

A tool for facilitating incremental recursive IMAP transfers from one mailbox to another (bugs, misuse, or source server limitations caused us to abandon this solution)

http://freshmeat.net/projects/imapsync/

 

On-line statistics of answered emails of support@ and billing@ accounts of Switzernet:

http://switzernet.com/public/090309-support-answers/last/

http://www.unappel.ch/public/090309-support-answers/last/

 

The application generating the on-line statistics of answers:

http://switzernet.com/public/090309-support-answers/

http://www.unappel.ch/public/090309-support-answers/

 

Zimbra collaboration suite:

http://www.zimbra.com/

 

*   *   *

Copyright © 2009, Switzernet

www.switzernet.com