CIX scratchpad to MBox mail store file conversion Howto, mark 2
Comments and feedback sought!
--
Liam P.
*Migrating from Ameol to Thunderbird*
A preliminary how-to guide on converting a messagebase of email in Ameol
or another CIX offline reader into MBox format for import into Mozilla
Thunderbird, Eudora or other Mbox-based email clients.
Based on the notes at
http://www.livejournal.com/users/lproven/65227.html
and
cix:bikers/14nonbike_qury:4149
==========================
This guide will help you convert your email store from a CIX offline
reader into a dedicated email client.
_Notes_
1. The source format is a CIX scratchpad, not your OLRs internal
storage format. You must export your messages to a scratchpad then work on
that file, not on the messagebase itself.
2. I am assuming that you are using Windows, starting from Ameol &
going to Thunderbird. However, the same procedure will work on any OS
you will just need appropriate tools
3. Do not start with your entire email store. Start with a small
subset of just a few messages perhaps a dozen or two. Once you know what
youre doing, try a larger number or if you confident, all of them.
4. Ameol strips attachments out of mail messages when it decodes
them. This leaves the messages malformed and they will not survive the
translation, appearing as blank messages. I welcome any suggestions as to
how to resolve this.
5. If you have sorted your Ameol inbox into subfolders, I suggest
exporting the whole thing to one big scratchpad & recreating the hierarchy
afterwards.
6. Note that Ameol message flags read/unread, priority, keep, lock
etc. will not survive the conversion. I believe, in fact, that they
dont survive exportation to a scratchpad.
While Microsoft Word is perfectly capable of doing the search and replace
operations, it struggles with large file sizes, such as an email archive
of hundreds of thousands of messages. It can bring Windows to its knees
with swapping.
I have found that the shareware TextPad editor is capable of handling
files of around the size of the PCs total memory without causing
excessive thrashing and it has the search & replace capabilities required.
Recommended, but if you have a preferred editor, by all means use it.
http://www.textpad.com/
_Steps_
[1] Create a scratchpad
To begin, select the email messages you want to work on and export them to
a scratchpad. In Ameol, to do this, open the File menu and choose Export.
Select the CIX Scratchpad format. (This will be written File | Export |
CIX Scratchpad from now on for brevity.)
[2] Make a backup copy of your scratchpad.
Work on the copy. You will not get this perfectly right first time!
[3] Open the scratchpad in your editor.
You might also wish to create a small MBox file in your email program
perhaps by sending yourself a few messages from a webmail account, for
example and open it side-by-side for comparison.
[4] Convert the message headers
The only difference between a scratchpad and an MBox file are the
delimiters which separate messages from one another. The email headers are
in standard Internet mail formats which do not change. To accomplish the
conversion, its merely necessary to convert CIXs message separator lines
into MBox-standard ones. This can be done with a couple of careful search
& replace commands, but it requires an editor whose search & replace
commands can handle strings containing carriage returns.
[5] Copy the converted file into the mail programs store directory.
This must be done with your email program shut down and not running. Use
Task Manager to ensure that it is not still in memory but invisible.
[6] Re-open the mail program and open the new folder
Most email programs today maintain separate index files for performance
reasons. The first time a newly-converted mail store file is opened, these
indices must be created. This takes some time.
[7] After moving around messages to taste, compact the MBox files.
MBox files are inefficient. If a message is moved or deleted, a blank gap
is left in the file; the files do not shrink. To prevent the email program
needing vast amounts of RAM to open these holey files, its necessary to
compress them after sorting and tidying messages.
_The file formats_
MBox mail messages are delimited by a line starting with the word From
followed by a space. The Thunderbird "From " header is just a date & it's
always the same one,
so the value is clearly not used.
CIX scratchpad messages begin with
!MF
or
!MFA
Followed by a header containing the CIX Memo number. If we wish to
preserve that information, since throwing data away is seldom a good idea,
we can make it into a extra MBox header by reformatting Memo [number] as
X-CIX-Memo [number].
So what you need to do is replace pairs of lines containing:
!MF
Memo
or
!MFA
Memo
With pairs saying:
From CIX scratchpad
X-CIX-Memo:
(I used this phrasing, in place of Mozillas date, just as a marker for
myself).
You need to search for exclamation mark M F, newline, Memo, space and
replace it with From CIX Scratchpad, newline, X-CIX-Memo, space, or
similar. The point is to replace both lines at once with the new pair of
lines.
Once you have done this search & replace, your working-copy scratchpad
should be an MBox format file.
For Thunderbird users, you can just copy the file into Thunderbirds mail
folders and next run the program should find it, display it as a message
folder and, when you click on it, index it and display the messages
within.
Thunderbird keeps its message store under a randomly-generated profile
name in the Application Data folder in a users Documents and Settings
folder (assuming WinXP). Under this is a folder for each configured email
service, named after the server itself, with a suffixed number for
multiple instances of the same server.
E.g. mine is:
C:\Documents and Settings\Liam Proven\Application
Data\Thunderbird\Profiles\97jxmos5.import\Mail\mail.cix.co.uk
This is a suitable directory to place the converted file into. If
Thunderbird does not recognize it, try moving an existing file out of
there such as Inbox and replacing it with your converted file,
renamed appropriately.
N.B. This process can be slow. I converted some 325,000 messages totalling
some 300MB on a 512MB dual-processor 2GHz machine. Saving the modified
file out of my editor took some ten minutes and Thunderbird took several
hours to index and display the resultant email folder. Start with small
test folders and be patient.
After a successful import, choose File | Compact Folders to reorganize
the MBox files for efficient use.
I think you need an extra step before this step
"So what you need to do is replace pairs of lines containing:
!MF
Memo
or
!MFA
Memo
With pairs saying:
From CIX scratchpad
X-CIX-Memo:
"
That finds the message delimiter used by the mbox format i.e.
[blank line]
From[space]
and replaces it with before the above step
[blank line]
>From[space]
Otherwise and messages with that sequence in will get split into two (bit
before the from and bit after)
See cix:email/clients:153 for a fuller description.
Andy
I don't quite follow.
> I think you need an extra step before this step [...]
All right. But then:
> That finds the message delimiter used by the mbox format i.e.
>
> [blank line]
> From[space]
>
> and replaces it with before the above step
>
> [blank line]
> >From[space]
The phrasing of that is giving me difficulties.
You're saying that before replacing CIX message delimiters with MBox ones,
it's necessary to quote any lines starting with "From" after a blank line?
(Where "quote" means insert a greater-than symbol.)
> Otherwise and messages with that sequence in will get split into two
> (bit before the from and bit after)
Right. I can see that. I've not had this problem, but I can see that.
> See cix:email/clients:153 for a fuller description.
--
Liam P.
>You're saying that before replacing CIX message delimiters with MBox
ones,
>it's necessary to quote any lines starting with "From" after
a blank line?
>(Where "quote" means insert a greater-than symbol.)
Yep, that's exactly what I'm saying, but perhaps not very clearly!
This might be a better explanation:
http://email.about.com/cs/standards/a/mbox_format.htm
Cheers
Andy
Right. Gotcha. TFTI!
--
Liam P.