Converting from mbox to Maildir
Introduction
This article describes how to convert from mbox to Maildir format. Maildirs have quite a few advantages over mbox format, especially when you have a lot of users with large mailboxes. Another big advantage is that with dovecot and Maildirs, users can create sub-folders.
These notes are specific to my situation. I wasn’t able to find a complete guide to doing this anywhere, so hopefully it’ll serve as a good guide to others. My system uses:
- Debian GNU/Linux 3.1r2
- dovecot (POP3 and IMAPS)
- exim4 (exim4-daemon-heavy)
- procmail (some users have a ~/.procmailrc and there’s a global /etc/procmailrc)
- Some users SSH in and use mutt.
Before the conversion
Before the conversion, everyone used the mbox mail format. Everyone’s mail spool was /var/mail/$USER. In addition, dovecot was setup with default_mail_env = mbox:%h/Mail/:INBOX=/var/mail/%u. So people also had mail in $HOME/Mail/, such as $HOME/Mail/Spam, $HOME/Mail/Sent etc.
Not every user on my system has a valid shell. Some users are mail only, so have a shell of /bin/false. Others are webspace only so have a shell of /usr/sbin/scponlyc. This makes the conversion process a little more complicated because I wasn’t able to su to these users to convert their mailboxes.
Preparation
Before doing on work on converting everything, I stopped dovecot and exim from running and then changed my firewall so only I had access to these services. I didn’t want any mail coming in or users checking their mail during the conversion. I also made sure all mail queues were cleared down.
It goes without saying that a full backup should be done and if possible, run a test restore. Just in case something goes horribly wrong.
Configuring exim
The first task was to reconfigure exim. By default it delivers mail to /var/mail/$USER, in mbox format. To tell exim that it should now deliver email to a Maildir you should change the local_delivery transport so that it looks like:
local_delivery:
driver = appendfile
create_directory = true
directory = ${home}/Maildir/
mode = 0600
directory_mode = 0700
maildir_format
delivery_date_add
envelope_to_add
return_path_add
Also check the rest of your exim configuration, to ensure nothing is configured to look at /var/mail/$USER or $HOME/Mail. One pitfall I had was an overcomplicated and poorly written procmail transport.
Remember to restart exim.
Configuring dovecot
dovecot is very easy to reconfigure. Simply change the default_mail_env line so it looks like:
default_mail_env = maildir:%h/Maildir
Remember to restart dovecot.
Check the global procmailrc file
This one caused me a headache for quite a while. If you have a global procmailrc file (/etc/procmailrc) then you should to add the following to the beginning:
DEFAULT=$HOME/Maildir/
Run a test conversion
To do this you need to install mb2md which is a Perl script that will convert any mail in an mbox format to in to a Maildir format.
Before attempting a “mass conversion”, I’d strongly suggest you sacrifice your own account as a test. Backup your /var/mail/$USER spool and your $HOME/Mail somewhere (just in case). Now run:
$ makemaildir.dovecot $HOME/Maildir $ mb2md -s /var/mail/$USER -d $HOME/Maildir $ mb2md -s $HOME/Mail -R -d $HOME/Maildir $ cp $HOME/Mail/.subscriptions $HOME/Maildir/ $ mv /var/mail/$USER /var/mail/$USER.preMaildir $ mv $HOME/Mail $HOME/Mail.preMaildir
Now you should have your mail in Maildir format. Give it a test by loading your mail client and making sure everything still works (all your mail is there, IMAP subscriptions are there and you can recieve a test email from yourself).
If something isn’t quite right, check:
- Permissions
- Your .forward or .procmailrc is OK (maybe mv them out of the way as a test)
- You configured exim and dovecot correctly and restarted them.
Once everything is working, undo the work you just did (because next we’ll be converting everyone, at once):
$ rm -rf $HOME/Maildir $ mv /var/mail/$USER.preMaildir /var/mail/$USER $ mv $HOME/Mail.preMaildir $HOME/Mail
Converting everyone at once
This is the scary bit. Ensure you do have a working backup. As I mentioned previously, not all my users had a valid shell so it made automating the conversion a bit more tricky. So I decided to run the conversions as root and then use chown afterwards. All of this was scripted and the script I wrote and used is as follows:
#!/bin/sh
# *** README *** README *** README *** README *** README *** README ***
# This conversion script worked fine for me (tm). YMMV.
#
# Things to look out for:
# 1. On my system, mail in mbox format was also stored in ~/Mail. Your
# system might be different (~/mail/ or just ~/mbox).
# 2. The maildirmake.dovecot may be called something else on your system.
# Things you should do before executing this:
# 1. Add an "echo" before any of the commands, so they don't execute
# which will allow you to make sure everything looks OK.
# 2. Change line 29 so that it only processes a test account, not
# everyone (i.e. change "-ge 500" to "-eq 500" to only process the
# user account with UID 500).
# *** README *** README *** README *** README *** README *** README ***
IFS=$' '
# Cycle through users.
for line in "$(getent passwd)"; do
# Only attempt to process a user with a UID > 500.
if [[ $(echo|"$line" | awk -F: '{print $3;}') -ge 500 ]]; then
# Get some information about the user.
username=$(echo $line | awk -F: '{print $1}')
uid=$(echo "$line" | awk -F: '{print $3}')
gid=$(echo "$line" | awk -F: '{print $4}')
homedir=$(echo "$line" | awk -F: '{print $6}')
# Do they have a mail spool or mbox directory?
if [[ -f|/var/mail/$username ]] || [[ -d|$homedir/Mail ]]; then
# For whatever strange reason, they may already have a
# ~/Maildir, when they probably shouldn't.
if [[ -d|$homedir/Maildir ]]; then
mv $homedir/Maildir $homedir/Maildir.preMaildir
fi
# Setup the Maildir structure.
maildirmake.dovecot $homedir/Maildir
# Convert their mail spool.
if [[ -f|/var/mail/$username ]]; then
mb2md -s /var/mail/$username -d $homedir/Maildir
mv /var/mail/$username /var/mail/$username.preMaildir
fi
# Convert their old mbox directory.
if [[ -d|$homedir/Mail ]]; then
mb2md -s $homedir/Mail/ -R -d $homedir/Maildir
cp -p $homedir/Mail/.subscriptions $homedir/Maildir/
mv $homedir/Mail $homedir/Mail.preMaildir
fi
# Uncomment the following to attempt to generate a
# subscriptions file (probably a bad hack).
#find $homedir/Maildir -type d -iname ".*" > $homedir/Maildir/.subscriptions
# Give the user back their rights.
chown -R $uid:$gid $homedir/Maildir
chmod 600 $homedir/Maildir/.subscriptions
if [[ -f|$homedir/.procmailrc ]] || [[ -f|$homedir/.forward ]]; then
echo "*** $username has a .procmail or .forward!"
fi
echo "*** $username has been converted."
fi
echo
fi
done
This script worked perfectly and converted all the users in one go without a problem. Everyone’s mail spool in /var/mail will be mv’d out of the way and have “.preMaildir” appended to it. The same applies for $HOME/Mail. Remember to investigate people’s .procmailrc or .forward files to make sure they’re compatible.
Changing the shell environment
The final part of the process is to change the shell environment. Without doing this, pam_mail (the PAM module that tells you if you’ve got new mail when you login) won’t work and neither will mutt.
First edit /etc/pam.d/login and /etc/pam.d/ssh and add dir=~/Maildir to the pam_mail.so line:
session optional pam_mail.so standard noenv dir=~/Maildir
Now edit /etc/login.defs and uncomment the QMAIL_DIR line.
I’d also suggest running makemaildir.dovecot /etc/skel/Maildir
Lastly, if you use mutt, edit /etc/Muttrc and add/edit the following:
set mbox_type=Maildir set folder="~/Maildir/" set mask="."
Open the flood gates
Do a final check. Make sure everything looks correct and then unfirewall the exim and dovecot services. Finally keep your fingers crossed that no one notices any changes.
Page written by DavidRamsden
Leave a Reply
You must be logged in to post a comment.