Reading IMAP Mail in Emacs on OSX

Working! Last Tested: Sep 30, 2016

Introduction

Reading mail in Emacs is tricky (unless you are using Emacs as your only mail reader), but it can have various advantages. This post summarizes how to use Emacs as one of the applications to read email from multiple IMAP accounts.

The solution I am using to read IMAP mail on Emacs is found is based on the following toolchain:

and, in this post, you will get information on how to setup them. Despite the number of tools involved, the setup is relatively fast and you can be up and running in an hour or so.

NOTE FOR GMAIL USERS. The only valid setup I have found requires you to enable access for less secure apps in your Google Account. See below for the details.

Workflow and Tool Alternatives

The simplest solution I could find requires to break the mail-reading process in the following steps:

  1. Mirroring email on the laptop: the goal is create a copy of the messages on the server on the local computer. Various tools perform the trick. The ones mentioned more often on the net seem to be offlineimap and isync. I chose isync.
  2. Indexing email: the goal is facilitating searches. These tools are different from mail readers, since their only task is allowing one to search and annotate email with tags. Tools mentioned on the net include: notmuch, mu, and mairix (mairix does not seem to be actively developed). I use mu.
  3. Reading email: the indexers come with tools to read and manage email. In Emacs, the most common options include nm, notmuch (if you are using notmuch), and mu4e. I use mu4e.

By default, Emacs comes with a mode to compose and send email. However, if you have multiple accounts, you probably want to use multiple SMTP servers. In this case, the simplest solution seems to be:

  1. Composing email: Emacs default mode for composing messages works just fine
  2. Sending email: I use msmtp, which allows one to deliver mail using different SMTP servers. Servers can be automatically chosen, for instance, according to the "from" field in the email being sent.

The workflow is summarized in the following diagram:

Mbsync workflow

(view a png version, if you browser does not support svg)

  • Email accounts are provided by Google (gmail) and Apple (icloud)
  • mbsync -a synchronizes email between the servers and my computer. In the process, it creates a local copy of all email on my computer using the maildir format
  • mu index is used to index the email on my local computer.
  • mu4e provides an interface to read mail in maildir format using mu as the "search" engine for email
  • Emacs message mode is used to compose email, which is then delivered with msmtp. A copy of the email sent is stored locally (and synced back to the server by mbsync -a).
  • msmtp manages the connection with the remote SMTP servers, selecting the server according to the email specified in the "from" field of the outgoing mail.

Since all changes made in Emacs can be pushed back to the server, I can still use Apple Mail or any other mail reader, if I wan to.

Notice also that the synchronization and indexing processes are transparent, since mu4e invokes mbsync -a and mu index directly from Emacs.

Configuration and Setup

Step 0: install the required packages

Install the required packages. On OSX, I use homebrew

brew install isync
brew install mu --with-emacs
brew install msmtp
brew install w3m

where:

  • isync is the package providing mbsync
  • w3m is used to read HTML mail (other options are also available).

If you want to mirror email from a Gmail account, your also need to install and make brew's openssl the default on OSX

brew install openssl
brew link openssl --force

On Linux you have similar requirements. In Ubuntu and LinuxMint, for instance:

sudo apt-get install isync
sudo apt-get install maildir-utils
sudo apt-get install msmtp

On all systems, as an optional step, you can also install the maildir gem, which provides a Ruby interface to maildir:

gem install maildir

Step 1: Configure isync

The isync package provides the mbsync application, which is responsible of synchronizing email between servers and the local computer.

To get mbsync up and running, we "only" need to write a .mbsyncrc configuration file, which specifies where mail is gotten from, where mail is stored, and a mapping between remote and local folders. The configuration is organized in four items:

  • accounts: where mail is gotten from (and synced to)
  • storage: where mail is stored to
  • channels: what folders are synced
  • groups: what synchronizations are made available (usually one per account)

A very good and fast tutorial on how to setup .mbsyncrc can be found on the iSync page of the Archlinux website. I copied and pasted their example configuration and it worked fine with me. Other tutorials on mbsync (including information on how to use SSL) can be found here and here.

Configure isync for Apple Mail

This is the portion of my .mbsyncrc file which allows mbsync to mirror Apple Mail. If you are wondering where the server name comes from, have a look at Determining the URL of CalDAV calendars:

IMAPAccount icloud
Host p04-imap.mail.me.com
User MYEMAIL@me.com
# UseIMAPS yes
AuthMechs LOGIN
SSLType IMAPS
SSLVersions TLSv1
PassCmd "security find-generic-password -s mbsync-icloud-password -w"

IMAPStore icloud-remote
Account icloud

MaildirStore icloud-local
Path ~/Maildir/icloud/
Inbox ~/Maildir/icloud/inbox
Trash trash

Channel icloud-folders
Master :icloud-remote:
Slave :icloud-local:
Patterns "INBOX" "Drafts" "Arch*" "Sent*" "Trash" "Junk" "Deleted*"
Create Both
Expunge Both
SyncState *

Group icloud
Channel icloud-folders

Configuring isync for Gmail

This recipe works for me, as of Sep 30, 2016 (I had to change the configuration given in the previous version of this post).

A. Allow access to Less Secure Apps. Google recently (= in the last year) changed its authentication mechanism, introducing XOAUTH2 and the mbsync configuration proposed in many tutorials does not work anymore.

The only way I have found starts by allowing access for less secure apps on my account., which basically "downgrades" the authentication mechanism to something mbsync can manage.

Remarks. Alternative solutions might be based on `offlineimap` or trying to have the `XOUATH2` authentication work with `mbsync`. I did not find an easy way to do it. Some information on `offlineimap` can be found on a Stack Overflow Post and on a Github Gist. If you do, please get in touch and I will update the post.

B. Install the Required Certificates. Gmail requires a secure connection. To establish it, you need to install the certificate chain on your computer.

Download and store the certificates by running:

openssl s_client -connect imap.gmail.com:993 -showcerts

which will produce something like: openssl output.

Copy the three chunks between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and store them in three different files. If you want to stick to my configuration:

  • store the certificates in /usr/local/etc/openssl/certs (which is the default location for openssl on brew)
  • use the following filenames gmail.crt, google.crt, Equifax.crt.

However, you can use any directory and any filename, as long as you reference them correctly in the .mbsyncrc configuration file.

On Linux the installation is even simpler: you can install the certificates with:

sudo apt-get install ca-certificates

C. Configure the mbsyncrc file. You can now add the relevant configuration to your .mbsyncrc file. This is, for instance, the configuration I currently use (please replace MYEMAIL with … YOUREMAIL!):

# ACCOUNT INFORMATION
IMAPAccount gmail
Host imap.gmail.com
User MYEMAIL@gmail.com
PassCmd "security find-generic-password -s mbsync-gmail-password -w"
# UseIMAPS yes
# AuthMechs LOGIN
AuthMechs PLAIN
SSLType IMAPS
# SSLVersions SSLv3
CertificateFile /usr/local/etc/openssl/certs/gmail.crt
CertificateFile /usr/local/etc/openssl/certs/google.crt
CertificateFile /usr/local/etc/openssl/certs/Equifax.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore gmail-remote
Account gmail

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore gmail-local
Path ~/Maildir/gmail/
Inbox ~/Maildir/gmail/inbox

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR
#
# FOR INSTANCE IN THE SPECIFICATION BELOW:
#
# gmail-inbox gets the folder INBOX, ARCHIVE, and everything under "ARCHIVE*"
# gmail-trash gets only the "[Gmail]/Trash" folder and stores it to the local "trash" folder

Channel gmail-inbox
Master :gmail-remote:
Slave :gmail-local:
Patterns "INBOX" "Arch*"
Create Both
Expunge Both
SyncState *

Channel gmail-trash
Master :gmail-remote:"[Gmail]/Trash"
Slave :gmail-local:trash
Create Both
Expunge Both
SyncState *

Channel gmail-sent
Master :gmail-remote:"[Gmail]/Sent Mail"
Slave :gmail-local:sent
Create Both
Expunge Both
SyncState *

# GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE
# MBSYNC ON A GROUP TO SYNC ALL CHANNELS
#
# FOR INSTANCE: "mbsync gmail" GETS MAIL FROM 
# "gmail-inbox", "gmail-sent", and "gmail-trash"
#
Group gmail
Channel gmail-inbox
Channel gmail-sent
Channel gmail-trash

Step 2: mu

Mu is the tool used to index email. It works out of the box: type mu index to index your email and then search and view email from the command line.

The query syntax is described on the following pages:

Pages related to my: homepage and github repository.

Step 3: mu4e

Mu4e is the official mu mail-reader for Emacs. It ships with mu.

This is my configuration (in .emacs) to use mu4e:

; add the source shipped with mu to load-path
(add-to-list 'load-path (expand-file-name "/usr/local/Cellar/mu/0.9.10/share/emacs/site-lisp/mu4e"))

; make sure emacs finds applications in /usr/local/bin
(setq exec-path (cons "/usr/local/bin" exec-path))

; require mu4e
(require 'mu4e)

; tell mu4e where my Maildir is
(setq mu4e-maildir "/Users/adolfo/Mail")
; tell mu4e how to sync email
(setq mu4e-get-mail-command "/usr/local/bin/mbsync -a")
; tell mu4e to use w3m for html rendering
(setq mu4e-html2text-command "/usr/local/bin/w3m -T text/html")

; taken from mu4e page to define bookmarks
(add-to-list 'mu4e-bookmarks
            '("size:5M..500M"       "Big messages"     ?b))

; mu4e requires to specify drafts, sent, and trash dirs
; a smarter configuration allows to select directories according to the account (see mu4e page)
(setq mu4e-drafts-folder "/work/drafts")
(setq mu4e-sent-folder "/work/sent")
(setq mu4e-trash-folder "/work/trash")

The important stuff is:

  • adding mu4e directory to the load-path (otherwise there is no way require will work)
  • adding /usr/local/bin to the exec-path, so that Emacs can find w3m and mbsync
  • specify the maildir location mu4e-maildir
  • specify the command to get email: mu4e-get-mail-command

Step 4: sending email with msmtp

msmtp is an application which delivers mail using different SMTP servers. This allows me to use gmail or icloud according to the "from" address I specify in the email I compose.

To get it working, I had to configure both msmtp and Emacs.

Concerning msmtp configuration, the example file described here is just fine: simply adapt it to match your SMTP server. If you want to have a look at my configuration file, here it is:

defaults
tls on
auto_from on
logfile ~/.msmtp.log

account gmail
host smtp.gmail.com
tls on
tls_certcheck off
auth on
from MY_EMAIL@gmail.com
user MY_EMAIL@gmail.com
password ********** (this is a security risk, see next section)
port 587

account icloud
host smtp.mail.me.com
tls on
tls_certcheck off
auth on
from MY_ICLOUD_EMAIL@me.com
user MY_ICLOUD_EMAIL@me.com
password *********** (this is a security risk, see next section)
port 587

Concerning Emacs, the configuration I use is the following:

; use msmtp
(setq message-send-mail-function 'message-send-mail-with-sendmail)
(setq sendmail-program "/usr/local/bin/msmtp")
; tell msmtp to choose the SMTP server according to the from field in the outgoing email
(setq message-sendmail-extra-arguments '("--read-envelope-from"))
(setq message-sendmail-f-is-evil 't)

Step 5. Storing Passwords using OSX Keychain

When I first started using mbsync and msmtp, I stored the password directly in the configuration files, because I did not want to spend too much time in securing an installation I was not even sure I would keep.

After I decided to keep the system, I also secured the configuration, using the OSX keychain. It is relatively simple both for mbsync and for msmtp.

MBSYNC Replace the Password directive with PassCmd, which allows one to specify a command to run to read the password. On OSX the directive looks something like:

PassCmd "security find-generic-password -s mbsync-gmail-password -w"

on the assumptions you have a Keychain entry named mbsync-gmail-password containing the password you need.

MSMTP Similar to mbsync, a passwordeval command allows one to specify the command to read the passwod. The instructions on the msmtp man page are very clear.

Emacs Mail Packages

As a final remark Emacs Wiki has a page dedicated to email, which lists various packages.

These are the ones I tried and the reason I ended up with the solution described above.

  • RMAIL is the "official" package to read email in Emacs. It works out of the box, but reading email is a one-way solution: RMAIL works on its own copy of email, after downloading it from the server; it is impossible to push back changes. Thus the solution is ok if you are using RMAIL as the only (or main) mail reader.
  • Gnus is the official Emacs news reader. Today it seems also the simplest and most used solution to read IMAP email. In fact, it can be configured to read mail from IMAP servers and instructions are available on various websites (see, for instance: Multiple Gmail Accounts in GNUS and GnusGmail). However, the interface is "unusual" for a mailer and I could not get myself to like it for reading email.
  • VM is another famous package to read email. Version 8.2.0b (from the Launchpad VM project page) compiles with no problems in Emacs 24.4. After a bit of configuration, I managed to have VM reading IMAP folders with vm-visit-imap-folder. The command, however, opens only one folder at a time. To get a "unified" view of emails you need to configure VM so that it downloads messages locally to a single inbox. Similar to RMAIL, however, this solution does not allow to push changes back to the server (if I got it right). The instructions to setup VM are available here.

Other packages I did not try include Wanderlust and Mew.

Other References on the Net

Post a comment

Note: Existing comments have been temporarily removed, due to a bug in my middleman template. They will come back!