Mutt mail indexing on steroids!

Update 31/03/2012: mutt-notmuch has been integrated upstream under the name notmuch-mutt.

I've been using external mail indexing with Mutt for quite a while now. Before now I haven't given Notmuch a try, as it seemed too much experimental at the time of my initial review of mail indexing tools.

As there is nothing better than a long oversea flight to perpetrate new hacks, I've now not only tested it, but also switched to Notmuch without looking back at maildir-utils for a split second.

Getting started with Notmuch is trivial:

  • Notmuch is notmuch in Debian (and maintained by upstream author, thanks!)
  • notmuch setup asks a few questions about the user and then prepare a template ~/.notmuch-config, which is good enough to start
  • notmuch new does a first run of indexing (which might take a while, depending on the size of your maildirs)

Some performance figures: (1) the size of the index is a bit larger than the size of the mails (on my laptop: 535 Mb of index vs 375 Mb of mails stored in 60 maildirs); (2) even if the underlying indexing engine is the same (Xapian) as maildir-utils, Notmuch is much faster. With my setup it is so fast that I've added it as a hook triggered by offlineimap at the end of each synchronization:

    zack@usha:~$ grep -B 1 hook .offlineimaprc
    [Account Upsilon]
    postsynchook = notmuch new

With the big round of new mail downloaded in the morning, it hardly takes more than a few seconds to update Notmuch index, while with other periodic downloads Notmuch almost invariably reports "... in almost no time" (i.e. less than 1 second).

Integrating Notmuch with Mutt

The basic integration I want between Mutt and a mail indexer is the ability of stating a search query interactively and then jump to a fresh Maildir containing search results only. Additionally, given that Notmuch has neat thread reconstruction abilities, I also want to be able to reconstruct on the fly the thread corresponding to the currently highlighted mail: it comes handy when a thread has been split across different maildirs, archived, or the like.

To that end, I've cooked up a little helper, called mutt-notmuch, that enables to trigger mail searches via a Mutt macro (F8 being my choice) and reconstruct threads via another (F9). Check the manpage for the 2-liner configuration snippet for your ~/.muttrc.

Arguably, some of the logics of mutt-notmuch could disappear by adding support for a --output=symlinks flag to Notmuch. A bug report requesting that is pending: too bad oversea flights are not as good for accessing the Internet as they are for perpetrating hacks!

Download


Update 10/04/2011: mutt-notmuch code is now available via Git. Resistence to propose your own patches is futile! The above link to mutt-notmuch has been changed to point to the live Git version.

Update 11/02/2012: Git repository has been moved to Gitorious; links updated accordingly

Notmuch seems interesting project. I my self tried different indexers couple of years a go, but finally found that best for my mail work flow was to just view all emails at once in Mutt. I did couple of neat limit key bindings and added colors to different mailing list messages and now my Mutt mail view has all messages quickly accessible.

Mutt is really fast with cache options enabled. For example my mutt status bar states: [Msgs:47453 New:53 531M] and the mailbox loads in 1 or 2 seconds.

Comment by Petteri Thu 03 Feb 2011 10:19:37 PM CET
It requires notmuch 0.5 because of the output file option. Otherwise, a sincere thank you for sharing this tip. I feel like I have my life back. :)
Comment by hendry [iki.fi] Fri 04 Feb 2011 12:59:56 PM CET

Thanks for this very nice recipe.

Took me a bit of twiddeling to find out that the "libmail-box-perl" package is needed and to use the newer version of "notmuch" from unstable.

Now it just works *g*

Kudos

Comment by Niels Sat 05 Feb 2011 12:36:31 AM CET

In the browser screen, change-folder-readonly doesn't work. You can use the following snippet instead:

    macro browser <F8> "<enter-command>unset wait_key<enter><shell-escape>mutt-notmuch --prompt search\
      <enter><change-dir>../.cache/<enter><search>mutt_results<enter><enter>" "search mail (using notmuch)"

Cheers

Comment by Enrico Tassi Sat 05 Feb 2011 06:58:47 PM CET

Hi Zack,

I think your script needs to live somewhere! :-)

I thought perhaps "moreutils", but it might not fit there; there is also the "bikeshed" package in Ubuntu (hmm, not in Debian yet though, something to put on my maybe-ITP list)

Comment by Jon Mon 07 Feb 2011 03:21:09 PM CET

If you use mutt-patched the F9 macro works, but if you use regular mutt and you have ignore message-id in your conf file, then you need to modify your lines as follows

    macro index <F9> "<enter-command>unset wait_key<enter><enter-command>unignore message-id\
      <enter><pipe-message>mutt-notmuch thread<enter><change-folder-readonly>~/.cache/mutt_results\
      <enter><enter-command>set wait_key<enter>" "search and reconstruct owning thread (using notmuch)"

othwise the script is fed with an incomplete header and cannot rebuild the thread.

Cheers

Comment by Enrico Tassi Mon 07 Feb 2011 04:23:18 PM CET

I guess the right package would be notmuch, since moreutils is more general purpose.

A completely new package for just one 100 lines scripts seems a bit overkilling, but is doable too.

Comment by Enrico Tassi Mon 07 Feb 2011 04:27:53 PM CET

Does anybody know how I could make mark email in that 'virtual mailbox as deleted or read?

I've tried replacing change-folder-readonly to change-folder, but that didn't help. That feature would make this setup perfect!

Comment by nomad Tue 08 Feb 2011 09:40:04 PM CET

The associated perl script was very helpful, now I'm cruising along with notmuch + mutt + offlineimap. For you Fedora (14+) users out there, the following perl dependencies were needed for notmuch-mutt: "perl-Email-Sender perl-MailTools"

These will pull in the following perl packages and spamassasin: http://pastebin.com/nkfB5bRQ

Comment by sadsfae Wed 09 Feb 2011 06:56:03 AM CET

Hi,

This is a great page! Integrating mutt with notmuch (and offlineimap) is fantastic. After doing this, one thing that I found I wanted was a way to tag and untag emails based on a selection made in mutt, i.e. tagging (in the notmuch sense) emails that have been tagged (in the mutt sense).

I can't write anything in perl so I can't contribute directly to your work, but was able to hack a small shell script together that does what I need. It is not pretty, but I thought I'd share it here. It looks like getting user input into mutt is not so easy, but using vi in this way seems to work.

# mutt-notmuch-tag.sh
# Simple and crude script to tag multiple messagess in mutt using notmuch.
grep "Message-ID" - > temp.txt
sed -e 's|Message-ID:.*<\(.*\)>.*|id:\1|' temp.txt > temp1.txt
sed '{:q;N;s/\n/ OR /g;t q}' temp1.txt > temp2.txt

## When vi opens up list the tagging commands, e.g. +tag1 -tag2
vi tags.txt

TAGS=`cat tags.txt`
SEARCHEXPR=`cat temp2.txt`

## For debugging
echo "notmuch tag $TAGS -- $SEARCHEXPR" >> temp2.txt

## Now apply the tags
notmuch tag $TAGS -- $SEARCHEXPR

# End of mutt-notmuch-tag.sh

Then in my .muttrc I have:

macro index <F7> "<tag-prefix><pipe-message>~/bin/mutt-notmuch-tag.sh<enter>" "Tag messages (using notmuch)"

Comment by David Mon 30 May 2011 06:39:09 PM CEST

On BSD systems the script fails because of gnu xargs and ln options. I changed the code to be pure perl.

--- mutt-notmuch.orig   2011-06-07 18:35:15.000000000 -0500
+++ mutt-notmuch    2011-06-07 18:40:51.000000000 -0500
@@ -1,4 +1,4 @@
-#!/usr/bin/perl -w
+#!/usr/bin/env perl -w
 #
 # mutt-notmuch - notmuch (of a) helper for Mutt
 #
@@ -12,6 +12,7 @@
 use warnings;

 use File::Path;
+use File::Basename;
 use Getopt::Long;
 use Mail::Internet;
 use Mail::Box::Maildir;
@@ -33,8 +34,12 @@
     my ($maildir, $query) = @_;

     empty_maildir($maildir);
-    system("notmuch search --output=files $query"
-      . " | xargs --no-run-if-empty ln -s -t $maildir/cur/");
+    my @filelist = `notmuch search --output=files $query`;
+    foreach(@filelist) {
+      chomp;
+      my $target = sprintf("$maildir/cur/%s", basename($_));
+      symlink($_, $target);
+    }
 }

 sub search_action($$@) {

Comment by Taylor Wed 08 Jun 2011 01:44:58 AM CEST

I think your script needs to live somewhere!

You're right!

I've now proposed it for inclusion into upstream notmuch. Upstream author seems positive about it and is preparing a contrib/ directory to include it and similar tools. As soon as that happens, I'll move maintenance of the script there.

The wishlist bug report where the discussion happened is Debian bug 628018. If all goes at planned, it will soon land in Debian as a new binary package "notmuch-mutt".

Cheers.

Comment by zack Fri 10 Jun 2011 11:02:40 AM CEST

On BSD systems the script fails because of gnu xargs and ln options. I changed the code to be pure perl.

Thanks a lot. Do you mind providing a patch against the current git repo?

I'll be happy to apply it if you do so, but note that at the moment it doesn't apply cleanly (as I've added in the meantime support for mailboxes containing spaces, which requires an extra sed pipe in between).

Many thanks in advance, Cheers.

Comment by zack Fri 10 Jun 2011 11:07:29 AM CEST
apart from 'qmutt' ?
Comment by yhager Thu 03 Nov 2011 06:50:40 PM CET

sure, but it has nothing which is mutt-notmuch-specific.

Search results are shown in a regular mailbox, so "go back" actually just means changing to another mailbox (which might happen to be the one you were before searching or not).

I'm using mutt-patched, so I move across mailboxes through the sidebar. In regular mutt you usually do that via the "c" keybinding.

Hope this helps.

Comment by zack Thu 03 Nov 2011 08:50:47 PM CET

Hi,

When I search I get lots of:

ln: failed to create symbolic link „/home/<username>/.cache/mutt_results/cur/<mail>: File exists

I'm guessing that this is due to my current setup, where I have a lot of symlinks to keep the maildrop sort rules very generic, but still conflate some folders (e.g. .debian.devel-announce -> .debian.devel). In the end it works, but the failures are annoying :)

Is this something that your script can handle or should it be handled in notmuch itself (assuming it makes sense to do so)?

Thanks,
Andrei

Comment by yetanotherpersonal [blogspot.com] Thu 15 Mar 2012 10:58:36 AM CET