/dev/blah

things i want to remember, things i want to share

Développeur Python et adepte Linux depuis 2005, Core développeur Kivy, passionné par beaucoup trop de choses.
Profil Github gpg signature bitcoin address

Entries tagged “offlineimap”

Mails, moving from offlineimap to mbsync (isync)

written by tshirtman, on 3/12/14 1:39 AM.

I’ve been mostly happy with the mail setup i documented some time ago, but i never was fully convinced by offlineimap, it was often slow, sometime blocking, and felt a bit unreliable. Last time i went out looking for a replacement, there wasn’t much of them around, and although i had read a bit about mbsync, it didn’t seem nearly as commonly used as offlineimap, the voices praising it were few.

Discussing my issues with offlineimap at work the other day, i discovered that a co-worker had been using mbsync for his setup for years, and seemed to have zero issues with it. So i decided to give it a whirl.

I based my configuration on information found in the arch linux wiki which is often a very nice source of information, even for users of other distributions, especially for users of uncommonly used, nerdy, pieces of software. I directly pointed my configuration to my existing Maildir, and after a few runs (the first one seemed to chock a bit on the number of mails to sync), it went well and have been running very reliably since.

I’m not running it into a crontab yet, i’m just keeping a terminal next to the mutt one, with a simple script like:

while true;
do; sleep 10m
mbsync -a; date
done

Which allows me to see when was the last sync, and what actually happens during the sync. I must say i’m really happy i don’t have to go forward and code my own IMAP synchronization tool, that would probably have been a very pointy experience.

Tip me if you like this :)

Mutt + offlineimap + notmuch + nottoomuch

written by tshirtman, on 3/17/13 4:00 PM.

Some time ago, I got tired of various bugs and the general slowness of thunderbird, and decided to try to go back to a more sensible setup. I’m a big CLI user, and trying mutt again made sense, even if my first experience with it two years ago had been inconclusive, I decided to do things right this time.

After some research, I concluded what I did wrong last time was to give to much responsibility to mutt itself, and that I should use it only for what it’s good at, displaying mails. So I went on to complete the various pieces of the puzzle.

Getting mails

For various reasons, even though my computer is always connected to the internet, I think it’s good idea to have a local copy of my mails, one of them is that although my ubiquitous mail provider has a “don’t be evil” moto, such a big company can still screw up, and some people have been trapped out of their mail in the past, if it happens to me, at least I have a backup.

So I set up offlineimap, as a way to retrieve emails to a local MailDir. Setting up offlineimap is not complex, you edit the .offlineimaprc file to declare your accounts, and it gets your mail each time you start it.

Here is the general structure of the file

[general]
# list of the configured accounts
accounts = Personal, Professional
fsync = False 

[Account Personal]
localrepository = Local
remoterepository = Remote
#autorefresh = 10 # no autorefresh, i run it one-shot each time
quick = 30
status_backend = sqlite
maxage = 900

[Repository Local]
type = Maildir
localfolders = /media/gabriel/stockage/Mail/Personal

[Repository Remote]
type = Gmail
remoteuser = example@gmail.com
realdelete = no
# google app-specific password
remotepass = 
ssl = yes
maxconnections = 5
holdconnectionopen = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
postsynchook = notmuch new

# then nearly the same thing for Professional and its 
# repositories

It’s easy, the general section reference to the account sections, and the account sections reference the repository sections, the local is where you store the files, the remote is where they are on the mail provider. If you use gmail (there, i said it) and 2factors authentication (which i recommend you to do) you can use an app-specific password here.

As there is a few gigs of mails, I decided to put them out of my home directory, you can put them wherever you want, as long as it’s mounted when offlineimap is started (heh).

Also, we tell it to export the mailbox names for mutt to be able to display them:

[mbnames]
enabled = yes
filename = ~/.mutt/mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

This works to show gmail’s tags.

I use cron to run offlineimap with the desired granularity

0 * * * * offlineimap

Displaying mails

This is mutt’s job, first we need to tell it where the mails are:

in muttrc

set mbox_type=Maildir
set folder=/media/gabriel/stockage/Mail

I tried to cut most configuration in separate files by usage:

source $HOME/.mutt/mailboxes
source $HOME/.mutt/folder_hooks
source $HOME/.mutt/macros
source $HOME/.mutt/colors
source $HOME/.mutt/sidebar
source $HOME/.mutt/statusbar
source $HOME/.mutt/pager
    source $HOME/.mutt/query

We’ll see them later (except mailboxes, which is generated by offlineimap as seen before).

Customize index view:

set index_format="%Z %{%a %d %b %Y} %3M %-15.15L │%s"

Then a few general options.

set realname = "Gabriel Pettier"

# initial box
set spoolfile=+Personal/INBOX

set header_cache =~/.mutt/cache/headers
set message_cachedir =~/.mutt/cache/bodies
set certificate_file =~/.mutt/certificates
set mailcap_path = $HOME/.mutt/mailcap

set move = no  #Stop asking to "move read messages to mbox"!

ignore headers *
unignore headers from to subject date cc

set sendmail_wait=-1
set wait_key = no        # shut up, mutt
set timeout = 3          # idle time before scanning
set mail_check = 0       # minimum time between scans
unset move               # gmail does that
set delete               # don't ask, just do
unset confirmappend      # don't ask, just do!
set quit                 # don't ask, just do!!
unset mark_old           # read/new is good enough for me
set smart_wrap

I can delete mails from INBOX, they are still in “All mails” (here “tous les messages”), so the effect is exactly the same as archiving them.

foder_hooks

folder-hook . 'unset trash'
folder-hook Personal source $HOME/.mutt/accounts/personnal
folder-hook Professional source $HOME/.mutt/accounts/Professional

both of these files being modified copies of accounts/dummy

unset from;
unset signature;
set from="Gabriel Pettier <gabriel.pettier@gmail.com>";
set smtp_url = "smtp://gabriel.pettier@smtp.gmail.com:587/"
set smtp_pass = ""
set trash="=[Gmail].Corbeille";
set mbox="=[Gmail].Tous les messages";
set postponed="=[Gmail].Brouillons"'
set signature="";

This is mostly for having the correct sending address and signature based on which mail box I’m in.

macros

    bind editor <Tab> complete-query
bind editor ^T complete

This is to have contact completion in to/cc/bcc inputs

macro index,pager c "<change-folder>?<toggle-mailboxes>" "open a 
different folder"

macro index p "<enter-command>set pager_index_lines=30"
macro index P "<enter-command>set pager_index_lines=0"

Using mutt with the sidebar patch, i don’t use these much.

bind index gg       first-entry
bind index G        last-entry

Faster vim-like navigation

bind index R        group-reply
bind index u        sync-mailbox
bind index <space>  collapse-thread
bind index _        collapse-all

bind compose p postpone-message

macro index,pager d "<save-message>=[Gmail].Corbeille<enter>"           "move message to the trash"
macro index,pager S "<save-message>=/[Gmail].Spam<enter>"                "mark message as spam"

Manage draft/trash/spam folders

colors

I kept things simple on this one

color header brightred default subject
color hdrdefault cyan default
color quoted green default
color status cyan default
color indicator red default
color normal white default
color tree cyan default

sidebar

This is only useful if you use the sidebar patch of mutt (mutt-sidebar on ubuntu).

# set up the sidebar, default visible
set sidebar_width=43
set sidebar_visible=yes
set sidebar_delim   = '  │'

# color of folders with new mail
color sidebar_new yellow default

# up/down to select next, prev folder
# right to open selected folder
bind index,pager <up> sidebar-prev
bind index,pager <down> sidebar-next
bind index,pager <right> sidebar-open

# I don't need these.  just for documentation purposes.  See below.
# sidebar-scroll-up
# sidebar-scroll-down

# b toggles sidebar visibility
macro index b '<enter-command>toggle sidebar_visible<enter>'
macro pager b '<enter-command>toggle sidebar_visible<enter>'

statusbar

I customized the status bar (at the bottom of the pager), to display more information.

set status_chars  = " *%A"
set status_format = "───[ Folder: %f ]─%V──[%r%m messages%?n? (%n new)?%?d? (%d to delete)?%?t? (%t tagged)? ]───%>─%?p?( %p postponed )?─── %l|%L"

%V shows if we currently have a filter (l) on the view, and %l/%L display the file size of the box, I’m not sure of the difference between both values, they are usually equal for me.

pager

This manage the display/navigation when a mail is open for reading.

set pager_index_lines = 10 # number of index lines to show
set pager_context = 3      # number of context lines to show
set pager_stop             # don't go to next message automatically
set menu_scroll            # scroll in menus
set tilde                  # show tildes like in vim
unset markers              # no ugly plus signs

set quote_regexp = "^( {0,4}[>|:#%]| {0,4}[a-z0-9]+[>|]+)+"
alternative_order text/plain text/enriched text/html

bind pager R  group-reply

# View attachments properly.
bind attach <return> view-mailcap

Also, the viewing of attachment files is managed using mailcap preferences.

query

This is to hook with nottoomuch-addresses and fixing the output format

set query_command="$HOME/.mutt/ntmw.sh %s"
set query_format="%t %-25.24n %a %e"

We’ll see the ntmw.sh script later, info on the query_format can be found here (took me quite some time to find how to fix the stupid column default width).

mailcap

#MS office documents
application/excel; $HOME/.mutt/mutt_bgrun libreoffice %s
application/ms-exc; $HOME/.mutt/mutt_bgrun libreoffice %s
application/msword; $HOME/.mutt/mutt_bgrun libreoffice %s
application/vnd.ms-exc; $HOME/.mutt/mutt_bgrun libreoffice %s
application/vnd.ms-excel; $HOME/.mutt/mutt_bgrun libreoffice %s
application/x-msexcel; $HOME/.mutt/mutt_bgrun libreoffice %s

I don’t get these often, but still nice to be able to open them

# Images
image/*; $HOME/.mutt/mutt_bgrun eog %s

# PDFs
application/pdf; $HOME/.mutt/mutt_bgrun evince %s

# HTML
text/html; $HOME/.mutt/mutt_bgrun firefox %s

text/*; vim %s

# Unidentified files
application/octet-stream; $HOME/.mutt/mutt_bgrun xdg-open %s

multipart/alternative; less %s

As you can see, I use the mutt_bgrun script so opening gui applications to display a file doesn’t block mutt, I found this script here, and it works well.

searching

Searching through mails is done using notmuch, notmuch is mainly an interface to the xapian search engine, it’s fast and reliable, you just need it configured and bound to a key in mutt to start a search.

Configuration (~/.notmuch-config):

[database]
path=/media/gabriel/stockage/Mail

[user]
name=Gabriel Pettier
primary_email=gabriel.pettier@gmail.com

[new]
tags=unread;inbox;
ignore=

[search]
exclude_tags=deleted;spam;

[maildir]
synchronize_flags=true

As configured before, offlineimap will start it after sync, so not much more to do (the first import will take some time, the next ones will be really fast).

Install notmuch-mutt (’apt-get install notmuch-mutt’ on ubuntu), this adds the F8 and F9 macros to search through your mails using a simple syntax (do F8 then help to get started), and then return the result as a pseudo mailbox you can manipulate as any mailbox, F9 allows you to get the entire thread the selected mail comes from. I usually use the sidebar to get back to the mailbox I’m interested in after.

Contacts completion

I didn’t have much success with the goobook script, so i searched more, and fund that there was a way to use notmuch to provide me all the addresses I know about, there are various scripts to do that, but nottoomuch-addresses.sh was hailed as the fastest one, so I gave it a try.

Get it from here and run it with –update to get your addresses list generated (again, first import will take some time, next ones much less).

However, it’s more designed to be used with emacs than with mutt, thus integration is not perfect, so I did a small wrapper:

ntmw.sh

#!/usr/bin/env sh
# nottoomuch-addresses.sh wrapper

$HOME/.mutt/nottoomuch-addresses.sh "$1" \
    |sed -s 's/\(.*\) \(<.*\)/\2\   \1/'\
    |sed -s 's/\"//g'\
    |sed -s '/buzz+.*/d'\

This reverses order of mail address and real name in the answer, and separates them with a tab (so mutt’s query-menu put them in different columns), it also removes “ caracters, as mutt adds some on its own, finally, I added a small filter to remove old adresses i won’t care about ever (generated by now-defunct google buzz).

So now when I type in a to/cc/bcc input, i get a list of all the addresses containing the typed input, I can select a few different ones with ‘t’ or just type enter to have the selected one inserted, the only issue i have with it is to insert multiple different addresses, once you have inserted an address, the search doesn’t work, the solution is to go to the beggining of the line again (home key), and to insert the new name, the search will now work, so you just have to do your additions in reverse order.

Conclusion

This may be not the perfect setup (yet), but it works great for me, I’m confident my mails are secure and easy to search, I have great (and fast!) contact completion, and can type my mails in vim. Finally, adding a new mailbox (for work or whatever) is easy thanks to offlineimap. I still go to gmail interface to setup filters, as I prefer these things to be server side, and i don’t see how to do these configuration from anything client-side, but it’s not an everyday operation, so no hurry with that.

Tip me if you like this :)

Tags

#FIXME 3G absurd ad_sense alterway aléatoire android animation anonymity atheism bach backlog bash bitcoins blog blogging boldness book books boulot bricolage bépo C canvas captcha captures carte SD censure christianity chroot CLI cli cloudwatt code colors comfort zone command line community company life conferences contest copwatch copwatchnord-idf core-devs cours ct705 culture deb debian debug deformation dehors dessin dev distribute distribution débutant déploiement développement ebooks eeepad eeepc effect ego empty en escher event firefly flappy bird flask fosdem foss fr fun game garden gdb geek culture git github goals graphics grrr gödel hack hackathon hacked hacking health hooks i3 images IMAP inspirational install isync java jeu jeu video jinja2 jni keyboard keynav kivy kv lame learning lib libre life linux lol macosx magnet mail mailing-list mails maths mbsync meetings memory leak mesh meta mint mirroir MIT module motivational mouse museomix mutt nexus7 no-mouse notmuch nottoomuch offlineimap onycroit opencourseware osc packaging paris passphrase password patch pentacatyl people perte de données ping pip planning plugin positioning pr procrastination programmation progress project projet property proudhon proxy psf publisher/consumer pull-down pygame pyjnius pypi python pythonar qtile raid rapsberry pi reading recorder references release religion responsive review reviews réseau réseaux sociaux résurection salon screenshots script self service shows shutil shyness sizing solib sortie sousous!!! spam spritz stash status systeme système templating terminal texture texture coordinates Thomas Paine thread thème tiling time time management. tip tips tools transformer tutorial tv twitter typematrix typing ubuntu ubuntu-fr ultimate-smash-friends unity update upload images useless usf utils value VDM video vie/mort vim virtualenv visite widget windows wm wmii work workflow workflow. zine études