Given that the main development workflow for most kernel maintainers is
with email, I spend a lot of time in my email client. For the past few
decades I have used (mutt),
but every once in a while I look around to see if there is anything else
out there that might work better.
One project that looks promising is (aerc)
which was started by (Drew DeVault).
It is a terminal-based email client written in Go, and relies on a lot
of other go libraries to handle a lot of the “grungy” work in dealing
with imap clients, email parsing, and other fun things when it comes to
free-flow text parsing that emails require.
aerc isn’t in a usable state for me just yet, but Drew asked if I could
document exactly how I use an email client for my day-to-day workflow to
see what needs to be done to aerc to have me consider switching.
Note, this isn’t a criticism of mutt at all. I love the tool, and spend
more time using that userspace program than any other. But as anyone
who knows email clients, they all suck, it’s just that mutt sucks less
than everything else (that’s literally their motto)
I did a
(basic overview of how I apply patches to the stable kernel trees quite a few years ago)
but my workflow has evolved over time, so instead of just writing a
private email to Drew, I figured it was time to post something showing
others just how the sausage really is made.
Anyway, my email workflow can be divided up into 3 different primary
things that I do:
- basic email reading, management, and sorting
- reviewing new development patches and applying them to a source
repository.
- reviewing potential stable kernel patches and applying them to a
source repository.
Given that all stable kernel patches need to already be in Linus’s
kernel tree first, the workflow of the how to work with the stable tree
is much different from the new patch workflow.
Basic email reading
All of my email ends up in either two “inboxes” on my local machine.
One for everything that is sent directly to me (either with To: or Cc:)
as well as a number of mailing lists that I ensure I read all messages
that are sent to it because I am a maintainer of those subsystems (like
(USB), or
(stable)).
The second inbox consists of other mailing lists that I do not read all
messages of, but review as needed, and can be referenced when I need to
look something up. Those mailing lists are the “big” linux-kernel
mailing list to ensure I have a local copy to search from when I am
offline (due to traveling), as well as other “minor” development mailing
lists that I like to keep a copy locally like linux-pci, linux-fsdevel,
and a few other smaller vger lists.
I get these maildir folders synced with the mail server using
(mbsync) which works really well and is much
faster than using (offlineimap), which I used
for many many years ends up being really slow for when you do not live on the
same continent as the mail server.
(Luis’s)
recent post of switching to mbsync finally pushed me to take the time to
configure it all properly and I am glad that I did.
Let’s ignore my “lists” inbox, as that should be able to be read by any
email client by just pointing it at it. I do this with a simple alias:
alias muttl='mutt -f ~/mail_linux/'
which allows me to type muttl at any command line to instantly bring
it up:
What I spend most of the time in is my “main” mailbox, and that is in a
local maildir that gets synced when needed in ~/mail/INBOX/. A simple
mutt on the command line brings this up:
Yes, everything just ends up in one place, in handling my mail, I prune
relentlessly. Everything ends up in one of 3 states for what I need to
do next:
- not read yet
- read and left in INBOX as I need to do something “soon” about it
- read and it is a patch to do something with
Everything that does not require a response, or I’ve already responded
to it, gets deleted from the main INBOX at that point in time, or saved
into an archive in case I need to refer back to it again (like mailing
list messages).
That last state makes me save the message into one of two local
maildirs, todo and stable. Everything in todo is a new patch that
I need to review, comment on, or apply to a development tree.
Everything in stable is something that has to do with patches that
need to get applied to the stable kernel tree.
Side note, I have scripts that run frequently that email me any patches
that need to be applied to the stable kernel trees, when they hit
Linus’s tree. That way I can just live in my email client and have
everything that needs to be applied to a stable release in one place.
I sweep my main INBOX ever few hours, and sort things out either quickly
responding, deleting, archiving, or saving into the todo or stable
directory. I don’t achieve a constant “inbox zero”, but if I only have
40 or so emails in there, I am doing well.
So, for this main workflow, I need an easy way to:
- filter the INBOX by a pattern so that I only see one “type” of message at a
time (more below)
- read an email
- write an email
- respond to existing email, and use vim as an editor as my hands have those
key bindings burned into them.
- delete an email
- save an email to one of two mboxes with a press of a few keystrokes
- bulk delete/archive/save emails all at once
These are all tasks that I bet almost everyone needs to do all the time,
so a tool like aerc should be able to do that easily.
A note about filtering. As everything comes into one inbox, it is
easier to filter that mbox based on things so I can process everything
at once.
As an example, I want to read all of the messages sent to the linux-usb
mailing list right now, and not see anything else. To do that, in mutt,
I press l (limit) which brings up a prompt for a filter to apply to
the mbox. This ability to limit messages to one type of thing is really
powerful and I use it in many different ways within mutt.
Here’s an example of me just viewing all of the messages that are sent
to the linux-usb mailing list, and saving them off after I have read
them:
This isn’t that complex, but it has to work quickly and well on
mailboxes that are really really big. As an example, here’s me opening
my “all lists” mbox and filtering on the linux-api mailing list messages
that I have not read yet. It’s really fast as mutt caches lots of
information about the mailbox and does not require reading all of the
messages each time it starts up to generate its internal structures.
All messages that I want to save to the todo directory I can do with a
two keystroke sequence, .t which saves the message there automatically
Again, that’s a binding I set up years ago, , jumps to the specific
mbox, and . copies the message to that location.
Now you see why using mutt is not exactly obvious, those bindings are
not part of the default configuration and everyone ends up creating
their own custom key bindings for whatever they want to do. It takes a
good amount of time to figure this out and set things up how you want,
but once you are over that learning curve, you can do very complex
things easily. Much like an editor (emacs, vim), you can configure them
to do complex things easily, but getting to that level can take a lot of
time and knowledge. It’s a tool, and if you are going to rely on it,
you should spend the time to learn how to use your tools really well.
Hopefully aerc can get to this level of functionality soon. Odds are
everyone else does something much like this, as my use-case is not
unusual.
Now let’s get to the unusual use cases, the fun things:
Development Patch review and apply
When I decide it’s time to review and apply patches, I do so by
subsystem (as I maintain a number of different ones). As all pending
patches are in one big maildir, I filter the messages by the subsystem I
care about at the moment, and save all of the messages out to a local
mbox file that I call s (hey, naming is hard, it gets worse, just
wait…)
So, in my linux/work/ local directory, I keep the development trees for
different subsystems like usb, char-misc, driver-core, tty, and
staging.
Let’s look at how I handle some staging patches.
First, I go into my ~/linux/work/staging/ directory, which I will stay
in while doing all of this work. I open the todo mbox with a quick ,t
pressed within mutt (a macro I picked from somewhere long ago, I don’t
remember where…), and then filter all staging messages, and save them
to a local mbox with the following keystrokes:
mutt
,t
l staging
T
s ../s
Yes, I could skip the l staging step, and just do T staging instead
of T, but it’s nice to see what I’m going to save off first before
doing so:
Now all of those messages are in a local mbox file that I can open with
a single keystroke, ’s’ on the command line. That is an alias:
alias s='mutt -f ../s'
I then dig around in that mbox, sort patches by driver type to see
everything for that driver at once by filtering on the name and then
save those messages to another mbox called ‘s1’ (see, I told you the
names got worse.)
s
l erofs
T
s ../s1
I have lots of local mbox files all “intuitively” named ‘s1’, ‘s2’, and
‘s3’. Of course I have aliases to open those files quickly:
alias s1='mutt -f ../s1'
alias s2='mutt -f ../s2'
alias s3='mutt -f ../s3'
I have a number of these mbox files as sometimes I need to filter even
further by patch set, or other things, and saving them all to different
mboxes makes things go faster.
So, all the erofs patches are in one mbox, let’s open it up and review
them, and save the patches that look good enough to apply to another
mbox:
Turns out that not all patches need to be dealt with right now (moving
erofs out of the staging directory requires other people to review it,
so I just save those messages back to the todo mbox:
Now I have a single patch that I want to apply, but I need to add some
acks from the maintainers of erofs provided. I do this by editing the
“raw” message directly from within mutt. I open the individual messages
from the maintainers, cut their reviewed-by line, and then edit the
original patch and add those lines to the patch:
Some kernel maintainers right now are screaming something like “Automate
this!”, “Patchwork does this for you!”, “Are you crazy?” Yeah, this is
one place that I need to work on, but the time involved to do this is
not that much and it’s not common that others actually review patches
for subsystems I maintain, unfortunately.
The ability to edit a single message directly within my email client is
essential. I end up having to fix up changelog text, editing the
subject line to be correct, fixing the mail headers to not do foolish
things with text formats, and in some cases, editing the patch itself
for when it is corrupted or needs to be fixed (I want a Linkedin skill
badge for “can edit diff files by hand and have them still work”)
So one hard requirement I have is “editing a raw message from within the
email client.” If an email client can not do this, it’s a non-starter
for me, sorry.
So we now have a single patch that needs to be applied to the tree. I
am already in the ~/linux/work/staging/ directory, and on the correct
git branch for where this patch needs to go (how I handle branches and
how patches move between them deserve a totally different blog post…)
I can apply this patch in one of two different ways, using
git am -s ../s1 on the command line, piping the whole mbox into git
and applying the patches directly, or I can apply them within mutt
individually by using a macro.
When I have a lot of patches to apply, I just pipe the mbox file to
git am -s as I’m comfortable with that, and it goes quick for multiple
patches. It also works well as I have lots of different terminal
windows open in the same directory when doing this and I can quickly
toggle between them.
But we are talking about email clients at the moment, so here’s me
applying a single patch to the local git tree:
All it took was hitting the L key. That key is set up as a macro in
my mutt configuration file with a single line:
macro index L '| git am -s'\n
This macro pipes the output of the current message to git am -s.
The ability of mutt to pipe the current message (or messages) to
external scripts is essential for my workflow in a number of different
places. Not having to leave the email client but being able to run
something else with that message, is a very powerful functionality, and
again, a hard requirement for me.
So that’s it for applying development patches. It’s a bunch of the same
tasks over and over:
- collect patches by a common theme
- filter the patches by a smaller subset
- review them manually and responding if there are problems
- saving “good” patches off to apply
- applying the good patches
- jump back to the first step
Doing that all within the email program and being able to quickly get
in, and out of the program, as well as do work directly from the email
program, is key.
Of course I do a “test build and sometimes test boot and then push git
trees and notify author that the patch is applied” set of steps when
applying patches too, but those are outside of my email client workflow
and happen in a separate terminal window.
Stable patch review and apply
The process of reviewing patches for the stable tree is much like the
development patch process, but it differs in that I never use ‘git am’
for applying anything.
The stable kernel tree, while under development, is kept as a series of
patches that need to be applied to the previous release. This series of
patches is maintained by using a tool called
(quilt).
Quilt is very powerful and handles sets of patches that need to be
applied on top of a moving base very easily. The tool was based on a
crazy set of shell scripts written by Andrew Morton a long time ago, and
is currently maintained by Jean Delvare and has been rewritten in perl
to make them more maintainable. It handles thousands of patches easily
and quickly and is used by many developers to handle kernel patches for
distributions as well as other projects.
I highly recommend it as it allows you to reorder, drop, add in the
middle of the series, and manipulate patches in all sorts of ways, as
well as create new patches directly. I do this for the stable tree as
lots of times we end up dropping patches from the middle of the series
when reviewers say they should not be applied, adding new patches where
needed as prerequisites of existing patches, and other changes that with
git, would require lots of rebasing.
Rebasing a git does not work for when you have developers working “down”
from your tree. We usually have the rule with kernel development that
if you have a public tree, it never gets rebased otherwise no one can
use it for development.
Anyway, the stable patches are kept in a quilt series in a repository
that is kept under version control in git (complex, yeah, sorry.) That
queue can always be found
(here).
I do create a linux-stable-rc git tree that is constantly rebased
based on the stable queue for those who run test systems that can not
handle quilt patches. That tree is found
(here)
and should not ever be used by anyone for anything other than automated
testing. See
(this email
for a bit more explanation of how these git trees should, and should
not, be used.
With all that background information behind us, let’s look at how I take
patches that are in Linus’s tree, and apply them to the current stable
kernel queues:
First I open the stable mbox. Then I filter by everything that has
upstream in the subject line. Then I filter again by alsa to only
look at the alsa patches. I look at the individual patches, looking at
the patch to verify that it really is something that should be applied
to the stable tree and determine what order to apply the patches in
based on the date of the original commit.
I then hit F to pipe the message to a script that looks up the
Fixes: tag in the message to determine what stable tree, if any, the
commit that this fix was contained in.
In this example, the patch only should go back to the 4.19 kernel tree,
so when I apply it, I know to stop at that place and not go further.
To apply the patch, I hit A which is another macro that I define in my
mutt configuration
macro index A |'~/linux/stable/apply_it_from_email'\n
macro pager A |'~/linux/stable/apply_it_from_email'\n
It is defined “twice” as you can have different key bindings when you
are looking at mailbox’s index of all messages from when you are looking
at the contents of a single message.
In both cases, I pipe the whole email message to my
apply_it_from_email
script.
That script digs through the message, finds the git commit id of the
patch in Linus’s tree, then runs a different script that takes the
commit id, exports the patch associated with that id, edits the message
to add my signed-off-by to the patch as well as dropping me into my
editor to make any needed tweaks that might be needed (sometimes files
get renamed so I have to do that by hand, and it gives me one final
change to review the patch in my editor which is usually easier than in
the email client directly as I have better syntax highlighting and can
search and review the text better.
If all goes well, I save the file and the script continues and applies
the patch to a bunch of stable kernel trees, one after another, adding
the patch to the quilt series for that specific kernel version. To do
all of this I had to spawn a separate terminal window as mutt does fun
things to standard input/output when piping messages to a script, and I
couldn’t ever figure out how to do this all without doing the extra
spawn process.
Here it is in action, as a video as
(asciinema) can’t show multiple windows at the
same time.
Once I have applied the patch, I save it away as I might need to refer
to it again, and I move on to the next one.
This sounds like a lot of different steps, but I can process a lot of
these relatively quickly. The patch review step is the slowest one here,
as that of course can not be automated.
I later take those new patches that have been applied and run kernel
build tests and other things before sending out emails saying they have
been applied to the tree. But like with development patches, that
happens outside of my email client workflow.
Bonus, sending email from the command line
In writing this up, I remembered that I do have some scripts that use
mutt to send email out. I don’t normally use mutt for this for patch
reviews, as I use other scripts for that (ones that eventually got
turned into git send-email), so it’s not a hard requirement, but it is
nice to be able to do a simple:
mutt -s "${subject}" "${address}" < ${msg} >> error.log 2>&1
from within a script when needed.
Thunderbird also can do this, I have used:
thunderbird --compose "to='${address}',subject='${subject}',message=${msg}"
at times in the past when dealing with email servers that mutt can not
connect to easily (i.e. gmail when using oauth tokens).
Summary of what I need from an email client
So, to summarize it all for Drew, here’s my list of requirements for me
to be able to use an email client for kernel maintainership roles:
- work with local mbox and maildir folders easily
- open huge mbox and maildir folders quickly.
- custom key bindings for any command. Defaults that are sane is
always good, but everyone is used to a previous program and training
fingers can be hard.
- create new key bindings for common tasks (like save a message to a
specific mbox)
- easily filter messages based on various things. Full regexes are
not needed, see the PATTERNS section of ‘man muttrc’ for examples of
what people have come up with over the years as being needed by an
email client.
- when sending/responding to an email, bring it up in the editor of my
choice, with full headers. I know aerc already uses vim for this,
which is great as that makes it easy to send patches or include
other files directly in an email body
- edit a message directly from the email client and then save it back
to the local mbox it came from
- pipe the current message to an external program
That’s what I use for kernel development.
Oh, I forgot:
- handle gpg encrypted email. Some mailing lists I am on send
everything encrypted with a mailing list key which is needed to both
decrypt the message and to encrypt messages sent to the list. SMIME
could be used if GPG can’t work, but both of them are probably
equally horrid to code support for, so I recommend GPG as that’s
probably used more often.
Bonus things that I have grown to rely on when using mutt is:
- handle displaying html email by piping to an external tool like w3m
- send a message from the command line with a specific subject and a
specific attchment if needed.
- specify the configuration file to use as a command line option. It
is usually easier to have one configuration file for a “work”
account, and another one for a “personal” one, with different email
servers and settings provided for both.
- configuration files that can include other configuration files.
Mutt allows me to keep all of my “core” keybindings in one config
file and just specific email server options in separate config
files, allowing me to make configuration management easier.
If you have made it this far, and you aren’t writing an email client,
that’s amazing, it must be a slow news day, which is a good thing. I
hope this writeup helps others to show them how mutt can be used to
handle developer workflows easily, and what is required of a good email
client in order to be able to do all of this.
Hopefully other email clients can get to state where they too can do all
of this. Competition is good and maybe aerc can get there someday.
