lore+lei: part 2, now with IMAP
This is the second installment in the series where we're looking at using the public-inbox
lei tool for interacting with remote mailing list archives such as lore.kernel.org. In the previous article we looked at delivering your search results locally, and today let's look at doing the same, but with remote IMAP folders. For feedback, send a follow-up to this message on the workflows list:
For our example query today, we'll do some stargazing. The following will show us all mail sent by Linus Torvalds:
f:torvalds AND rt:1.month.ago..
I'm mostly using it because it's short, but you may want to use something similar if you have co-maintainer duties and want to automatically receive a copy of all mail sent by your fellow subsystem maintainers.
Note on saving credentials
When accessing IMAP folders,
lei will require a username and password. Unless you really like typing them in manually every time you run
lei up, you will probably want to have them cached on your local system.
Lei will defer to
git-credential-helper for this purpose, so if you haven't already set this up, you will want to do that now.
The two commonly used credential storage backends on Linux are “libsecret” and “store”:
libsecret is the preferred mechanism, as it will work with your Desktop Environment's keyring manager to store the credentials in a relatively safe fashion (encrypted at rest).
store should only be used if you don't have any other option, as it will record the credentials without any kind of encryption in the
~/.git-credentialsfile. However, if nothing else is working for you and you are fairly confident in the security of your system, it's something you can use.
Simply run the following command to configure the credential helper globally for your environment:
git config --global credential.helper libsecret
For more in-depth information about this topic, see
Getting your IMAP server ready
Before you start, you should get some information about your IMAP server, such as your login information. For my examples, I'm going to use Gmail, Migadu, and a generic Dovecot IMAP server installation, which should hopefully cover enough ground to be useful for the vast majority of cases.
What you will need beforehand:
- the IMAP server hostname and port (if it's not 993)
- the IMAP username
- the IMAP password
It will also help to know the folder hierarchy. Some IMAP servers create all subfolders below INBOX, while others don't really care.
We happen to be running Dovecot on mail.codeaurora.org, so I'm going to use it as my “generic Dovecot” system and run the following command:
lei q -I https://lore.kernel.org/all/ -d mid \ -o imaps://mail.codeaurora.org/INBOX/torvalds \ <<< 'f:torvalds AND rt:1.month.ago..'
<<< bit above is a Bash-ism, so if you're using a different shell, you can use the POSIX-compliant heredoc format instead:
lei q -I https://lore.kernel.org/all/ -d mid \ -o imaps://mail.codeaurora.org/INBOX/torvalds <<EOF f:torvalds AND rt:1.month.ago.. EOF
The first time you run it, you should get a
password: prompt, but after that the credentials should be cached and no longer required on each repeated access to the same imaps server.
NOTE: some IMAP servers use the dot
. instead of the slash
/ for indicating folder hierarchy, so if
INBOX/torvalds is not working for you, try
Refreshing and subscribing to IMAP folders
If the above command succeeded, then you should be able to view the IMAP folder in your mail client. If you cannot see
torvalds in your list of available folders, then you may need to refresh and/or subscribe to the newly created folder. The process will be different for every mail client, but it shouldn't be too hard to find.
The same with Migadu
If you have a linux.dev account (see https://korg.docs.kernel.org/linuxdev.html), then you probably already know that we ask you not to use your account for subscribing to busy mailing lists. This is due to Migadu imposing soft limits on how much incoming email is allowed for each hosted domain — so using
lei + IMAP is an excellent alternative.
To set this up with your linux.dev account (or any other account hosted on Migadu), use the following command:
lei q -I https://lore.kernel.org/all/ -d mid \ -o imaps://imap.migadu.com/lei/torvalds \ <<< 'f:torvalds AND rt:1.month.ago..'
Again, you will need to subscribe to the new
lei/torvalds folder to see it in your mail client.
The same with Gmail
If you are a Gmail user and aren't already using IMAP, then you will need to jump through a few additional hoops before you are able to get going. Google is attempting to enhance the security of your account by restricting how much can be done with just your Google username and password, so services like IMAP are not available without setting up a special “app password” that can only be used for mail access.
Enabling app passwords requires that you first enable 2-factor authentication, and then generate a random app password to use with IMAP. Please follow the process described in the following Google document: https://support.google.com/mail/answer/185833
Once you have the app password for use with IMAP, you can use
lei and imaps just like with any other IMAP server:
lei q -I https://lore.kernel.org/all/ -d mid \ -o imaps://imap.gmail.com/lei/torvalds \ <<< 'f:torvalds AND rt:1.month.ago..'
It requires a browser page reload for the folder to show up in your Gmail web UI.
Automating lei up runs
If you're setting up IMAP access, then you probably want IMAP updates to happen behind the scenes without your direct involvement. All you need to do is periodically run
lei up --all (plus
-q if you don't want non-critical output).
If you're just getting started, then you can set up a simple
screen session with a
watch command at a 10-minute interval, like so:
watch -n 600 lei up --all
You can then detach from the screen terminal and let that command continue behind the scenes. The main problem with this approach is that it won't survive a system reboot, so if everything is working well and you want to make the command a bit more permanent, you can set up a systemd user timer.
Here's the service file to put in
[Unit] Description=lei up --all service ConditionPathExists=%h/.local/share/lei [Service] Type=oneshot ExecStart=/usr/bin/lei up --all -q [Install] WantedBy=mail.target
And the timer file to put in
[Unit] Description=lei up --all timer ConditionPathExists=%h/.local/share/lei [Timer] OnUnitInactiveSec=10m [Install] WantedBy=default.target
Enable the timer:
systemctl --user enable --now lei-up-all.timer
You can use
journalctl -xn to view the latest journal messages and make sure that the timer is running well.
CAUTION: user timers only run when the user is logged in. This is not actually that bad, as your keyring is not going to be unlocked unless you are logged into the desktop session. If you want to run
lei up as a background process on some server, you should set up a system-level timer and use a different git-credential mechanism (e.g.
store) — and you probably shouldn't do this on a shared system where you have to worry about your account credentials being stolen.
Coming up next
In the next installment we'll look at some other part of lei and public-inbox... I haven't yet decided which. :)