Konstantin Ryabitsev

kernel.org administrator

Linux development depends on the ability to send and receive emails. Unfortunately, it is common for corporate gateways to post-process both outgoing and incoming messages with the purposes of adding lengthy legal disclaimers or performing anti-phishing link quarantines, both of which interferes with regular patch flow.

While it is possible to configure free options like GMail to work well with sending and receiving patches, Google services may not be available in all geographical locales — or there may be other reasons why someone may prefer not to have a gmail.com address.

For this reason, we have partnered with Migadu to provide a mail hosting service under the linux.dev domain. If you're a Linux subsystem maintainer or reviewer and you need a mailbox to do your work, we should be able to help you out.

We hope to expand the service to include other kernel developers in the near future.

Please see our service documentation page for full details.

asciicast

One of the side-effects of the recent UMN Affair has been renewed scrutiny of the kernel development process that continues to rely on patches sent via email. This prompted me to revisit my end-to-end patch attestation work and get it to the point where I consider it to be both stable for day-to-day work and satisfactory from the point of view of underlying security and usability.

Goals of the project

These were the goals at the outset:

  • make it opt-in and don't interfere with existing tooling and workflows
  • be as behind-the-scenes and non-intrusive as possible
  • be simple and easy to understand, explain, and audit

I believe the proposed solution hits all of these points:

  • the implementation is very similar to DKIM and uses email headers for cryptographic attestation of all relevant content (“From:” and “Subject:” headers, plus the message body). Any existing tooling will simply ignore the unrecognized header.
  • cryptographic signing is done via a git hook invoked automatically by git-send-email (sendemail-validate), so it only needs to be set up once and doesn't require remembering to do any extra steps
  • the library doing the signing is only a few hundred lines of Python code and reuses the DKIM standard for most of its logic

Introducing patatt

The library is called “patatt” (for Patch Attestation, obviously), and can be installed from PyPi:

  • pip install --user patatt

It only requires PyNaCl (Python libsodium bindings), git, and gnupg (if signing with a PGP key). The detailed principles of operation are described on the PyPi project page, so I will not duplicate them here.

The screencast linked above shows patatt in action from the point of view of a regular patch contributor.

If you have an hour or so, you can also watch my presentation to the Digital Identity Attestation WG:

Youtube video

Support in b4

Patatt is fully supported starting with version 0.7.0 of b4 — here it is in action verifying a patch from Greg Kroah-Hartman:

$ b4 am 20210527101426.3283214-1-gregkh@linuxfoundation.org
[...]
---
  ✓ [PATCH] USB: gr_udc: remove dentry storage for debugfs file
  ---
  ✓ Signed: openpgp/gregkh@linuxfoundation.org
  ✓ Signed: DKIM/linuxfoundation.org
---
Total patches: 1
---
[...]

As you see above, b4 verified that the DKIM header was valid and that the PGP signature from Greg Kroah-Hartman passed as well, giving double assurance that the message was not modified between leaving Greg's computer and being checked on the end-system of the person retrieving the patch.

Keyring management

Patatt (and b4) also introduce the idea of tracking contributor public keys in the git repository itself. It may sound silly — how can the repository itself be a source of trusted keys? However, it actually makes a lot of sense and isn't any worse than any other currently used public key distribution mechanism:

  • git is already decentralized and can be mirrored to multiple locations, avoiding any single points of failure
  • all contents are already versioned and key additions/removals can be audited and “git blame’d”
  • git commits themselves can be cryptographically signed, which allows a small subset of developers to act as “trusted introducers” to many other contributors (mimicking the “keysigning” process)

Contributor public keys can be added either to the main branch itself, along with the project codebase (perhaps in the .keys toplevel subdirectory), or it can be managed in a dedicated ref, such as refs/meta/keyring). The latter can be especially useful for large projects where patches are collected by subsystem maintainers and then submitted as pull requests for inclusion into the mainline repository. Keeping the keyring in its own ref assures that it stays out of the way of regular development work but is still centrally managed and tracked.

Further work

I am hoping that people will now start using cryptographic attestation for the patches they send, however I certainly can't force anyone's hand. If you are a kernel subsystem maintainer or a core developer of some other project that relies on mailed-in patches for the submission and code review process, I hope that you will give this a try.

If you have any comments, concerns, or improvement suggestions, please reach out to the tools list.

We have recently announced the availability of our new mailing list platform that will eventually take on the duties currently performed by vger. Off the bat, there were a few questions about how it works under the hood — especially regarding DMARC-friendly cofiguration.

Under the hood

There is nothing fancy about the setup — mailing lists are managed by mlmmj, while all delivery operations are handled by Postfix. All outgoing mail is delivered via kernel.org mirror edge nodes (see the output of dig -t txt +short _spf.kernel.org if you are curious what they are), which is mostly done to speed up delivery by spreading out the queue across several systems.

When mlmmj writes to the archive directory of the mailing list, the message is immediately picked up by public-inbox-watch and appended to the public-inbox archive for that list. The archive is then replicated to lore.kernel.org (using grokmirror integration), which usually happens within 60 seconds. This replication is parallel to Postfix delivering mail to list subscribers, so is not dependent on the size of the mail queue. In theory, it shouldn't take longer than a few minutes for a message sent to a lists.linux.dev address to show up on lore.kernel.org. Similarly, messages should never go missing from the public-inbox archive if they got accepted by mlmmj for delivery (I know, famous last words).

Appeasing DMARC

It's a common misconception that mailing lists are somehow incompatible with DMARC. There are two key principles to follow:

  1. The Envelope-From should be that of the mailing list domain. For example, if I send an email to linux-staging, the envelope-from of the outgoing message will be changed to linux-staging+bounces-x@lists.linux.dev, with some subscriber bounce tracking information in place of x. This way, when MTAs are looking at DMARC for “konstantin@linuxfoundation.org”, the SPF check will be performed against “lists.linux.dev” instead of the domain of the original sender (“linuxfoundation.org”).

  2. There should be no changes to any of the existing message headers and no modifications to the message body. This is actually the part that generally trips up mailing list operators, as it is a long-standing practice to do two things when it comes to mailing lists: modify the subject to insert a terse list identifier (e.g. Subject: [linux-staging] [PATCH] ...) and append a footer to the message body with mailing list administrative info. Doing either of these things will almost certainly invalidate DKIM signatures and therefore cause the message to fail the DMARC check. It is correct to add proper List-Id/List-Subscribe/etc headers, though — and hopefully the domain of the original sender isn't misconfigured to include these headers into the DKIM signature (true story, saw this happen).

Following the above advice will work for nearly all cases except where a domain sets a DMARC policy, but the message is sent without a DKIM signature. If this happens, DMARC validators are supposed to use a kludgy “alignment” check where the envelope-from must match the From: header. In that particular case the messages we send out will fail DMARC checks, unfortunately. As far as I'm concerned, this is the fault of domain owners and is properly fixed by setting up proper DKIM signing and giving users a way to send outgoing mail via proper SMTP gateways.

(There is a way to work around this by rewriting the “From: “ header so that it matches the list domain as well, but let's just not go there, as rewriting the From: header is not an acceptable solution for lists working with code reviews.)

Here's a write-up I randomly found while writing this post that goes into some more detail regarding DMARC and mailing lists.

Why no ARC headers?

We don't currently add ARC headers — as far as I can tell, they aren't required for operating a mailing list that properly sets the envelope-from. In theory, using ARC signing may help with the “DMARC with no DKIM” corner-case above, but I'm not convinced this is worth the crazy header bloat. Who knows, I may change my mind about this in the future.

Parting words

In short, the best way to assure that a message sent via subspace.kernel.org is delivered to all subscribers is to send it from a domain that properly DKIM-signs all mail. If you run your own server, you can either set up OpenDKIM on your own (it's not complicated, honest), or you can pay some money to a company like Mailgun to do it for you.

Many people know that you can PGP-sign git objects — such as tags or commits themselves — but very few know of another attestation feature that git provides, which is signed git pushes.

Why sign git pushes? And how are they different from signed tags/commits?

Signed commits are great, but one thing they do not indicate is intent. For example, you could write some dangerous proof-of-concept code and push it into refs/heads/dangerous-do-not-use. You could even push it into some other fork hosted on a totally different server, just to make it clear that this is not production-ready code.

However, if your commits are PGP-signed, someone could take them and replay over any other branch in any other fork of your repository. To anyone checking the commit signatures, everything will look totally legitimate, as the actual commits are signed by you — never mind that they contain dangerous vulnerable code and were never intended to be pushed into something like refs/heads/next. At the very least, you will look reckless for pushing bad code, even though you were just messing around in a totally separate environment set up specifically for experimentation.

To help hedge against this problem, git provides developers a way to sign their actual pushes, as a means to attest “yes, I actually did intend to push these commits into this ref in this repository on this server, and here's my PGP signature to prove it.” When a push is signed, git will both check the signature it received against a trusted keyring and generate a “push certificate” that can be logged in something like a transparency log:

https://git.kernel.org/pub/scm/infra/transparency-logs/gitolite/git/1.git/plain/m?id=c06eebe4875d6103d580efcf8cd78cc9cc4b5192

Now, before you rush to enable signed pushes, please keep in mind that this functionality needs to first be enabled on the server side, and the vast majority of public git hosting forges do NOT have this turned on. Thankfully, git provides an if-asked setting, which will first check if the remote server supports signed pushes, and only generate the push certificate if the remote server accepts them. To enable this feature for yourself, simply add the following to your ~/.gitconfig:

[push]
    gpgSign = if-asked

Enabling on the server side

If you are running your own git server, then it is easy to enable this on the server side. Add the following either to each repository config file, or to /etc/gitconfig to enable it globally:

[receive]
    advertisePushOptions = true
    certNonceSeed = "<uniquerandomstring>"

You should set the certNonceSeed setting to some randomly generated long string that should be kept secret. It is combined with the timestamp to generate a one-time value (“nonce”) that the git client is required to sign and provides both a mechanism to prevent replay attacks and to offer proof that the certificate was generated for that specific server (though for others to verify this, they would need to know the value of the nonce seed).

Once you have this feature enabled, it is up to you what you do with the generated certificates. You can simply opt to record them, just like we do with our transparency log, or you can actually reject pushes that do not come with valid push certificates. I refer you to the git documentation and to our post-receive-activity-feed hook, which we use to generate the transparency log:

This week we made public all of our git commit logs, going back to 2013, in hopes to increase the transparency of high-importance kernel.org operations. All writes performed on public git repositories are now recorded in a public-inbox feed, which is immediately replicated to multiple worldwide servers. This is done with the goal do make it difficult for someone to make changes to any git repository hosted on kernel.org without it generating a verifiable, tamper-evident record.

The transparency logs are available at the following address:

https://git.kernel.org/pub/scm/infra/transparency-logs/gitolite/git/

You can read more detailed documentation here:

https://korg.docs.kernel.org/gitolite/transparency-log.html

You can have lore.kernel.org mailing lists delivered right into your inbox straight from the git archive (in fact, this will work for any public-inbox server, not just for lore.kernel.org). It's efficient and (optionally) preserves a full copy of entire list archives on your system — should you wish to keep them.

Note: this requires grokmirror-2.0.2+, as earlier versions do not come with the grok-pi-piper utility.

Installing grokmirror-2.0

Easiest is to install from pip:

pip install --user grokmirror~=2.0.2

You may have grokmirror available from your distro packages, too, but make sure it's version 2.0.2 or above.

Installing procmail

Procmail should be available with your distribution, so install it like any other package.

Configuring procmail

Procmail configuration can easily be a topic for a whole book in itself, but if you just want to have messages delivered into your inbox, all you have to do is create a ~/.procmailrc with the following contents:

DEFAULT=$HOME/Mail/

# Don't deliver duplicates sent to multiple lists
:0 Wh: .msgid.lock
| formail -D 8192 .msgid.cache

If your mailbox is not in ~/Mail, then you should adjust the above accordingly.

Configuring grokmirror

Create a ~/.config/lore.conf with the following contents. We'll use three lists as examples: git, linux-hardening, and linux-doc, but you'll obviously want to use the lists you care about. You can see which lists are available from https://lore.kernel.org/lists, or the exact git repositories on https://erol.kernel.org/.

[core]
toplevel = ~/.local/share/grokmirror/lore
log = ${toplevel}/grokmirror.log

[remote]
site = https://lore.kernel.org
manifest = https://lore.kernel.org/manifest.js.gz

[pull]
post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf
refresh = 300
include = /git/*
          /linux-hardening/*
          /linux-doc/*

The above assumes that you installed grokmirror with pip install --user. Now make the toplevel directory for the git repos:

$ mkdir -p ~/.local/share/grokmirror/lore

Configuring pi-piper

The last step is to create ~/.config/pi-piper.conf:

[DEFAULT]
pipe = /usr/bin/procmail
shallow = yes

The important bit here is shallow = yes. Public-inbox stores every mail message as a separate commit, so once a message is piped to procmail and delivered, we usually don't care about keeping a copy of that commit any more. If you set shallow = yes, pi-piper will prune all but the last successfully processed commit out of your local git copy by turning those repos into shallow git repositories. This helps to greatly save disk space, especially for large archives.

If you do want to keep full archives, then don't set shallow. You can change your mind at any time by running git fetch _grokmirror master --unshallow in each underlying git repository (you can find them in ~/.local/share/grokmirror/lore/).

You can also specify the shallow option per list:

[DEFAULT]
pipe = /usr/bin/procmail

[linux-hardening]
shallow = yes

Running grok-pull

You can now run grok-pull to get the initial repo copies. Note, that during the first run grokmirror will perform full clones even if you specified shallow = yes in the pi-piper config, so it may take some time for large archives like those for the git list. However, once the pi-piper hook runs, they will be repacked to almost nothing. Future versions of grokmirror may become smarter about this and perform shallow clones from the beginning.

During the initial pi-piper run, there will be no mail delivered, as it will just perform initial setup and make a note where the HEAD is pointing. If you run grok-pull again, two things may happen:

  1. There will be no changes and grok-pull will exit right away
  2. If there are changes, they will be fetched and the hook will deliver them to procmail (and to your inbox)

Running in the background

You can run grok-pull in the background, where it will check for updates as frequently as the refresh setting says (300 seconds in the example above).

You can either background it “the old way”:

grok-pull -o -c ~/.config/lore.conf &

Or the new way, using a systemd user service:

$ cat .config/systemd/user/grok-pull@.service

[Unit]
Description=Grok-pull service for %I
ConditionPathExists=%h/.config/%i.conf

[Service]
ExecStart=%h/.local/bin/grok-pull -o -c %h/.config/%i.conf
Type=simple
Restart=on-failure

[Install]
WantedBy=default.target

$ systemctl --user enable grok-pull@lore
$ systemctl --user start grok-pull@lore

If you make changes to ~/.config/lore.conf, for example to add new lists, you will need to restart the service:

$ systemctl --user restart grok-pull@lore

Combining with mbsync

You can totally combine this with mbsync and deliver into the same local inbox. As a perk, any messages injected from grokmirror will be uploaded to your remote imap mailbox. See this post from mcgrof about configuring mbsync:

Troubles

Email tools@linux.kernel.org if you have any trouble getting the above to work. The grok-pi-piper utility is fairly new, so it's entirely possible that it's full of bugs.

With git, a cryptographic signature on a commit provides strong integrity guarantees of the entire history of that branch going backwards, including all metadata and all contents of the repository, all the way back to the initial commit. This is possible because git records the hash of the previous commit in each next commit's metadata, creating an unbreakable cryptographic chain of records. If you can verify the cryptographic signature at the tip of the branch, you effectively verify that branch's entire history.

For example, let's take a look at linux.git, where the latest tag at the time of writing, v5.8-rc7, is signed by Linus Torvalds. (Tag signatures are slightly different from commit signatures — but in every practical sense they offer the same guarantees.)

If you have a git checkout of torvalds/linux.git, you are welcome to follow along.

$ git cat-file -p v5.8-rc7
object 92ed301919932f777713b9172e525674157e983d
type commit
tag v5.8-rc7
tagger Linus Torvalds <torvalds@linux-foundation.org> 1595798046 -0700

Linux 5.8-rc7
-----BEGIN PGP SIGNATURE-----

iQFSBAABCAA8FiEEq68RxlopcLEwq+PEeb4+QwBBGIYFAl8d8h4eHHRvcnZhbGRz
QGxpbnV4LWZvdW5kYXRpb24ub3JnAAoJEHm+PkMAQRiGd0sH/2iktYhMwPxzzpnb
eI3OuTX/mRn4vUFOfpx9dmGVleMfKkpbvnn3IY7wA62Qfv7J7lkFRa1Bd1DlqXfW
yyGTGDSKG5chiRCOU3s9ni92M4xIzFlrojyt/dIK2lUGMzUPI9FGlZRGQLKqqwLh
2syOXRWbcQ7e52IHtDSy3YBNveKRsP4NyqV+GxGiex18SMB/M3Pw9EMH614eDPsE
QAGQi5uGv4hPJtFHgXgUyBPLFHIyFAiVxhFRIj7u2DSEKY79+wO1CGWFiFvdTY4B
CbqKXLffY3iQdFsLJkj9Dl8cnOQnoY44V0EBzhhORxeOp71StUVaRwQMFa5tp48G
171s5Hs=
=BQIl
-----END PGP SIGNATURE-----

If we have Linus's key in our GnuPG keyring, we can easily verify that this tag is valid:

$ git verify-tag v5.8-rc7
gpg: Signature made Sun 26 Jul 2020 05:14:06 PM EDT
gpg:                using RSA key ABAF11C65A2970B130ABE3C479BE3E4300411886
gpg:                issuer "torvalds@linux-foundation.org"
gpg: Good signature from "Linus Torvalds <torvalds@kernel.org>" [unknown]
gpg:                 aka "Linus Torvalds <torvalds@linux-foundation.org>" [full]

The entire contents of this tag are signed, so this tells us that when Linus signed the tag, the “object hash” on his system was 92ed301919932f777713b9172e525674157e983d. But what exactly is that “object hash?” What are the contents that are hashed here? We can find out by asking git to tell us more about that object:

$ git cat-file -p 92ed301919932f777713b9172e525674157e983d
tree f16e3e4bcea2d875a17d2278ff67364b3277b10a
parent 1c8594b8427290c178c5d39885eacd9e41f68743
author Linus Torvalds <torvalds@linux-foundation.org> 1595798046 -0700
committer Linus Torvalds <torvalds@linux-foundation.org> 1595798046 -0700

Linux 5.8-rc7

The above contents in their entirety (slightly differently formatted) is what gives us the sha1 hash 92ed301919932f777713b9172e525674157e983d. So, thus far, we have unbroken cryptographic attestation from Linus's PGP signature to two other important bits about his git repository:

  • information about the state of his source code (tree)
  • information about the previous commit in the history (parent)
  • information about the author of the commit and the committer, which are the one and the same in this particular case
  • information about the date and time when the commit was made

Let's take a look a the tree line — what contents were hashed to arrive at that checksum? Let's ask git:

$ git cat-file -p f16e3e4bcea2d875a17d2278ff67364b3277b10a
100644 blob a0a96088c74f49a961a80bc0851a84214b0a9f83    .clang-format
100644 blob 43967c6b20151ee126db08e24758e3c789bcb844    .cocciconfig
100644 blob a64d219137455f407a7b1f2c6b156c5575852e9e    .get_maintainer.ignore
100644 blob 4b32eaa9571e64e47b51c43537063f56b204d8b3    .gitattributes
100644 blob d5f4804ed07cd36336a5e80f2a24e45104f902cf    .gitignore
100644 blob db4f2295bd9d792b47eb77aab179a9db0d968454    .mailmap
100644 blob a635a38ef9405fdfcfe97f3a435393c1e9cae971    COPYING
100644 blob 0787b5872906c8a92a63cde3961ed630e2ec93b6    CREDITS
040000 tree 37e1b4166d912d69738beca645d3d539da4bbf30    Documentation
[...]
040000 tree ba6955ee6228666d9ef117fdd45df2e53ba0e221    virt

This is the entirety of the top-level Linux kernel directory contents. The blob entries are sha1sum's of the actual file contents in that directory, so these are straightforward. Subdirectories are represented as other tree entries, which also consist of blob and tree records going all the way down to the last sublevel, which will only contain blobs.

So, tree f16e3e4bcea2d875a17d2278ff67364b3277b10a in the commit record is a checksum of other checksums and it allows us to verify that each and every file in linux.git is exactly the same as it was on Linus Torvalds' system when he created the commit. If any file is changed, the tree checksum would be different and the whole repository would be considered invalid, because the object hash would be different than in the commit.

Finally, if we look at the object mentioned in parent 1c8594b8427290c178c5d39885eacd9e41f68743, we will see that it is a hash of another commit, containing its own tree and parent records:

$ git cat-file -p 1c8594b8427290c178c5d39885eacd9e41f68743
tree d56de40028d9ecdbebfc2121fd1ce1213fa09fa2
parent 40c60ac32174f0c0c090cd31d0d1712f2478e689
parent ca9b31f6bb9c6aa9b4e5f0792f39a97bbffb8c51
author Linus Torvalds <torvalds@linux-foundation.org> 1595796417 -0700
committer Linus Torvalds <torvalds@linux-foundation.org> 1595796417 -0700
mergetag object ca9b31f6bb9c6aa9b4e5f0792f39a97bbffb8c51
[...]

If we cared to, we could walk each commit all the way back to the beginning of Linux git history, but we don't need to do that — verifying the checksum of the latest commit is sufficient to provide us all the necessary assurances about the entire history of that tree.

So, if we verify the signature on the tag and confirm that it matches the key belonging to Linus Torvalds, we will have strong cryptographic assurances that the repository on our disk is byte-for-byte the same as the repository on the computer belonging to Linus Torvalds — with all its contents and its entire history going back to the initial commit.

The difference between signed tags and signed commits is minimal — in the case of commit signing, the signature goes into the commit object itself. It is generally a good practice to PGP-sign commits, particularly in environments where multiple people can push to the same repository branch. Signed commits provide easy forensic proof of code origins (e.g. without commit signing Alice can fake a commit to pretend that it was actually authored by Bob). It also allows for easy verification in cases where someone wants to cherry-pick specific commits into their own tree without performing a git merge.

If you are looking to get started with git and PGP signatures, I can recommend my own Protecting Code Integrity guide, or its kernel-specific adaptation that ships with the kernel docs: Kernel Maintainer PGP Guide.

Obligatory note: sha1 is not considered sufficiently strong for hashing purposes these days, and this is widely acknowledged by the git development community. Significant efforts are under way to migrate git to stronger cryptographic hashes, but they require careful planning and implementation in order to minimize disruption to various projects using git. To my knowledge, there are no effective attacks against sha1 as used by git, and git developers have added further precautions against sha1 collision attacks in git itself, which helps buy some time until stronger hashing implementations are considered ready for real-world use.

For the past few weeks I've been working on a tool to fetch patches from lore.kernel.org and perform the kind of post-processing that is common for most maintainers:

  • rearrange the patches in proper order
  • tally up various follow-up trailers like Reviewed-by, Acked-by, etc
  • check if a newer series revision exists and automatically grab it

The tool started out as get-lore-mbox, but has now graduated into its own project called b4 — you can find it on git.kernel.org and pypi.

To use it, all you need to know is the message-id of one of the patches in the thread you want to grab. Once you have that, you can use the lore.kernel.org archive to grab the whole thread and prepare an mbox file that is ready to be fed to git-am:

$ b4 am 20200312131531.3615556-1-christian.brauner@ubuntu.com
Looking up https://lore.kernel.org/r/20200312131531.3615556-1-christian.brauner@ubuntu.com
Grabbing thread from lore.kernel.org
Analyzing 26 messages in the thread
Found new series v2
Will use the latest revision: v2
You can pick other revisions using the -vN flag
---
Writing ./v2_20200313_christian_brauner_ubuntu_com.mbx
  [PATCH v2 1/3] binderfs: port tests to test harness infrastructure
    Added: Reviewed-by: Kees Cook <keescook@chromium.org>
  [PATCH v2 2/3] binderfs_test: switch from /dev to a unique per-test mountpoint
    Added: Reviewed-by: Kees Cook <keescook@chromium.org>
  [PATCH v2 3/3] binderfs: add stress test for binderfs binder devices
    Added: Reviewed-by: Kees Cook <keescook@chromium.org>
---
Total patches: 3
---
 Link: https://lore.kernel.org/r/20200313152420.138777-1-christian.brauner@ubuntu.com
 Base: 2c523b344dfa65a3738e7039832044aa133c75fb
       git checkout -b v2_20200313_christian_brauner_ubuntu_com 2c523b344dfa65a3738e7039832044aa133c75fb
       git am ./v2_20200313_christian_brauner_ubuntu_com.mbx

As you can see, it was able to:

  • grab the whole thread
  • find the latest revision of the series (v2)
  • tally up the Reviewed-by trailers from Kees Cook and insert them into proper places
  • save all patches into an mbox file
  • show the commit-base (since it was specified)
  • show example git checkout and git am commands

Pretty neat, eh? You don't even need to know on which list the thread was posted — lore.kernel.org, through the magic of public-inbox, will try to find it automatically.

If you want to try it out, you can install b4 using:

pip install b4

(If you are wondering about the name, then you should click the following links: V'ger, Lore, B-4.)

The same, but now with patch attestation

On top of that, b4 also introduces support for cryptographic patch attestation, which makes it possible to verify that patches (and their metadata) weren't modified in transit between developers. This is still an experimental feature, but initial tests have been pretty encouraging.

I tried to design this mechanism so it fulfills the following requirements:

  • it must be unobtrusive and not pollute the mailing lists with attestation data
  • it must be possible to submit attestation after the patches were already sent off to the list (for example, from a different system, or after being asked to do so by the maintainer/reviewer)
  • it must not invent any new crypto or key distribution routines; this means sticking with PGP/GnuPG — at least for the time being

If you are curious about the technical details, I refer you to my original RFC where I describe the implementation.

If you simply want to start using it, then read on.

Submitting patch attestation

If you would like to submit attestation for a patch or a series of patches, the best time to do that is right after you use git send-email to submit your patches to the list. Simply run the following:

b4 attest *.patch

This will do the following:

  • create a set of 3 hashes per each patch (for the metadata, for the commit message, and for the patch itself)
  • add these hashes to a YAML-style document
  • PGP-sign the attestation document using the PGP key you set up with git
  • connect to mail.kernel.org:587 and send the attestation document to the signatures@kernel.org mailing list.

If you don't want to send that attestation right away, use the -n flag to simply generate the message and save it locally for review.

Verifying patch attestation

When running b4 am, the tool will automatically check if attestation is available by querying the signatures archive on lore.kernel.org. If it finds the attestation document, it will run gpg --verify on it. All of the following checks must pass before attestation is accepted:

  1. The signature must be “good” (signed contents weren't modified)
  2. The signature must be “valid” (not done with a revoked/expired key)
  3. The signature must be “trusted” (more on this below)

If all these checks pass, b4 am will show validation checkmarks next to the patches as it processes them:

$ b4 am 202003131609.228C4BBEDE@keescook
Looking up https://lore.kernel.org/r/202003131609.228C4BBEDE@keescook
Grabbing thread from lore.kernel.org
...
---
Writing ./v2_20200313_keescook_chromium_org.mbx
  [✓] [PATCH v2 1/2] selftests/harness: Move test child waiting logic
  [✓] [PATCH v2 2/2] selftests/harness: Handle timeouts cleanly
  ---
  [✓] Attestation-by: Kees Cook <keescook@chromium.org> (pgp: 8972F4DFDC6DC026)
---
Total patches: 2
---
...

These checkmarks give you assurance that all patches are exactly the same as when they were generated by the developer on their system.

Trusting on First Use (TOFU)

The most bothersome part of PGP is key management. In fact, it's the most bothersome part of any cryptographic attestation scheme — you either have to delegate your trust management to some shadowy Certification Authority, or you have to do a lot of decision making of your own when evaluating which keys to trust.

GnuPG tries to make it a bit easier by introducing the “Trust on First Use” (TOFU) model. The first time you come across a key, it is considered automatically trusted. If you suddenly come across a different key with the same identity on it, GnuPG will mark both keys as untrusted and let you decide on your own which one is “the right one.”

If you want to use the TOFU trust policy for patch attestation, you can add the following configuration parameter to your $HOME/.gitconfig:

[b4]
  attestation-trust-model = tofu

Alternatively, you can use the traditional GnuPG trust model, where you rely on cross-certification (“key signing”) to make a decision on which keys you trust.

Where to get help

If either b4 or patch attestation are breaking for you — or with any questions or comments — please reach out for help on the kernel.org tools mailing list:

  • tools@linux.kernel.org

If Greg KH ever writes a book about his work as the stable kernel maintainer, it should be titled “Everyone must upgrade” (or it could be a Dr. Who fanfic about Cybermen, I guess). Today, I'm going to borrow a leaf out of that non-existent book to loudly proclaim that all patch submissions must include base-commit info.

What is a base-commit?

When you submit a single patch or a series of patches to a kernel maintainer, there is one important piece of information that they need to know in order to properly apply it. Specifically, they need to know what was the state of your git tree at the time when you wrote that code. Kernel development moves very quickly and there is no guarantee that a patch written mid-January would still apply at the beginning of February, unless there were no significant changes to any of the files your patch touches.

To solve this problem, you can include a small one-liner in your patch:

base-commit: abcde12345

This tells the person reviewing your patch that, at the time when you wrote your code, the latest commit in the git repository was abcde12345. It is now easy for the maintainer to do the following:

git checkout -b incoming_patch abcde12345
git am incoming_patch.mbx

This will tell git to create a new branch using abcde12345 as the parent commit and apply your patches at that point in history, ensuring that there will be no failed or rejected hunks due to recent code changes.

After reviewing your submission the maintainer can then merge that branch back into master, resolving any conflicts during the merge stage (they are really good at that), instead of having to modify patches during the git am stage. This saves maintainers a lot of work, because if your patches require revisions before they can be accepted, they don't have to manually edit anything at all.

Automated CI systems

Adding base-commit info really makes a difference when automated CI systems are involved. With more and more CI tests written for the Linux kernel, maintainers are hoping to be able to receive test reports for submitted patches even before they look at them, as a way to save time and effort.

Unfortunately, if the CI system does not have the base-commit information to work with, it will most likely try to apply your patches to the latest master. If that fails, there will be no CI report, which means the maintainers will be that much less likely to look at your patches.

How to add base-commit to your submission

If you are using git-format-patch (and you really should be), then you can already automatically include the base commit information. The easiest way to do so is by using topical branches and git format-patch --base=auto, for example:

$ git checkout -t -b my-topical-branch master
Branch 'my-topical-branch' set up to track local branch 'master'.
Switched to a new branch 'my-topical-branch'

[perform your edits and commits]

$ git format-patch --base=auto --cover-letter -o outgoing/ master
outgoing/0000-cover-letter.patch
outgoing/0001-First-Commit.patch
outgoing/...

When you open outgoing/0000-cover-letter.patch for editing, you will notice that it will have the base-commit: trailer at the very bottom.

Once you have the set of patches to send, you should run them through checkpatch.pl to make sure that there are no glaring errors, and then submit them to the proper developers and mailing lists.

You can learn more by reading the submitting patches document, which now includes a section about base-commit info as well.

WriteFreely recently added support for creating and editing posts via the command-line wf tool and this functionality is available to all users at people.kernel.org.

On the surface, this is easy to use — you just need to write out a markdown-formatted file and then use wf publish myfile.md to push it into your blog (as draft). However, there are some formatting-related caveats to be aware of.

Line-breaks

Firstly, WriteFreely's MD flavour differs from GitHub's in how it treats hard linebreaks: specifically, they will be preserved in the final output. On GitHub, if you write the following markdown:

Hello world! Dis next line. And dis next line.

And dis next para. Pretty neat, huh?

GitHub will collapse single linebreaks and only preserve the double linebreak to separate text into two paragraphs. On the contrary, WriteFreely will preserve all newlines as-is. I was first annoyed by difference from other markdown flavours, but then I realized that this is actually more like how email is rendered, and found zen and peace in this. :)

Therefore, publishing via wf post will apply stylistic markdown formatting and properly linkify all links, but will preserve all newlines as if you were reading an email message on lore.kernel.org.

There's some discussion about making markdown flavouring user-selectable, so if you want to add your voice to the discussion, please do it there.

Making it behave more like GitHub's markdown

If you do want to make it behave more like GitHub's markdown, you need to make sure that:

  1. You aren't using hard linebreaks to wrap your long lines
  2. You are publishing using --font serif

E.g.:

  $ gedit mypost.md
  $ cat mypost.md | wf post --font serif

This will render things more like how you get them by publishing from the WriteFreely's web interface.

Using “post” and “publish” actually puts things into drafts

I found this slightly confusing, but this is not a bad feature in itself, as it allows previewing your post before putting it out into the world. The way it works is:

  $ vim myfile.md
  $ cat myfile.md | wf post
  https://people.kernel.org/abcrandomstr

You can then access that URL to make sure everything got rendered correctly. If something isn't quite right, you can update it via using its abcrandomstr preview URL:

  $ vim myfile.md
  $ cat myfile.md | wf update abcrandomstr

After you're satisfied, you can publish the post using the “move to Yourblog” link in the Drafts view.

Read the friendly manual

Please read the user guide and the markdown reference to try things out.