From the Homebrew post about the incident:
“The security researcher also recommended we consider using GPG signing for Homebrew/homebrew-core. The Homebrew project leadership committee took a vote on this and it was rejected non-unanimously due to workflow concerns.”
How is PGP signing not a no-brainer. What kind of workflow concerns would prevent them from signing commits?!
If you make PGP signing easy enough, at some point you end up with a Jenkins with a trusted PGP signing key, and you haven't actually solved anything.
The problem isn't making it easy to sign things, the problem is making it sufficiently hard for unauthorized parties to sign things without affecting any workflows you'd like to preserve - that is, the real problem is a workflow problem. The real problem is figuring out how to secure automation so it has the privileges to do what it needs but isn't leaking access. The Jenkins instance was designed to do authenticated pushes - it needs automated write access to the Homebrew repos.
Also, signing commits doesn't help you if the risk is unauthorized pushes to master. You can pick up someone's test commit and push that to master, or push a rollback of OpenSSL to a vulnerable version, or something, and still ruin many people's days.
That is true for git's definition of "history," but it is not helpful here. If I force-push a commit that was signed a year ago, then the signature does not cover the fact that master was just rolled back by a year (the signature does not cover the reflog, in git parlance). You have a valid signature of the previous version of history, and clients cannot tell that history was rolled back without authorization.
If I push a pull-request to master that wasn't approved to be on master (e.g., a maintainer did a build of "disable signature validation to narrow down why tests are failing", and signed that commit and pushed it to a PR with the intention of rejecting the PR), then I also have a valid and completely signed "history", and it probably isn't even a force-push to get it onto master.
Git commit signatures authenticate exactly one thing: that at some point, the holder of this PGP key committed this commit. They say nothing about the suitability or future suitability of that commit to be used for any purpose. They don't solve the problem Homebrew had here, and they cause other problems (like breaking rebases).
Git tag signatures are significantly more useful, since they include the name of the tag. So you're not vulnerable to the second attack, and you're mostly not vulnerable to the first since a client wouldn't intentionally request tag 1.0 after getting tag 1.5. But you still have the problem of the client knowing which tag is current, and frequent tagging isn't a great replacement for a workflow where you want people to follow master.
Technically `git push --signed` also exists which could fix the issue of rolling back commits. It would verify that the person doing the push also holds the GPG key at least. But as far as I can tell you have to manually do something with it in the post-receive hook and GitHub doesn't support it at all.
Are there any OSS maintainers who use air-gapped computers to sign packages (at least for major versions)? I would expect this level of precautions be taken for projects that may have large-scale repercussions in the event of a security breach.
I don't think air-gapping is particularly helpful in practice - for large and active codebases, you'd need to read all of the changed source code on the air-gapped machine to look for subtle back doors, which is difficult. Alternatively, if you want to do air-gapped builds and binary signatures, you're going to have to copy over the build-dependencies, which for most build environments are assumed trusted (i.e., they can compromise your air-gapped machine).
For small and intermittently-active ones, the primary development constraint is the OSS maintainer having free time, and doing dev or builds on an air-gapped machine is a big time sink. I maintain very little open-source software and I have still tried to at least use a separate machine for builds + PGP signatures and not my day-to-day machine, and maintaining this machine is just overhead that eats into my, what, one evening every two months that I get to spend on my project.
The solution I'd like to see is a) better tracking of community code reviews/audits; if every line of code has been read by multiple people (perhaps identified with a PGP key, but something simpler would be fine), you're more confident that there are no subtle backdoors than if you make one person stare at the entire diff between major versions on an air-gappped machine until their eyes glaze over, and b) better ways to do builds on multiple clouds and verify that they're identical. The Reproducible Builds effort is a good approach here; if you do that plus a CI infrastructure that runs on two different clouds with two different signing keys, and client systems require both signatures, you can be reasonably assured that the build process wasn't compromised.
How much review does 'cperciva do of the source code and build-dependencies that are copied to the air-gapped machine, which presumably originate from internet-connected machines?
Also, how secure is the kernel on the air-gapped machine against malicious filesystems on the USB stick? (If it's running Linux, the answer is almost certainly "not;" I could imagine FreeBSD is better but I don't know how much people have explored that.)
To be clear I'm not opposed to air-gapping if the maintainer is excited about it, I just suspect there are many much weaker links on the way to/from the air-gapped system, and fixing those is a much harder project that almost nobody is excited about.
How much review does 'cperciva do of the source code and build-dependencies that are copied to the air-gapped machine, which presumably originate from internet-connected machines?
I verify that the source code being compiled is the source code which is published in a signed tarball. Yes, someone could have tampered with the internet-connected system where I do Tarsnap development, but their tampering would be visible in the source code.
Build dependencies are verified to be the packages shipped by Debian. If someone has tampered with the Debian gcc package, we've lost even without Tarsnap binary packages.
Also, how secure is the kernel on the air-gapped machine against malicious filesystems on the USB stick? (If it's running Linux, the answer is almost certainly "not;" I could imagine FreeBSD is better but I don't know how much people have explored that.)
I don't use a filesystem on the USB stick I use for sneakernet, for exactly this reason -- I write and read files to it using tar. (Yes, you can write to a USB stick as a raw device just like you would write to a tape drive.)
The 'malicious file system' on a USB stick is not something to worry about - the firmware of your USB stick is. People (ie for-fun hackers) modified firmwares on USB sticks to make them look like HID keyboards and send commands to target computers - it is well enough in the capabilities of a determined adversary to own your internet machine and implant something on your USB stick.
For secure airgapped computers I'd use one way low tech comm channels with no side bands, maybe IR or sound? (if you trust the device drivers)
Good point... stupid of me but I didn't think about the input side of things. Could be handled with a tar dump dd-ed straight to the USB device. (That's probably not happening, but it's good to think about these things.)
Or a virtual throw away machine just for mounting a USB filesystem, extracting the files, place them in some dump directory and then the whole virtual machine is wiped.
Heck, the virtual machine for copying should run BSD. :)
I'll happily state publicly that I voted for this proposal but in this case the issue is that Homebrew/homebrew-core does not follow a GitHub Flow process but pulls binary packages in with a custom tool (`brew pull`) and generally relies on a rebase workflow which GitHub does not sign (understandably as it's modifying the original commit and would have GitHub signing non-merge-commits). I'm still optimistic we can figure out a way to do this in future.
General rule of thumb for secure package distribution:
1. Is the identifier mutable? Make sure it points to a content addressable identifier (SHA2), and sign that link.
2. Is it a content addressable identifier? Nothing to do.
When it comes to signing in git, signing tags is usually where you see the most value (mutable identifier that points to a git tree, which is content addressable).
You’re just trying to improve the trust in saying “Hey, v1.2 is this SHA digest”.
He seems to be discouraging signing every commit's individual data but encouraging signing the actual commit ID (SHA1) which should be perfectly feasible for something like homebrew.
You're still getting a signature directly from the developer's machine, not from the repository server and as such you're still vastly shrinking the attack surface.
You have no idea how creative people get when faced with minor nuisances. I've seen devs/admins go to great lengths to avoid doing more than one 2FA per day.
Code signing is important, but artifact signing is even more important, because that's what you end up trusting at the end of the chain. So not only do you have to sign your code and secure all your code signing keys, your build agent has to have a build signing key to sign builds. If any of this is compromised, there goes your build integrity.
How is PGP signing not a no-brainer. What kind of workflow concerns would prevent them from signing commits?!