The point release that upgrades your entire OS

By
in SecureDrop
tagged rust, securedrop, ubuntu

For the past two weeks, we've been putting out SecureDrop point releases that, as the title gives away, upgraded the underlying server's entire operating system. Definitely the kind of change you expect when upgrading from 2.12.3 to 2.12.4.

(An alternative title for this post could've been "Committing semver crimes" — thankfully we don't strictly follow semver.)

We've previously written about how the migration works on a technical level, I wanted to write a little bit about how we rolled it out. The hook about versioning was just clickbait, I couldn't care less about version numbers in this context.

A few relevant pieces of context:

SecureDrop is set up with two servers: the Application Server hosts the web interface as an onion service, while the Monitor Server runs an intrusion detection system and fires off email notifications. Both servers run unattended upgrades to pick up new updates and reboot every night.

We released 2.12.0 in mid-March; after upgrading each server generated a random number between 1 and 5 (inclusive) and saved it. This effectively grouped the servers in 5 buckets, so we could target ~20% of all servers at a time.

SecureDrop administrators had slightly over a month to manually do what we called the "semiautomated migration". It was the same code as the normal migration, except it was initiated by the administrator, at their convenience.

After that period ended, we started issuing point releases that started the migration in a fully automated manner. Even though we were relatively confident in the code after seeing a number of successful semiautomated migrations, we still did a staged rollout just as a safety measure.

It would be really bad if there was some unexpected edge case that caused a significant amount of SecureDrops to go down all at once! And we did end up catching a bug after the first round of fully automated upgrades, so the caution paid off.

We started out slowly and then quickly ramped up the cadence of releases:

  • April 22: 2.12.2 released, upgrading 20% of Application Servers.
  • April 28: 2.12.3 released, upgrading 40% of Application Servers.
  • April 30: 2.12.4 released, upgrading 60% of Application Servers.
  • May 5: 2.12.5 released, upgrading 100% of Application Servers.
  • May 6: 2.12.6 released, upgrading 20% of Monitor Servers (in addition to all Application Servers).
  • May 7: 2.12.7 released, upgrading 40% of Monitor Servers
  • May 8: 2.12.8 released, upgrading 100% of Monitor Servers

The official Focal end-of-life was scheduled for May 29, so we made with just a few weeks to spare. Phew!