Adventures of a YAML engineer

I want to brag about a bit of YAML code I wrote back in March for SecureDrop's completed migration to Ubuntu Noble that I neglected to mention in the blog post explaining the technical details. Yes, YAML, is a programming language.

We offered SecureDrop Administrators the option for a "semiautomated" upgrade: they run one command, ./securedrop-admin noble_migration, and it'll take care of the rest. The main advantage for doing so was that the upgrade would happen at the time you chose, and if something happened to go wrong, you were already on hand to deal with it!

Under the hood the semiautomated upgrade was starting an Ansible playbook that edited our JSON control file to mark the server as ready to be upgraded and then started the systemd service. And then it just waits until the upgrade completes, which ended up being the harder part to implement.

During the upgrade, the server reboots twice (once before installing updates and once after), which means Ansible will lose its SSH connection. I used Ansible's wait_for_connection module to reconnect instead of error out, and naively had it wait for that to happen twice before checking if the upgrade had finished.

But during testing we found a problem when using SSH-over-Tor, in which Ansible would disconnect three times. It disconnected on the first pre-upgrade reboot, then during the upgrade when the Tor package was restarted, and then again during the second post-upgrade reboot.

And, to make it even more fun, this was subject to a race condition. In at least one instance, it took long enough for Tor to come back that the server rebooted before it reconnected, so there were only two disconnections.

Knowing that, a naive solution wasn't going to cut it anymore, so I implemented the same state machine as the Rust code, just in the YAML playbook. It now parsed the JSON state file, looked up where in the overall process it was, and then calculated how many reboots are likely remaining. Once it disconnected and reconnected, it looked at the state file again, so it knew how many more to expect.

Here's the end result, it ended up being just over 200 lines of YAML (including comments).

Alternative clickbait titles for this post include: "Porting some of my Rust code to YAML" and "Writing a state machine in YAML".