Implementing search for my blog in WebAssembly

If you visit my blog (most likely what you're reading now) and have JavaScript enabled, you should see a magnifying glass in the top right, next to the feed icon. Clicking it should open up a search box that lets you perform a very rudimentary full-text search of all of my blog posts.

It's implemented fully client-side using Rust compiled to WebAssembly (WASM), here's all the code I added to implement it.

At a high level, it:

1. Splits all blog posts into individual words, counts them, and dumps it into a search index that is a JSON blob.
2. Installs a click handler (using JavaScript) that displays the search bar and lazy-loads the rest of the WASM code and search index.
3. Installs an input handler (using WASM) that takes the user's input, searches through the index, and returns up to 10 matching articles.

The search algorithm is pretty basic, it gives one point per word-occurence in the blog post, and 5 points if the word is in the title or a tag. Then it sorts by score, and if there's a tie, by most recently published.

There's no stemming or language processing, the only normalization that happens is treating everything as lowercase.

I've played with WASM before but this is the first time I've actually deployed something using it. As much as I enjoyed writing it in Rust, the experience left something to be desired. I had to use a separate tool (wasm-bindgen) and load a pre-built JavaScript file first that then let me initialize the WASM code.

The payload is also ...heavy:

• search.js: 5.53kB (23.63kB before gzip)
• search_bg.wasm: 53.78kB (122.82kB before gzip)
• search_index.json: 323.13kB (322.76kB before gzip)

I'm not sure why the index compresses so poorly with Apache, locally it goes down to 100kB. (I had briefly considered using a binary encoding like MessagePack but thought it wouldn't be more efficient than JSON after compression.) And of course, the more I write, the bigger the index gets, so it'll need to be addressed sooner rather than later. I think any pure-JavaScript code would be much much smaller than the WASM bundle.

Wiki burnout

Yeah, I burned out from wiki things. I flew too close to the sun, tried to take on too many projects and cool ideas and then crashed and let people down. I'm sorry.

I participate in wiki communities because, aside from believing in free knowledge, etc., it's really fun. And then it stopped being fun, so I just...stopped. In some aspects it was nice, I spent my time doing a lot of other IRL things (I bought a bike), but I also missed doing wiki things.

And so I'm slowly starting to get back into things. I'll try to get back to everyone...eventually. I'm not really sure what's next in my queue. I'm trying to not jump back into what I was doing previously because that doesn't really solve the burnout problem but also I owe people some work.

I'll be at the "All-Day Hacking Sunday" this weekend in New York City, it should be fun and hope to hang out with people there.

Updates to my blog

It's been nearly 10 years since I created this blog and for that entire time it's been using the Pelican static site generator. It's been pretty good, but lately I've wanted to improve and change some things, so I've taken the opportunity to rewrite it from mostly scratch.

b2 is a Rust program that takes an input folder of markdown files and spits out a complete HTML directory of the blog. The templates and theme are taken from my fork of pelican-sober. For the most part nothing has changed, I've just taken the opportunity to adjust the CSS and improve some of the HTML output.

b2 is pretty specific for my usecase, I'm not planning to turn it into a general purpose static site generator, though you're more than welcome to fork it for your own needs.

"Skip Mobile Wikipedia" add-on available on Android (again)

In 2018 I announced the creation of my "Skip Mobile Wikipedia" Firefox add-on, which automatically redirects you to the desktop version of the site. At the time it worked on both standard desktop Firefox and in Firefox for Android, life was great.

Unfortunately at some point, Firefox stopped allowing arbitrary add-ons on Android and it was disabled. It was still usable in the "Nightly" edition, but that wasn't very user friendly.

Anyways, fast-forward to earlier this month, Firefox has re-enabled general add-on support, so "Skip Mobile Wikipedia" is installable and works again on Android. The source code is the same, I haven't had any need to update it since 2020.

I personally use the responsive Timeless skin, which I think provides a nicer viewing and editing experience from my phone. Happy browsing!

Support my work, please

I am privileged to work for a non-profit organization, the Freedom of the Press Foundation, which allows me to work full-time on free and open-source software, namely SecureDrop. It also pays the bills and lets me spend me free time doing other stuff that's good for humanity, like contributing to wikis, running a Mastodon server and riding bikes.

FPF accomplished a lot of important things this year, but like every other non-profit, we're currently doing our end of the year fundraising drive, hence this blog post.

If you like my work, whether on SecureDrop or just in general, and have the means to do so, I'd appreciate you donating to FPF: https://freedom.press/donate/ (if you live in the US, this donation is tax deductible). Click the checkbox that says "This donation is in tribute of someone", and then say the gift is in honor of me, "Kunal". (You're also welcome to type other stuff, like "legos" or "boba tea".)

Hate my work? That's fine too, just pick the option that says "This gift is in opposition to" and mention my name. You could also type other stuff here, like "the Kragle" or "boba tea haters".

Now that you've read this far, I'll also mention that we have a merch store with FPF and SecureDrop swag, including stickers! If you use the coupon code "FPF2023" you'll get a 15% discount (only on Nov. 27 aka Cyber Monday).

Thanks.

Migrating SecureDrop’s PGP backend from GnuPG to Sequoia

I haven't been very good with writing about what I've been working on in SecureDrop-land, but here is: Migrating SecureDrop’s PGP backend from GnuPG to Sequoia over on the SecureDrop website. You should read it, and then come back to read the rest of this.

We had been discussing Sequoia internally for a while, but I officially filed an issue titled "Use Sequoia-PGP instead of gpg/pretty_bad_protocol" in April 2022. I pushed a first WIP of the Rust code to a branch named oxidize on July 8, 2022 and had it working in the development environment by the 14th. TBH that was the easy part, migrating existing sources from GPG to Sequoia was going to be the hard part.

Small tangent, I was very fortunate to grow up around a lot of Sequoia trees, and I have very fond memories of visiting Muir Woods National Monument as a kid. So when I had to pick a name for the Rust crate, I tried to stay with the theme, and went to the Wikipedia disambiguation page for Sequoia, clicked on the first link, Sequoioideae, read "...commonly referred to as redwoods", and then immediately ran cargo new redwood --lib. I anticipated we'd switch to a generic, boring, name like rust but that didn't end up happening :-).

Exporting public keys is pretty easy, but GPG stores secret keys in its own custom s-expression thing (documented as "...easier to read and edit by human beings", okay.), which we would have to write something custom to read. I flailed around trying to do so but never really got anywhere. Sequoia opened a ticket on their end for supporting GPG's format, but as of this writing, it hasn't seen any real progress (which is fine and understandable!).

Also I got busy with other SecureDrop things (e.g. "Reorganize Debian packaging, have debhelper do most of the work", which deserves its own blog post).

At the end of December 2022, Sequoia announced their chameleon project to replace the gpg CLI. In January 2023 (this year!) I discussed whether using that was an option in the Sequoia IRC channel, and Neal (who is fantastic), suggested just using GPG directly. Great idea!

Now that we had a real migration plan, February and March had some internal discussions on whether to move forward with this, and I started working on it for real in May. The first PR was up by the end of May and landed the first week of June.

The second major PR was posted mid-July and merged at the beginning of October! There were a number of smaller PRs in the middle, but that was "step 2" in my high-level roadmap. There was a lot of good back and forth during review from SecureDrop team members, Sequoia developers and other volunteer contributors. And related PRs from other SecureDrop team members.

Everything else landed in October, and then this week I finished running some benchmarking, which, surprise surprise, showed that calling the Sequoia Rust code was faster than shelling out to GPG.

SecureDrop 2.7.0 powered by Sequoia should be (knocks on wood) released in a few days, which will be the largest milestone to date of this multi-year project. And as the blog post pointed out, there's still a lot of work to do.

This is the first Rust project that I've worked on that is being shipped to production users (all the wiki bots I've written really don't count), and it's been a blast. It certainly won't be the last ;-)