Writing a new MediaWiki tarball release script

Last week's security release of MediaWiki 1.27.5 / 1.29.3 / 1.30.1 / 1.31.1 mentioned a small hint of a new release script being used for this release. Chad came up with the concept/architecture of the new script, I wrote most of the code, and Reedy did the actual release, providing feedback on missing functionality and other feature requests.

Before I explain the new script, let me explain how the old script worked (source). First, the script would clone MediaWiki core, extensions, skins, and vendor for you. Except it wanted one directory per version being released, so if you wanted to do a security release for 4 MediaWiki versions, then you'd need to have MediaWiki core cloned 4 times! Oh, but since we need to make patch files against the previous release, it'll need to recreate those tarballs (a separate problem), so you now have 8 clones of MediaWiki core. Ouch.

Then comes security patches. These patches are not yet published on Gerrit, and currently only exist as git patch files. The old script required these be in a patches directory, but in a specific naming pattern so the script would know which branch they should be applied to. Mostly this confused releasers and wasn't straightforward.

There were definitely other issues with the old script, but those two were the main motivation for me at least.

Enter makerelease2.py (inital commit). The theory behind this script is to simply archive whatever exists in git. We added the bundled extensions and skins plus vendor as submodules, so we did not have to maintain separate configuration on which extensions should be bundled for which MediaWiki version. This also had the added benefit of making the build more reproducible, as each tag now has a pointer to specific extension commits instead of always using the tip of the release branch.

Excluded files can be maintained with .gitattributes rather than by the release script (yet another plus for reproducibility, maybe you can see a pattern :)).

If you're not already familiar, there's a git-archive command, which creates tarballs (or zipballs) based on what is in your repository. Notably, GitHub uses this for their "Download tarball/zipball" feature.

There's only one drawback - git-archive doesn't support submodules. Luckily other people have also run into this limitation, and Kentzo on GitHub wrote a library for this: git-archive-all. It respects .gitattributes, and had nearly all the features we needed. It was missing the ability to unset git attributes, which I submitted a pull request for, and Kentzo fixed up and merged!

So, running the new script: ./makerelease2.py ~/path/to/mw-core 1.31.1. This will spit out two tarballs, one of the mediawiki-core variant (no extensions or skins bundled), and the full mediawiki tarball. You can create a tarball of whatever you want, a tag, branch, a specific commit, etc., and it'll run. Additional checks will kick in if it is a tag, notably that it will verify that $wgVersion matches the tag you're trying to make.

To create a security release, you take a fresh clone of MediaWiki core, apply the security patches to the git tree, and create the new tags. Using the native git tools makes it straightforward to apply the patches, and then once the release has been announced, it can easily be pushed to Gerrit.

If you pass --previous 1.31.0, then it will additionally create a patch file against the previous tarball that you specified. However, instead of trying to recreate that tarball if it doesn't exist, we download that tarball from releases.wikimedia.org. So regardless of any changes to the release scripts, the patch file will definitely apply to the previous tarball (this wasn't true in the past).

The following bugs were fixed by this rewrite:

What's next? This was really only step 1 in the "streamline MediaWiki releases" project. The next step (as outlined by Chad) is to continuously be generating tarballs, and then be generating secret tarballs that also include the current security patches. I don't think any of this is especially technically hard, it will mostly require process improvements with how we handle and manage security patches.

Goodbye PHPStorm, hello Atom

I've been using the JetBrains IDE PHPStorm ever since I really got started in MediaWiki development in 2013. Its symbol analysis and autocomplete is fantastic, and the built-in inspections generally caught most coding issues while you were still writing the code.

But, it's also non-free software, which has always made me feel uncomfortable using it. I used to hope that they would one day make a free/libre community version, like they did with their Python IDE, PyCharm. But after five years of waiting, I think it's time to give up on that hope.

So, about a year ago I started playing with replacements. I evaluated NetBeans, Eclipse, and Atom. I quickly gave up on NetBeans and Eclipse because it took too long for me to figure out how to create a project to import my code into. Atom looked promising, but if I remember correctly, it didn't have the symbol analysis part working yet.

I gave Atom a try again two weeks ago, since it looked like the PHP 7 language server was ready (spoiler: it isn't really). I like it. Here's my intial feelings:

  • The quick search bar (ctrl+t) has to re-index every time I open up Atom, which means I can't use it right away. It only searches filenames, but that's not a huge issue since now most of MediaWiki class names match the filenames.
  • Everything that is .gitignore'd is excluded from the editor. This is smart but also gets in the way, when I have all MediaWiki extensions cloned to extensions/, which is gitignored'd in core.
  • Theme needs more contrast, I need to create my own or look through other community ones.
  • Language server regularly takes up an entire CPU when I'm not even using the editor. I don't know what it's doing - definitely not providing good symbol analysis. It really can't do anything more advanced than things that are in the same file. I'm much less concerned about this since phan tends to catch most of these errors anyways.
  • The PHPCS linter plugin doesn't work. I need to spend some time understanding how it's supposed to work still, because I think I'm using it wrong.

Overall I'm pretty happy with Atom. I think there are still some glaring places where it falls short, but now I have the power to actually fix those things. I'd estimate that my productivity loss in the past two weeks has been 20%, but now it's probably closer to 10-15%. And as time goes on, I expect I'll start making productivity gains since I can customize my editor significantly more. Hooray for software freedom!

Day 19: The End

Part of a series on my journalism faculty-led program through Italy and Greece.

It's over. Tonight was our last night on the trip, and we head back to Athens tomorrow. I think I learned a lot more about myself than the skills and knowledge I picked up. I spent some time reflecting by talking to people in person, so I'm not going to write anything up tonight. Goodbye (for now!) ^.^

"Boyz of FLP"

"Boyz of FLP"

P.S.: My team won in Jeopardy! tonight, $2,600 - $500 - $400 - $200. It was fun.

Day 17 & 18: Dignity and human rights

Part of a series on my journalism faculty-led program through Italy and Greece.

We visited the Kara Tepe and Moria refugee camps over the past two days. The director of Kara Tepe really spoke to me when he started talking about dignity, freedom, and other human rights. I ended up re-reading the Universal Declaration of Human Rights, and some of the background behind it's creation.

I'm still processing the Moria camp, and how to tell its story. There's a slight possibility that we might go back tomorrow, so I'll just post a short snippet from the interview we conducted today with a 23 year old Syrian photojournalism student (basically me, but from a war-torn country and not from immense privilege). One of his friends translated the Arabic to English for us, and I'm paraphrasing the quote.

"Would you rather be at Moria or in Syria?"

"Syria. We're not safe here, we weren't safe there. But at least in Syria we had the freedom to go wherever we wanted and we could see our family."

Salam, Ibrahim.

Day 16: Lost and found

Part of a series on my journalism faculty-led program through Italy and Greece. Note: I wrote this halfway through day 17, which definitely affected what I'm writing now.

Today was mostly a stay-at-the-hotel-and-work day, so this is another reflection.

For a while now I've felt lost in where I want to go in the future. I have about a year and a little bit more of school, but after that I've been pretty unsure. I don't think I want to become a real journalist, but I've also been contemplating whether I should be continuing at my current job (in a full time capacity). At least one person I talked to on this trip suggested that other work experience outside of Wikimedia would be good for me so I can broaden my horizons and grow as a person, and I think I generally agree with him.

So the main question I've had is, what should I do? I think it's pretty clear that I want to do something that will improve the world and people's lives. I have the privledge to not have to worry that much about money, so I feel the obligation and desire to help other people.

A few people have suggested law school to me, which is pretty appealing. It seems like a lot of problems today need lawyers to fight them, and I've always had a fascination with civil rights litigation. But that requires going through law school, and I don't think I have the patience to wait that long. I want to start improving the world now.

It's funny sometimes how one small event can turn your entire life over. It was pretty dangerous, reckless, stupid, but incredibly fun. I don't think I've had my adrenaline running like that in years.

I didn't say anything to her about what happened, but my mom immediately noticed the difference in my mental state when I talked to her for 30 seconds on the phone.

We talked a lot about values, relationships, and most importantly failure last night. I don't think I've exactly found where I want to go yet, but I feel much better that I've found the right track.

The sunrise

Maylea's picture of the sunrise

Day 15: The knife, the port, the prison, and the boy

Part of a series on my journalism faculty-led program through Italy and Greece.

There are four stories today.

The knife

We were going to the lifejacket graveyard. It's a dump that has boats and lifejackets from refugees that made it here. But each lifejacket is a good thing - it represents a refugee that successfully completed the journey, and no longer have a need for it.

We got out of the car, started looking around, opening up tripods, when he came. He works at the state-owned dump, and was driving a bulldozer.

"What are you doing here?" he shouted. "This is private property!"

He got out of the bulldozer, and started waving a giant butcher's knife at us, while telling us to leave. We retreated back to our van, but our local driver walked up to him, and stood her ground.

Knife guy

She later told us that she suspects he was a member of the far-right Golden Dawn party, since he told her that "we [Greeks] are being colonized by refugees and migrants" (translated, paraphrased).

The port

Molyvos is a tourist town. But there simply weren't enough tourists for the amount of shops and services they had. We had heard that the refugee crisis had scared away tourists from the island, but this seemed eerie at times. I wasn't sure whether people were being extremely friendly because that was just the way they were, or whether they were desperate to have people buy their merchandise.

We interviewed a shopkeeper there, who is from Belgium, but has lived and worked on Lesvos for 25 years, meeting her husband here as well. She had an interesting take on the fear of tourists that the island was overrun by refugees. (By the way, it's not.) She explained that people who want to come here for the tourism could just stay in the tourist areas, and those who wnated to come here to help the refugees should do that. It was as if she wanted to treat the island as two parts: a refugee crisis area and a beautiful tourism center.

As we were leaving lunch, I briefly talked with a member of the Hellenic Coast Guard about his experiences. We talked about a few different things, but one quote stuck with me:

"The people who come here are refugees, leaving war, but some are migrants, looking for a better life than home. But to me, it doesn't matter. I save everyone."

The prison

Call it what you want - Moria is a prison. We drove past it, and it looks like a prison. The walls are prison walls. There's barbed wire on top of the walls. If they didn't want it to be a prison, they would have torn that stuff down.

We'll be going to Moria again on Monday, this time to conduct interviews, get pictures, etc. I'm pretty nervous.

The boy

We came back to the hotel, I quickly changed, and jumped into the ocean, taking a kayak with me. One of them was a two person kayak, so I taught two of my classmates how to kayak before he came up to me, asking for a ride.

He looked at me and said, "you're black", to which I nodded. Then he looked down into the water at my feet and said, "you're white in the water," to which I responded with "Probably. But why does it matter?"

And he didn't say anything, but I knew why it mattered to him.

He is a five year old boy from the Congo. He spoke decent English, and was usually understandable. But he's a refugee, and had to leave his country for well...whatever this island is. No one else at the resort at the time was as dark skinned as me or him, so hopefully he felt a bit more comfortable.

I didn't see his parents around, so I didn't feel comfortable asking any other questions. We rode out into the ocean, talking about the waves and the kayak. If there was one thing that put a smile on my face today, it was seeing the smile on his face.