Proposal for a dedicated archive site for Wikipedia

In February when English Wikipedia (and a few months later, all of Wikimedia) banned usage of archive.today (aka archive.is), there was a bit of a crisis in needing to find alternative archives.

For reasons, archive.today was especially good at archiving some difficult websites that weren't available in the Wayback Machine. I know a lot of people used it to bypass paywalls, and for related or unrelated reasons, there was an FBI investigation into the site.

I don't really have much sympathy for the archive.today operator after they abused open proxies and/or a botnet to linkspam across Wikipedia over a decade ago, but I certainly get why they want to remain anonymous.

Wikipedia editors have started discussing what a better solution to archiving webpages could be, with much of the discussion focusing on copyright.

As far as I know, no solution has actually materialized, so I thought I'd at least write down my idea. I am not actually planning to implement this, so feel free to steal this idea, my only ask is that you give it a hella sick name.

Overview#

While a having a generic web archive is incredibly useful, I want to just focus on building a web archive that meets what Wikipedia needs.

For basic features we want that:

• editors can add new URLs to the archive
• readers and editors can verify claims cited to that URL after the link is dead
• editors should be able to use the archived copy to add other claims to articles

And that editors can reuse their Wikipedia accounts for authentication via OAuth.

The workflow of my proposed archive would look like:

1. Editors visit a website, and take a copy.
2. When uploading their copy to the archive, editors need to annotate the specific parts that are being used for citations.
3. Readers can view only the annotated snippets that back up those citations.
4. After paying a non-monetary fee, editors can view the full copy of an article in order to annotate other snippets to use as further citations.

I think this meets both Wikipedia's requirements and minimizes legal risk.

User-generated content#

Unlike most other archives I know of, I would have users upload the archived content. By treating the archives as user-generated content, we get to take advantage of a number of legal protections, like Section 230 in the U.S.

Editors would install a browser extension that when asked, would capture a WARC (or similar) of the URL, and then upload it to the archive. It would be nice if there's some cryptographic way to verify the request wasn't tampered with (e.g. looking at the TLS connection?) but even without that, uploads would still be associated with specific editors, so if someone was caught falsifying, we'd be able to track it, just like you would deal with on Wikipedia.

This also conviently works around the drastic increase in scraper blockers that have started to lock down the internet by doing it in the user's browser. For some very dynamic content, we might also want to capture a screenshot, so if it relies on some complex JS or APIs, we can at least display what it looked like.

Annotated snippets#

The biggest change would be that editors need to annotate specifically which part of the webpage is being cited. I think this is a good practice for citations in general.

But because we'd only publicly display the limited snippet, it would work around the archive being used to bypass paywalls. Displaying a limited amount greatly works in our favor for a fair use claim; to quote Wikipedia:

In general, the less that is used in relation to the whole, the more likely the use will be considered fair.

We'd need to develop some system to keep these snippets in sync with the on-wiki citations; I don't imagine it would be that complex to figure out.

Paying for full access#

While the snippets are sufficient for readers, editors might want to reference back to the full text for the purpose of expanding an article or just doing a more thorough fact check. I think we can still discourage the paywall bypass use-case by requiring editors to pay a small fee for access.

But instead of charging money, editors can pay with Wikipedia edits, e.g. 5 edits per URL. I think this would provide a small bit of friction that editors would only use it when they actually need it, but not enough to actually stop someone who genuinely needs access.

And of course if someone did choose to "game the system" to use it to bypass paywalls, they'd only be able to do so by actually contributing to Wikipedia, so... still a win.

Moderation#

I think it's quite difficult to launch a new site that accepts user-generated content, since you need to have a plan for content moderation. By building a Wikipedia-focused archive, we can piggyback on the existing Wikipedia community processes to help with this.

If you get blocked on Wikipedia, then you're blocked on the archive too. Have advanced administrator permissions on Wikipedia? We can automatically give you extra permissions on the archive. And so on.

I'm not sure if that would be sufficient long-term or just a temporary thing to help while it bootstraps and gets established.

Last thoughts#

I think the social and legal parts of this are far harder to get right; in comparison the technology seems pretty straightforward. I think it would be beneficial to have this be a separate LLC (i.e. not part of the Wikimedia Foundation) to isolate risk and allow for more flexibility.

While I think I've covered the primary legal bases, IANALY of course, it's entirely possible I've missed something. This is also a rather conservative proposal, and may have the consequence of shifting legal risk from the archive to individual users (though arguably that's how Wikipedia is already set up).

There are probably refinements to be made, or if you think I'm totally off, please, pitch your own proposal!