Blog:The LWP’s handling of mathematical formulae evolves: Difference between revisions

As a sophisticated system which is developed by a lively community, MediaWiki is [[mediawikiwiki:Version lifecycle|constantly updated]]: a new version is released every six months, and one in four of those is a “long-term support” (LTS) version, which is guaranteed to receive security updates for three years. Updating MediaWiki from one version to the next is not entirely trivial, and so we at the LWP only use LTS versions: the website [[Special:Version|currently runs]] MediaWiki 1.39, the legacy LTS version, which will be supported until November 2025, and it will be upgraded to MediaWiki 1.43, the current LTS version, before then.

Up until 1.39, in the configuration which was adopted on our website, it relied on Wikimedia’s so-called “Mathoid” service to convert the LaTeX source code into the visual rendition of the formula. In other words, the MediaWiki site (for example, the LWP’s site) would send a request to the Wikimedia servers via an API; the Mathoid service, [[mediawikiwiki:RESTBase|running on the Wikimedia servers]], would perform the rendering; the rendered formula would be sent back to the MediaWiki site as a PNG image; and it would then simply be displayed as an image.

The way SimpleMathJax works is different from the process I described above in that there is no communication between the local site and remote servers. When the page is loaded, the rendering of the formula is performed client-side, in the user’s web browser, thanks to the “[https://www.mathjax.org/ MathJax]” JavaScript library.

As I often repeat when I talk about the LWP’s, technical infrastructure and distribution strategy, the website is king. This means that the LWP’s editions are best consulted directly on the website, where they are:

The result of Frederic’s work was [[Project:Downloading, exporting, and manipulating the texts|the LWP’s ebook export feature]], which is not a MediaWiki extension but is also free and open-source software ([https://github.com/wittgenstein-project/wittgenstein-project.github.io the code is hosted on GitHub]). It is mostly written in Python and works as follows: it uses the [https://pypi.org/project/beautifulsoup4/ Beautifulsoup] web scraping library to retrieve the list of books to be converted from our “All texts” page; for each of those, it retrieves the plain HTML version which MediaWiki generates when the string <code>?action=render</code> is appended to the page’s URL; it uses a custom-made parser to convert that HTML code into [[wikipedia:Markdown|MarkDown]] code, a simple markup language which is designed to only preserve the formatting which is semantically relevant; it then uses the [https://pandoc.org/ Pandoc] library to convert the MarkDown code into PDF, EPUB, and MOBI files. In addition to those, the MarkDown file remains available as a very clean, plain-text version of the ebook.

At the time this system was first deployed, the website used the “Math” extension and the Mathoid service to handle the formulae; therefore, in Frederic’s version of the export feature, the formulae were handled as images.

In order to do this, we were helped with great professionalism and kindness by another friend and LWP volunteer, Liu Ruoxiao. She managed to integrate the [https://matplotlib.org/ Matplotlib] Python library into Frederic’s code, so that, upon receiving the raw LaTeX markup as it is embedded in the new version of the HTML source code (the one generate by MediaWiki with extension “SimpleMathJax”), the application could perform the rendering of the formulae itself, save them locally as images, and include these in the PDF, EPUB, and MOBI files.

I thought all of this was worth sharing for two reasons: first, this blog post may serve as a piece of technical documentation (for us to track some of the changes in the website’s infrastructure, and for others to potentially build upon our code to create a similar export feature for their own website); second, it tells the story of a successful cooperation between several people who didn’t necessarily know each other very well, but were brought together by the common goal of making a valuable collection of texts more widely available and easily accessible—something that always warms my heart.