And that’s what the LaTeX template does, using the mathptmx LaTeX package. As the package name suggests, it also provides a version of the Times font for math typesetting, which is quite important for most LaTeX users. One of the many flaws of mathptmx is that it’s extremely obsolete and everyone strongly suggests against it:

1. It’s CTAN page says “reckoned to be obsolete”.
2. The 3rd edition of “The LaTeX Companion” published in 2023 warns against it.
3. The esteemed TeX StackExchange user egreg has said “remove mathptmx, which is a 25-year-old hack” in 2024.
4. etc.

One reason against mathptmx is that its math font is an inconsistent mess, as described by CTAN page:

[…] provides maths support using glyphs from the Symbol, Chancery and Computer Modern fonts together with letters, etc., from Times Roman.

The successor of mathptmx is txfonts which is also obsolete by now. And its successor is newtx which isn’t yet obsolete. Another non-obsolete alternative is to use the TeX Gyre Termes Math font. In this post, I will try switching my PhD thesis to both modern alternatives to compare them and find the best one to use instead of the ancient mathptmx.

Setup

Before diving into the comparison, here are the three LaTeX setups I will be comparing (click each tab).

• mathptmx
• newtx
• TeX Gyre Termes

• Under pdfLaTeX:

  \usepackage{mathptmx}
  \usepackage{amsmath}
  \usepackage{amssymb}
  \usepackage{mathtools}
  \usepackage{stmaryrd}
  \renewcommand{\sfdefault}{phv}
  \usepackage[mono]{inconsolata}
  

  This setup is the relevant part from the PhD thesis template, only inconsolata is my addition. It might not even show up any of the examples below but its placement in the package loading sequence matters.

  Helvetica (phv) loaded like this is visibly larger than Times, both for lower- and uppercase letters, but this doesn’t seem to have bothered anyone using the template?!

• Under pdfLaTeX:

  \usepackage[largesc]{newtxtext}
  \usepackage[mono]{inconsolata}
  \usepackage{newtxmath}
  \usepackage{mathtools}
  

  The newtx packages take care to load a Helvetica clone (TeX Gyre Heros) at a more reasonable scale (0.94 in newer versions).

• Under LuaLaTeX:

  \usepackage{fontspec}
  \setmainfont{TeXGyreTermesX}
  \setsansfont{TeX Gyre Heros}[Scale=0.94]
  \setmonofont{inconsolata}
  \usepackage{amsmath}
  \usepackage{amssymb}
  \usepackage{mathtools}
  \usepackage{unicode-math}
  \setmathfont{TeX Gyre Termes Math}
  \AtBeginDocument{%
      \NewCommandCopy{\llbracket}{\lBrack}%
      \NewCommandCopy{\rrbracket}{\rBrack}%
  }
  

  Since TeX Gyre Termes Math only exists in the OpenType Format, I tried it out under LuaLaTeX. This requires a different way of loading text and math fonts (with fontspec and unicode-math respectively).

  In order to minimize the visual differences with the newtx setup, I’m using TeXGyreTermesX from newtx instead of TeX Gyre Termes for the text. Moreover, I’m using the same scaling for Heros and Inconsolata as the newtx setup does implicitly. This scaling might not be optimal, though: neither lowercase nor uppercase letter height is matched.

  Since unicode-math doesn’t define \llbracket and \rrbracket (like stmaryrd and newtxmath do), they’re defined using the corresponding unicode-math symbols to make the rest of my thesis easily compile.

Text mode

Although this post isn’t about Times in text mode, there is one difference worth pointing out.

mathptmx

newtx

TeX Gyre Termes

(Click on images to zoom.)

The small caps (\textsc) in newtx is a bit heavier than in mathptmx, but the small capitals are a bit shorter. And this already is newtx loaded with the largesc option to get large small caps; by default, it offers petite caps which are even smaller.

Since Termes in the rightmost figure is actually TeXGyreTermesX from newtx but in OpenType format, the small caps look almost the same as in newtx, but not quite! Somehow LuaLaTeX and the OpenType version allow for overly close kerning between certain letter pairs, causing uneven spacing across the whole word:

• “ac” in “RacerF”,
• “ag” in “Deagle”,
• “Da” in “Dartagnan”,
• “pa” in “UTaipan”,
• “Ac” in “CPAchecker”.

On a side note, unicode-math in LuaLaTeX redefines \vdots to only work in math mode, which is why it’s missing in the figure (but easily fixed by just using it in text mode). The LaTeX default \vdots works in both modes because it’s not actually using a symbol from the font.

Math mode

With text mode out of the way, the rest of the post compares Times in math mode.

Parenthesis kerning

mathptmx

newtx

TeX Gyre Termes

Kerning between the left parenthesis and certain letters is bad in Termes. The parenthesis outright collides with ‘f’. The parenthesis doesn’t collide with ‘p’, although at normal text size it feels suspiciously close, even though the other fonts actually have a similar gap.

The latter comes down to the different parentheses:

1. In mathptmx they are the lightest but also tallest, extending below the descenders of ‘p’ and ‘f’.
2. In newtx they are the heaviest but of medium height, in line with the descenders.
3. In Termes they are of medium weight but the shortest, not reaching all the way around the descenders.

On a side note, in mathptmx the \mathsf used for “unique” and “create” doesn’t actually use Helvetica (or its clone) but just Computer Modern Sans. This causes a dissonance with the sans-serif font in the text mode (e.g. \textsf), which does use a version of Helvetica. Thus, the same document mixes two different sans-serif fonts.

Subscript kerning

mathptmx

newtx

TeX Gyre Termes

Ignoring the exact letterforms, subscript kerning is quite similar in mathptmx and newtx. Both have a bit too much space between the ‘x’ and the subscript ‘j’. This issue is acknowledged by newtx, which offers the package option subscriptcorrection. Using some LaTeX hacking, it implements special behavior to reduce the left kerning of certain letters appearing first in subscripts. This fixes the issue with ‘j’ but for some strange reason unnecessarily reduces the spacing in front of other non-problematic subscripts which aren’t even declared in its default subscriptcorrectionfile.

Subscript kerning in Termes is all over the place:

1. It gets the kerning in x_j correct out of the box.
2. It leaves too little space in \mathrm{st}_f.
3. It leaves way too much space for subscripts on \mathcal letters.

(Double) brackets

mathptmx

newtx

TeX Gyre Termes

The double brackets (\llbracket and \rrbracket) in newtx are noticeably heavier than all other symbols and really stand out on a full page. Like in mathptmx, they consist of two normal (single) brackets close together (duh), but it becomes too much with the thicker single brackets (which match the parenthesis) in newtx. Termes has solved the weight issue by having the single brackets in a double bracket slightly lighter, giving a more uniform look overall.

Furthermore, the double brackets in newtx are slightly taller than the normal brackets, which is not the case for the other two fonts. Given the bracket nesting, it might even be a good thing in this example, but it’s a strange mismatch in general.

Additionally, bracket kerning is quite generous in newtx. In mathptmx letters almost look like they reach into the brackets (although they actually don’t, but there’s no additional space either), but in newtx there is additional space. The gap between the double bracket and the single bracket is particularly visible in newtx.

On a side note, in mathptmx the \mathbb{T} in the subscript is clearly distinguishable from a normal T, but not so well in the others. In newtx this situation can perhaps be improved by choosing a different \mathbb variant using package options.

\leq vs \preceq

mathptmx

newtx

TeX Gyre Termes

The \preceq relation in Termes is mistakably similar to \leq, especially at normal text size, because it doesn’t bend as much as the others. The same probably applies to \prec vs <, etc.

The \preceq in newtx has a particularly small gap between \prec and the bottom equality line, but I can live with that over Termes.

On a side note, in mathptmx the ‘i’ appears much closer to the \leq relation than the ‘j’.

\setminus

mathptmx

newtx

TeX Gyre Termes

The \setminus operator in Termes is unusually wide: both in terms of its angle and its (left) kerning. I suppose it’s intended to match the width of other set operators (e.g. \cup, \cap) but I’m so used to the thinner one from Computer Modern. On the other hand, Termes has the most balanced left vs right kerning for it, with mathptmx being particularly uneven.

\nabla and \Delta

mathptmx

newtx

TeX Gyre Termes

The \nabla and \Delta are specifically used as binary operators in abstract interpretation and, thus, properly wrapped in \mathbin here. Surprisingly, they have different width (or kerning) in mathptmx, as clearly seen from the misalignment of their right arguments.

Annoyingly, \nabla in newtx is heavier than \Delta by having two thick sides instead of one. Not only are they asymmetric, the heaviness of \nabla stands out on a full page. Newtx offers \laplace as a similarly heavier version of \Delta, but that would make the latter stand out as well. Instead, I would like the opposite: a lighter version of \nabla that matches the \Delta, but newtx doesn’t offer that.

On a side note, in Termes \mathrm{\Delta} does not work, despite \Delta being already upright. This is a bit strange since unicode-math documentation explicitly mentions \mathup\Delta and that \mathup is just an alias for \mathrm. I’m not sure why I had used \mathrm{\Delta} in the first place, perhaps it’s a relic from using the operator macro with a different font which defaulted to italic uppercase Greek letters.

\times

mathptmx

newtx

TeX Gyre Termes

The visual perception may be somewhat due to the not-so-common use of the \times operator, but:

1. In mathptmx it is a bit light compared to the rest.
2. In newtx it is heavier but really touches the baseline and doesn’t look as good.
3. In Termes it is smaller (but suitably heavy) and closer to the middle line of the numbers, which looks the best in this context.

\texttimes

newtx

After switching to newtx I noticed that another similar instance looked different because it used the Unicode × in the LaTeX source. Turns out this is mapped to \texttimes which differs from $\times$ in newtx and fixes my problem.

Miscellany

mathptmx

newtx

TeX Gyre Termes

First, there is no \coloneqq in Termes. However, there is \coloneq, so the former omission is odd.

Second, there is no \square in Termes. The one in the Termes figure comes from amssymb and is thus the same one as in the mathptmx figure. It is too light for Times.

Third, differences between text mode and math mode parentheses are visible:

1. In mathptmx the math mode ones are too light.
2. In newtx they appear to be the same, which is great for consistency.
3. In Termes they are quite similar, although the math mode ones are slightly shorter but not too light.

Fourth, the \bowtie operators are very different in size:

1. In mathptmx it is the largest and most clearly visible.
2. In newtx it is scaled down and heavier. This kind of matches with the smaller \square in newtx though.
3. In Termes it is microscopic and horizontally squashed. The holes in the bowtie are hardly visible at normal text size.

\Join

newtx

After digging into newtx font files with FontForge I accidentally noticed that it contains two different bowties! Turns out the other one is is less squashed and provided as \Join in newtx.

Conclusion

Clearly mathptmx inferior to both newtx and Termes. Based on my observations above, I think newtx is the better one of the two. Termes needs some improvements to kerning and math operators.

Services

Let’s look at a bunch of services for getting BibTeX entries by DOI. I’ll use the DOI 10.1007/978-3-031-50524-9_4 (one of my papers) as the example.

Click on each tab to see the BibTeX entry from each service and my comments about it.

• DOI
• DOI formatter
• doi2bib
• Springer
• ACM
• DBLP condensed
• DBLP standard

•  @inbook{Saan_2023, title={Correctness Witness Validation by Abstract Interpretation}, ISBN={9783031505249}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-031-50524-9_4}, DOI={10.1007/978-3-031-50524-9_4}, booktitle={Verification, Model Checking, and Abstract Interpretation}, publisher={Springer Nature Switzerland}, author={Saan, Simmo and Schwarz, Michael and Erhard, Julian and Seidl, Helmut and Tilscher, Sarah and Vojdani, Vesal}, year={2023}, month=dec, pages={74–97} }
  

  This is returned by DOI Content Negotiation which simply means making an HTTP(S) request to the usual DOI URL https://doi.org/10.1007/978-3-031-50524-9_4 but with the Accept: application/x-bibtex HTTP header, i.e.

  curl -LH "Accept: application/x-bibtex" https://doi.org/10.1007/978-3-031-50524-9_4
  

  For this particular DOI, this actually delegates to the Crossref API at

  curl -L https://api.crossref.org/works/10.1007/978-3-031-50524-9_4/transform/application/x-bibtex
  

  Comments

  1. The entry type is @inbook, although @inproceedings would be more precise for this work.
  2. The url field has value http://dx.doi.org/10.1007/978-3-031-50524-9_4. There are two things wrong with that:
     1. It’s HTTP, not HTTPS.
     2. It uses dx.doi.org, not just doi.org.

     The former options in both points are no longer preferred, yet the official DOI metadata service doesn’t follow its own recommendations.

  3. The booktitle field is actually not specified for @inbook in BibTeX. It is specified for @inproceedings, so it really should be that. In BibLaTeX, booktitle is also specified for @inbook but only because BibLaTeX gives @inbook a slightly different meaning than BibTeX.
  4. The whole result is on one line (fine) and has a spurious single space in the beginning (which is odd).

•  @misc{Saan_Schwarz_Erhard_Seidl_Tilscher_Vojdani_2023, title={Correctness Witness Validation by Abstract Interpretation}, url={http://dx.doi.org/10.1007/978-3-031-50524-9_4}, DOI={10.1007/978-3-031-50524-9_4}, journal={Lecture Notes in Computer Science}, publisher={Springer Nature Switzerland}, author={Saan, Simmo and Schwarz, Michael and Erhard, Julian and Seidl, Helmut and Tilscher, Sarah and Vojdani, Vesal}, year={2023}, month=dec, pages={74–97}, language={en} }
  

  This is returned by the DOI Citation Formatter for the style bibtex, which can also be accessed through an API:

  curl 'https://citation.doi.org/format?doi=10.1007%2F978-3-031-50524-9_4&style=bibtex&lang=en-US'
  

  Comments

  It’s quite similar to the previous one from DOI Content Negotiation, but objectively worse:

  1. The entry type is now just @misc.
  2. The booktitle field is missing (it’s not specified for @misc anyway), and the title “Verification, Model Checking, and Abstract Interpretation” isn’t in any other field either.
  3. The journal field is now present (it’s not specified for @misc either!) and has value “Lecture Notes in Computer Science”, which isn’t a journal but a book series (which belongs to the series field, if it wasn’t for @misc).

• @inbook{Saan2023,
    title = {Correctness Witness Validation by Abstract Interpretation},
    ISBN = {9783031505249},
    ISSN = {1611-3349},
    url = {http://dx.doi.org/10.1007/978-3-031-50524-9_4},
    DOI = {10.1007/978-3-031-50524-9_4},
    booktitle = {Verification,  Model Checking,  and Abstract Interpretation},
    publisher = {Springer Nature Switzerland},
    author = {Saan,  Simmo and Schwarz,  Michael and Erhard,  Julian and Seidl,  Helmut and Tilscher,  Sarah and Vojdani,  Vesal},
    year = {2023},
    month = dec,
    pages = {74–97}
  }
  

  This is returned by doi2bib at https://www.doi2bib.org/bib/10.1007/978-3-031-50524-9_4. doi2bib is just a browser frontend for DOI Content Negotiation and performs client-side reformatting. As far as I have seen, many other tools actually do this under the hood.

  Comments

  It has all the issues of DOI Content Negotiation and only the following differences:

  1. The formatting is generally more human-friendly.
  2. The formatting adds double spaces after commas in field values. This shouldn’t affect Bib(La)TeX, but is odd nevertheless.

• @InProceedings{10.1007/978-3-031-50524-9_4,
  author="Saan, Simmo
  and Schwarz, Michael
  and Erhard, Julian
  and Seidl, Helmut
  and Tilscher, Sarah
  and Vojdani, Vesal",
  editor="Dimitrova, Rayna
  and Lahav, Ori
  and Wolff, Sebastian",
  title="Correctness Witness Validation by Abstract Interpretation",
  booktitle="Verification, Model Checking, and Abstract Interpretation",
  year="2024",
  publisher="Springer Nature Switzerland",
  address="Cham",
  pages="74--97",
  abstract="Witnesses record automated program analysis results and make them exchangeable. To validate correctness witnesses through abstract interpretation, we introduce a novel abstract operation unassume. This operator incorporates witness invariants into the abstract program state. Given suitable invariants, the unassume operation can accelerate fixpoint convergence and yield more precise results. We demonstrate the feasibility of this approach by augmenting an abstract interpreter with unassume operators and evaluating the impact of incorporating witnesses on performance and precision. Using manually crafted witnesses, we can confirm verification results for multi-threaded programs with a reduction in effort ranging from 7{\%} to 47{\%} in CPU time. More intriguingly, we discover that using witnesses from model checkers can guide our analyzer to verify program properties that it could not verify on its own.",
  isbn="978-3-031-50524-9"
  }
  

  This is returned by the “Download citation (.BIB)” feature of Springer Link which the particular DOI points to:

  curl 'https://citation-needed.springer.com/v2/references/10.1007/978-3-031-50524-9_4?format=bibtex&flavour=citation'
  

  Comments

  1. This is completely different from the previous ones based on DOI Content Negotiation. I guess because those actually come from Crossref’s database, while this one comes from Springer’s own database, but as a user I shouldn’t have to know or care. It’s still Springer submitting data to Crossref and the DOI URL itself redirects to Springer under normal conditions (standard HTTP request).
  2. The entry type is @InProceedings, which is more accurate than all the previous ones.
  3. The doi field is missing. The DOI is in the entry key, although that doesn’t help to have the DOI show up in a Bib(La)TeX bibliography.
  4. The url field is also missing. Thus, there would be no digital reference in a rendered bibliography.
  5. The formatting is multiline, but not indented.

• @inproceedings{10.1007/978-3-031-50524-9_4,
  author = {Saan, Simmo and Schwarz, Michael and Erhard, Julian and Seidl, Helmut and Tilscher, Sarah and Vojdani, Vesal},
  title = {Correctness Witness Validation by Abstract Interpretation},
  year = {2024},
  isbn = {978-3-031-50523-2},
  publisher = {Springer-Verlag},
  address = {Berlin, Heidelberg},
  url = {https://doi.org/10.1007/978-3-031-50524-9_4},
  doi = {10.1007/978-3-031-50524-9_4},
  abstract = {Witnesses record automated program analysis results and make them exchangeable. To validate correctness witnesses through abstract interpretation, we introduce a novel abstract operation unassume. This operator incorporates witness invariants into the abstract program state. Given suitable invariants, the unassume operation can accelerate fixpoint convergence and yield more precise results. We demonstrate the feasibility of this approach by augmenting an abstract interpreter with unassume operators and evaluating the impact of incorporating witnesses on performance and precision. Using manually crafted witnesses, we can confirm verification results for multi-threaded programs with a reduction in effort ranging from 7\% to 47\% in CPU time. More intriguingly, we discover that using witnesses from model checkers can guide our analyzer to verify program properties that it could not verify on its own.},
  booktitle = {Verification, Model Checking, and Abstract Interpretation: 25th International Conference, VMCAI 2024, London, United Kingdom, January 15–16, 2024, Proceedings, Part I},
  pages = {74–97},
  numpages = {24},
  keywords = {Correctness Witness, Witness Validation, Software Verification, Program Analysis, Abstract Interpretation},
  location = {London, United Kingdom}
  }
  

  This is returned by the “Export Citation” feature of ACM Digital Library at https://dl.acm.org/doi/10.1007/978-3-031-50524-9_4. Although the particular work is published by Springer, ACM seems to index it.

  Comments

  1. The title field value includes  , which is inappropriate for Bib(La)TeX.
  2. The publisher and address field values “Springer-Verlag” and “Berlin, Heidelberg” seem wrong because Springer itself returned “Springer Nature Switzerland” and “Cham”. (Although personally I don’t care: I would drop the address and simplify publisher to “Springer”.)
  3. The booktitle field value includes the book’s subtitle “25th International Conference, VMCAI 2024, London, United Kingdom, January 15–16, 2024, Proceedings, Part I”. In BibTeX, there’s no other way (except omitting it like in all previous services). BibLaTeX specifies the booksubtitle field, and even more appropriate ones like eventtitle, venue and eventdate (as also pointed out in this TeX StackExchange answer).
  4. The formatting is multiline, but not indented.

• @inproceedings{DBLP:conf/vmcai/SaanSESTV24,
    author       = {Simmo Saan and
                    Michael Schwarz and
                    Julian Erhard and
                    Helmut Seidl and
                    Sarah Tilscher and
                    Vesal Vojdani},
    title        = {Correctness Witness Validation by Abstract Interpretation},
    booktitle    = {{VMCAI} {(1)}},
    series       = {Lecture Notes in Computer Science},
    volume       = {14499},
    pages        = {74--97},
    publisher    = {Springer},
    year         = {2024}
  }
  

  This is returned by the “export record (BibTeX)” feature of DBLP at https://dblp.org/rec/conf/vmcai/SaanSESTV24.html?view=bibtex&param=0. DBLP offers multiple BibTeX formats, this being the condensed one.

  Comments

  1. The doi field is missing and, unlike Springer, it’s not in the entry key either.
  2. The url field is also missing.

  3. The volume field value is “14499” which actually corresponds to the series “Lecture Notes in Computer Science”. This is wrong in both BibTeX and BibLaTeX: it should instead be the number field with the value “14499”.

     BibTeX specifies:

     number

     The number of […] a work in a series. […] sometimes books are given numbers in a named series.

     volume

     The volume of a journal or multivolume book.

     BibLaTeX specifies:

     number

     […] the volume/number of a book in a series.

     volume

     The volume of a multi-volume book or a periodical.

     This has also been pointed out in this TeX StackExchange answer.

  4. The booktitle field value is essentially “VMCAI (1)”, where the 1 refers to the part. The latter is what actually should go into the volume field according to the specifications above.

     Alternatively, BibLaTeX also specifies:

     part

     The number of a partial volume. This field applies to books only, not to journals. It may be used when a logical volume consists of two or more physical ones. In this case the number of the logical volume goes in the volume field and the number of the part of that volume in the part field.

     The distinction between logical and physical is a bit hazy in this case. Even Springer cannot make up their mind about the terminology:

     1. The subtitle of the book ends with “Part I”.
     2. The Springer Link page for the book has the section “Other volumes”.
     3. The “About this book” section on the same page mentions both, while starting with “The two-volume set LNCS 14499 and 14500 […]”.
  5. The formatting is the nicest of them all. Although, when copying the BibTeX code from the DBLP website, the copied text includes two empty leading and trailing lines for some reason. The empty lines are not present in the downloadable .bib file.

• @inproceedings{DBLP:conf/vmcai/SaanSESTV24,
    author       = {Simmo Saan and
                    Michael Schwarz and
                    Julian Erhard and
                    Helmut Seidl and
                    Sarah Tilscher and
                    Vesal Vojdani},
    editor       = {Rayna Dimitrova and
                    Ori Lahav and
                    Sebastian Wolff},
    title        = {Correctness Witness Validation by Abstract Interpretation},
    booktitle    = {Verification, Model Checking, and Abstract Interpretation - 25th International
                    Conference, {VMCAI} 2024, London, United Kingdom, January 15-16, 2024,
                    Proceedings, Part {I}},
    series       = {Lecture Notes in Computer Science},
    volume       = {14499},
    pages        = {74--97},
    publisher    = {Springer},
    year         = {2024},
    url          = {https://doi.org/10.1007/978-3-031-50524-9\_4},
    doi          = {10.1007/978-3-031-50524-9\_4},
    timestamp    = {Sat, 10 Feb 2024 18:04:44 +0100},
    biburl       = {https://dblp.org/rec/conf/vmcai/SaanSESTV24.bib},
    bibsource    = {dblp computer science bibliography, https://dblp.org}
  }
  

  This is returned by the “export record (BibTeX)” feature of DBLP at https://dblp.org/rec/conf/vmcai/SaanSESTV24.html?view=bibtex&param=1. DBLP offers multiple BibTeX formats, this being the standard one.

  Comments

  It is mostly an extension of the previous one from DBLP, but with additional fields which can be (and are) treated incorrectly:

  1. The doi field value has the underscore escaped. This is unnecessary and even wrong: DOI lookup returns “DOI Not Found”.
  2. The url field value also has the underscore escaped. This is again unnecessary and even wrong: the url is broken.
  3. The booktitle field value is uncondensed, but has the same issues as with ACM.

The tab content ends here.¹

Comparison

Here’s a table to summarize some aspects of the entries returned by the services. The values I consider acceptable are in bold² and the values I prefer are in italic.

The table also compares the year field values which weren’t discussed above. Surprisingly, there even isn’t consensus about such a basic fact. It probably has to do with “First Online: 30 December 2023”. The Crossref data for the DOI seems to correspond to that, while Springer itself considers the publication to be in 2024, which is also when the conference took place. This just goes to show that the DOI Content Negotiation data, which gets used by many other services, may be inaccurate w.r.t. the very basics.

Conclusion

I learned about DOI Content Negotiation and how bad it actually is for BibTeX. The databases (Springer, ACM, DBLP) are better, but none is perfect or even good enough, as the comparison table reveals. I guess I’ll end up doing a lot of manual work, although some is semi-automatable using BibLaTeX source maps (which are a story for another time).

Package registry search

One can search for the entire barcode of a product to see whether it’s included in the registry and some extra data about it. But one can also search for a part of the barcode to find products whose barcodes include it as a substring. However, there’s an important restriction: the search returns at most 10 products (like SQL’s LIMIT 10). Additionally, the empty substring cannot really be searched (misleadingly, it returns zero products).

Scraping problem

With the search at hand, let’s consider scraping the entire package registry, i.e. finding barcodes of all the registered packages and the extra data about them.² The main focus is on the barcodes because the extra data is a simple byproduct: when the barcode of a product is confirmed to be in the registry by seeing it in the results of some search query, then we also get its extra data and can record it on the side.

The package registry is essentially rate-limited to 1 query per second (technically, 60 queries per minute). Therefore, the goal is to minimize the number of queries needed to completely scrape the registry.

Naïve solution

Since the barcodes are numeric and (mostly) of length 13, there’s an obvious algorithm to scrape the registry: just query each complete barcode to see whether zero or one results are returned. Obviously, this algorithm is completely impractical because it requires \(10^{13}\) queries. (This could be reduced by exploiting the structure of EAN-13 barcodes but that’s not enough to make it practical.)

To further complicate things, the registry also contains small numbers of 12-digit UPC-A and 8-digit EAN-8 barcodes, which require additional queries to scrape. Furthermore, a priori there’s no indication which lengths of barcodes exist in the registry. The 10 result limit for a query means that an extra assumption is needed to make the scraping problem properly solvable at all. Namely, for every complete barcode \(b\) in the registry, the registry contains at most 10 barcodes which have \(b\) as a substring (including \(b\) itself). If this wasn’t the case, then there would be no guarantee that querying \(b\) would necessarily return \(b\). This assumption (probably severely) limits the potential search space.³

Basic online solution

The naïve algorithm, besides being impractical, has another limitation: it first fixes all the many queries to be made and then performs them all. For a feasible solution, we can use an online algorithm, which determines new queries to make based on the results of previous queries.

My first idea for such an algorithm was the following. First, query 0: if it returns less than 10 results (which is unlikely), then we immediately know all barcodes in the registry which include 0; otherwise, recursively query 00, 01, …, 09. For each of those, behave similarly: if there are less than 10 results, then stop recursing; otherwise, continue recursing to all 10 extensions of the query. And so on…. Once all of this is done, repeat the same process starting with 1, 2, …, 9. (The general approach would be to start one recursion from the empty string, but due to the empty query corner case that wouldn’t actually work.)

I have not proven this algorithm to be correct (nor defined what correctness means, yet). So if it isn’t, then this post has already gone off the rails. Comment below if you believe that’s the case!

Suffix tree solution

The basic online algorithm can be slightly improved using a (generalized) suffix tree. To this end, every barcode returned by any query (including those with at least 10 results) is added to a generalized suffix tree, e.g. efficiently using Ukkonen’s algorithm. (For everything related to suffix trees, I would recommend reading (Gusfield, 1997).) Before making any actual query, we can efficiently simulate the query on the suffix tree to check if we have already seen at least 10 barcodes with the query as substring. If this is the case, then we can skip the actual query and immediately proceed with recursion, because the actual query would also return 10 results and force us to recurse anyway.

I used this approach to scrape the actual registry on 2025-08-22, which yielded 10171 barcodes (9535 EAN-13, 377 UPC-A and 259 EAN-8).⁴ Hoping this algorithm is correct, I have a copy of the entire registry at hand locally, which makes it much easier and faster to experiment with different solutions: the registry can be simulated without any rate limiting.⁵ On this local simulation, the suffix tree algorithm uses 73713 queries to scrape the registry. With the actual rate limiting, this requires almost ~20.5 hours. In comparison, the basic algorithm uses 81250 queries, which is ~10% more.

Better solutions?

I suspect the suffix tree solution is far from optimal because it discovers each barcode in the registry numerous times, corresponding to each suffix of each barcode. Intuitively, it should be possible to do better. For example, if 0000 and its extensions have already been scraped, then later it would be a waste to scrape 10000 and its extensions, because all of them have already been found. Except, that’s not true! If the 0000 query has at least 10 results and 00001, …, 00009 have all been scraped, then we might not have seen barcodes that end with 0000. For this tricky reason, my attempts at further optimization (e.g. also using a suffix tree of all performed queries to find such query inclusions) have failed to correctly scrape the entire registry. Such pruning can reduce the number of queries by over 50%, but also cause it to miss a handful of barcodes, which is no good (“It doesn’t work, but it’s fast”).

Comment below if you have (ideas for) a better correct solution!

Verification problem

With the scraped registry at hand, let’s consider checking the result, i.e. the scraped dataset has exactly the same barcodes as the actual registry. This set equality can be viewed as two set inclusions: all the scraped barcodes exist in the actual registry, but also all the barcodes in the actual registry were scraped. Let’s call these properties soundness and completeness, respectively.

In a way, verification is like solving the scraping problem while knowing the answer to begin with. So intuitively, it should be doable with fewer queries.

Soundness

Naïve solution

The obvious algorithm is to just query each scraped barcode from the actual registry and confirm that it is included in the results. (If any of the results contains new barcodes, we’ve inadvertently disproven completeness.) Thus, this requires as many queries as there are barcodes in the registry, which can’t be the most optimal. For the registry scraped above, it requires 10171 queries.

Suffix tree solution

Since the answer is already known, a generalized suffix tree of all the barcodes can be constructed to begin with. The verification algorithm is then to traverse the suffix tree and query from the actual registry the substring corresponding to each suffix tree path (from its root) where the subtree under that node has at most 10 barcodes (while its parent has more). Unlike suffix tree solution to the scraping problem, nodes representing exactly 10 barcodes should also be fine here.

As with the suffix tree solution to the scraping problem, this also checks each barcode multiple times, corresponding to each suffix of each barcode. Having implemented this approach, for the registry scraped above, it requires 29794 queries, which is much worse than even the naïve algorithm.⁴

Suffix tree and set cover solution

Since we have the answer, we can do some optimization to avoid verifying each barcode numerous times. Like the previous solution, the suffix tree provides all substrings we could query and their expected results. Minimizing the number of queries to cover all the barcodes is an instance of the set cover problem, which, somewhat unfortunately, is NP-hard. So curiously, to efficiently verify the solution to the scraping problem, we have to solve (not verify) an instance of the difficult set cover problem.

Luckily, the greedy algorithm for the set cover problem isn’t too shabby in this case: having implemented it, for the registry scraped above, it requires just 2159 queries, which is almost 5 times less than the naïve solution.⁴ Theoretical results about the greedy algorithm show that its over-approximation ratio here is \(H(10) \approx 2.93\), where 10 is the size of the largest set (picked from the suffix tree). In other words, the greedy algorithm can be at most ~2.93 times worse than the optimal set cover size. Or put another way, for the registry scraped above, the optimal number of verification queries is at least 738.

As far as I managed to find, similar ideas have been proposed to solve a slightly different string barcoding problem (Rash & Gusfield, 2002). Coincidentally, it also mentions barcodes by name, albeit with a completely different meaning: it’s a problem in computational biology for constructing some kind of probes to distinguish a set of DNA sequences. Nevertheless, their solution also processes nodes of a suffix tree in relation to the strings contained in their corresponding subtrees. However, they use it to construct an integer linear programming (ILP) problem to then solve (i.e. optimize). Also coincidentally, the set cover problem can be stated as an ILP problem. So, obviously, ILP is also NP-hard, but can be approximated via relaxation to non-integer linear programming. That approximation doesn’t necessarily ensure as good of a solution than the greedy algorithm does for the scraped database because here it has approximation ratio of 11.

Completeness

Checking completeness is not as easy: we have to check whether every barcode in the actual registry is in our scraped answer, but knowing the former is the scraping problem itself! It seems more promising to check the contrapositive: whether every barcode not in our scraped answer also is not in the actual registry. This doesn’t sound much easier at first: most barcodes are not in the answer and iterating over them would be unrealistic.

Suffix tree solution

Although I have not fully worked out the algorithm for this, I believe the suffix tree used for soundness can also be useful for completeness. Namely, every branch which is missing in the suffix tree corresponds to a partial barcode which isn’t a substring of any barcode in the scraped answer. The suffix tree path (from its root) to this hypothetical missing branch serves as a query which should return zero results from the actual registry, otherwise our answer is incomplete.

Better solutions?

As always, such suffix-tree–based checking incurs some redundancy. For example, to prove that the barcode 000008 isn’t in the registry, it’s enough that the query 0000 returns zero results, making it unnecessary to also query 0008, which might also be missing. However, the latter might still be necessary to rule out other substrings.

Removing the redundancy using set cover isn’t as simple for the negative case: each of the missing branches (likely) represents a very large (complete) suffix subtree. Thus, the resulting set cover problem instance would not be realistically solvable.

Comment below if you have (ideas for) a better solution!

Joint verification

It is possible to exploit a single query for both soundness and completeness (as mentioned during soundness). If a query returns less than 10 barcodes, we both can confirm that our scraped ones are included, but also that no others containing the queried substring is present. Thus, a joint set cover problem to cover all barcodes in the answer and all missing branches in the suffix tree might yield a not-too-bad solution overall.

Update problem

I scraped the registry at some point in time (or more precisely, over a duration of time) and hypothetically even verified it to be correct soon after. However, the actual registry will inevitably change: new products with new barcodes will be added (and perhaps some old ones removed, although I’m not sure if the registry would ever do that — it still contains some quite old products). Hence, another interesting problem arises: how to update the scraped registry without scraping it from scratch?

I haven’t put much thought into this, but I suspect the suffix tree will again come in handy. Maybe one can try verifying the current scraped registry and, when a mismatch is detected, do some scraping from that point, but it’s just a vague idea. I would expect the amount of extra scraping to be somewhat proportional to the size of the registry change. Of course, if the registry is completely replaced with a disjoint set of barcodes (a hypothetical worst case scenario), then it’s probably hopeless to beat from scratch scraping.

Comment below if you have (ideas for) a solution!

Conclusion

What started as a silly exercise in scraping a registry turned into a set of quite complicated algorithmic problems. And in no way can I claim any of these to be solved. Therefore, I challenge the reader to come up with better solutions and quickly try them out on the locally simulated registry.⁴

References

Nevertheless, it may be desirable to make them accessible only to authenticated users. Luckily, oauth2-proxy is a (reverse) proxy service that lets us put authentication in front of any such application by delegating the authentication part to another service of choice, e.g. Google, GitHub, etc. In order to not depend on any external authentication service, it’s also possible to use the Synology DSM and its user management for authentication, just like it is for Synology’s own applications. And this post will show how to do exactly that.

Introduction

This post assumes that you have the Synology DSM accessible over HTTPS at https://example.com:5001, i.e. “example.com” stands for your own domain and Synology DSM is already configured with a valid HTTPS certificate for it. For example, this can be achieved using Let’s Encrypt or Tailscale.

The tutorial will set up the following:

• For example’s sake, the insecure unauthenticated application will be httpbin, which will not be exposed directly. This can be replaced with the insecure unauthenticated application of choice.
• The insecure oauth2-proxy will be exposed at http://example.com:5400 and will serve as a reverse proxy to httpbin once the user is authenticated. It is insecure only in the sense of HTTP — it is still authenticated.
• The secure oauth2-proxy will be exposed at https://example.com:5401 via a reverse proxy to the insecure version. This allows the HTTPS certificate for “example.com” to be reused without having to also set up a HTTPS certificate in oauth2-proxy itself.
• The authentication will be provided by Synology DSM at https://example.com:5001.

Step-by-step

All of the following steps are to be performed in the Synology DSM.

Reverse Proxy

1. Navigate to Control Panel → Login Portal → Advanced → Reverse Proxy.

2. Create a new reverse proxy (Create) with the following details:

   • Reverse Proxy Name: “oauth2-proxy”.
   • Source:
     • Protocol: HTTPS.
     • Hostname: “example.com”.
     • Port: 5401.
   • Destination:
     • Protocol: HTTP.
     • Hostname: “localhost”.
     • Port: 5400.
3. Press “Save” and close the Reverse Proxy window.

SSO Server

1. Install “SSO Server” from Package Center.
2. Open SSO Server.
3. Under General Settings, set Server URL: “example.com:5001” (the “https://” prefix will be implicitly there).
4. Under Service → OIDC, Enable OIDC server.

5. Under Application, add a new application (Add) with the following details:

   1. Select an SSO protocol: OIDC.
   2. Application Name: “oauth2-proxy”.
   3. Redirect URI: “https://example.com:5401/oauth2/callback”.
6. Select the just-added application from the list and click “Edit” in the toolbar.
7. Make note of the generated “Application ID” and “Application secret”, which will be needed in the next step. You can just keep the Edit window open and come back to copy them when needed.

oauth2-proxy

Configuration

1. In File Station, create a folder for oauth2-proxy configuration files, e.g. /docker/oauth2-proxy.
2. Using Text Editor, create the files oauth2-proxy.cfg and authenticated_emails with the contents described below, and save them into the previously-created folder.

oauth2-proxy.cfg

Copy the following:

http_address="0.0.0.0:5400"
reverse_proxy="true"
upstreams="http://httpbin"

# cookies
cookie_secret="TODO"
cookie_secure="true"
cookie_domains=["example.com"]
whitelist_domains=["example.com:5401"]

# Synology
provider="oidc"
oidc_issuer_url="https://example.com:5001/webman/sso"
client_id="TODO"
client_secret="TODO"
redirect_url="https://example.com:5401/oauth2/callback"
code_challenge_method="S256"
skip_provider_button="true"

# authentication
oidc_email_claim="sub"
authenticated_emails_file="/authenticated_emails"

Replace all TODOs as follows:

• Replace cookie_secret with a freshly-generated cookie secret, as described here.
• Replace client_id with “Application ID” from SSO Server.
• Replace client_secret with “Application secret” from SSO Server.

authenticated_emails

Write a list of authorized Synology usernames, one on each line. For example:

myuser

Despite the file and the oauth2-proxy configuration calling them emails, we don’t use emails here. Using emails from Synology users would be insecure because users can arbitrarily change their email. Hence, we make oauth2-proxy use usernames in place of emails using oidc_email_claim="sub". I learned about this trick from Okta’s tutorial Add Auth to Any App with OAuth2 Proxy.

Permissions

1. In File Station, navigate to the previously-created folder for oauth2-proxy configuration files.
2. Right-click on oauth2-proxy.cfg and select “Properties”.
3. Switch to “Permission” tab.

4. Press “Create” in the toolbar and enter the following details:

   • User or group: Everyone.
   • Type: Allow.
   • Permission: Read.
5. Press “Done” and then “Save”.
6. Repeat for authenticated_emails.

This is recommended by Synology to fix permissions issues with files mapped into Docker containers (in the next step).

Container

1. Install “Container Manager” from Package Center.
2. Navigate to Container Manager → Project.

3. Create a new project (Create) with the following details:

   • Project name: “oauth2-proxy”.
   • Path: “/docker/oauth2-proxy”.
   • Source: Create docker-compose.yml.
4. Paste the following Docker Compose file into the text box below:

   version: '3.0'
   services:
     oauth2-proxy:
       container_name: oauth2-proxy
       image: quay.io/oauth2-proxy/oauth2-proxy:v7.10.0
       command: --config /oauth2-proxy.cfg
       hostname: oauth2-proxy
       volumes:
         - "./oauth2-proxy.cfg:/oauth2-proxy.cfg:ro"
         - "./authenticated_emails:/authenticated_emails:ro"
       restart: unless-stopped
       ports:
         - 5400:5400/tcp
       networks:
         httpbin: {}
       depends_on:
         - httpbin
     httpbin:
       container_name: httpbin
       image: kennethreitz/httpbin
       restart: unless-stopped
       ports: []
       networks:
         httpbin: {}
   networks:
     httpbin: {}
   

5. Press “Next”, “Next” and “Done”.
6. Wait for the Docker containers to be downloaded and started.

Conclusion

If all went well, then you can now navigate to https://example.com:5401 in your browser. This should first redirect to Synology DSM login. However, since you should already be logged into Synology DSM, you will not be prompted to log in again. Finally, you will be redirected back to https://example.com:5401, but this time you’ll see the httpbin application instead.

You can replace the httpbin container with your desired Docker-based application. Importantly, it should not expose any ports. Instead, it will only be accessed via the Docker network (also called httpbin in this case) by oauth2-proxy, which has access to the same network. You may need to adapt upstreams in oauth2-proxy.cfg for your application and restart the oauth2-proxy container for changes to take effect.

\begin{minted}[linenos]{c}
for (int i = 0; i < 100; i++) {
    printf("i = %d\n", i);
}
\end{minted}

This is rendered as:

For completeness, here’s the preamble used for examples throughout this post:

\usepackage{minted}
\setminted{style=tango}
\usepackage[svgnames]{xcolor}
\usepackage{inconsolata}

Full line highlight

Occasionally it is useful to highlight a particular line of code. Conveniently, minted provides the highlightlines option to do so by line number:

\begin{minted}[linenos,highlightlines=1]{c}
for (int i = 0; i < 100; i++) {
    printf("i = %d\n", i);
}
\end{minted}

The following image comparison illustrates the change in rendering compared to no highlight (hover/swipe across the image):¹

Partial line highlight

Occasionally it would be useful to not highlight the entire line of code but only a part of it. Unfortunately, minted does not provide a straightforward way to do so. So let’s try to come up with a custom solution!

Attempt 1

The obvious way is to use a \colorbox within minted’s escapeinside:

\newcommand{\codehighlight}[1]{\colorbox{LightCyan}{#1}}
\begin{minted}[linenos,escapeinside=||]{c}
for (int i = 0; |\codehighlight{i < 100}|; i++) {
    printf("i = %d\n", i);
}
\end{minted}

The color LightCyan is just what minted has as default for highlightcolor.

The following image comparison illustrates the change in rendering compared to full line highlight:

This has multiple issues. For one, the characters in the \colorbox don’t align with the rest of the columns of monospaced text.

Attempt 2

This is caused by the default value of \fboxsep, which is used to pad the contents of the \colorbox on all sides. To avoid the padding causing misalignment, it can be set to 0pt:

\newcommand{\codehighlight}[1]{{\setlength{\fboxsep}{0pt}\colorbox{LightCyan}{#1}}}
\begin{minted}[linenos,escapeinside=||]{c}
for (int i = 0; |\codehighlight{i < 100}|; i++) {
    printf("i = %d\n", i);
}
\end{minted}

The following image comparison illustrates the change in rendering compared to the first attempt:

The columns of monospaced characters now align, but the highlighting is very tight, also vertically (unlike highlightlines).

Attempt 3

We could do some trickery to effectively get different horizontal and vertical \fboxsep. However, it’s much easier to just insert a \strut, which is an invisible zero-width box that (more-or-less) accounts for the line height:

\newcommand{\codehighlight}[1]{{\setlength{\fboxsep}{0pt}\colorbox{LightCyan}{\strut #1}}}
\begin{minted}[linenos,escapeinside=||]{c}
for (int i = 0; |\codehighlight{i < 100}|; i++) {
    printf("i = %d\n", i);
}
\end{minted}

The following image comparison illustrates the change in rendering compared to the second attempt:

This fixes the alignment and padding issues,² but there’s still a noticeable problem: the text, which we have highlighted for the user using the background color, is not being code highlighted by minted at all.

Attempt 4 (solution)

As far as I know, it’s not really possible to put \mintinline inside the minted environment to somehow try to fix this. Instead, we can do something strange with how our \codehighlight macro is used:

\newcommand{\codehighlight}[1]{{\setlength{\fboxsep}{0pt}\colorbox{LightCyan}{\strut #1}}}
\begin{minted}[linenos,escapeinside=||]{c}
for (int i = 0; |\codehighlight{{|i < 100|}}|; i++) {
    printf("i = %d\n", i);
}
\end{minted}

Previously, we had a single escapeinside that contained the macro applied to its contents. Now, there are two escapeinsides: one before the contents and one after:

1. The one before only calls the macro and starts its argument. Importantly, this needs double { because minted puts the escapeinside contents into a group (i.e. between braces like {\codehighlight{{}). The first brace starts the macro argument, the second is only there to cancel out the group-closing brace. Without the latter, \codehighlight would just be given an empty argument.
2. The one after only ends the macro argument. Analogously, this needs double }} (i.e. grouped like like {}}}). The first brace is only there to cancel out the group-opening brace and the second ends the macro argument.

The following image comparison illustrates the change in rendering compared to the third attempt:

The strange amalgamation fixes the code highlighting “within” \codehighlight.

Finally, this image comparison illustrates the change in rendering compared to no highlight:

Luckily, minted builds on the much older fancyvrb package, which provides basic support for referencing lines. For example, the desired line can be marked using \label using escapeinside:

\begin{minted}[linenos,escapeinside=||]{python}
print("foo")
print("bar")|\label{ln:bar}|
print("baz")
\end{minted}

Then elsewhere \ref{ln:bar} expands to “2”.

Problem

Unfortunately, that’s also where the support ends. In particular, fancyvrb is incompatible with two other often-used packages:

1. The hyperref package makes \ref{ln:bar} a hyperlink, but it’s a link to the wrong place! The link doesn’t go to the specific line of code or not even the beginning of the minted environment it is in. Instead, it goes to whatever happens to be the previous hyperref anchor at the point of \label{ln:bar}, e.g. a previous section heading, figure caption, etc. If you’re lucky, this might at least be on the same page as the code, but it could also be many pages before.

2. The cleveref package adds \cref and friends, which prefix the reference number with its kind, e.g. \cref{sec:introduction} might expand to “section 1” instead of just “1”. The incompatibility with fancyvrb is similar to the one with hyperref: \cref{ln:bar} expands to a complete reference to something before the code, e.g. “section 1”. Not only is the “section” prefix wrong, but the number isn’t even the line number “2”. (At least it’s self-consistent: the number is for whatever is being referenced instead.)

TeXnical details

Deep inside LaTeX, the problem stems from the fact that both hyperref and cleveref achieve their functionality by redefining \refstepcounter. However, fancyvrb does not use that standard command and instead defines its own version:

\def\FV@refstepcounter#1{
  \stepcounter{#1}
  \protected@edef\@currentlabel{\csname p@#1\endcsname\arabic{FancyVerbLine}}
}

It explicitly uses \arabic{FancyVerbLine} instead of \theFancyVerbLine, which the standard definition would use. As far as I can tell, it needs to do that because it defines the latter as:

\def\theFancyVerbLine{\rmfamily\tiny\arabic{FancyVerbLine}}

which is weird because it puts the number formatting (for in the code environment) into the counter value formatting itself, which no other sensible counter would do. And \FV@refstepcounter is explicitly there to not make the number tiny at \ref{ln:bar}.

Non-solutions

There have been some attempts at avoiding the incompatibilities:

1. A reddit thread suggests to use |\phantomsection\label{ln:bar}| in the minted environment instead. The \phantomsection provided by hyperref inserts a fresh anchor which \ref{ln:bar} will link to. This comes with two problems:
   1. It’s outright annoying to have to manually insert \phantomsection before every \label in code.
   2. The anchor inserted by \phantomsection is not at the beginning of the line of code, but at the specific column where the escaped label is placed. One could put all code line labels at the beginning of lines, but that makes the code even less readable: at the end of lines the code at least maintains its indentation in the LaTeX sources.
2. One file on GitHub uses \let\FV@refstepcounter\refstepcounter to make fancyvrb use the standard command and thus allow hyperref/cleveref to modify it as usual. In order to avoid the tiny numbers at \ref{ln:bar}, it goes on to patch fancyvrb in various places to move the \tiny to a more appropriate place by adding numberstyle customization option. This is morally the right approach, but unfortunately also comes with two problems:
   1. It breaks fancyvrb’s firstnumber option and causes the first two lines to have the same number. Seems like fancyvrb’s logic is very particular to its oddities.
   2. The resulting hyperref anchors are based on the FancyVerbLine counter, which isn’t globally unique. So pdflatex warns about duplicate destinations and all of them link into the first minted environment, not the one where the \label actually is.
3. Another file on GitHub only patches \FV@refstepcounter to add \refstepcounter of an additional global counter, avoiding the last issue. But it’s also not well-behaved, at least with minted: extra empty space appears before minted environments. I believe this is because minted works in two phases:
   1. The contents of a minted environment are not typeset, but written to a file (to run Pygments on). While doing so, fancyvrb still steps the counter which inserts spurious hyperref anchors but nothing else is typeset yet.
   2. After running Pygments, its result is somehow input into some fancyvrb environment for actual typesetting. This is what actually displays the syntax-highlighted code along with the line numbers (which are produced by stepping the line counter again).
4. A TeX StackExchange answer defines alternative \label and \ref commands to use just for lines of code, which isn’t entirely satisfactory (one doesn’t need separate commands for sections, figures, etc.). It bypasses the usual hyperref anchor mechanism and uses \hypertarget and \hyperlink to directly work with custom PDF destinations. It also tries to provide a hint for cleveref, but admits that it still produces a wrong reference.

Solution

After digging deep into the implementation of fancyvrb, minted, hyperref and cleveref to (try to) understand how they work, I managed to put together a solution which seems to achieve the desired functionality without any of the downsides listed above. I wrapped my solution into my new fancyvrbref package. I haven’t (yet) published it on CTAN, but you can just copy it into your project for the time being.

The solution is the following:

1. For hyperref compatibility, the fancyvrb internal command for actually typesetting a line is patched to insert a globally unique anchor with FancyVerbLine* prefix. Since this isn’t in \FV@refstepcounter, it doesn’t screw up minted.
2. For cleveref compatibility, the counter FancyVerbRefLine is defined as an alias for FancyVerbLine, but without tiny font, and the \FV@refstepcounter command is extended to define \cref@currentlabel (for cleveref), \@currentlabel (for consistency with \ref), and \@currentcounter (for cleveref with LaTeX2e format since 2024-11-01). This avoids modifying \theFancyVerbLine.

Let me know if you find this package useful or find any issues with it!

As of July 6, 2025,¹ the register contained 2555 theses in total. I have excluded some from the following analysis:

• 8 theses from before 2010, because it seems like the data for those years is incomplete.

This leaves 2547 theses for the analysis. Note that the data for 2025 is not yet final.

Word processors used

The institute provides thesis templates for Microsoft Word and LaTeX. I wanted to find out how much each of them is used by the students.

This is complicated by the fact that theses are submitted as PDFs. Luckily, PDF file metadata contains two fields which give a lot of insight: PDF creator and PDF producer. Although, the content of these fields is not standardized, it’s not as messy as web browser User-Agent headers (yet). Working with the data, I reached the following classification:

• Microsoft Word if the PDF creator matches Microsoft®? (Office )?Word|Acrobat PDFMaker .* for Word.
• TeX if the PDF creator contains TeX.
• Google Docs if the PDF producer matches Google Docs|Skia/PDF.
• LibreOffice if the PDF producer matches (Libre|Open)Office.
• Quartz if the PDF creator contains Quartz PDFContext. These are somehow created by MacOS, but it’s unclear to me how.
• Print if the PDF producer matches Microsoft: Print To PDF|Foxit Reader (PDF )?Printer|PDF Printer. These are various PDF printers.
• Unknown otherwise.

Overall

First, let’s look at the overall word processor breakdown:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "aggregate": [{
        "op": "count",
        "as": "count"
      }],
      "groupby": ["pdf_classification"]
    },
    {
      "joinaggregate": [{
        "op": "sum",
        "field": "count",
        "as": "total_count"
      }],
      "groupby": []
    },
    {
      "calculate": "datum.count / datum.total_count",
      "as": "fraction"
    },
    {
      "lookup": "pdf_classification",
      "from": {
        "data": {
          "values": [
            {"pdf_classification": "Microsoft Word", "classification_order": 0},
            {"pdf_classification": "LibreOffice", "classification_order": 1},
            {"pdf_classification": "Google Docs", "classification_order": 2},
            {"pdf_classification": "Quartz", "classification_order": 3},
            {"pdf_classification": "Print", "classification_order": 4},
            {"pdf_classification": "Unknown", "classification_order": 5},
            {"pdf_classification": "TeX", "classification_order": 6}
          ]
        },
        "key": "pdf_classification",
        "fields": ["classification_order"]
      }
    }
  ],
  "width": "400",
  "mark": "arc",
  "encoding": {
    "theta": {
      "field": "count",
      "type": "quantitative",
      "aggregate": "sum",
      "stack": "normalize",
      "title": "Theses"
    },
    "color": {
      "field": "pdf_classification",
      "title": "PDF creator"
    },
    "order": {
      "field": "classification_order"
    },
    "tooltip": [
      {
        "field": "fraction",
        "format": ".0%",
        "title": " Theses"
      },
      {
        "field": "count",
        "title": "Theses"
      }
    ]
  }
}

This shows that Microsoft Word is used slightly more than LaTeX. Notably, Word-like WYSIWYG editors make up the majority.

Now, let’s dig a little deeper to see how the breakdown depends on the year and the curriculum.

By year

Second, let’s look at the word processor breakdown across the years:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "aggregate": [{
        "op": "count",
        "as": "count"
      }],
      "groupby": ["pdf_classification", "defence_year"]
    },
    {
      "joinaggregate": [{
        "op": "sum",
        "field": "count",
        "as": "year_count"
      }],
      "groupby": ["defence_year"]
    },
    {
      "calculate": "datum.count / datum.year_count",
      "as": "year_fraction"
    },
    {
      "lookup": "pdf_classification",
      "from": {
        "data": {
          "values": [
            {"pdf_classification": "Microsoft Word", "classification_order": 0},
            {"pdf_classification": "LibreOffice", "classification_order": 1},
            {"pdf_classification": "Google Docs", "classification_order": 2},
            {"pdf_classification": "Quartz", "classification_order": 3},
            {"pdf_classification": "Print", "classification_order": 4},
            {"pdf_classification": "Unknown", "classification_order": 5},
            {"pdf_classification": "TeX", "classification_order": 6}
          ]
        },
        "key": "pdf_classification",
        "fields": ["classification_order"]
      }
    }
  ],
  "width": "725",
  "mark": "bar",
  "encoding": {
    "x": {
      "field": "defence_year",
      "title": "Year",
      "axis": {
        "labelAngle": 0
      }
    },
    "y": {
      "field": "count",
      "type": "quantitative",
      "stack": "normalize",
      "title": "Theses"
    },
    "color": {
      "field": "pdf_classification",
      "title": "PDF creator"
    },
    "order": {
      "field": "classification_order"
    },
    "tooltip": [
      {
        "field": "year_fraction",
        "format": ".0%",
        "title": "Theses (of year)"
      },
      {
        "field": "count",
        "title": "Theses"
      }
    ]
  }
}

This reveals two main trends:

1. OpenOffice/LibreOffice usage has mostly diminished.
2. Google Docs usage has become widespread.

The latter is worrying because Google Docs is (in my opinion) inadequate for typesetting a thesis. Having supervised and reviewed a number of theses (although relatively few compared to senior staff members), it’s often obvious that a thesis has been typeset in Google Docs based on poor and inconsistent formatting. Importing the Microsoft Word template into Google Docs is a lossy conversion because Docs has limited features and customizability, even compared to Word.

By curriculum

Third, let’s look at the relationship between word processor usage and curricula:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "lookup": "curriculum",
      "from": {
        "data": {
          "values": [
            {"curriculum": "bsc_computer_science", "curriculum_name": "BSc - Computer Science"},
            {"curriculum": "msc_computer_science", "curriculum_name": "MSc - Computer Science"},
            {"curriculum": "msc_software_engineering", "curriculum_name": "MSc - Software Engineering"},
            {"curriculum": "msc_data_science", "curriculum_name": "MSc - Data Science"},
            {"curriculum": "msc_data_science_exam", "curriculum_name": "MSc - Data Science (exam)"},
            {"curriculum": "msc_cyber_security", "curriculum_name": "MSc - Cyber Security"},
            {"curriculum": "msc_conversion_master_in_it", "curriculum_name": "MSc - Conversion Master in IT"},
            {"curriculum": "msc_innovation_and_technology_management", "curriculum_name": "MA - Innovation and Technology Management"},
            {"curriculum": "ma_maths_and_informatics_teacher", "curriculum_name": "MA - Teacher of Mathematics and Informatics"},
            {"curriculum": "other", "curriculum_name": "Other"}
          ]
        },
        "key": "curriculum",
        "fields": ["curriculum_name"]
      }
    }
  ],
  "width": {
    "step": "50"
  },
  "height": {
    "step": "50"
  },
  "mark": "rect",
  "encoding": {
    "x": {
      "field": "curriculum_name",
      "title": "Curriculum",
      "axis": {
        "labelAngle": -45
      },
      "sort": [
        "BSc - Computer Science",
        "MSc - Computer Science",
        "MSc - Software Engineering",
        "MSc - Cyber Security",
        "MSc - Data Science",
        "MSc - Data Science (exam)",
        "MSc - Conversion Master in IT",
        "MA - Innovation and Technology Management",
        "MA - Teacher of Mathematics and Informatics",
        "Other"
      ]
    },
    "y": {
      "field": "pdf_classification",
      "title": "PDF creator",
      "sort": [
        "TeX",
        "Unknown",
        "Print",
        "Quartz",
        "Google Docs",
        "LibreOffice",
        "Microsoft Word"
      ]
    },
    "color": {
      "aggregate": "count",
      "type": "quantitative",
      "scale": {
        "type": "log"
      },
      "title": "Theses"
    },
    "tooltip": {
      "aggregate": "count",
      "type": "quantitative"
    }
  }
}

Although the heatmap shows some trends, the logarithmic color scale² makes exact comparisons difficult. Thus, let’s look at the word processor breakdown across different curricula in a different way:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "aggregate": [{
        "op": "count",
        "as": "count"
      }],
      "groupby": ["pdf_classification", "curriculum"]
    },
    {
      "joinaggregate": [{
        "op": "sum",
        "field": "count",
        "as": "curriculum_count"
      }],
      "groupby": ["curriculum"]
    },
    {
      "calculate": "datum.count / datum.curriculum_count",
      "as": "curriculum_fraction"
    },
    {
      "lookup": "pdf_classification",
      "from": {
        "data": {
          "values": [
            {"pdf_classification": "Microsoft Word", "classification_order": 0},
            {"pdf_classification": "LibreOffice", "classification_order": 1},
            {"pdf_classification": "Google Docs", "classification_order": 2},
            {"pdf_classification": "Quartz", "classification_order": 3},
            {"pdf_classification": "Print", "classification_order": 4},
            {"pdf_classification": "Unknown", "classification_order": 5},
            {"pdf_classification": "TeX", "classification_order": 6}
          ]
        },
        "key": "pdf_classification",
        "fields": ["classification_order"]
      }
    },
    {
      "lookup": "curriculum",
      "from": {
        "data": {
          "values": [
            {"curriculum": "bsc_computer_science", "curriculum_name": "BSc - Computer Science"},
            {"curriculum": "msc_computer_science", "curriculum_name": "MSc - Computer Science"},
            {"curriculum": "msc_software_engineering", "curriculum_name": "MSc - Software Engineering"},
            {"curriculum": "msc_data_science", "curriculum_name": "MSc - Data Science"},
            {"curriculum": "msc_data_science_exam", "curriculum_name": "MSc - Data Science (exam)"},
            {"curriculum": "msc_cyber_security", "curriculum_name": "MSc - Cyber Security"},
            {"curriculum": "msc_conversion_master_in_it", "curriculum_name": "MSc - Conversion Master in IT"},
            {"curriculum": "msc_innovation_and_technology_management", "curriculum_name": "MA - Innovation and Technology Management"},
            {"curriculum": "ma_maths_and_informatics_teacher", "curriculum_name": "MA - Teacher of Mathematics and Informatics"},
            {"curriculum": "other", "curriculum_name": "Other"}
          ]
        },
        "key": "curriculum",
        "fields": ["curriculum_name"]
      }
    }
  ],
  "width": "725",
  "mark": "bar",
  "encoding": {
    "x": {
      "field": "curriculum_name",
      "title": "Curriculum",
      "axis": {
        "labelAngle": -45
      },
      "sort": [
        "BSc - Computer Science",
        "MSc - Computer Science",
        "MSc - Software Engineering",
        "MSc - Cyber Security",
        "MSc - Data Science",
        "MSc - Data Science (exam)",
        "MSc - Conversion Master in IT",
        "MA - Innovation and Technology Management",
        "MA - Teacher of Mathematics and Informatics",
        "Other"
      ]
    },
    "y": {
      "field": "count",
      "type": "quantitative",
      "stack": "normalize",
      "title": "Theses"
    },
    "color": {
      "field": "pdf_classification",
      "title": "PDF creator"
    },
    "order": {
      "field": "classification_order"
    },
    "tooltip": [
      {
        "field": "curriculum_fraction",
        "format": ".0%",
        "title": "Theses (of curriculum)"
      },
      {
        "field": "count",
        "title": "Theses"
      }
    ]
  }
}

This reveals the following:

1. Over 70% of “BSc - Computer Science” students use Word-like WYSIWYG editors and only 20% use LaTeX.
2. LaTeX usage is most popular among “MSc - Computer Science” and “MSc - Data Science” students. This is probably motivated by the need to typeset more mathematics or do more data visualization.
3. LaTeX usage is (almost) nonexistent among “MSc - Conversion Master in IT” and “MA - Teacher of Mathematics and Informatics” students. This is probably because students in those curricula are less technical.

Page count by curriculum

There’s another piece of PDF file metadata that can be analyzed: PDF page count. It only makes sense to consider curricula separately for this because the expected page counts (set by the guidelines) differ:

• Bachelor’s theses should be ~20 pages (excluding appendices).
• Master’s theses should be 40-50 pages (excluding appendices).

Since the PDF files also contain the appendices, the PDF page count can be expected to be higher.

From the following plots I’ve additionally excluded the following outliers:

• A 373-page thesis, because it would screw with the scale of the plots.
• All theses with under 11 pages, because these appear to be abstracts for theses with publishing restrictions and would skew the results.

Overall

First, let’s look at the thesis page count statistics by curricula:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "filter": "datum.pdf_pages != 373 && datum.pdf_pages >= 11"
    },
    {
      "lookup": "curriculum",
      "from": {
        "data": {
          "values": [
            {"curriculum": "bsc_computer_science", "curriculum_name": "BSc - Computer Science"},
            {"curriculum": "msc_computer_science", "curriculum_name": "MSc - Computer Science"},
            {"curriculum": "msc_software_engineering", "curriculum_name": "MSc - Software Engineering"},
            {"curriculum": "msc_data_science", "curriculum_name": "MSc - Data Science"},
            {"curriculum": "msc_data_science_exam", "curriculum_name": "MSc - Data Science (exam)"},
            {"curriculum": "msc_cyber_security", "curriculum_name": "MSc - Cyber Security"},
            {"curriculum": "msc_conversion_master_in_it", "curriculum_name": "MSc - Conversion Master in IT"},
            {"curriculum": "msc_innovation_and_technology_management", "curriculum_name": "MA - Innovation and Technology Management"},
            {"curriculum": "ma_maths_and_informatics_teacher", "curriculum_name": "MA - Teacher of Mathematics and Informatics"},
            {"curriculum": "other", "curriculum_name": "Other"}
          ]
        },
        "key": "curriculum",
        "fields": ["curriculum_name"]
      }
    }
  ],
  "width": "725",
  "height": "350",
  "mark": {
    "type": "boxplot",
    "size": 30,
    "ticks": true
  },
  "encoding": {
    "x": {
      "field": "curriculum_name",
      "title": "Curriculum",
      "axis": {
        "labelAngle": -45
      },
      "sort": [
        "BSc - Computer Science",
        "MSc - Computer Science",
        "MSc - Software Engineering",
        "MSc - Cyber Security",
        "MSc - Data Science",
        "MSc - Data Science (exam)",
        "MSc - Conversion Master in IT",
        "MA - Innovation and Technology Management",
        "MA - Teacher of Mathematics and Informatics",
        "Other"
      ]
    },
    "y": {
      "field": "pdf_pages",
      "type": "quantitative",
      "title": "Pages"
    },
    "tooltip": {
      "field": "pdf_pages",
      "type": "quantitative"
    }
  }
}

By year

Second, let’s look at the thesis page count average across the years, still by curricula: (click on a curriculum name in the legend for a more focused view)

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "url": "/assets/2025-07-06-unitartucs-theses-blog.csv",
    "format": {
      "type": "csv",
      "parse": {
        "pdf_pages": "number"
      }
    }
  },
  "transform": [
    {
      "filter": "datum.defence_year >= 2010 && datum.defence_year <= 2025"
    },
    {
      "filter": "datum.pdf_pages != 373 && datum.pdf_pages >= 11"
    },
    {
      "lookup": "curriculum",
      "from": {
        "data": {
          "values": [
            {"curriculum": "bsc_computer_science", "curriculum_name": "BSc - Computer Science"},
            {"curriculum": "msc_computer_science", "curriculum_name": "MSc - Computer Science"},
            {"curriculum": "msc_software_engineering", "curriculum_name": "MSc - Software Engineering"},
            {"curriculum": "msc_data_science", "curriculum_name": "MSc - Data Science"},
            {"curriculum": "msc_data_science_exam", "curriculum_name": "MSc - Data Science (exam)"},
            {"curriculum": "msc_cyber_security", "curriculum_name": "MSc - Cyber Security"},
            {"curriculum": "msc_conversion_master_in_it", "curriculum_name": "MSc - Conversion Master in IT"},
            {"curriculum": "msc_innovation_and_technology_management", "curriculum_name": "MA - Innovation and Technology Management"},
            {"curriculum": "ma_maths_and_informatics_teacher", "curriculum_name": "MA - Teacher of Mathematics and Informatics"},
            {"curriculum": "other", "curriculum_name": "Other"}
          ]
        },
        "key": "curriculum",
        "fields": ["curriculum_name"]
      }
    }
  ],
  "width": "650",
  "height": "350",
  "mark": {
    "type": "line",
    "point": true
  },
  "params": [{
    "name": "curriculum_name",
    "select": {"type": "point", "fields": ["curriculum_name"]},
    "bind": "legend"
  }],
  "encoding": {
    "x": {
      "field": "defence_year",
      "title": "Year",
      "axis": {
        "labelAngle": 0
      }
    },
    "y": {
      "field": "pdf_pages",
      "type": "quantitative",
      "aggregate": "average",
      "title": "Pages (average)"
    },
    "color": {
      "field": "curriculum_name",
      "type": "nominal",
      "title": "Curriculum",
      "sort": [
        "BSc - Computer Science",
        "MSc - Computer Science",
        "MSc - Software Engineering",
        "MSc - Cyber Security",
        "MSc - Data Science",
        "MSc - Data Science (exam)",
        "MSc - Conversion Master in IT",
        "MA - Innovation and Technology Management",
        "MA - Teacher of Mathematics and Informatics",
        "Other"
      ]
    },
    "tooltip": {
      "field": "pdf_pages",
      "type": "quantitative",
      "aggregate": "average",
      "format": ".1f"
    },
    "opacity": {
      "condition": {"param": "curriculum_name", "value": 1},
      "value": 0.2
    }
  }
}

1. There’s a well-known issue with the A52’s plastic back panel coming off. This issue started to develop on my phone also in July 2024, but I alleviated it by getting a phone case (for the first time in my life).¹
2. The A52 got its last Android update (to 14) in January 2024.
3. I’ve increasingly found it frustratingly sluggish.
4. Its virtual proximity sensing is quite inaccurate: the screen often activates in the pocket and even triggers presses. In particular, such presses would often mess with media controls which are directly accessible from the lock screen (e.g. randomly seeking, skipping to next podcast).

Anyway, I digress. I used Samsung Smart Switch to transfer everything from the old phone to the new one. Unfortunately, everything does not get automatically transferred and I’m writing this post to document/complain/rant about everything that did not go smoothly. This wasn’t my first time switching Android phones by the way: in 2021 I went from a Samsung Galaxy S6 edge+ to the A52. Although I don’t recall that switch being as painful as this one, but I may have blocked out those memories or I simply wasn’t yet using many of the problematic apps at the time.

Before getting into the bad, I want to briefly touch on the good. I was particularly glad to see that all of my workout data in GymRun was smoothly automatically transferred. I prepared myself for something worse and had made explicit backups on my old phone, such that I could restore them on the new one. Luckily that didn’t turn out to be necessary.

But now to the bad…

Meta apps

WhatsApp

Transferring WhatsApp data, particularly chat history, between phones is notorious for being problematic and I was aware of that. Even Smart Switch has an explicit screen about it. So, on the old phone I made sure to have chat history backed up with a Google Drive account. But that didn’t prepare me for what was going to happen.

Following very reasonable advice online, I moved my SIM card to my new phone before ever starting it up, such that I could set it up with everything in place. After logging into WhatsApp with my phone number on the new phone, my chats were there, but all with empty history! Digging into the respective menus (Settings → Chats → Chat backup), I see that there isn’t any backup.

Naturally, I go back to WhatsApp on my old phone to check. Surprisingly, this turns into a whole fiasco, although I cannot recall the exact sequence of events and all the details. Basically, because I’ve already logged into WhatsApp on my new phone, the old one doesn’t let me into the app to even check the backup (or do any backup-related things) and wants to log in again. But at this point my SIM is in the new phone. Somehow the login still works, at least initially. I guess WhatsApp is fine with not re-verifying the phone number or even having the SIM?

Chat backup view

Anyway, here’s roughly what the Chat backup view looked like on the old phone:

I had pressed “Back up”, I had selected Google for backup storage, I had selected the Google account and authorized it. But that’s not enough! Given the default Frequency setting, I should’ve pressed “Back up” again after setting up Google storage for WhatsApp backups.

Perhaps I should’ve noticed the “Last Backup: Never” but I would argue that this view is weird and unintuitive. Following it top-down yields the following order of operations:

1. It first describes backing up to Google for the purpose of switching phones. Great, that’s exactly what I want!
2. Then there’s a big green button to supposedly do it. But at this point only a local backup would be made. (There’s no description of local backups anywhere, by the way.)
3. Then I can choose and authorize a Google account.

In my opinion, the sensible design would be to have Google account selection before doing the backup because it’s necessary for the use case described at the top.

Furthermore, the backup status text above the “Back up” button is confusing. The “Last Backup” there seems to refer to the last Google backup. While “Local” seems to refer to the last local backup (again, unexplained).

Anyway, I pressed “Back up” again to have chat history backed up to Google for real this time.

Race condition

Now I want to restore the backup I just made on my new phone. Because I logged into WhatsApp on my old phone again, it logged the new phone out. Since I already got into the old phone, I think I triggered the login process on the new phone before I was done with all the backup stuff on the old one. And this seems to have caused a race condition.

The new phone was in the middle of the login process: my phone number was already entered (probably just autofilled from before) and I pressed some button to continue. And then it just got stuck “Initializing…“:

• Killing WhatsApp didn’t help: it opened right back up in the middle of “Initializing…”.
• Deleting WhatsApp data didn’t help either: the “Clear data” button in Android’s settings for WhatsApp just took me back to WhatsApp, which was “Initializing…”. I expected to see the usual warning about causing loss of data and unexpected behavior before agreeing, but somehow WhatsApp does something I had never seen before. I guess it tried to send me into WhatsApp itself to do some data clearing, but that didn’t go to the right screen because it was “Initializing…”.

Not sure if I did anything more or just waited enough, but sometime later opening WhatsApp again, it had given up on “Initializing…” and allowed me to do the login from scratch. This time without a race condition, the login worked instantly.

Transfer chats

But what still didn’t work was restoring the backup. It didn’t happen automatically: maybe because I hadn’t gone into WhatsApp settings on the new phone to authorize it for Google Drive? (Although it’s impossible to get into those settings without logging in first, which would be a cyclic dependency.) Surprisingly, I couldn’t do it manually either: there’s no “Restore” button! So however it’s supposed to work, it didn’t and I went through all the Google storage backup trouble for nothing.

What I ended up doing is using the “Transfer chats” feature instead: it does the transfer without Google via some local wireless connection. And that actually worked! (Although it took surprisingly long given how few messages of chat history I even have on WhatsApp.)

No excuses

To be honest, I could’ve moved on fine without having my WhatsApp chat history. I don’t really use WhatsApp and have two small (dead) chats. However, I wanted to do the transfer purely by principle: it’s 2025 and it’s supposed to be possible.

Not just possible, it’s not supposed to be this hard. The fact that WhatsApp is end-to-end encrypted and thus doesn’t store chat history on servers but only client devices is no excuse for the process being so painful. Signal is very similar to WhatsApp (login by phone number, end-to-end encrypted chats, only local chat history), but its transfer process was very smooth. It’s unbelievable that in 2025 Meta cannot get this right.

Google apps

Google Maps Timeline

Google Maps has had the Timeline feature, constantly tracking location for later viewing, for a long time. Recently, it was changed to no longer store location history on Google servers, but only on the phone itself. Thus, to not lose all that location data (almost a decade of it in my case), it needs to be manually transferred when switching phones.

Conveniently, Google Maps offers a way to (automatically) backup that on Google servers² and I had already set that up. Restoring that backup on the new phone wasn’t as easy as it should’ve been:

1. On the new phone the “Your Timeline” button just wasn’t there. After some time it just appeared.
2. Once it appeared, I could select “Import” for the backup from my old phone. It instantly said the backup was imported, but no old location data was actually present in the Timeline! Again, some time later trying again, it actually imported the data. So, manually check that the import worked! Some people on Samsung Community forums have reported similar issues with the Timeline on the S25 series, but it’s weird that Google Maps would have a bug that only appears on S25 phones: there shouldn’t be anything specific to these models.

Google Calendar

At least some (if not all) Google Calendar settings failed to transfer:

1. The “Start of the week” day went back to Sunday from my chosen Monday.
2. The selection of which calendars are synced to and shown on the phone.

Gboard

I’ve grown accustomed to Google’s Gboard over Samsung’s default keyboard. Annoyingly, nothing about the keyboard was transferred:

1. The default keyboard changed to Samsung’s.
2. All the languages and layouts in Gboard reset to one default one.
3. All personal dictionaries were empty on the new phone.
4. Probably all other Gboard settings went to default as well.

Google accounts

I have two Google accounts set up on my phone: a personal one and a work one. As expected, the personal one was transferred fully automatically, no login even needed if I remember correctly. Oddly enough, the work one did not get transferred at all.

Authentication apps

Microsoft Authenticator

I use Microsoft Authenticator for 2FA of my work account. Turns out, transferring Microsoft Authenticator data to a new phone, which can only be done via cloud backup, requires a personal Microsoft account. Andy Wegner has written a blog post about this very issue: Moving MS Authenticator to a new phone without a personal account. The only alternative seems to be manually re-adding all the accounts on the new phone…

Lucky for me, since I only use this app for my work account, there were only two accounts to re-add. However, the personal account restriction is arbitrary and unnecessary: my workplace already pays Microsoft (probably exorbitant amounts of) money to store emails and OneDrive files in regulatorily-compliant ways. Surely, forcing users to use personal accounts for work-related authentication is not acceptable for organization security and data-compliance.

Smart-ID

Smart-ID is a digital identification and signing service used in Estonia. It offers a convenient alternative to the same features provided by the national ID card without requiring a computer with an ID card reader, while having equivalent legal binding in almost all cases.³

Surprisingly, “transferring” a Smart-ID account from one device to another (Using your existing Smart-ID account to register a new account in FAQ) requires that

1. you registered your previous active account in a bank office, with the help of a bank teller

That isn’t the case for me: I just registered for Smart-ID online on my own computer using an ID card.⁴ If there’s a reason for this weird restriction, then nowhere is that explained.⁵ The suggested alternative is just registering for a new Smart-ID account for the new phone on a computer, again using an ID card.

Samsung apps

Galaxy Watch

I had my Galaxy Watch 6 paired with my old phone. After Samsung Smart Switch, the watch suggested resetting itself before pairing with the new phone. It’s probably possible to skip the reset, however, some Reddit threads suggest doing so to unlock Galaxy Watch features that are (for no good reason) locked to Samsung S-series phones. For example, I couldn’t use my old A52 to access the Camera Controller on my watch, but the new S25+ would allow it.

Resetting the watch is supposed to be a seamless experience: its data is backed up⁶ and automatically restored after reset. It’s not perfect though, at least two things failed to back up/restore correctly:

1. It restored the correct watch face, but not the complication customizations I had made.
2. It did not restore the three favorite exercises on the Samsung Health Exercises tile.

Other apps

AntennaPod

When Google killed Google Podcasts (or more precisely, announced to), I switched to AntennaPod for all my podcasting needs. It’s decentralized and open source, so I could be reasonably confident that my data is not held hostage. Hence, it’s another one of those apps whose data needs manual transferring to a new phone. Luckily AntennaPod provides import/export of its internal database for my podcast subscriptions and listening statistics (very important), so I could get those across to my new phone without a problem. Unfortunately, not everything about AntennaPod gets transferred this way.

For one, its database does not include app settings (which I don’t understand) nor does it provide any alternative way to transfer them. The only way is a manual transfer: have two phones side-by-side and navigate through all settings menus in parallel, changing settings on the new phone to match the old one. Either the database should include app settings or they should be separately transferrable (e.g. via usual Google app settings backup).

The other thing which does not get transferred is downloaded podcast episodes. The argument seems to be that it’s too much data (easily gigabytes) to transfer via Google app backup (which has some size limit). However, there’s also no way to do it offline or manually! By default, it stores downloads in /Android/data/ which on modern Android is inaccessible by file browsers (without root).

/Android/data/

There is a workaround: Android’s own file chooser has access to those directories. Thus, there are apps whose sole purpose is to open that file chooser (as if it’s a file browser). With that, it’s possible to go into the right directory and see all the downloaded podcast audio files, organized into subdirectories. It seems to even be possible to copy an individual file out of there. Weirdly enough, directories cannot be copied: its copying is non-recursive and only creates an empty directory at the target. There is an option to compress files and directories, but that also works for individual files, while only an empty directory gets compressed. With dozens of downloads in different subdirectories, it would be tedious to copy them one by one.

There’s one more workaround described online: when Android’s file chooser is opened in split screen with itself, then one can move data by dragging it from one half of the split into the other. Well, I tried that, with catastrophic consequences. Perhaps it would’ve worked with individual files, but I dragged and dropped a directory containing all the podcast downloads, which ended up creating yet another empty directory. Because this moves (not copies), all the original files were actually deleted, even though they weren’t successfully created into the target directories.⁷

At this point I no longer had any downloads to transfer, so the only solution was redownloading them all on my new phone. Luckily all of them were still available, although there’s no guarantee (RSS feeds or individual episode files can easily disappear from the internet). A big reason to keep some (favorite) downloaded episodes is to safeguard against their disappearance. Therefore, AntennaPod should really provide a proper migration path. That is, just store downloads in a sensible accessible place where they can be manually transferred (or even automatically by Smart Switch, which automatically transfers all user-accessible files (outside /Android/data) regardless of apps) by default.

The redownloading isn’t entirely straightforward either: the transferred database contains the list of downloaded episodes and their paths (in the inaccessible directory). Thus, on the new phone, AntennaPod shows them as downloaded, and without a download button. Only when you go play the episode does it realize that the file is missing and a download button re-appears. Alternatively, you can delete the downloads in AntennaPod (although there’s nothing to actually delete), which then also allows redownloading. However, this is also very problematic: you have to remember what you had downloaded (or at least favorite those episodes just for this purpose).

Non–Play-Store apps

Although Smart Switch transfers many things directly phone-to-phone, it doesn’t seem to do that with the installed apps themselves. Rather, the new phone just installs them from Google Play Store (and perhaps Galaxy Store). Any apps previously installed on the old phone which are not available from those stores are not installed on the new phone:

1. A handful of Play Store apps have been removed from the store (for whatever reason). Their names and icons are transferred to the new phone and show up in the apps list but grayed out. Launching them just goes to a Play Store view saying “Item not found”.⁸
2. Samsung has a group of apps collectively known as “Good Lock” which allow additional customization of their One UI. For some reason they are not available via the app stores in Estonia, but they can be installed via APKs found online.⁹
3. Any other apps installed unofficially from APKs, like YouTube ReVanced.

Sure, Google and Samsung want people to only install apps from official sources to avoid malware, but not transferring others doesn’t help with that. I already installed them on my old phone and can just as well re-install them on the new phone, it’s just unnecessarily annoying. Arguably, it would be better to use APKs from the old phone which I know to be safe than forcing me to go online and download new APKs, risking new malware.

App logins

A general annoyance is that pretty much all apps are logged out of my accounts on the new phone, even though the same apps are automatically installed. It’s such a tedious process to log in to each one again manually.

Settings

Time

After the transfer, which I did late enough in the day, I noticed that the new phone used the 12-hour format for the clock. So for some reason the 24-hour clock format setting did not get transferred, which seems like an obvious one.

Home screen

Smart Switch does transfer home screen and apps screen layouts, although there was a catch. My old phone used 5×5 grid but the new one defaults to 5×6 and Smart Switch went to that default, instead of what I had before. At least in this direction it’s harmless because only an extra empty row appeared.

Although it transfers home screen widgets, their settings do not get transferred. In particular:

1. For the AntennaPod widget I had configured which buttons it shows. For quite a while I thought the widget looked different because its slightly different size caused it to go with a different layout.
2. For the GitHub widget I had chosen my one and only GitHub account to show. Thus, it only showed “Sign in” which took me to GitHub login (again), even though I had already logged into the GitHub app with the same account. Doing that login didn’t even set it to be shown on the widget. I had to manually choose it in the widget settings again.

Wi-Fi

The names and passwords (or whatever login details) of known Wi-Fi networks are transferred automatically, which is very convenient. However, eduroam did not work out of the box: all of its (enterprise) login details seemed to have been transferred and authentication to eduroam seemed successful, but I had no network access. If I recall correctly, Android reported that it could not obtain an IP address. Not sure what that was about, so I just ended up reconfiguring eduroam from scratch and succeeded. Perhaps eduroam didn’t like some transferred certificate or whatever suddenly being used with a different MAC address?

Data usage

Android records data usage both (and separately) for Wi-Fi and mobile data. These monthly statistics were not transferred from the old phone to the new one. Moreover, the related settings (mobile data warning, etc.) also were not. Although this didn’t pose a problem for me, it could be annoying for some people, especially when switching phones mid–billing-cycle.

NOTE: If you set a new course start date, then all course dates will be shifted by the same amount.

Goal

For example, suppose that the old course start date (and time) is set to 14.02.2024 10:00 and activity’s availability/due date (and time) is 14.02.2024 12:00 (i.e. 2 hours after the course start). Both intuitively and according to the quoted documentation, the new course start date should be 12.02.2025 10:00 if the same activity’s date should become 12.02.2025 12:00 (i.e. shifted forward by 2 days less than 1 year).

Problem

Unfortunately (and unsurprisingly), Moodle is not intuitive nor properly documented. If you actually reset the course and choose 12.02.2025 10:00 as the new start date, then the same activity’s date becomes 12.02.2025 22:00 (note the incorrect hour).

Good luck figuring out why! Also, it’s not so easy to just try again by offsetting the new start date, because resetting changed the old start state. This offers hours of fun trial and error just to reset a Moodle course.

Code

Alternatively, one can dig into Moodle source code (something every teacher using Moodle definitely can/wants to do) and eventually find the corresponding logic in moodlelib.php:

// Time part of course startdate should be zero.
$data->timeshift = $data->reset_start_date - usergetmidnight($data->reset_start_date_old);

For some reason which is completely beyond me,¹ the date (and time) shift is not calculated using the old course start date directly. Instead, the usergetmidnight changes the old 14.02.2024 10:00 into 14.02.2024 00:00 (note the midnight hour) before calculating the difference (with the new course start date that still has user-set hour). In this case, that causes the shift to be 10 hours greater, causing the activity’s date to be shifted 10 hours later.

This means that the undesired extra time shift depends on the old course start time instead of being constant. If it were the latter (e.g. because of some timezone issue), then it would be much simpler to compensate for by trial and error.

According to git blame, that midnight calculation has been there for 17 years! It took 12 years for anyone to complain in a Moodle issue (MDL-65233), but has been repeatedly reported since (MDL-76882, MDL-82206, …). I wonder how many more years it will take to get fixed.

Fixed

The answer is one month! I went ahead and submitted a patch to remove the usergetmidnight. It was accepted by Moodle developers and the fix should ship in Moodle versions 4.4.7 and 4.5.3. Nevertheless, I suggest following the recommended workaround below until you can be sure that your institution has updated.

Workaround

Anyone who has a course to run cannot wait, so here are a few workarounds, illustrated on the example.

Workaround 1

1. Reset the course to have new start date 12.02.2025 00:00 (the 10 hours less from the time part of the old start date). The activity’s date will get shifted as intended.
2. The course start date after the reset will have the wrong time, so change it to 12.02.2025 10:00 in course settings (not by resetting the course!).

Workaround 2 (recommended)

1. Change the old course start date to 14.02.2024 00:00 in course settings (not by resetting the course!). Already having the time be midnight cancels the weirdness of usergetmidnight.
2. Reset the course to have new start date 12.02.2025 00:00 (also midnight). Both the new course start date and activity’s date will be as intended.

This workaround is a bit more future-proof:

1. By always having the course start date at midnight, you hopefully avoid the issue during future course resets because the time shift will be intuitive then.
2. If Moodle ever fixes the calculation (to remove usergetmidnight) and your institution finally updates Moodle, then this does not impact your course resetting workflow. (With workaround 1 you’d have to know about the change and stop manually compensating on each reset, because then you’d be overcompensating.)

Suppose the goal is to simultaneously achieve all of the following for typesetting URLs with the \url command:¹

1. URLs are clickable.
2. URLs are (line-)breakable.
3. URLs are colored.
4. URLs are underlined.²

Many StackOverflow questions and answers cover a subset of these, but none that I’ve found does all four. Moreover, such answers usually don’t combine to get all of the above.

A rant

The esteemed typographer Robert Bringhurst states in his “The Elements of Typographic Style”:

3.5.1 Change one parameter at a time.

Moreover, underlining is shunned in general, e.g. by the TeX FAQ and the soul-ori package documentation (with references to other typographic texts).

Hence, requiring URLs in texts to be underlined (and colored) is madness, but this is what happens when Word users make the rules…

LuaLaTeX

\usepackage{hyperref} % provides clickable \url command
\hypersetup{
    breaklinks, % allow line breaks in links
    colorlinks, % allow colors for links
    allcolors=black, % disable obnoxious colors for \ref, \cite, etc. (optional)
    urlcolor=blue, % choose color for \url (default)
}
\usepackage{xurl} % allows arbitrary line breaks in \url (optional)

\usepackage{lua-ul} % provides underlining for LuaLaTeX
\makeatletter % allow accessing \@underLine below
\DeclareUrlCommand{\Hurl}{% redefine hyperref's internal \Hurl instead of \url to preserve clickability and color
    \def\UrlLeft##1\UrlRight{\@underLine##1} % underline \url, use internal command instead of \underLine to preserve breakability
}
\makeatother

pdfLaTeX

Currently I am not aware of a way to simultaneously achieve breakability and underlining with pdfLaTeX. Neither the soul nor the ulem package for underlining with pdfLaTeX provides a command like \@underLine which can be used to underline without forcing the argument into an unbreakable box. Both of these packages and the url package (loaded by hyperref) for the \url command do some low-level \catcode trickery with their arguments, but they don’t seem to play well together. The href-ul package outright seems to do nothing.

Comment below if you have solution for pdfLaTeX!

Allowing all incoming IPv6 traffic

For a long time my home NAS was publically accessible from the internet under my own subdomain. Although risky, I did my best to secure things. First, in my TP-Link router I had forwarded to my NAS only a handful of ports for the Synology DSM and Home Assistant. Second, both of these used HTTPS using Let’s Encrypt certificates using Synology built-in functionality.

As I was removing all the port forwarding, I noticed something odd: I had never forwarded port 80 to my NAS, yet my Synology had been renewing Let’s Encrypt certificates for years using its HTTP-01 validation², which requires port 80 to be exposed. Moreover, even after removing all the port forwarding, those ports were still accessible from the internet (e.g., a DigitalOcean VPS). I hadn’t DMZ-ed the NAS in the TP-Link router, so how is this even possible?

To my absolute horror, I eventually realized that my TP-Link Archer C6 v2.0³ simply allows all incoming IPv6 traffic to all LAN devices. While IPv4 uses NAT, the port forwarding (or the lack of it) forms a firewall, TP-Link must’ve been thinking that IPv6 not needing NAT means that there’s no need for any kind of firewall whatsoever. This is unbelievably insecure because non-expert (but also quite advanced) users would never realize this. Furthermore, there’s no way to change it other than disabling IPv6 altogether. So much for IPv6 adoption…

Some deep digging reveals, that this massive security flaw has been noticed a few times on TP-Link Community forums and Reddit before. Some other Reddit posts don’t mention the allow-all behavior explicitly, but just wonder about an IPv6 firewall of any sort.

Blocking all incoming IPv6 traffic

Luckily (not for me), someone at TP-Link must’ve realized their stupidity at some point but only for some newer router models — no firmware updates are available for mine. More often, people on TP-Link Community forums and various Reddit posts have complained about their TP-Link routers blocking all incoming IPv6 traffic without any configurability (unlike IPv4). At least this is secure enough to not expose everything to the internet, but it doesn’t help IPv6 adoption either…

Providing non-functional IPv6 firewall configuration

For some even newer router models, it seems that TP-Link tried to also solve that problem by making the IPv6 firewall finally configurable. Judging by posts on TP-Link Community forums, however, this configurability doesn’t seem to actually work and all incoming IPv6 traffic is still blocked.

In order to secure web connections to the Synology DSM and various Docker-based services, I had set up Let’s Encrypt on Synology under my own subdomain. Since my NAS is no longer publically accessible, it cannot obtain new Let’s Encrypt certificates for the subdomain¹. Instead, I needed HTTPS certificates for the Tailscale full domain of the NAS.

Tailscale has a guide for setting Tailscale itself up on Synology and a guide for obtaining HTTPS certificates using tailscale cert. Surprisingly, neither documents the best solution, which is the undocumented command

tailscale configure synology-cert

Prior to its introduction, under this Tailscale issue users came up with their own scripts, but using the official command is now the easiest way.

Step-by-step

1. Set up Tailscale on your Synology NAS or update it to at least version 1.64.0.
2. Navigate in the Synology DSM to Control Panel → Task Scheduler.

3. Create a new scheduled task with an user-defined script (Create → Scheduled Task → User-defined script) with the following details:

   • General:
     • Task (name): “Tailscale Certificate” (or whatever you want).
     • User: root (the Tailscale command needs that).
   • Schedule:
     • “Run on the following days”: “Weekly”, “Monday” (seems “Monthly” is not frequent enough such that the 90 day Let’s Encrypt certificate is renewed automatically because months and 90 days may not remain nicely in sync).
   • Task Settings:
     • User-defined script: tailscale configure synology-cert (the magic command).
4. Press “OK” and follow on-screen instructions for setting up the root script.
5. Right click on the created task and select “Run” to get the first certificate immediately.
6. Navigate in the Synology DSM to Control Panel → Security → Certificate.
7. You should now see a certificate for your ts.net subdomain in this list.
8. Use the Tailscale certificate in one of the two ways, depending on your use case:
   1. Right click on the certificate and select “Edit”. Then tick “Set as default certificate” and press “OK”.
   2. Click “Settings” in the toolbar. Change the certificate on a per-service basis.

However, it might be desirable to undo this enforcement for valid and safe reasons, e.g., during web development and testing. In my case, I needed to override the protection after disabling “Automatically redirect HTTP connection to HTTPS for DSM desktop” in my Synology NAS settings.

While other browsers (Chrome/Edge) provide a way to power users to bypass HSTS for a site, Firefox insists not offering any means to do so due to “No User Recourse” from the HSTS RFC. Yet, the “solutions” presented by Mozilla employees still allow users to do just that, while also deleting other data and being significantly less secure than just bypassing for a single site…

Non-solutions

There are a few supposed solutions online, however, I consider each one a non-solution:

1. The official solution is to find the site in Firefox History and select “Forget this site” for it.

   This is a non-solution because it deletes all data related to the site, not just its HSTS state.

2. Editing SiteSecurityServiceState.txt to remove the HSTS entry for a specific site.

   While this only deletes the HSTS state, this is a non-solution because it no longer works: recent versions of Firefox use a proprietary binary file SiteSecurityServiceState.bin instead.

3. Deleting SiteSecurityServiceState.bin to remove HSTS entries for all sites.

   This is a non-solution because it deletes HSTS data related to other unrelated sites and unnecessarily giving up security provided by HSTS. It’s the most insane “solution” of them all.

The solution

1. Find your Firefox profile path. You can do this as follows:

   1. Navigate to the “URL” about:profiles.
   2. Find your profile from the list. This is likely the one with “This is the profile in use and it cannot be deleted.” under it.
   3. Copy the “Root Directory” or click “Open Directory” after it.

2. Close Firefox.

3. Back up the SiteSecurityServiceState.bin file in your Firefox profile path. For example, copying it as SiteSecurityServiceState.bin.bak. This is in case the binary file somehow ends up corrupted when modifying it in the next step.

4. Open the SiteSecurityServiceState.bin file in a hex editor. I used GHex on Linux.

5. Use the hex editor’s “Find” feature to find the desired site’s domain from the file.

6. Change the Unix timestamp in milliseconds (like 1723280965123) after it (there are many NUL/zero bytes in between) with one in the past. I changed it to 1696969696969.

   The file seems to have a similar format to the old SiteSecurityServiceState.txt file, but since it’s in a proprietary binary format, it’s not as simple as deleting a line from it. So the safest way is to just change the HSTS expiry timestamp in-place.

7. Save the file in the hex editor.

8. Open Firefox.

Proof 1

The first pair of proofs for the two papers were near-perfect. In both papers they just removed spaces between authors’ email addresses, i.e., {a, b}@c.d was replaced with {a,b}@c.d, which is harder to read in typewriter font but whatever. (They also did this for the paper in part 1.)

However, Springer still managed to introduce inconsistencies. Both papers are open access and have a paragraph about CC BY 4.0 after the references. In the Goblint Validator paper they inserted a dot after the bolded “Open Access” which begins the paragraph and changed the license URL into typewriter font (unlike all other URLs in the paper which were unchanged).

Proof 2

After accepting the first proofs, Springer emailed a second pair of proofs because they forgot to add artifact evaluation badges to both papers. Apparently their fabulous proof checking system cannot be used a second time so this had to be done completely by email.

In the Goblint (verifier) paper, the only change was indeed the addition of the badges. In the Goblint Validator paper, they intentionally screwed everything else up while adding those badges. Here’s the worst of what they did.

Misencoding editor names

In the year 2024, Springer still struggles with encodings: name of the proceedings editor Laura Kovács appeared like this:

Replacing small caps font in title

It’s common for tool names to be in small caps (\textsc) in paper titles, especially for these SV-COMP tool papers in TACAS proceedings. For some reason, small caps wasn’t enough for Springer and they made it bold italic small caps. I have never seen this in a paper title (or anywhere for that matter).

The following image comparisons illustrate the pessimization from our camera-ready version to Springer’s proof (it’s very easy to tell which side is which version). Hover/swipe across the images to fully appreciate the horror.

Expanding author emails

In this iteration they went a step further and expanded all author emails, i.e., {a,b}@c.d was replaced with a@c.d, b@c.d, which takes up more space. I don’t understand why it was necessary for this version of this paper, but not any other.

Reformatting tables entirely

Just like in part 1, Springer redid booktabs tables with the following changes:

1. All columns are left-aligned (again). That is especially bad for numeric data spanning multiple orders of magnitude. Our tables with siunitx columns that properly align digits and decimal points were ruined. Columns of centered checkmarks and crosses became awkward. Centered \multicolumn spans became odd.
2. Line breaks from multi-line column headers were removed. This makes some columns overly wide.
3. Row spacing was increased.
4. The font for tables was changed to something not matching the rest of the paper (again).

This time they didn’t add vertical column-separating rules between all columns!

Table 1

Note how they changed “2,015” to “2015” in the bottom right cell, thinking it’s a year or something.

Table 2

Yes, they moved this table onto a separate page and rotated it 90° (in addition to all of the above).

Proof 3

After painstakingly listing all the unnecessary changes they made as typesetting errors in the Goblint Validator paper, Springer gave up on (most of) their stupidity and reverted to the first proof (with artifact badges correctly added this time).

And yet, they still had to mess up something. In two places, end-of-line punctuation was shifted from the baseline to above the text. It is beyond me, how to do this accidentally.

Conclusion

At least this time Springer listened and properly undid all the ugliness, so I don’t have to feel shame about how the publisher’s versions of my papers look. But clearly they haven’t stopped making arbitrary unnecessary changes, which wasted everyone’s time, and unfathomable accidental changes.

Stay tuned for part 3! (It’s bound to happen…)

Discrediting other authors

In our paper the citation “Beyer and Strejček [23]” was edited by Springer to simply “Strejček [23]”, discrediting Dirk Beyer. We used \citet{Beyer2022} in LaTeX and both authors are still listed in the corresponding References entry. The cited paper is published via Springer, so they should have no doubt about the authorship. What reason would Springer have to replace \citet with a reduced authors list which is no longer consistent with References? Or how would one do that accidentally?

Reformatting tables entirely

We use the booktabs LaTeX package to typeset beautiful professional tables. For whatever reason Springer entirely reformats tables:

1. Vertical column-separating rules are added between all columns.
2. All columns are left-aligned. That is especially bad for numeric data spanning multiple orders of magnitude. Our tables with siunitx columns that properly align digits and decimal points were ruined. Columns of centered checkmarks and crosses became awkward. Centered \multicolumn spans became odd.
3. The font for tables was changed to something not matching the rest of the paper.

Nothing in the Springer guidelines requires any of such changes to tables, instead requiring:

It is essential that all illustrations are clear and legible.

By unnecessarily reformatting all tables, Springer editors have done the complete opposite of their own guidelines.

Comparisons

The following image comparisons illustrate the pessimization from our camera-ready version to Springer’s proof (it’s very easy to tell which side is which version). Hover/swipe across the images to fully appreciate the horror.

Table 1

Extra ugly is how the vertical rules have gaps in them and some cells are not completely colored.

Table 2

History

Apparently, this behavior is far from new: Dmytro Mishkin already complained about it in 2018. Meanwhile in the first half of 2023, booktabs tables were fine again, as evidenced by some of our papers. Clearly, Springer lacks any kind of policy on this and their typesetters are rulers of a wild west, each making up their own rules.

Making references inconsistent

Abbreviating partially

We explicitly edited our BibTeX bibliography to be consistent across all entries, in particular about booktitles/journals. As provided by Springer Link, we used unabbreviated names like “Static Analysis” and “Tools and Algorithms for the Construction and Analysis of Systems”. Springer says:

References are also modified to make them compatible with CrossRef, which will permit cross referencing within SpringerLink […]

Fair enough, they abbreviated such names to, e.g., SAS and TACAS. However, their editing is completely inconsistent: some entries use abbreviations, while others still have full names for the very same proceedings/journals.

Editing titles

We cite two papers which have the tool name Goblint set in small caps in their title. As provided by Springer Link, we did not have small caps in our bibliography. Springer edited one to use small caps, but left the other (right after the first) in normal font. If there were any policy, then it should be enforced consistently by Springer Link and Springer editors, or at minimum be consistent within a single References section.

Furthermore, Springer typesetters like to remove all crucial capitalization from cited titles:

• replacing “C programs” with “c programs”,
• replacing “Frama-C” with “frama-c”,
• replacing “CommuHash” with “commuhash”.

Adding random dots

We reference items in a prior enumerate like “Items 2 and 4 illustrate”, which was edited by Springer to “Items 2 and 4. illustrate”. The additional dot was consistently (!) throughout the paper added after item number 4 (multiple instances of “Item 4 from Example 4” were changed to “Item 4. from Example 4”). But all other item references into the same list did not get a dot — Springer randomly did it to Item 4.

Conclusion

After listing every occurrence of all of these issues explicitly in the response to typesetting proofs, Springer did manage to fix (i.e., undo) most, but not all, of these issues. In the publisher’s version the tables still aren’t as nice as our original version.

Stay tuned for Check out part 2!

Automata-theoretic approach

Let a rectangular regex crossword with \(h\) rows and \(w\) columns be defined by:

1. row regexes \(R_0, R_1, \dots, R_{h-1}\) (matching left-to-right),
2. column regexes \(C_0, C_1, \dots, C_{w-1}\) (matching top-to-bottom).

Let’s view the \(w \times h\) rectangle as a string of length \(w \times h\) where the rows have been concatenated (i.e., row-major order).

Example

Consider Intermediate: Always remember from regexcrossword.com as an example:

Row automata

First, construct a width automaton \(W\) which accepts all strings of length \(w\). This automaton corresponds to the regex .{w}.

Then, for each row regex \(R_i\) construct a row automaton \(R_i'\) as follows:

where

• \(\circ\) is the binary operator for concatenating two automata,
• exponentation self-concatenates indicated number of copies of the automaton (power 0 gives the automaton whose language contains only the empty string),
• \(\cap\) is the binary operator for intersection of two automata.

Example

For the above example, the automata are the following.

\(W\) is the width automaton corresponding to the regex .{3}:

graph LR
    start:::hidden
    w0(($$w_0$$))
    w1(($$w_1$$))
    w2(($$w_2$$))
    w3((($$w_3$$)))
    start-->w0-->|.|w1-->|.|w2-->|.|w3

    classDef hidden display: none;

For brevity, a single character regex is used to describe the possible transitions, as opposed parallel transitions for each character in the alphabet. Dead/trap states and transitions are also omitted.

1. \(R_0\) is the automaton corresponding to the regex [NOTAD]*:

    graph LR
        start:::hidden
        r0((($$r_0$$)))
        start-->r0-->|"[NOTAD]"|r0
   
        classDef hidden display: none;
   

   For brevity, a single character regex is used to describe the possible transitions, as opposed to 5 self-loops.

   \(R_0' = (R_0 \cap W) \circ W\) is the row automaton (which happens to correspond to the regex [NOTAD]{3}.{3}):

    graph LR
        start:::hidden
        q0(( ))
        q1(( ))
        q2(( ))
        q3(( ))
        q4(( ))
        q5(( ))
        q6((( )))
        start-->q0-->|"[NOTAD]"|q1-->|"[NOTAD]"|q2-->|"[NOTAD]"|q3-->|.|q4-->|.|q5-->|.|q6
   
        classDef hidden display: none;
   

2. \(R_1\) is the automaton corresponding to the regex WEL|BAL|EAR:

    graph LR
        start:::hidden
        r0(($$r_0$$))
        r1(($$r_1$$))
        r2(($$r_2$$))
        r3(($$r_3$$))
        r4(($$r_4$$))
        r5(($$r_5$$))
        r6(($$r_6$$))
        r7((($$r_7$$)))
        start-->r0-->|W|r1-->|E|r2-->|L|r7
        r0-->|B|r3-->|A|r4-->|L|r7
        r0-->|E|r5-->|A|r6-->|R|r7
   
        classDef hidden display: none;
   

   (For symmetry, this has not been minimized.)

   \(R_1' = W \circ (R_1 \cap W)\) is the row automaton (which happens to correspond to the regex .{3}(WEL|BAL|EAR)):

    graph LR
        start:::hidden
        w0(( ))
        w1(( ))
        w2(( ))
        w3(( ))
        start-->w0-->|.|w1-->|.|w2-->|.|w3
        r1(( ))
        r2(( ))
        r3(( ))
        r4(( ))
        r5(( ))
        r6(( ))
        r7((( )))
        w3-->|W|r1-->|E|r2-->|L|r7
        w3-->|B|r3-->|A|r4-->|L|r7
        w3-->|E|r5-->|A|r6-->|R|r7
   
        classDef hidden display: none;
   

Column automata

While the construction of automata for each row regex is rather intuitive, it’s significantly more involved for column regexes. That is because the characters in the row-major string that make up the subsequence which needs to match the column regex are not consequtive. Nevertheless, it is possible using a novel¹ construction.

First, for each column regex \(C_i\) construct a guard automaton \(O^i \circ W^*\) which is in an accepting state at every position in the row-major string that belongs to column \(i\). Here, \(O\) (for offset) is the automaton corresponding to the regex .. The guard automaton is a repetition of the width automaton \(W\), because every \(w\)-th position in the row-major string is in the same column, prefixed by an offset automaton \(O^i\) to start the repetition from the corresponding column. This automaton corresponds to the regex .{i}(.{w})*.

Then, the column regex \(C_i\) construct a column automaton \(C_i'\) as follows:

where \(A \triangleleft G\) is a special product-like guarded automaton (\(A\) guarded by \(G\)), where \(A\) transitions only if \(G\) is in an accepting state. In general \(A \triangleleft G\) is defined as:

1. Its states \((a, g)\) are from the product set of states \(A \times G\).
2. Its initial state consists of the initial states of \(A\) and \(G\).
3. Its state \((a, g)\) is accepting if \(a\) is an accepting state of \(A\).
4. In state \((a, g)\) with input character \(c\) the automaton steps to
   1. \((a', g')\) if \(g\) is an accepting state of \(G\), \(A\) steps from \(a\) to \(a'\) with \(c\) and \(G\) steps from \(g\) to \(g'\) with \(c\),
   2. \((a, g')\) if \(g\) is not an accepting state of \(G\) and \(G\) steps from \(g\) to \(g'\) with \(c\).

(This has some similarities to stretching of automata.)

Example

For the above example, the automata are the following.

1. \(C_0\) is the automaton corresponding to the regex UB|IE|AW:

    graph LR
        start:::hidden
        c0(($$c_0$$))
        c1(($$c_1$$))
        c2(($$c_2$$))
        c3(($$c_3$$))
        c4((($$c_4$$)))
        start-->c0-->|U|c1-->|B|c4
        c0-->|I|c2-->|E|c4
        c0-->|A|c3-->|W|c4
   
        classDef hidden display: none;
   

   \(W^*\) is the guard automaton corresponding to the regex (.{3})*:

    graph LR
        start:::hidden
        w0((($$w_0$$)))
        w1(($$w_1$$))
        w2(($$w_2$$))
        start-->w0-->|.|w1-->|.|w2-->|.|w0
   
        classDef hidden display: none;
   

   \(C_0' = C_0 \triangleleft W^*\) is the column automaton (which happens to correspond to the regex (U..B|I..E|A..W)..):

    graph LR
        start:::hidden
        c0w0(($$c_0,w_0$$))
        c1w1(($$c_1,w_1$$))
        c1w2(($$c_1,w_2$$))
        c1w0(($$c_1,w_0$$))
        c4w1((($$c_4,w_1$$)))
        c2w1(($$c_2,w_1$$))
        c2w2(($$c_2,w_2$$))
        c2w0(($$c_2,w_0$$))
        c3w1(($$c_3,w_1$$))
        c3w2(($$c_3,w_2$$))
        c3w0(($$c_3,w_0$$))
        c4w2((($$c_4,w_2$$)))
        c4w0((($$c_4,w_0$$)))
        start-->c0w0-->|U|c1w1-->|.|c1w2-->|.|c1w0-->|B|c4w1
        c0w0-->|I|c2w1-->|.|c2w2-->|.|c2w0-->|E|c4w1
        c0w0-->|A|c3w1-->|.|c3w2-->|.|c3w0-->|W|c4w1
        c4w1-->|.|c4w2-->|.|c4w0
   
        classDef hidden display: none;
   

2. \(C_1\) is the automaton corresponding to the regex [TUBE]*:

    graph LR
        start:::hidden
        c0((($$c_0$$)))
        start-->c0-->|"[TUBE]"|c0
   
        classDef hidden display: none;
   

   \(O^1 \circ W^*\) is the guard automaton corresponding to the regex .(.{3})*:

    graph LR
        start:::hidden
        o0(($$o_0$$))
        w0((($$w_0$$)))
        w1(($$w_1$$))
        w2(($$w_2$$))
        start-->o0-->|.|w0-->|.|w1-->|.|w2-->|.|w0
   
        classDef hidden display: none;
   

   \(C_1' = C_1 \triangleleft (O^1 \circ W^*)\) is the column automaton (which happens to correspond to the regex .(([TUBE]..)*([TUBE].?)?)? – this is uglier than the automaton):

    graph LR
        start:::hidden
        c0o0((($$c_0,o_0$$)))
        c0w0((($$c_0,w_0$$)))
        c0w1((($$c_0,w_1$$)))
        c0w2((($$c_0,w_2$$)))
        start-->c0o0-->|.|c0w0-->|"[TUBE]"|c0w1-->|.|c0w2-->|.|c0w0
   
        classDef hidden display: none;
   

   (Note that although all states shown are accepting, \(C_1'\) does not accept all strings – the dead state for mismatches at [TUBE] is not shown.)

3. \(C_2\) is the automaton corresponding to the regex [BORF].:

    graph LR
        start:::hidden
        c0(($$c_0$$))
        c1(($$c_1$$))
        c2((($$c_2$$)))
        start-->c0-->|"[BORF]"|c1-->|.|c2
   
        classDef hidden display: none;
   

   \(O^2 \circ W^*\) is the guard automaton corresponding to the regex ..(.{3})*:

    graph LR
        start:::hidden
        o0(($$o_0$$))
        o1(($$o_1$$))
        w0((($$w_0$$)))
        w1(($$w_1$$))
        w2(($$w_2$$))
        start-->o0-->|.|o1-->|.|w0-->|.|w1-->|.|w2-->|.|w0
   
        classDef hidden display: none;
   

   \(C_2' = C_2 \triangleleft (O^2 \circ W^*)\) is the column automaton (which happens to correspond to the regex ..[BORF]...):

    graph LR
        start:::hidden
        c0o0(($$c_0,o_0$$))
        c0o1(($$c_0,o_1$$))
        c0w0(($$c_0,w_0$$))
        c1w1(($$c_1,w_1$$))
        c1w2(($$c_1,w_2$$))
        c1w0(($$c_1,w_0$$))
        c2w1((($$c_2,w_1$$)))
        start-->c0o0-->|.|c0o1-->|.|c0w0-->|"[BORF]"|c1w1-->|.|c1w2-->|.|c1w0-->|.|c2w1
   
        classDef hidden display: none;
   

Solution

Finally, construct the solution automaton \(S\) as an intersection of all row and column automata:

This automaton describes all solutions to the regex crossword. If the regex crossword has a unique solution, this automaton is linear and describes exactly one accepted string.

Example

For the above example, the solution automaton is \(S = R_0' \cap R_1' \cap C_0' \cap C_1' \cap C_2'\) (which happens to correspond to the regex ATOWEL):

graph LR
    start:::hidden
    q0(( ))
    q1(( ))
    q2(( ))
    q3(( ))
    q4(( ))
    q5(( ))
    q6((( )))
    start-->q0-->|A|q1-->|T|q2-->|O|q3-->|W|q4-->|E|q5-->|L|q6

    classDef hidden display: none;

The solution to the regex crossword can be read off the automaton: the only accepted string is ATOWEL.

Performance

Although, I have implemented it in Java, I don’t have useful information about its performance (especially compared to other approaches). The runtimes on small test cases are neglible.

This approach involves a lot of product automata constructions (for intersections and guarded) which, at least in theory, can yield quite large automata. My hunch is that the intermediate automata, when minimized, are relatively small compared to the theoretical bounds (as also seen in the example). Conventionally regex crosswords have unique solutions, so as more automata are intersected more restrictions are combined, converging towards a smaller language with a smaller automaton.

Related work

The following table gives an overview of various approaches to the regex crossword problem. Most seem to use more brute force (backtracking, search, SMT), but also target non-regular regexes which cannot be expressed as finite automata.

Once the neat implementation was done, I wanted to test it, especially with pairs of regular expressions that aren’t trivially equivalent. I scoured the internet (mostly Stack Overflow) and the literature for such examples. Eventually I stumbled upon “Regular Algebra and Finite Machines” (Conway, 1971).

The error

Deep into the book, on page 121 it contains the following exercises:

(Here \(+\) indicates the regex operator |, \(1\) is the empty string aka \(\varepsilon\) and \([]\) are just nested parenthesis for grouping, not a regex character class.)

Exercise 3 asks us to prove that (xy)*(x|xy(yy*x)*)* is equivalent to ((xy*y)*yx|x)*(xy)*. However, my checker didn’t agree and spat out the counterexample yx. Indeed, it’s not too hard to verify by hand that the first regex does not match yx while the second regex does. Hence, they aren’t equivalent!

The fix

The correct formulation of exercise 3 would be:

The difference compared to the book is shown in bold: beginning of the left-hand side should have yx instead of xy. My checker agrees that the two are now equivalent.

This is indirectly corroborated later in the book as well. Solutions to the exercises say:

A proof of 3 (not from C1-14) is implicit in later exercises.

And exercise 9 includes the fixed left-hand side regular expression.

References

Tools

The following table summarizes all OCaml linting tools I managed to find; active or dead, general or special-purpose, standalone or Ppx, monolithic or modular. In this post I focus on linting (based on syntax and possibly types) and exclude program analyzers like reanalyze and Salto. Ocamllint and ocp-lint are the most universal attempts at OCaml linting, however they’re long dead and no replacement seems to have emerged.

There are two general execution modes:

1. Ppx, which are OCaml AST preprocessors, executed by the build system similarly to other Ppx-es like @@deriving features. These are relatively easy to integrate into modern dune-based workflows.
2. Standalone, which are to be executed outside of the usual compilation process. These don’t integrate into modern dune-based workflows due to very limited linting support in dune.

There are three general structures to these linters:

1. Monolithic, where new rules would have to be implemented intertwined with already existing rules. This is reasonable for special-purpose linters and allows all checks to be performed in a single AST pass.
2. Modular (but not extensible), where rules are implemented independently from others but form a fixed ruleset. These can be more difficult to combine into a single AST pass and might mean multiple passes in some cases. They are non-extensible because new rules must be integrated into the core tool itself.
3. Modular and extensible, which has the benefits from the previous point, but also allows custom rules to be added without modifying the tool itself. Thus, they feature some sort of plugin system.

Non-Ppxlib tools

The following table provides more details about the non-Ppxlib tools. Notably, some support type information in rules, which allows more expressive and accurate checks, but also that they cannot be part of the usual Ppx preprocessing step.

Ppxlib tools

The following table provides more details about the Ppxlib-based tools. All of these integrate with dune in one way or another. In some cases, different parts of the same linter from the first table work by slightly different means.

There are two possible ways of dune integration:

1. (preprocess) stanza, which is the usual way to add Ppx preprocessors to the build of a library/executable. This runs unconditionally during the normal build process.
2. (lint) stanza, which has similar syntax but is undocumented. This doesn’t run by default in dune, but rather requires dune build @lint to be executed, which is very easy to forget. A rare example of this exists in dune’s test suite.

Either way, a major inconvenience is that the linter has to be added to every dune library and executable. There’s no way right now to define entire-project linters, which is error-prone as one may simply forget to add the linter to a new library.

There are two main Ppxlib phases used for such linters:

1. ~impl (or ~intf), which is usually used for defining AST transformations. However, linters wouldn’t actually transform the program, but just output warnings during such pass.
2. ~lint_impl (or ~lint_intf), which run before any transformations take place. In fact, this phase cannot even transform the AST, but only return a list of Lint_errors.

There are various ways in Ppxlib to traverse the AST and each linter uses one based on what needs to be returned from the phase and how the output is done. Note that Ppxlib’s context-free (rewriting) rules aren’t suitable for linting as-is: they can only match extension nodes, special functions, custom constants and attribute-annotated nodes. In particular, arbitrary Ast_pattern-based matching is not offered by Ppxlib. This is what bene-gesselint tries to provide as a thin wrapper, however it doesn’t neatly combine multiple Ast_pattern-matching rules into a single AST pass.

There are five means of output for Ppxlib-based linters:

1. Driver.register_correction, which proposes a code change that can be promoted using dune. Since this must propose a change, it cannot simply produce a warning, but multiple changes can also be registered.¹
2. Lint_error.of_string, which yields a preprocessor warning. These can only be returned from ~lint_impl, but many warnings can be returned from a single run.
3. Location.raise_errorf, which crashes the preprocessor with an error. Hence, multiple errors cannot be produced from a single linter run. Ppxlib also discourages the use of exceptions for error handling.
4. Location.error_extensionf, which creates a special error extension node to be put into the AST. Hence, this requires a map traversal, but also allows multiple errors to be returned. Ppxlib recommends this for error handling, at least for usual Ppxlib expanders, derivers and transformers. However, it seems to me that the OCaml compiler will still only print the error from the first error extension node.
5. eprintf, which is just very ad hoc.

Ppxlib techniques

Many combinations of dune integration, Ppxlib phase, traversal and output exist, but not all of them are compatible and sensible. Worse yet, some simply don’t even work, either silently or loudly. The following table gives an overview of the reasonable combinations and which to avoid.

This GitHub repository includes examples of all of these setups in the corresponding subdirectories. See the Cram test run.t files for example outputs.

The purpose of these unique jobs is to check whether the lower bounds of the package’s dependencies (i.e. their minimal versions), if declared at all, are right. That is, does the package actually compile when the oldest allowed versions of its dependencies are installed. All the other checks follow the default behavior of the opam package manager and install the newest allowed dependencies, essentially checking the upper bounds (at the time of submission at least).

The lower-bounds check works by first installing the dependencies of the package normally. The dependency constraint solver of opam is then reconfigured to instead downgrade and remove as many packages as possible (while still satisfying your package’s lower bounds). Having installed the up-to-date versions first, this step nicely shows the version ranges being downgraded, which makes related issues easier to debug.

Problem

These lower-bounds jobs can be quite annoying when trying to submit a package to opam, because package developers usually don’t test for that. You only find missing or too relaxed lower bounds after doing a release of the package and submitting a PR to opam-repository, just to find out it fails on their extensive CI.

There are two main ways to fix these issues:

1. Tighten the lower bound for a particular dependency (or add a lower bound if it doesn’t have one).
2. If possible, change the usage of a dependency to not require features it only introduced in newer versions.

If you follow the recommendations of dune-release, then after fixing the lower bound, instead of re-releasing the exact same version number of the package (and replacing the archive in-place), you release a new patch version of it. This might go on for a while: you release a patched version, submit that to opam-repository, see another lower-bounds failure, fix that – rinse and repeat.

Solution

It would be much quicker and less hassle, if you could somehow run a similar lower-bounds job on your own GitHub repository’s Actions.

Updated

The post has been updated with the recommended modern approach. For reference, the old version is kept below.

With opam ≥ 2.1 (recommended)

In fact, you can by using the 0install solver built into opam 2.1 and above. Its dependency solver --criteria argument allows configuring the preference for old versions (and possibly removing packages). The complete GitHub Actions workflow using setup-ocaml is the following¹:

on:
  push:
  pull_request:

jobs:
  lower-bounds:
    strategy:
      matrix:
        os:
          - ubuntu-latest
        ocaml-compiler:
          - 4.14.2

    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up OCaml ${{ matrix.ocaml-compiler }}
        uses: ocaml/setup-ocaml@v3
        with:
          ocaml-compiler: ${{ matrix.ocaml-compiler }}

      - name: Install dependencies
        run: opam install . --deps-only --with-test

      - name: Downgrade dependencies
        # Option 1: optimize for removing packages and downgrades (like opam-ci)
        run: opam install --solver=builtin-0install --criteria="+removed,+count[version-lag,solution]" . --deps-only --with-test
        # Option 2: don't optimize for removing packages, only downgrades (unlike opam-ci); will also remove depopts
        # run: opam install --solver=builtin-0install --criteria="+count[version-lag,solution]" . --deps-only --with-test

      - name: Build
        run: opam exec -- dune build

      - name: Test
        run: opam exec -- dune runtest

With opam-0install (legacy)

In fact, you can by using opam-0install and its --prefer-oldest argument to downgrade the dependencies to their lower bounds². The complete GitHub Actions workflow using setup-ocaml is the following (replace MY_PACKAGE with your package name)¹:

on:
  push:
  pull_request:

jobs:
  lower-bounds:
    strategy:
      matrix:
        os:
          - ubuntu-latest
        ocaml-compiler:
          - 4.14.2

    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up OCaml ${{ matrix.ocaml-compiler }}
        uses: ocaml/setup-ocaml@v2
        with:
          ocaml-compiler: ${{ matrix.ocaml-compiler }}

      - name: Install dependencies
        run: opam install . --deps-only --with-test

      - name: Install opam-0install
        run: opam install opam-0install

      - name: Downgrade dependencies
        # Option 1: allow OCaml version downgrade
        run: opam install --unlock-base $(opam exec -- opam-0install --prefer-oldest --with-test MY_PACKAGE)
        # Option 2: forbid OCaml version downgrade (specify ocaml-base-compiler again to prevent it from being downgraded)
        # run: opam install $(opam exec -- opam-0install --prefer-oldest --with-test MY_PACKAGE ocaml-base-compiler.${{ matrix.ocaml-compiler }})

      - name: Build
        run: opam exec -- dune build

      - name: Test
        run: opam exec -- dune runtest

The feature can be enabled for a repository with

git config rerere.enabled true

Training

When you first enable it on an existing repository which contains merge conflict resolutions, git rerere won’t automatically know about them because they haven’t been recorded while the feature is active.

Luckily there is an useful rerere-train.sh script to automatically record past merge conflict resolutions into its database. On Ubuntu this is also included in the git apt package at /usr/share/doc/git/contrib/rerere-train.sh (although without the execute bit set).

After enabling git rerere, you can use the script to learn conflict resolutions from a range of commits commit1..commit2 using the following command:

bash /usr/share/doc/git/contrib/rerere-train.sh commit1..commit2

If only one commit is given instead of a range, the script will train git rerere on the entire history back to the initial commit, which is probably undesired and useless.

Nowadays machine learning, and neural networks specifically, are used to solve a wide spectrum of tasks. Besides simple real-valued vectors as data, successful techniques and architectures have been developed to also work with images, natural language and audio. As a programmer and a programming languages enthusiast the obvious question is how program source code can be input into a neural network to solve tasks which would benefit us the programmers.

First, I will describe what differentiates source code from the other mentioned types of input. Second, I will explain the path-based representation for inputting code into neural networks. Third, I will give a short summary of code2vec and code2seq, which are based on this representation. Last, I will discuss some of their limitations.

This gives an overview of the following articles:

1. A General Path-Based Representation for Predicting Program Properties¹,
2. Code2Vec: Learning Distributed Representations of Code²,
3. code2seq: Generating Sequences from Structured Representations of Code³.

How source code is represented?

Plainly put, source code is just text, so natural language is quite related. The symbols of both make lexical tokens and the token sequences of both contain grammatical structure, which can be represented by syntax trees. So naturally, one could take the token sequence of source code, as given by an existing lexer, and feed it into a recurrent neural network (RNN) just like a token (word) sequence of text.

Natural language processing (NLP) techniques have been developed for decades and are very successful, so this likely isn’t too bad. Unfortunately, it completely ignores the inherent structure of source code. Thus, to be successful, such neural network would have to learn the structure from scratch by example from the training data. Not only may this be inaccurate, it also requires a significant amount of training data and time.

Unlike natural language, where ambiguities are possible, code has very strictly defined structure which is enforced by a parser. Such parsers are fully precise and very fast to extract the structure of code in the form of a syntax tree. Although a similar thing exists in NLP, it’s a whole separate (machine learning) task on its own.

Therefore, instead of requiring a machine learning model to learn the structure of programs, we can just get it using a perfect parser. The result is a syntax tree or, after some additional processing, an abstract syntax tree (AST), which omits certain irrelevant details. ASTs are the fundamental representation of source code used by interpreters, compilers, program analyzers, etc. Hence, it makes sense to use the structure provided as an AST.

Example

For example, consder the following Java method, which checks if a list contains an element:

boolean f(Object target) {
    for (Object elem: this.elements) {
        if (elem.equals(target)) {
            return true;
        }
    }
    return false;
}

It has the following AST (ignore the colors and numbers for now)²:

How to input trees into neural networks?

Given an AST for a program, the problem is far from solved because mainstream techniques don’t take input in the form of trees. RNNs allow a network to consume linear input of varying length, but trees are nonlinear, have varying size and varying number of children nodes. Although techniques for tree input have been proposed, they won’t be considered here.

Instead, the people behind code2vec suggest the following: don’t input the tree directly, but input paths, which are linear. This works as follows¹:

1. Pick any two leaves of the AST.
2. Represent them by their tokens including their data (e.g. variable name, integer value, etc).
3. Trace the unique path which connects them in the AST.
4. Represent the path as a sequence of AST intermediate node types delimited by up/down arrows, which indicate whether the path moves up or down the tree.
5. Form a path-context triple: (first leaf, connecting path, second leaf).

A single path-context represents only one part of the AST, and by extension only one part of the code. To represent it all, just construct the bag/set of all path-contexts between all pairs of leaves.

Although a path-context can be constructed for every pair of leaves, it may be too inefficient and not really necessary. A limited number of paths (code2vec uses 200) may be randomly sampled or a maximum length limit may be set.

Example

In the previous example, consider the red path connecting the leaves elements and true. Its representation as a path-context is the following triple:

(elements, Name ↑ FieldAccess ↑ Foreach ↓ Block ↓ IfStmt ↓ Block ↓ Return ↓ BooleanExpr, true).

Reading this from left to right while following the arrows, it encodes a notable amount of structural information: elements is a field, which is iterated over using a foreach loop, which checks something for each element and if true, returns true.

This single path-context already captures the key idea of this method. Other also important parts of this AST are captured by the blue, green and yellow paths in the previous figure.

How code2vec works?

Based on the bag of path-contexts representation of source code, code2vec² does for code what word2vec, etc do for natural language: represent the input as a single dense fixed-size embedding vector. As with word2vec, the goal for the vectors is to somehow capture their semantic properties, such that close vectors correspond to semantically similar inputs, and far vectors to semantically dissimilar ones. These are also known as distributed representations.

As word2vec has proven in NLP, such representations of the input are extremely useful for downstream machine learning tasks, which then don’t have to relearn all the semantic properties themselves. Hence code2vec does this for source code, so that code can be input into the downstream neural network like any other vector.

Training

In order to do this, the neural network part of code2vec takes as input the bag of path-contexts (which are extracted from the source code, as explained above). To train the embedding and have something to optimize, the downstream task of predicting the method name is used. This is apparently one of the most challenging tasks on source code while the method name should capture the semantics of its code.

The following neural network is thus trained²:

In order, the layers work as follows:

1. Each path-context triple consists of two tokens and a path. A simple embedding maps the tokens to token vectors and a simple embedding maps the entire path to a path vector. These simple embeddings are trained simultaneously with the network. The three vectors are then concatenated into a context vector.
2. Each context vector is reduced in dimensionality via a fully-connected layer (with tanh activation) into a combined context vector.
3. The bag of combined context vectors is aggregated into a single code vector, which is the desired distributed representation. A global attention weight vector is used to compute a weight for each combined context vector by dot product. This attention weight vector is trained simultaneously with the network. Then the combined context vectors are aggregated into a code vector using the (normalized) attention weights. Attention allows the network to choose, which of the many remotely incoming path-contexts are more important and which less important.
4. Finally softmax is used to predict the method name from the code vector.

In the above example program and AST, the latter shows the top four path-contexts by their attention weight:

1. red path from elements to true (weight: 0.23),
2. blue path from target to false (weight: 0.14),
3. green path from boolean to ? (actual function name is removed for to make prediction non-trivial), (weight: 0.09)
4. yellow path from Object to target (weight: 0.07).

Many other path-contexts also exist here but they are already too insignificant according to attention. The use of attention also improves the explainability of the model.

Results

This code2vec network achieves state-of-the-art performance in the method name predicition task. It shows that both the path-context–based representation and the learned distributed representation are very useful for working with source code.

The final softmax layer can be chopped off from the above network to simply get the distributed embeddings for other tasks. It still needs the preprocessing of parsing the code, constructing the AST and extracting the bag of path-contexts from it.

Famously word2vec embeddings have beautiful properties between the semantics of the words and their vectors. Surprisingly, this is also the case for code2vec! For example, adding the embedding vectors for equals and toLowerCase methods gives a vector closest to the embedding of equalsIgnoreCase. Many other similar examples were noticed by the authors and they can be explored on the code2vec website. This empirically proves that the learned distributed embeddings indeed capture some semantic properties of source code, which means that the trained code2vec network should also work well for tasks other than predicting method names.

How code2seq works?

The follow-up work code2seq³ improves upon code2vec in a number of ways. It is still based on the bag of path-contexts extracted from the AST and uses (combined) context vectors in the middle of the network.

The following neural network is used³:

Differences from code2vec are the following:

1. Tokens like ArrayList are decomposed into subtokens like Array and List, which are embedded separately and summed. This allows the network to better exploit naming schemes present in source code.
2. Paths are not embedded directly and monolithically, but instead using a bidirectional LSTM layer. A simple embedding is used for the individual path components separately. This is highly preferred to embedding the paths of varying-length monolithically because a simple embedding may be quite sparse and not make use of state-of-the-art RNNs.
3. No single code vector is constructed, so no distributed representation can be extracted for downstream tasks!
4. The network is still trained to predict method names, but now as a sequence of subtokens, e.g. equals, Ignore, Case for the name equalsIgnoreCase. It uses a decoder, which uses attention to attend over all the individual combined context vectors at every step.

How limited are the models?

Both code2vec and code2seq provide their trained models (for Java code) for download. My initial goal was to reuse these (e.g. by extracting distributed representations) and apply them to some other task which has source code as input, for example related to teaching duties of the Laboratory of Software Science. Unfortunately, this turned out to be more complicated than expected.

Firstly, both trained models only handle the source code of a single Java method, not an entire file, which contains a class with likely many methods, among other things. Although a file/class can be split up into methods to separately pass into a model, it would be very limited. The same class may implement all the logic in a single method or have it nicely organized into multiple methods. They would semantically be the same, but comparing a single distributed representation with multiple is not meaningful and loses the semantic properties of the embedding.

This is not a restriction of the path-based representation because the AST of the entire file/class could be used instead. But new models would have to be trained from scratch. Not only that, the training task itself would have to change from method name prediction to maybe class name prediction (which usually doesn’t capture the semantics of all the methods in the class) or something else. And of course the new training task would still need to be such that the learned distributed representations have semantic properties.

Secondly, while code2vec provides distributed representations of code suitable for downstream tasks, the more advanced code2seq architecture and model do not. Rather, the final output is a sequence and the layer before that is just the entire bag of combined context vectors. Therefore, it is possible to combine the best of both worlds:

1. Use the first part of code2seq, which uses a RNN for path embedding.
2. Use the middle part of code2vec, which aggregates the combined context vectors into a single distributed representation using attention.

Thirdly, there is not enough labelled data available for semantic downstream tasks. Although there is a lot of source code data available on GitHub (code2vec used 32GB, code2seq used 125GB), there are no labels for semantic properties. Both of these models were just trained to predict method names, something which syntactically already exists in the same code. It is more likely, that the distributed representations could be used in unsupervised tasks instead, e.g. detecting similar but not duplicate code by clustering.

How to continue from here?

Path-based representations are a promising generic and practical means of inputting source code into neural networks. Although introduced by code2vec and code2seq, others have started using them as well. Notably, JetBrains Research has developed astminer for extracting path-contexts from various languages (and can be extended further) and used them to build representations of authors’ coding style⁴.

Instead of an AST, another idea would be to extract such path-contexts from a control-flow graph (CFG). Control-flow of a method is more closely related to the runtime behavior and semantics of it. It would even be possible to inline method calls with CFGs to construct paths spanning across methods, regardless how well the code is structured. Although parsers exist for all programming languages, control-flow graph generators are almost impossible to come by because most languages have complex features, which significantly impact the flow, e.g. exceptions. Moreover, CFGs can only be constructed for executable code like method bodies, but not class fields or type declarations.

As described on the GitHub Blog, this isn’t just limited to such high-profile open-source contributors but “nearly 12000” GitHub users got the badge. It is determined by having contributed to particular open-source projects before particular versions, with the full list available in GitHub documentation. Most of these seem to be Python projects, but one non-Python project caught my eye: OpenCV.

Immediately I just had to check my GitHub profile and to my great surprise I also got the “Mars 2020 Helicopter Contributor” badge:

I’m honored to be one of the 12000, but my single contribution is very likely unrelated to the NASA mission: “Allow V4L, V4L2 to be used as preferred capture API”. As far as I remember, I encountered and fixed the issue while working on a Robotex soccer robot called Cryptex, which relied on OpenCV for its cameras and vision. So, no, I don’t secretly work for NASA.