A six-part article series on \expandafter, TeX tokens and expansion

Overview

The TeX primitive \expandafter is a versatile command, used by LaTeX/TeX programmers to create sophisticated macros—such as the commands (macros) provided by LaTeX packages. Superficially, \expandafter is straightforward to use but that mask of simplicity soon slips when you encounter more complex constructions such as multiple consecutive \expandafter commands used to perform all sorts of “token juggling” ...\expandafter\expandafter\expandafter....

This article series has an ambitious goal: to explain how the \expandafter command, and TeX expansion, actually work—from basic principles through to exploring processes which take place deep inside the TeX engine software. Readers can jump straight to the articles or read an introduction and the reason and rationale for the series.

Accessing articles: the navigation bar

Each article has the following navigation bar before and after the text so that you can quickly jump to another article in the series:

Here are links to the 6 articles in the series:

1. An introduction to TeX tokens
2. The meaning of expansion
3. TeX uses temporary token lists
4. From basic principles to exploring TeX's source code
5. A detailed macro case study
6. A detailed study of consecutive \expandafter commands

Introduction and who is this series for?

\expandafter: basic principles of operation

The purpose of \expandafter is to force premature expansion of a token in the input: given two consecutive tokens in the input, \(\mathrm{T_1}\) and \(\mathrm{T_2}\), \expandafter will save token \(\mathrm{T_1}\), expand token \(\mathrm{T_2}\) and place token \(\mathrm{T_1}\) back in front of the expansion of \(\mathrm{T_2}\):

\[\verb*\expandafter*\ \mathrm{T_1}\mathrm{T_2} \rightarrow \mathrm{T_1}\text{<expansion of }\mathrm{T_2}\text{>}\]

Following the standard description of \expandafter’s behaviour we noted that it “saves token \(\mathrm{T_1}\)”, forces the “expansion” of token \(\mathrm{T_2}\) (before TeX would normally do so) and then puts token \(\mathrm{T_1}\) “back into the input” for TeX to read again. The author of this article recalls being confused by some aspects of \expandafter’s “token juggling” behaviour; in particular:

• What does it mean to “save a token” or “put a token back into the input”—where do these saved tokens “go” and how does TeX re-read them?
• What does expansion “look like” or really mean?
• How does TeX process multiple consecutive \expandafter commands—how does that really work?

Objectives of the article series

This series seeks to address the questions posed above through documenting the results of research into the inner workings of TeX. Overleaf built a customized version of one of the TeX engines and used the Eclipse IDE to observe the TeX source code in operation as it processed a .tex file containing example uses of \expandafter.

We'll share the results of that research to examine:

• the true nature/meaning of TeX tokens;
• TeX’s generation/use of temporary token lists;
• what “expansion” really means.

Through an exploration of these topics we’ll piece together the big picture to explain how \expandafter performs its “token juggling” behaviour. This article series is written for anyone who is interested to develop an insight these topics: to take a deeper look inside TeX itself, examining those parts of TeX’s internal operations which enable \expandafter to work its magic.

Writing articles about TeX and TeX-related concepts/principles is challenging because TeX is such an extraordinarily complex piece of software—its typeset source code runs to many hundreds of pages. For example, see this Overleaf project which, by default, typesets the source code of \(\mathrm{\varepsilon}\text{-}\mathrm{\TeX}\): the typeset PDF file contains over 600 pages.

Internally, TeX is an intricate web of interrelated actions and processes which present challenges when attempting to discuss isolated aspects of its behaviour: dissecting out a specific “component”, such as \expandafter, and trying to explain it without constantly needing to reference other aspects of the TeX software. By its very nature, the \expandafter command engages with TeX’s process of expansion—a process which, for many of us, is a vague and somewhat difficult concept to fully understand. However, expansion is a truly fundamental aspect of TeX's behaviour, interwoven throughout the inner workings of TeX and taking place deep inside the TeX software itself. In this aricle series we will explore the real meaning of expansion, starting out by assembling a collection of related principles and behaviours sufficient to construct a basic understanding of expansion. Building on those foundations we conclude by taking a “deep dive” to look inside of TeX itself, exploring the actual source code which implements TeX’s expansion mechanism.

In conclusion

In these articles we have tried to provide content that is of use/value or interest to the maximum number of readers: attempting to balance brevity with unavoidable detail whilst simultaneously trying not to risk the reader’s patience by providing too much, or too little, contextual material. It is our hope to have contributed something of use/value to the TeX literature, which present and, perhaps, future members of the TeX community will enjoy reading.

Happy \(\mathrm{\TeX}\text{ing!}\), from Graham Douglas and the Overleaf team.