# A six-part article series on \expandafter, TeX tokens and expansion

## Overview

The TeX primitive `\expandafter`

is a versatile command, used by LaTeX/TeX programmers to create sophisticated macros—such as the commands (macros) provided by LaTeX packages. Superficially, `\expandafter`

is straightforward to use but that mask of simplicity soon slips when you encounter more complex constructions such as multiple consecutive `\expandafter`

commands used to perform all sorts of “token juggling” `...\expandafter\expandafter\expandafter...`

.

This article series has an ambitious goal: to explain *how* the `\expandafter`

command, and TeX expansion, actually work—from basic principles through to exploring processes which take place deep inside the TeX engine software. Readers can jump straight to the articles or read an introduction and the reason and rationale for the series.

Each article has the following navigation bar before and after the text so that you can quickly jump to another article in the series:

Here are links to the 6 articles in the series:

- An introduction to TeX tokens
- The meaning of expansion
- TeX uses temporary token lists
- From basic principles to exploring TeX's source code
- A detailed macro case study
- A detailed study of consecutive
`\expandafter`

commands

## Introduction and who is this series for?

`\expandafter`

: basic principles of operation

The purpose of \expandafter is to force premature expansion of a token in the input: given two consecutive tokens in the input, \(\mathrm{T_1}\) and \(\mathrm{T_2}\), `\expandafter`

will save token \(\mathrm{T_1}\), expand token \(\mathrm{T_2}\) and place token \(\mathrm{T_1}\) back in front of the expansion of \(\mathrm{T_2}\):

\[\verb*\expandafter*\ \mathrm{T_1}\mathrm{T_2} \rightarrow \mathrm{T_1}\text{<expansion of }\mathrm{T_2}\text{>}\]

Following the standard description of `\expandafter`

’s behaviour we noted that it “saves token \(\mathrm{T_1}\)”, forces the “expansion” of token \(\mathrm{T_2}\) (before TeX would normally do so) and then puts token \(\mathrm{T_1}\) “back into the input” for TeX to read again. The author of this article recalls being confused by some aspects of `\expandafter`

’s “token juggling” behaviour; in particular:

- What does it mean to “save a token” or “put a token back into the input”—where do these saved tokens “go” and how does TeX re-read them?
- What does expansion “look like” or really mean?
- How does TeX process multiple consecutive
`\expandafter`

commands—how does that*really*work?

### Objectives of the article series

This series seeks to address the questions posed above through documenting the results of research into the inner workings of TeX. Overleaf built a customized version of one of the TeX engines and used the Eclipse IDE to observe the TeX source code in operation as it processed a `.tex`

file containing example uses of `\expandafter`

.

We'll share the results of that research to examine:

- the true nature/meaning of TeX tokens;
- TeX’s generation/use of temporary token lists;
- what “expansion” really means.

Through an exploration of these topics we’ll piece together the big picture to explain *how* `\expandafter`

performs its “token juggling” behaviour. This article series is written for anyone who is interested to develop an insight these topics: to take a deeper look inside TeX itself, examining those parts of TeX’s internal operations which enable `\expandafter`

to work its magic.

Writing articles about TeX and TeX-related concepts/principles is challenging because TeX is such an extraordinarily complex piece of software—its typeset source code runs to many hundreds of pages. For example, see this Overleaf project which, by default, typesets the source code of \(\mathrm{\varepsilon}\text{-}\mathrm{\TeX}\): the typeset PDF file contains over 600 pages.

Internally, TeX is an intricate web of interrelated actions and processes which present challenges when attempting to discuss isolated aspects of its behaviour: dissecting out a specific “component”, such as `\expandafter`

, and trying to explain it without constantly needing to reference other aspects of the TeX software. By its very nature, the `\expandafter`

command engages with TeX’s process of *expansion*—a process which, for many of us, is a vague and somewhat difficult concept to fully understand. However, expansion is a truly fundamental aspect of TeX's behaviour, interwoven throughout the inner workings of TeX and taking place deep inside the TeX software itself. In this aricle series we will explore the real meaning of expansion, starting out by assembling a collection of related principles and behaviours sufficient to construct a basic understanding of expansion. Building on those foundations we conclude by taking a “deep dive” to look inside of TeX itself, exploring the actual source code which implements TeX’s expansion mechanism.

## In conclusion

In these articles we have tried to provide content that is of use/value or interest to the maximum number of readers: attempting to balance brevity with unavoidable detail whilst simultaneously trying not to risk the reader’s patience by providing too much, or too little, contextual material. It is our hope to have contributed something of use/value to the TeX literature, which present and, perhaps, future members of the TeX community will enjoy reading.

Happy \(\mathrm{\TeX}\text{ing!}\), from Graham Douglas and the Overleaf team.

## Overleaf guides

- Creating a document in Overleaf
- Uploading a project
- Copying a project
- Creating a project from a template
- Using the Overleaf project menu
- Including images in Overleaf
- Exporting your work from Overleaf
- Working offline in Overleaf
- Using Track Changes in Overleaf
- Using bibliographies in Overleaf
- Sharing your work with others
- Using the History feature
- Debugging Compilation timeout errors
- How-to guides

## LaTeX Basics

- Creating your first LaTeX document
- Choosing a LaTeX Compiler
- Paragraphs and new lines
- Bold, italics and underlining
- Lists
- Errors

## Mathematics

- Mathematical expressions
- Subscripts and superscripts
- Brackets and Parentheses
- Matrices
- Fractions and Binomials
- Aligning equations
- Operators
- Spacing in math mode
- Integrals, sums and limits
- Display style in math mode
- List of Greek letters and math symbols
- Mathematical fonts

## Figures and tables

- Inserting Images
- Tables
- Positioning Images and Tables
- Lists of Tables and Figures
- Drawing Diagrams Directly in LaTeX
- TikZ package

## References and Citations

- Bibliography management with bibtex
- Bibliography management with natbib
- Bibliography management with biblatex
- Bibtex bibliography styles
- Natbib bibliography styles
- Natbib citation styles
- Biblatex bibliography styles
- Biblatex citation styles

## Languages

- Multilingual typesetting on Overleaf using polyglossia and fontspec
- Multilingual typesetting on Overleaf using babel and fontspec
- International language support
- Quotations and quotation marks
- Arabic
- Chinese
- French
- German
- Greek
- Italian
- Japanese
- Korean
- Portuguese
- Russian
- Spanish

## Document structure

- Sections and chapters
- Table of contents
- Cross referencing sections, equations and floats
- Indices
- Glossaries
- Nomenclatures
- Management in a large project
- Multi-file LaTeX projects
- Hyperlinks

## Formatting

- Lengths in LaTeX
- Headers and footers
- Page numbering
- Paragraph formatting
- Line breaks and blank spaces
- Text alignment
- Page size and margins
- Single sided and double sided documents
- Multiple columns
- Counters
- Code listing
- Code Highlighting with minted
- Using colours in LaTeX
- Footnotes
- Margin notes

## Fonts

## Presentations

## Commands

## Field specific

- Theorems and proofs
- Chemistry formulae
- Feynman diagrams
- Molecular orbital diagrams
- Chess notation
- Knitting patterns
- CircuiTikz package
- Pgfplots package
- Typesetting exams in LaTeX
- Knitr
- Attribute Value Matrices

## Class files

- Understanding packages and class files
- List of packages and class files
- Writing your own package
- Writing your own class