2 The rendering pipeline

Rendering is where post.tmd stops being text and becomes the block list everything else keys off: its heading, its paragraph, and its code cell, each a separate block with its own identity. The entry point is render_document_scoped_with_site(src, base_dir, chapter, site) in render/mod.rs; it runs a fixed sequence of stages, each a small, testable function.

flowchart TB
  src[".tmd source"] --> inc["includes::resolve<br/>{{< include >}} + source map"]
  inc --> sc["expand_shortcodes<br/>extension {{< name >}} templates"]
  sc --> spans["scan_div_spans<br/>record ::: fenced divs"]
  spans --> pre["preprocess<br/>replace ::: markers (line-preserving)"]
  pre --> ast["comrak parse_document<br/>(+ sourcepos)"]
  ast --> walk["walk top-level nodes<br/>→ FlatBlock (emit HTML, hash id)"]
  walk --> group["group_divs<br/>rewrap into callouts/grids"]
  group --> cite["table captions + cite::process<br/>citations, @refs"]
  cite --> blocks["RenderedDoc { blocks, title, … }"]

2.1 Stage by stage

1. Include resolution (includes.rs). {{< include path >}} shortcodes are expanded inline, and a Vec<LineOrigin> source map records, for every line of the expanded text, which file + line it came from. A cycle (a page that includes itself too) is left literal with a warning, and a page stops expanding at a budget (1,000 includes, 4 MiB of source) that only a runaway include tree reaches. Click-to-source relies on this map to work across files.
2. Shortcode expansion (render/extension/). The built-in {{< input >}} tag expands to its HTML template, the only shortcode that expands here since {{< include >}} was resolved a pass earlier. It is line-preserving (each tag opens and closes on one line, the template’s newlines collapsed) so the source map from stage 1 stays valid.
3. Front matter, fenced-div scan + preprocess (frontmatter.rs, render/divs.rs). frontmatter::front_matter_block is the one front-matter splitter: the block is read once as YAML (render/fm_extract.rs), then its lines are blanked, so comrak (whose own front-matter extension is off) parses the body alone. The ::: fenced divs aren’t CommonMark, so their spans are recorded first, then each marker line is replaced without changing the line count (again preserving sourcepos): by a thematic break (***), which ends any list, quote or code block open above it and which the walk skips, or by a blank line where raw HTML sits directly above it. The inner content then parses as ordinary blocks.
4. Parse (comrak, with sourcepos on). One AST for the whole document, with every node carrying its startLine:col-endLine:col. The parse and the walk that follows run on a worker thread with a 256 MB stack (render_internal hands the render to a worker from render/workers.rs, which parks between renders rather than being spawned for each one). The large stack is there because deeply nested Markdown (blockquotes, lists) drives deep recursion in both comrak’s parser and Taliesin’s block emission, and on the default ~8 MB thread stack that recursion overflows at around 3000 levels. A stack overflow is not a catchable panic: it aborts the whole process, so one pathological document would crash a build or take down the live preview server. The big stack absorbs any realistic nesting; an ordinary panic inside the worker is propagated to the caller unchanged. (If spawning the big-stack worker fails, for instance under a strict ulimit -v, render falls back to the current default-stack thread rather than failing the render.)
5. Walk → FlatBlock (render/mod.rs + render/emit.rs). Each top-level node becomes one block: the emitter renders it to HTML (code highlighted by highlight.rs, math by math.rs), and the block gets a content-hash id and a data-sourcepos mapped back through the source map. A ```{lang} fence is recognised as an executable cell and recorded for the server to run later.
6. Regroup divs (render/divs.rs). The flat blocks are wrapped back into callout / layout-grid / generic container blocks using the spans from stage 3. Inner blocks keep their own ids and sourcepos, so click-to-source works inside a callout.
7. Captions + citations (apply_table_captions, then cite/). A Pandoc table caption (: caption {#tbl-x} on the paragraph directly after a table) is not a sibling block in the output: apply_table_captions numbers it, folds it into the table as a <caption> first child, registers tbl-x for @tbl- references, and then removes the caption paragraph block so the caption lives as metadata on the table block. After that, [@key] citations and @fig-/@sec-/… cross-references are resolved against the bibliography and the cross-reference registry.

Stages 1 to 3 read source line by line, and none of them decides for itself what is code. lines.rs does, by parsing the text with the renderer’s own comrak options and labelling every line: markdown, front matter, fenced or indented code, or raw HTML (an HTML block of types 1 to 5, such as a comment or <pre>), with its block-quote and list-item depth. An include directive, a shortcode or a ::: marker counts only on a markdown line, and a ::: marker only at depth 0. The site’s anchor scan and the prose count ask the same question the same way (render::rendered_lines).

The result is a RenderedDoc: an ordered Vec<Block> plus the title, subtitle, TOC flag, the cross-reference number map, and any non-fatal warnings. Every block in it carries the three data attributes the rest of the system keys off, data-block-id (hashed from the block’s raw Markdown source by make_id), data-sourcepos, and data-source-file on included blocks. The block model is the chapter on them; crates/core/tests/corpus.rs enforces the contract over every corpus document.

2.2 Two line coordinate systems

After stage 1 there are two kinds of line number, and confusing them is a recurring bug. A post-include buffer line (what group_divs matches ::: spans in, what comrak’s sourcepos reports) is a BufLine: a newtype in render/model.rs with no Display and no conversion, so it cannot be formatted into a data-sourcepos or passed to Warning::at. The way out is map_origin / map_span, which return the author’s own file and line; past that point there is only one system left, so the source side stays a bare number.

The compiler enforces the separation because both halves of the mistake are invisible: an unmapped line paired with a data-source-file lands N lines off inside a real, openable file, and mapping the two ends of a range separately emits spans like 39:1-6:25 on a paragraph comrak merged across an include boundary, which the client silently skips. map_span maps both ends together, which prevents both mistakes.

2.3 Citations & cross-references

Citations and cross-references (cite/) are the last pipeline stage, and they run as a post-process over the already-rendered block HTML: they transform only plain-text runs, never the inside of a tag, code, or math span, so block sourcepos is left untouched. [@key] citations become numbered links to a References block appended to the document (the one structural change), formatted from the parsed BibTeX. @fig- / @sec- / @tbl- / @lst- / @eq- references become links to their anchor, carrying the resolved number when it is known. It is a simple formatter, not a full CSL engine. Cross-page references (a @fig- that lives in another chapter) are resolved a layer up, in the site subsystem (crates/core/src/site/xref.rs).

Markdown footnotes ([^name]) are the one construct whose content is written in one place and rendered in another:

• A footnote reference ([^name]) renders in place, as a superscript link inside its surrounding block (emit.rs).
• A footnote definition ([^name]: …) does not.¹1This note is an example. Its text was written under the bullet list in the source, and it renders here, in the margin beside the bullet that references it.Back It is spliced in as a margin sidenote immediately after its own reference, the only position CSS can float into the margin from: no selector can relocate an end-of-document element to sit beside an arbitrary earlier one, so the move happens at render time. There is no gathered endnote section.

comrak moves every definition to the end of the document, so a walk that met a reference first would not yet have seen its definition. render_internal_impl therefore runs a pre-pass over the root’s children collecting definitions into a map before the main walk begins, the same shape as the heading-level pre-scan just above it.

A block id is hashed from the block’s source lines, not from its emitted HTML. A note’s source lives somewhere else while its text renders inside the referencing block, so the definitions a block displays are folded into that block’s hash input. Without that fold, editing a note would leave every block id identical, the diff would emit no operation, and the live preview would keep showing the old note, with no error and no other symptom. Only the notes a block displays are folded in, so editing one note does not churn the ids of blocks that reference a different one.

Because the note renders as phrasing content inside another block, a definition carrying block content (a list, a quote, a code block) is flattened to its text with a warning: an <ul> inside a <span> inside a <p> makes the HTML parser close the paragraph early, which would leave the block with two root elements and break the one-root-element invariant the block swap depends on.

2.4 Incremental updates

On a save the server re-runs this whole pipeline and hands the fresh block list to diff_blocks(old, new), which turns it into the handful of ops that cross the websocket. The block model describes that algorithm.

2.5 Output assembly

A RenderedDoc becomes a full page through one assembler, render/page.rs (assemble_html_page). The blocks are joined into the body, and that body is dropped into a single fixed HTML shell (the PAGE_TEMPLATE format! string) that supplies the page chrome the reader never authored:

• A pre-paint theme bootstrap in <head> (theme_head): a tiny inline <script> that reads the reader’s device preference (prefers-color-scheme) and sets data-theme on <html> before the first paint, so a dark-mode reader never sees a white flash, then keeps following the OS live. Both palettes always ship and the device alone decides: there is no author theme key and no reader toggle in a built page.
• The bundled offline assets: base.css, the dark layer, the optional site CSS, and (only when the document uses math) the KaTeX stylesheet, all include_str!d into the binary so a built page is self-contained with no CDN.
• A skip-to-content link when the body carries the focusable <main id="tali-main">, plus the chrome’s own head markup (per-page OpenGraph/SEO meta and the feed links). No author-configured markup reaches the <head>.
• For a page with {js} cells, the vendored d3 + Observable Plot libs and the {js} enhancer (gated on the rendered body, so a page without {js} carries none of it).

A site/book page adds chrome from site/, but the same assembler builds the page shell for the static build and for the live-preview server (preview just injects the websocket client into that shell), so they can’t drift apart. The same block list backs every output (see Architecture).

2.6 Interactive cells ({js})

{js} cells are client-side, not kernel-executed, so they sit outside the execution path. A native enhancer (tali-js.js) runs each cell’s source against a per-cell scope (tali.get/set/value, onInput, container, invalidation); it is a small scheduler, not a full reactive VM. Cell kinds come from //| options: //| viewof: name returns a DOM input registered under name, //| name: x publishes the return value into a shared scope, and //| input: a, b re-runs the cell when input a or b fires. From the cells’ defines/inputs the enhancer builds a dependency graph (buildGraph): a name → consumers map plus a global topological order via Kahn’s algorithm. When an input fires, scheduleFrom re-runs exactly the cell’s transitive-downstream closure (following each hit cell’s own defines, so a n → squared → … chain re-runs end to end), once each, in that topological order, rather than cascading listener fires. The initial run is a single document-order (producer-before-consumer) awaited pass; cells left over after Kahn’s are in a cycle and show a diagnostic instead of running. A Python cell’s define(...) bridges values across via a <script type="tali-define"> blob the enhancer ingests, re-running dependents when it lands after first paint.

That graph is running in this page. The slider is a {{< input >}}, the cell under it declares //| input: tau, and dragging the slider re-runs only that cell, redrawing our running example’s cooling curve in the browser rather than in the kernel: