feat: Module 11 becomes 13; diff-tracks fuzzer; align the two reference tracks

- Rename course/11-break-your-engine to 13-break-your-engine and update every
  link and mention; start-module explains that Module 13 has no tests to start.
- tools/diff-tracks.mjs (npm run diff:tracks): stages both references of a
  module, generates a seeded random SpellGrid level and script per seed, pipes
  it to the Rust and TypeScript CLIs and diffs their output; crashes on one or
  both sides fail. stagePlay takes a module number.
- register_assembled: the TypeScript track compared only the text of an
  existing Assembled card, so a base re-registered with a new cost (or a
  different split) silently rewrote the earlier card. It now compares every
  field the engine reads, as Rust does (Modules 5-10).
  New test_05_assemble_never_rewrites_an_earlier_card.
- SpellGrid CLIs: coordinates and chooser numbers are 1-9 plain digits in both
  tracks (Rust accepted signs, TypeScript accepted "+2", "0x2", "1e0"); Rust
  quotes card names verbatim instead of with Debug escapes; Rust joins the
  minion list with "; " instead of rewriting ", ". Two new CLI tests per track.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-11-11 10:08:56 +01:00
1 parent b36f4605cc
commit 06a1b34c5d
32 files changed
+645 -55

No files matched your search

+3 -3
View File
@@ -5,7 +5,7 @@ A free, hands-on course on **systemic game design**. You learn by rebuilding the
- **Part I: Hearthstone** (cards, zones, triggers, death phases, auras, text assembly)
- **Part II: Baba Is You** (rules as pushable text, parsing, movement, win/lose)
- **Part III: SpellGrid**, a small original game that combines both systems
- **Module 11: Break your own engine**, an optional closing module on testing what the tests miss: mutation testing, invariants, metamorphic relations, undo and replay, and a level solver
- **Module 13: Break your own engine**, an optional closing module on testing what the tests miss: mutation testing, invariants, metamorphic relations, undo and replay, and a level solver
Every module follows the same beats: Observe, Predict, Spec, Build, Name it, Expansion Drop, Stress it, Further reading. The Expansion Drop is the test of your design: you add new content without touching the engine. Stress it is optional: edge cases and stretch exercises that push the design past what the module's tests check.
@@ -40,9 +40,9 @@ cd ~/my-engine && git init && git add -A && git commit -m "start from template"
node tools/start-module.mjs 01 ~/my-engine
```
`start-module` copies a module's tests for your track (from your workspace's `.course.json`) and refuses to start a module if you have not yet taken the previous one. Module 00 uses the `naive-engine/` project and is not started this way.
`start-module` copies a module's tests for your track (from your workspace's `.course.json`) and refuses to start a module if you have not yet taken the previous one. Module 00 uses the `naive-engine/` project and is not started this way, and Module 13 has no tests to start.
Other tools: `tools/coupling-audit.mjs <workspace>` (engine code must not name specific cards or nouns), `tools/expansion-check.mjs <workspace> <baseRef>` (an Expansion Drop must change content only), `tools/verify-references.mjs [NN]` (reference solutions pass the tests; `--expansion` also checks every Expansion Drop solution), `tools/play-spellgrid.mjs <rust|ts> [l1|l2|l3]` (play the Module 10 capstone, SpellGrid, on the command line with the reference engine), `tools/solve-spellgrid.mjs [l1|l2|l3|level.json]` (search a SpellGrid level for its shortest solution by brute force; `--never-rule "X IS Y"` asks whether it can be solved without a given rule; see Module 11).
Other tools: `tools/coupling-audit.mjs <workspace>` (engine code must not name specific cards or nouns), `tools/expansion-check.mjs <workspace> <baseRef>` (an Expansion Drop must change content only), `tools/verify-references.mjs [NN]` (reference solutions pass the tests; `--expansion` also checks every Expansion Drop solution), `tools/play-spellgrid.mjs <rust|ts> [l1|l2|l3]` (play the Module 10 capstone, SpellGrid, on the command line with the reference engine), `tools/solve-spellgrid.mjs [l1|l2|l3|level.json]` (search a SpellGrid level for its shortest solution by brute force; `--never-rule "X IS Y"` asks whether it can be solved without a given rule; see Module 13), `tools/diff-tracks.mjs [--seeds N] [--seed S] [--module NN] [--dir D] [--keep]` (differential fuzzer: plays random SpellGrid levels and scripts on the Rust and the TypeScript reference CLIs and fails on the first seed where they print different text or one of them crashes; `npm run diff:tracks`).
## Licence
+1 -1
View File
@@ -310,7 +310,7 @@ repeat 200 times:
on failure: print the seed and the operations so far
```
One thing a tempting fourth invariant gets wrong: "the board list is sorted by order of play" is false. It holds while every minion arrives at the right end, and stops holding the first time one arrives anywhere else (Module 3's Deathrattle summons). Position and order of play are different facts, which is the point of this module. [Module 11](../11-break-your-engine/lesson.md) takes this style of testing further.
One thing a tempting fourth invariant gets wrong: "the board list is sorted by order of play" is false. It holds while every minion arrives at the right end, and stops holding the first time one arrives anywhere else (Module 3's Deathrattle summons). Position and order of play are different facts, which is the point of this module. [Module 13](../13-break-your-engine/lesson.md) takes this style of testing further.
</details>
+1 -1
View File
@@ -381,7 +381,7 @@ The snapshot differs from a live scan only when a multiplier's presence on the b
Module 4 has no card with both a Battlecry and a Battlecry multiplier, so the difference is invisible here. Module 5 can write one as data, and its `test_05_own_aura_misses_own_battlecry` does: "Battlecry: Draw a card. Your Battlecries trigger twice." draws one card under the snapshot and two under a live scan. The real game sides with the snapshot: "The double Battlecry effect will only take place if Brann was already present on the board." ([Brann Bronzebeard](https://hearthstone.wiki.gg/wiki/Brann_Bronzebeard)).
The method matters more than the card. You changed a design decision, the tests stayed green, and that told you the tests didn't pin the decision. Then you worked out the smallest scenario that tells the two designs apart. That is **mutation testing** done by hand, and [Module 11](../11-break-your-engine/lesson.md) does it to the whole course.
The method matters more than the card. You changed a design decision, the tests stayed green, and that told you the tests didn't pin the decision. Then you worked out the smallest scenario that tells the two designs apart. That is **mutation testing** done by hand, and [Module 13](../13-break-your-engine/lesson.md) does it to the whole course.
</details>
+4 -3
View File
@@ -111,6 +111,7 @@ The full statements are in `course/05-assemble/scenarios.md`.
- `test_05_assemble_bottom_half`: Harvest Golem with Dread Infernal's bottom half, under Baron, next to a `Test Filler` (1/1) and a `Test Sponge` (1/3). Destroy it. Exact log: two activations of "Deal 1 damage to ALL other characters", the Filler dies afterwards, both heroes at 28.
- `test_05_assembled_keeps_base_stats`: the same card in hand costs 3 and is 2/3.
- `test_05_assemble_must_change_text`: Loot Hoarder with Loot Hoarder fails, top and bottom.
- `test_05_assemble_never_rewrites_an_earlier_card`: Harvest Golem is re-registered with a new cost, and the same Assemble is made again. The second card gets a new name, and the first card in hand keeps its cost.
- `test_05_top_half_swap`: Loot Hoarder with Novice Engineer's top half. Playing it draws once; destroying it draws nothing.
- `test_05_assemble_effect_with_chooser`: cast `Test Workshop` ("Assemble a Battlecry minion. Give all minions in your hand +1/+1.") with an injected chooser. The base options are exactly the Battlecry minions in `cards/`; the new minion and a `Test Filler` already in hand both get +1/+1.
- `test_05_additive_multipliers_stack`: two `Test Chorus` ("Your minions trigger their Deathrattles an additional time.", stacking `additive`) and a Loot Hoarder: three activations. Module 4 built the `additive` branch; this is the first card that uses it, and it needed no code.
@@ -195,7 +196,7 @@ Run `cargo test` or `npm test`. The Module 5 tests fail to compile (Rust) or fai
## Build
Make the nine tests pass, and keep Modules 1–4 green at every step. Where code goes:
Make the ten tests pass, and keep Modules 1–4 green at every step. Where code goes:
- **No card in code.** Your content directory becomes a loader: it reads `cards/*.json` and hands the definitions to the engine. Delete the card functions from Modules 2–4 once the data versions pass the old tests.
- **The engine interprets.** `src/engine/` gets types that mirror the card format, and one routine that turns an effect plus its context (owner, source, target, position) into the actions you already have. That routine is the only code that knows what `draw` or `dealDamage` means. It names effect kinds and targets, never cards.
@@ -271,7 +272,7 @@ Decisions in the reference that the tests don't force:
- **Where a summon goes.** Until now only Deathrattles summoned, at the dead minion's remembered position (Module 3). With Assemble, "Battlecry: Summon a 2/1 Damaged Golem." exists, and its minion is still on the board. The reference summons to the right of a source still in play, at the remembered position for a source that has left, and at the right end for a spell. The course's research notes don't cover Battlecry summons, so treat this as the course's choice.
- **The pool** is every registered minion with text, minus Assembled cards. That includes the fictional `Test` cards. The real pool is curated: see below.
- **Two Assembles of the same base with different donors** would both be named "Harvest Golem (Assembled)", and registering the second would silently rewrite the first. The reference names the second "Harvest Golem (Assembled 2)". The harness contract only fixes the first name.
- **Two Assembles of the same base with different donors** would both be named "Harvest Golem (Assembled)", and registering the second would silently rewrite the first. The reference names the second "Harvest Golem (Assembled 2)". The harness contract fixes the first name, and `test_05_assemble_never_rewrites_an_earlier_card` fixes only that the second differs. "Different" means anything the engine reads: the text, the `split`, or a base re-registered with a new cost. The two reference engines once disagreed here, because one of them compared only the text.
- **The default chooser** takes the first option. A real game would ask the player or roll; the tests never rely on the default.
- **Unknown keywords do nothing.** Unstable Ghoul's data says `taunt`; the course has no combat, so the engine keeps the clause (it is part of the text Assemble moves) and ignores it.
@@ -465,7 +466,7 @@ The same log, provided you also recorded the *inputs*: every harness call that s
Two things break replay, and both are worth trying. Iterating over an unordered map somewhere in the engine (TypeScript objects keep insertion order; a Rust `HashMap` does not, so on the Rust track this can bite) and ids that depend on something outside the game. If your replayed log differs, the first differing event points at the place.
Replay is the foundation for three things the course doesn't build: saving a game as its inputs, undo by replaying all but the last input, and reproducing a bug report exactly. Robert Nystrom's [*Command*](https://gameprogrammingpatterns.com/command.html) chapter makes the same observation about recorded input: "We can take the player's input, push it over the network to another machine, and then replay it." [Module 11](../11-break-your-engine/lesson.md) uses replay to build a solver.
Replay is the foundation for three things the course doesn't build: saving a game as its inputs, undo by replaying all but the last input, and reproducing a bug report exactly. Robert Nystrom's [*Command*](https://gameprogrammingpatterns.com/command.html) chapter makes the same observation about recorded input: "We can take the player's input, push it over the network to another machine, and then replay it." [Module 13](../13-break-your-engine/lesson.md) uses replay to build a solver.
</details>
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
+8
View File
@@ -51,6 +51,14 @@ The harness adds `registerCards(defs)`, `assemble(base, donor, half)`, `cost(id)
- Action: `assemble("Loot Hoarder", "Loot Hoarder", "bottom")`, then `assemble("Loot Hoarder", "Loot Hoarder", "top")`.
- Assertions: each fails with `AssembleError` (TypeScript: throws an `AssembleError`; Rust: returns `Err`). A minion assembled from itself has the same text.
### test_05_assemble_never_rewrites_an_earlier_card
- Setup: `assemble("Harvest Golem", "Dread Infernal", "bottom")` (name N1); `addToHand(N1, 0)` (id A). Then `registerCards` with Harvest Golem's definition from `cards/` with `cost` 5 instead of 3: the same text, a different card.
- Action: `assemble("Harvest Golem", "Dread Infernal", "bottom")` again (name N2); `addToHand(N2, 0)` (id B).
- Assertions:
- N2 is not N1. The new Assembled card differs from the one already registered as N1, so registering it under N1 would silently rewrite the card A already is.
- `cost(A)` is 3 and `cost(B)` is 5.
### test_05_top_half_swap
- Setup: `newGame` with a deck of 5 `Test Filler` for player 0. `assemble("Loot Hoarder", "Novice Engineer", "top")` (name N): Loot Hoarder's top half `[trigger deathrattle]` is replaced by Novice Engineer's `[trigger battlecry]`, so N is `[trigger battlecry | effect draw 1]`, "Battlecry: Draw a card." `addToHand(N, 0)` (id E).
+26
View File
@@ -172,6 +172,32 @@ fn test_05_assemble_must_change_text() {
assert!(top.is_err(), "{top:?}");
}
#[test]
fn test_05_assemble_never_rewrites_an_earlier_card() {
let mut g = new_game(GameOpts::default());
let first = g
.assemble("Harvest Golem", "Dread Infernal", Half::Bottom)
.unwrap();
let a = g.add_to_hand(&first, 0);
// The base changes (a new cost, the same text), then the same Assemble is made again.
let (_, mut golem) = card_files()
.into_iter()
.find(|(_, def)| def.name == "Harvest Golem")
.unwrap();
golem.cost = 5;
g.register_cards(vec![golem]);
let second = g
.assemble("Harvest Golem", "Dread Infernal", Half::Bottom)
.unwrap();
let b = g.add_to_hand(&second, 0);
// A different card, so a different name: the first card in hand is still the one it was.
assert_ne!(second, first);
assert_eq!(g.cost(a), 3);
assert_eq!(g.cost(b), 5);
}
#[test]
fn test_05_top_half_swap() {
let mut g = new_game(GameOpts {
@@ -109,6 +109,23 @@ test("test_05_assemble_must_change_text", () => {
expect(() => g.assemble("Loot Hoarder", "Loot Hoarder", "top")).toThrow(AssembleError);
});
test("test_05_assemble_never_rewrites_an_earlier_card", () => {
const g = newGame();
const first = g.assemble("Harvest Golem", "Dread Infernal", "bottom");
const a = g.addToHand(first, 0);
// The base changes (a new cost, the same text), then the same Assemble is made again.
const golem = cardFiles().find((c) => c.name === "Harvest Golem")!;
g.registerCards([{ ...golem, cost: 5 }]);
const second = g.assemble("Harvest Golem", "Dread Infernal", "bottom");
const b = g.addToHand(second, 0);
// A different card, so a different name: the first card in hand is still the one it was.
expect(second).not.toBe(first);
expect(g.cost(a)).toBe(3);
expect(g.cost(b)).toBe(5);
});
test("test_05_top_half_swap", () => {
const g = newGame({ decks: [fillers(5), []] });
const name = g.assemble("Loot Hoarder", "Novice Engineer", "top");
+1 -1
View File
@@ -364,7 +364,7 @@ rules(board) == rules(slide(board, dx, dy))
rules(board) == rules(swapXY(board))
```
This is **metamorphic testing**: you don't know the right answer for a random board, but you know how the answers for two related boards must relate. A metamorphic relation is a necessary property "of the intended functionality" that "must involve multiple executions of the software" ([Metamorphic testing, Wikipedia](https://en.wikipedia.org/wiki/Metamorphic_testing)). The swap relation is a sharp one for this module, because a parser that handles rows and columns in two separate pieces of code can easily be right in one and wrong in the other, and no hand-written test compares them. Module 1's seeded generator is all you need. [Module 11](../11-break-your-engine/lesson.md) collects the relations that hold across the course.
This is **metamorphic testing**: you don't know the right answer for a random board, but you know how the answers for two related boards must relate. A metamorphic relation is a necessary property "of the intended functionality" that "must involve multiple executions of the software" ([Metamorphic testing, Wikipedia](https://en.wikipedia.org/wiki/Metamorphic_testing)). The swap relation is a sharp one for this module, because a parser that handles rows and columns in two separate pieces of code can easily be right in one and wrong in the other, and no hand-written test compares them. Module 1's seeded generator is all you need. [Module 13](../13-break-your-engine/lesson.md) collects the relations that hold across the course.
</details>
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
+1 -1
View File
@@ -369,7 +369,7 @@ The general point: once two names can refer to one thing, every rule about a *na
Not by pushing: the three words that spell it are text, and no text is pushable while it stands. The rule protects itself. Another rule could still move text some other way, which is what the wiki's rulebook alludes to ("Text can be stacked if TEXT IS NOT PUSH can be formed with another way to move text"), but the course has no other way.
A level that lets the player form this sentence with nothing else in reach is unwinnable from that point, and nothing tells the player so. That is a property of the *level*, not the engine, and you can check it mechanically: [Module 11](../11-break-your-engine/lesson.md) builds a search that finds every position a level can reach, including the dead ones.
A level that lets the player form this sentence with nothing else in reach is unwinnable from that point, and nothing tells the player so. That is a property of the *level*, not the engine, and you can check it mechanically: [Module 13](../13-break-your-engine/lesson.md) builds a search that finds every position a level can reach, including the dead ones.
</details>
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
+1 -1
View File
@@ -433,7 +433,7 @@ Make the level wrap around (step off the right edge, arrive at the left) and the
Very. Sokoban, which is this module with one property and no text, is the standard example: "deciding whether any given Sokoban puzzle is solvable is a problem known to be NP-hard and PSPACE-complete" ([Sokoban, Wikipedia](https://en.wikipedia.org/wiki/Sokoban), citing Dor and Zwick 1999 and Culberson 1997). Pushes are easy to do and hard to take back, so positions that look alike differ in what is still reachable, and a solver can't avoid exploring them.
That is not a reason to give up on checking levels. Complexity results are about how the cost grows with the size of the puzzle, and a designer's levels are small. A brute-force search over a 9 × 7 level finishes in seconds, and it will tell you two things no amount of play-testing can: that a level has *no* solution other than the one you intended, or that it has one you didn't. [Module 11](../11-break-your-engine/lesson.md) builds that search.
That is not a reason to give up on checking levels. Complexity results are about how the cost grows with the size of the puzzle, and a designer's levels are small. A brute-force search over a 9 × 7 level finishes in seconds, and it will tell you two things no amount of play-testing can: that a level has *no* solution other than the one you intended, or that it has one you didn't. [Module 13](../13-break-your-engine/lesson.md) builds that search.
</details>
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
+1 -1
View File
@@ -143,7 +143,7 @@ The full statements are in `course/09-turn-pipeline/scenarios.md`.
- `test_09_hazard_order_melt_before_defeat`: Predict question 11.
- `test_09_hazard_order_sink_before_melt`: a key that is `HOT` and `MELT` lies on water that is `SINK`. Both are destroyed, because SINK runs first.
The last five look like a lot of tests for one list. Each exists because the turn makes a separate ordering decision there, and an engine with that one decision changed passed every earlier test in the course. [Module 11](../11-break-your-engine/lesson.md) shows the experiment.
The last five look like a lot of tests for one list. Each exists because the turn makes a separate ordering decision there, and an engine with that one decision changed passed every earlier test in the course. [Module 13](../13-break-your-engine/lesson.md) shows the experiment.
The first test:
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
+2 -2
View File
@@ -258,7 +258,7 @@ node tools/play-spellgrid.mjs ts l1 # TypeScript: npx tsx src/cli.ts leve
The level is `l1`, `l2` or `l3` (or a path to a level file of your own). `--dir <dir>` stages into a directory you choose and keeps it, and `--stage-only` prints the command without running it. The tool prints where it staged, so you can run the command there again.
`w`, `a`, `s`, `d` play a turn; `c <card> [x y]` casts a spell (at the minion on that tile, if given) or plays a minion (onto that empty tile); `q` quits. Graft asks for its base, donor and half; answer with a number or a name. Writing your own is optional: the course's tests don't call it.
`w`, `a`, `s`, `d` play a turn; `c <card> [x y]` casts a spell (at the minion on that tile, if given) or plays a minion (onto that empty tile); `q` quits. Graft asks for its base, donor and half; answer with a number or a name. Writing your own is optional: the course's tests don't call it. The Rust and the TypeScript command lines print the same text for the same input, and `node tools/diff-tracks.mjs` checks that on random levels and scripts.
## Name it
@@ -440,7 +440,7 @@ What it reports for the shipped levels:
- **Level 2** has no solution shorter than 4 inputs, and among some 31,000 positions within 4 inputs exactly one wins: the intended graft, played and pushed into the lava.
- **Level 3** is solved in 6 inputs, and under `--naive` the search exhausts all 11 reachable positions with no win. "The knight dies" is not only what the one scripted test shows; no other line of play saves the naive engine.
An unintended solution is called a *cook*, and a level with one doesn't teach what it was built to teach. [Module 11](../11-break-your-engine/lesson.md) explains how the solver works and what it can't tell you. Then design a fourth level, and try to cook it before the tool does.
An unintended solution is called a *cook*, and a level with one doesn't teach what it was built to teach. [Module 13](../13-break-your-engine/lesson.md) explains how the solver works and what it can't tell you. Then design a fourth level, and try to cook it before the tool does.
</details>
+14 -12
View File
@@ -82,26 +82,29 @@ pub fn render(g: &SpellGrid, level: &LevelDef) -> String {
}
}
}
let or = |list: Vec<String>, none: &str| {
let or = |list: Vec<String>, separator: &str, none: &str| {
if list.is_empty() {
none.to_string()
} else {
list.join(", ")
list.join(separator)
}
};
lines.push(format!(
"Minions: {}",
or(minions, "none").replace(", ", "; ")
));
lines.push(format!("Hand: {}", or(g.hand(), "empty")));
lines.push(format!("Minions: {}", or(minions, "; ", "none")));
lines.push(format!("Hand: {}", or(g.hand(), ", ", "empty")));
lines.join("\n")
}
/// A coordinate, or a chooser's number: one to nine digits, nothing else (no sign, no point).
fn number(word: &str) -> Option<i32> {
let digits = (1..=9).contains(&word.len()) && word.bytes().all(|b| b.is_ascii_digit());
digits.then(|| word.parse().expect("nine digits fit an i32"))
}
/// A `c` command: `c <card words...> [x y]`.
fn parse_cast(args: &[&str]) -> (String, Option<(i32, i32)>) {
let n = args.len();
if n >= 3 {
if let (Ok(x), Ok(y)) = (args[n - 2].parse(), args[n - 1].parse()) {
if let (Some(x), Some(y)) = (number(args[n - 2]), number(args[n - 1])) {
return (args[..n - 2].join(" "), Some((x, y)));
}
}
@@ -131,10 +134,9 @@ pub fn run(level_text: &str, input: Box<dyn BufRead>, output: Box<dyn Write>) ->
if options.contains(&answer) {
return answer;
}
let picked = answer
.parse::<usize>()
.ok()
.and_then(|n| options.get(n.wrapping_sub(1)));
let picked = number(&answer)
.and_then(|n| usize::try_from(n - 1).ok())
.and_then(|i| options.get(i));
picked.unwrap_or(&options[0]).clone()
});
play(&mut g, &level, &io)
@@ -200,7 +200,7 @@ impl SpellGrid {
)
});
match target {
None if chooses => Err(format!("{card:?} needs a target minion")),
None if chooses => Err(format!("\"{card}\" needs a target minion")),
None => Ok(()),
Some(id) => {
// Every id in `entities` was issued by the board, so `find` may ask about it.
@@ -218,7 +218,9 @@ impl SpellGrid {
pub fn check_play(&self, card: &str, x: i32, y: i32) -> Result<(), String> {
self.side.check_hand(card, CardType::Minion)?;
if self.side.world.card(card).def.noun.is_none() {
return Err(format!("{card:?} has no noun: it can't stand on the grid"));
return Err(format!(
"\"{card}\" has no noun: it can't stand on the grid"
));
}
if !self.level.board().in_bounds(x, y) {
return Err(format!("({x}, {y}) is off the grid"));
@@ -303,14 +305,14 @@ impl CardSide {
fn check_hand(&self, card: &str, card_type: CardType) -> Result<(), String> {
if self.in_hand(card).is_none() {
return Err(format!("{card:?} is not in the hand"));
return Err(format!("\"{card}\" is not in the hand"));
}
if self.world.card(card).def.card_type != card_type {
let kind = match card_type {
CardType::Minion => "minion",
CardType::Spell => "spell",
};
return Err(format!("{card:?} is not a {kind}"));
return Err(format!("\"{card}\" is not a {kind}"));
}
Ok(())
}
@@ -64,3 +64,26 @@ fn cli_reports_errors_and_quits() {
assert!(out.contains("\"Spark\" is not in the hand"));
assert!(out.contains("Commands:"));
}
// The two reference CLIs print the same text for the same input (tools/diff-tracks.mjs checks it).
#[test]
fn cli_prints_card_names_as_typed() {
let level = include_str!("../levels/l3-simultaneous-paradox.json");
let (_, out) = play(level, &["c a\"b", "q"]);
assert!(out.contains("\"a\"b\" is not in the hand"), "{out}");
}
#[test]
fn cli_numbers_are_plain_digits() {
// "-1" is not a coordinate, so it is part of the card's name.
let level = include_str!("../levels/l1-double-tap.json");
let (_, out) = play(level, &["c Whirlwind -1 0", "q"]);
assert!(
out.contains("\"Whirlwind -1 0\" is not in the hand"),
"{out}"
);
// "+2" is not a number, so the chooser takes the first option.
let level = include_str!("../levels/l2-spliced-corpse.json");
let (_, out) = play(level, &["c Graft", "+2", "1", "1", "q"]);
assert!(out.contains("Hand: Ember Moth (Assembled)"), "{out}");
}
+6 -2
View File
@@ -47,10 +47,13 @@ export function render(g: SpellGrid, level: LevelDef): string {
return lines.join("\n");
}
/** A coordinate, or a chooser's number: one to nine digits, nothing else (no sign, no point). */
const NUMBER = /^\d{1,9}$/;
/** A `c` command: `c <card words...> [x y]`. */
function parseCast(args: string[]): { card: string; at?: [number, number] } {
const n = args.length;
if (n >= 3 && /^\d+$/.test(args[n - 2]) && /^\d+$/.test(args[n - 1])) {
if (n >= 3 && NUMBER.test(args[n - 2]) && NUMBER.test(args[n - 1])) {
return { card: args.slice(0, n - 2).join(" "), at: [Number(args[n - 2]), Number(args[n - 1])] };
}
return { card: args.join(" ") };
@@ -67,7 +70,8 @@ export function run(levelText: string, readLine: () => string | undefined, write
g.setChooser((kind, options) => {
write(`Choose the ${kind} (a number or a name):\n${options.map((o, i) => ` ${i + 1}) ${o}`).join("\n")}\n? `);
const answer = readLine()?.trim() ?? "";
return options.includes(answer) ? answer : (options[Number(answer) - 1] ?? options[0]);
if (options.includes(answer)) return answer;
return (NUMBER.test(answer) ? options[Number(answer) - 1] : undefined) ?? options[0];
});
for (;;) {
write(`${render(g, level)}\n`);
@@ -34,6 +34,12 @@ const canonical = (value: unknown): string =>
: v,
);
const sameText = (a: CardDef["text"], b: CardDef["text"]): boolean => canonical(a) === canonical(b);
/** Two definitions the engine can't tell apart (it ignores `fictional` and `game`). */
const sameCard = (a: CardDef, b: CardDef): boolean => {
const read = (d: CardDef) =>
canonical([d.name, d.type, d.cost, d.attack ?? 0, d.health ?? 0, d.noun ?? null, d.text, d.split]);
return read(a) === read(b);
};
/**
* The base card with `half` of its text replaced by the same half of `donor`.
@@ -66,8 +72,8 @@ const changes = (base: CardDef, donor: CardDef, half: Half): boolean => {
export function registerAssembled(world: World, def: CardDef): string {
const stem = def.name.replace(/\)$/, "");
let name = def.name;
// Same base, so the same name and stats: only the text can differ.
for (let n = 2; world.hasCard(name) && !sameText(world.card(name).def.text, def.text); n++) {
// A different card is any difference the engine reads: the text, the split, or a base re-registered since.
for (let n = 2; world.hasCard(name) && !sameCard(world.card(name).def, { ...def, name }); n++) {
name = `${stem} ${n})`;
}
world.registerAssembled({ ...def, name });
@@ -30,3 +30,18 @@ test("cli_reports_errors_and_quits", () => {
expect(out).toContain('"Spark" is not in the hand');
expect(out).toContain("Commands:");
});
// The two reference CLIs print the same text for the same input (tools/diff-tracks.mjs checks it).
test("cli_prints_card_names_as_typed", () => {
const { out } = play("l3-simultaneous-paradox", ['c a"b', "q"]);
expect(out).toContain('"a"b" is not in the hand');
});
test("cli_numbers_are_plain_digits", () => {
// "-1" is not a coordinate, so it is part of the card's name.
const cast = play("l1-double-tap", ["c Whirlwind -1 0", "q"]);
expect(cast.out).toContain('"Whirlwind -1 0" is not in the hand');
// "+2" is not a number, so the chooser takes the first option.
const graft = play("l2-spliced-corpse", ["c Graft", "+2", "1", "1", "q"]);
expect(graft.out).toContain("Hand: Ember Moth (Assembled)");
});
@@ -1,4 +1,4 @@
# Module 11: Break your own engine
# Module 13: Break your own engine
The course is finished at Module 10. This module is about what to do with an engine once its tests pass, which is where most of an engine's life is spent.
@@ -97,7 +97,7 @@ This repository has a Rust reference and a TypeScript reference for every module
This is **differential testing**. It is also how [Module 6](../06-parser/lesson.md) suggests you protect an optimisation: keep the slow version and compare. If you did both tracks of the course, you own two engines; if you did one, the reference in your language is the second. The command-line front ends from Module 10 (`tools/play-spellgrid.mjs`) read inputs from standard input and print the board, the rules, the minions and the hand after each one, so two engines can be compared as text.
The course has not done this systematically, and the two references are known to differ in at least one place the tests don't reach (how each decides that an Assembled card already exists). Finding the rest is an open exercise, and a good one.
`node tools/diff-tracks.mjs` does this for the two references. For each seed it makes a random SpellGrid level and a random script, runs both command lines, and prints a diff for the first seed on which they disagree. Its first run of 500 seeds found nothing. The references did differ, though: in how each decides that an Assembled card already exists, and in how each command line reads a signed number or quotes a card name. The fuzzer could reach none of these. The first needs a card registered again in the middle of a game, and the others need input outside the command grammar it generates. A differential tester finds only what its generator can reach. All three are now fixed, and `test_05_assemble_never_rewrites_an_earlier_card` and the reference command-line tests pin them. Widening the generator is an open exercise, and a good one.
## State, undo and replay
@@ -153,7 +153,7 @@ Three practical consequences:
The pattern is the **oracle problem**, and the techniques in this module are ways round it. A test needs something that knows the right answer. Hand-written expected values are one such oracle, the most precise and the most expensive. The others trade precision for coverage:
| Oracle | Knows | Module 11 section |
| Oracle | Knows | Module 13 section |
|---|---|---|
| An expected value written by a person | the exact answer, for one input | (Modules 1–10) |
| An invariant | a fact about every answer | Invariants |
+1 -1
View File
@@ -51,7 +51,7 @@ Read the module's `lesson.md` and `scenarios.md` before starting it. Teach the l
3. Run `expansion-check.mjs <workspace> <expansionBase>`. If it fails, the engine changed: do not offer to fix it. Ask what the engine lacked and what data would have let the content express the card. The learner reverts the content change, makes the engine generic in its own commit, records that commit as the new `expansionBase`, and repeats the drop.
4. Run the coupling audit again. Then open the lesson's "what this drop measures" discussion.
**Stress it.** Optional; offer it, do not assign it. Treat each question like a Predict question: the learner commits to an answer before you open its discussion. Where a question asks them to change their engine or write a test, they do it (Rule 1), and experiments that change the engine are made on a throwaway branch or reverted, never left in place before the next module. Module 11 is made only of this kind of work: it has no tests to start and no drop, so teach its sections in whatever order the learner picks.
**Stress it.** Optional; offer it, do not assign it. Treat each question like a Predict question: the learner commits to an answer before you open its discussion. Where a question asks them to change their engine or write a test, they do it (Rule 1), and experiments that change the engine are made on a throwaway branch or reverted, never left in place before the next module. Module 13 is made only of this kind of work: it has no tests to start and no drop, so teach its sections in whatever order the learner picks.
**Further reading.** Offer the list; do not assign it. Then set `module` to the next number, `beat` to `Observe`, `hintsUsed` to 0.
+2 -1
View File
@@ -12,6 +12,7 @@
"test:site": "node --test site/scripts/*.test.mjs",
"check:links": "node tools/check-links.mjs",
"play:spellgrid": "node tools/play-spellgrid.mjs",
"solve:spellgrid": "node tools/solve-spellgrid.mjs"
"solve:spellgrid": "node tools/solve-spellgrid.mjs",
"diff:tracks": "node tools/diff-tracks.mjs"
}
}
+326
View File
@@ -0,0 +1,326 @@
#!/usr/bin/env node
// Differential fuzzer: plays the same random SpellGrid level and script on the Rust and the
// TypeScript reference CLIs of one module and compares what they print.
//
// node tools/diff-tracks.mjs [--seeds N] [--seed S] [--module NN] [--dir D] [--keep]
// npm run diff:tracks -- --seeds 500
//
// Stages both references of module NN (default: the highest module whose references have a
// SpellGrid CLI) with their cards, as play-spellgrid does, into <D>/rust and <D>/ts (default D:
// a new temp directory), builds the Rust CLI once and installs the TypeScript dependencies.
// Then, for each seed (1..N, or only S), it generates a level (<D>/cases/seed-S.json) and a
// script of commands (<D>/cases/seed-S.txt), pipes the script to both CLIs, and compares stdout.
// Nothing is printed for a seed on which they agree. The first divergence prints the seed, the
// level path, the script and a unified diff, and exits 1. A crash (panic, uncaught exception,
// 10 s timeout) on one side is a divergence; on both sides it is reported as "both crashed" and
// exits 1. A temp directory is removed at the end unless --keep is given or a case failed; a
// directory given with --dir is always kept, and re-running there reuses the Rust build.
import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, existsSync, writeFileSync } from "node:fs";
import { spawn, spawnSync } from "node:child_process";
import { availableParallelism, tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { fail, isMain, moduleDirs } from "./lib.mjs";
import { stagePlay } from "./play-spellgrid.mjs";
const TIMEOUT_MS = 10_000;
const OPERATORS = ["IS", "AND", "NOT"];
const DIRS = ["w", "a", "s", "d"];
const MARKER = ".diff-tracks";
export function parseArgs(argv) {
const opts = { seeds: 100, seed: null, module: null, dir: null, keep: false };
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--seeds") opts.seeds = Number(argv[++i]);
else if (a === "--seed") opts.seed = Number(argv[++i]);
else if (a === "--module") opts.module = String(argv[++i]).padStart(2, "0");
else if (a === "--dir") opts.dir = argv[++i];
else if (a === "--keep") opts.keep = true;
else fail(`diff-tracks: unknown argument ${a}\nusage: diff-tracks.mjs [--seeds N] [--seed S] [--module NN] [--dir D] [--keep]`);
}
if (!Number.isInteger(opts.seeds) || opts.seeds < 1) fail("diff-tracks: --seeds needs a positive whole number");
if (opts.seed !== null && !Number.isInteger(opts.seed)) fail("diff-tracks: --seed needs a whole number");
return opts;
}
/** A seeded PRNG (mulberry32): `rng(seed)()` is a number in [0, 1). The same seed gives the same sequence. */
export function rng(seed) {
let a = seed >>> 0;
return () => {
a = (a + 0x6d2b79f5) >>> 0;
let t = a;
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
/**
* The words and cards a staged reference reads: `words` from its words.json, `cards` (by name,
* a later directory replacing an earlier card of the same name) as { name, type, noun, chooses },
* where `chooses` marks a card whose effect asks the chooser (Assemble).
*/
export function loadVocab(wordsFile, cardsDirs) {
const words = JSON.parse(readFileSync(wordsFile, "utf8"));
const byName = new Map();
for (const dir of cardsDirs) {
for (const f of readdirSync(dir).filter((f) => f.endsWith(".json")).sort()) {
const def = JSON.parse(readFileSync(join(dir, f), "utf8"));
const chooses = (def.text ?? []).some((t) => t.kind === "effect" && t.effect?.type === "assemble");
byName.set(def.name, { name: def.name, type: def.type, noun: def.noun ?? null, chooses });
}
}
const cards = [...byName.values()].sort((a, b) => a.name.localeCompare(b.name));
return { words, cards };
}
/** One random case: a level in the course's level format and a script of CLI commands. Pure. */
export function generateCase(seed, vocab) {
const r = rng(seed);
const int = (lo, hi) => lo + Math.floor(r() * (hi - lo + 1));
const pick = (list) => list[Math.floor(r() * list.length)];
const chance = (p) => r() < p;
const nouns = vocab.words.filter((w) => w.role === "noun").map((w) => w.name);
const objectNouns = vocab.words.filter((w) => w.role === "noun" && !w.selects).map((w) => w.name);
const properties = vocab.words.filter((w) => w.role === "property").map((w) => w.name);
const placeable = vocab.cards.filter((c) => c.type === "minion" && c.noun);
const holdable = vocab.cards.filter((c) => c.type === "spell" || (c.type === "minion" && c.noun));
const width = int(4, 8);
const height = Math.min(8, Math.max(4, width + int(-2, 2)));
const free = [];
for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) free.push([x, y]);
const takeFree = () => (free.length ? free.splice(Math.floor(r() * free.length), 1)[0] : null);
const isFree = (x, y) => free.some(([fx, fy]) => fx === x && fy === y);
const claim = (x, y) => free.splice(free.findIndex(([fx, fy]) => fx === x && fy === y), 1);
const objects = [];
const cards = [];
// The cards on the grid first: their nouns are good sentence subjects.
for (let i = int(0, 3); i > 0 && placeable.length; i--) {
const at = takeFree();
if (at) cards.push({ card: pick(placeable).name, x: at[0], y: at[1] });
}
const objectKinds = [pick(objectNouns), pick(objectNouns)];
const cardNouns = cards.map((c) => vocab.cards.find((v) => v.name === c.card).noun);
const subjects = [...objectKinds, ...cardNouns];
// Whole sentences, in a row or a column: NOUN [AND NOUN] IS [NOT] (PROPERTY | NOUN) [AND ...].
const sentence = (first) => {
// The first is the player's: the first object noun IS YOU (that noun always has an object).
if (first) return [objectKinds[0], "IS", "YOU"];
const line = [chance(0.7) ? pick(subjects) : pick(nouns)];
if (chance(0.15)) line.push("AND", pick(subjects));
line.push("IS");
if (chance(0.15)) line.push("NOT");
line.push(chance(0.75) ? pick(properties) : pick(nouns));
if (chance(0.2)) line.push("AND", pick(properties));
return line;
};
const sentences = [];
if (chance(0.9)) sentences.push(sentence(true));
for (let i = int(1, 2); i > 0; i--) sentences.push(sentence(false));
for (const line of sentences) {
for (let tries = 0; tries < 50; tries++) {
const across = chance(0.5);
const [dx, dy] = across ? [1, 0] : [0, 1];
const [maxX, maxY] = [width - 1 - dx * (line.length - 1), height - 1 - dy * (line.length - 1)];
if (maxX < 0 || maxY < 0) continue;
const [x0, y0] = [int(0, maxX), int(0, maxY)];
const cells = line.map((_, i) => [x0 + dx * i, y0 + dy * i]);
if (!cells.every(([x, y]) => isFree(x, y))) continue;
cells.forEach(([x, y], i) => {
claim(x, y);
objects.push({ kind: `text:${line[i]}`, x, y });
});
break;
}
}
// Loose words, the objects of the nouns, and now and then one object on an occupied tile.
const allWords = [...vocab.words.map((w) => w.name), ...OPERATORS];
for (let i = int(0, 3); i > 0; i--) {
const at = takeFree();
if (at) objects.push({ kind: `text:${pick(allWords)}`, x: at[0], y: at[1] });
}
for (let i = int(1, 6); i > 0; i--) {
const at = takeFree();
if (at) objects.push({ kind: (i === 1 ? objectKinds[0] : pick(objectKinds)).toLowerCase(), x: at[0], y: at[1] });
}
if (objects.length && chance(0.2)) {
const { x, y } = pick(objects);
objects.push({ kind: pick(objectKinds).toLowerCase(), x, y });
}
const hand = [];
for (let i = int(0, 3); i > 0 && holdable.length; i--) hand.push(pick(holdable).name);
// The script: moves, casts and plays (mostly of cards in the hand), and the chooser's answers.
const answer = () => {
const k = r();
return k < 0.5 ? String(int(1, 3)) : k < 0.8 ? pick(vocab.cards).name : pick(["top", "bottom"]);
};
const coords = () => {
if (cards.length && chance(0.5)) {
const c = pick(cards);
return [c.x, c.y];
}
return [int(0, width), int(0, height)];
};
const minions = vocab.cards.filter((c) => c.type === "minion");
const script = [];
const length = int(5, 25);
let base = null; // the base the last Assemble asked for by name, if any
while (script.length < length) {
const k = r();
if (k < 0.6) script.push(pick(DIRS));
else if (k < 0.93) {
// After an Assemble, now and then play what it may have made: "<minion> (Assembled)".
if (base && chance(0.4)) {
script.push(`c ${chance(0.8) ? base : pick(minions).name} (Assembled) ${int(0, width - 1)} ${int(0, height - 1)}`);
continue;
}
const name = hand.length && chance(0.75) ? pick(hand) : pick(vocab.cards).name;
const card = vocab.cards.find((c) => c.name === name);
const at = chance(card.type === "spell" ? 0.5 : 0.85) ? coords() : null;
script.push(at ? `c ${card.name} ${at[0]} ${at[1]}` : `c ${card.name}`);
if (card.chooses) {
const answers = Array.from({ length: int(0, 3) }, answer);
if (answers.length && minions.length && chance(0.5)) answers[0] = pick(minions).name;
if (minions.some((m) => m.name === answers[0])) base = answers[0];
script.push(...answers);
}
} else script.push(answer());
}
return { level: { width, height, objects, cards, hand }, script: script.slice(0, 25) };
}
/** A unified diff of two texts, line by line, with 3 lines of context. */
export function unifiedDiff(a, b, nameA = "rust", nameB = "ts") {
const x = a.split("\n");
const y = b.split("\n");
const lcs = Array.from({ length: x.length + 1 }, () => new Uint32Array(y.length + 1));
for (let i = x.length - 1; i >= 0; i--) {
for (let j = y.length - 1; j >= 0; j--) {
lcs[i][j] = x[i] === y[j] ? lcs[i + 1][j + 1] + 1 : Math.max(lcs[i + 1][j], lcs[i][j + 1]);
}
}
// Each op: [sign, line, lines of a before it, lines of b before it].
const ops = [];
for (let i = 0, j = 0; i < x.length || j < y.length;) {
if (i < x.length && j < y.length && x[i] === y[j]) ops.push([" ", x[i], i++, j++]);
else if (i < x.length && (j === y.length || lcs[i + 1][j] >= lcs[i][j + 1])) ops.push(["-", x[i], i++, j]);
else ops.push(["+", y[j], i, j++]);
}
const out = [`--- ${nameA}`, `+++ ${nameB}`];
const changed = ops.flatMap((op, k) => (op[0] === " " ? [] : [k]));
for (let k = 0; k < changed.length;) {
const first = Math.max(0, changed[k] - 3);
let last = changed[k];
while (k < changed.length && changed[k] <= last + 6) last = changed[k++];
const hunk = ops.slice(first, Math.min(ops.length, last + 4));
const aLen = hunk.filter((op) => op[0] !== "+").length;
const bLen = hunk.filter((op) => op[0] !== "-").length;
out.push(`@@ -${hunk[0][2] + (aLen ? 1 : 0)},${aLen} +${hunk[0][3] + (bLen ? 1 : 0)},${bLen} @@`);
for (const [sign, line] of hunk) out.push(`${sign}${line}`);
}
return out.join("\n");
}
/** Run `cmd args` in `cwd` with `input` on stdin; resolves to { out, err, code, signal }. */
function runCli(cmd, args, cwd, input) {
return new Promise((done) => {
const child = spawn(cmd, args, { cwd, timeout: TIMEOUT_MS, killSignal: "SIGKILL" });
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d));
child.stderr.on("data", (d) => (err += d));
child.stdin.on("error", () => {});
child.on("close", (code, signal) => done({ out, err, code, signal }));
child.stdin.end(input);
});
}
/** Did a CLI run crash? Both CLIs exit 0 (won) or 1 (lost, quit) and print nothing on stderr otherwise. */
export const crashed = (r) => r.signal != null || (r.code !== 0 && r.code !== 1) || r.err.trim() !== "";
/** Compare a Rust run with a TypeScript run: "agree", "outputs differ", or which side crashed. */
export function verdict(rust, ts) {
const [rc, tc] = [crashed(rust), crashed(ts)];
if (rc && tc) return "both crashed";
if (rc) return "the Rust CLI crashed";
if (tc) return "the TypeScript CLI crashed";
return rust.out === ts.out ? "agree" : "outputs differ";
}
const why = (r) =>
r.signal === "SIGKILL" ? `timed out after ${TIMEOUT_MS / 1000} s`
: r.signal ? `killed by ${r.signal}`
: r.err.trim().split("\n").slice(0, 8).join("\n") || `exit code ${r.code}`;
/** The highest module whose Rust and TypeScript references both have the SpellGrid CLI. */
function defaultModule() {
const withCli = moduleDirs().filter((m) =>
existsSync(join(m.path, "reference", "ts", "src", "cli.ts")) && existsSync(join(m.path, "reference", "rust", "src", "cli.rs")));
if (!withCli.length) fail("diff-tracks: no module has a SpellGrid CLI in both references");
return withCli.at(-1).num;
}
async function main() {
const opts = parseArgs(process.argv.slice(2));
const num = opts.module ?? defaultModule();
const base = opts.dir ? resolve(opts.dir) : mkdtempSync(join(tmpdir(), "diff-tracks-"));
if (existsSync(base) && readdirSync(base).length && !existsSync(join(base, MARKER))) {
fail(`diff-tracks: ${base} is not empty and was not staged by diff-tracks; give an empty or new directory`);
}
mkdirSync(base, { recursive: true });
writeFileSync(join(base, MARKER), `${num}\n`);
const rust = stagePlay("rust", join(base, "rust"), num);
const ts = stagePlay("ts", join(base, "ts"), num);
const cases = join(base, "cases");
mkdirSync(cases, { recursive: true });
const build = spawnSync("cargo", ["build", "--quiet", "--bin", "spellgrid"], { cwd: rust, stdio: ["ignore", "inherit", "inherit"] });
if (build.status !== 0) fail(`diff-tracks: cargo build failed in ${rust}`);
if (!existsSync(join(ts, "node_modules"))) {
const r = spawnSync("npm", ["install", "--no-audit", "--no-fund", "--loglevel=error"], { cwd: ts, stdio: ["ignore", "ignore", "inherit"] });
if (r.status !== 0) fail(`diff-tracks: npm install failed in ${ts}`);
}
const vocab = loadVocab(join(ts, "src", "content", "words.json"), [join(ts, "cards")]);
const seeds = opts.seed !== null ? [opts.seed] : Array.from({ length: opts.seeds }, (_, i) => i + 1);
const width = Math.max(1, Math.min(8, availableParallelism()));
const runCase = async (seed) => {
const { level, script } = generateCase(seed, vocab);
const levelPath = join(cases, `seed-${seed}.json`);
writeFileSync(levelPath, `${JSON.stringify(level, null, 2)}\n`);
writeFileSync(join(cases, `seed-${seed}.txt`), `${script.join("\n")}\n`);
const input = `${script.join("\n")}\n`;
const [a, b] = await Promise.all([
runCli(join(rust, "target", "debug", "spellgrid"), [levelPath], rust, input),
runCli(process.execPath, ["--import", "tsx", "src/cli.ts", levelPath], ts, input),
]);
return { seed, levelPath, script, rust: a, ts: b };
};
let bad = null;
for (let k = 0; k < seeds.length && !bad; k += width) {
const results = await Promise.all(seeds.slice(k, k + width).map(runCase));
bad = results.find((c) => verdict(c.rust, c.ts) !== "agree") ?? null;
}
if (!bad) {
console.log(`diff-tracks: module ${num}, ${seeds.length} seed(s), Rust and TypeScript agree`);
if (!opts.dir && !opts.keep) rmSync(base, { recursive: true, force: true });
return;
}
console.log(`diff-tracks: module ${num}, seed ${bad.seed}: ${verdict(bad.rust, bad.ts)}`);
console.log(`level: ${bad.levelPath}`);
console.log(`script:\n${bad.script.map((l) => ` ${l}`).join("\n")}`);
if (crashed(bad.rust)) console.log(`Rust:\n${why(bad.rust)}`);
if (crashed(bad.ts)) console.log(`TypeScript:\n${why(bad.ts)}`);
if (bad.rust.out !== bad.ts.out) console.log(unifiedDiff(bad.rust.out, bad.ts.out));
console.log(`(staged in ${base}; kept)`);
process.exit(1);
}
if (isMain(import.meta.url)) await main();
+5 -5
View File
@@ -46,10 +46,10 @@ export function resolveLevel(levelsDir, level) {
return { path: join("levels", match), staged: true };
}
/** Copy the reference, its cards and the levels into `dir`. Returns `dir`. */
export function stagePlay(lang, dir) {
const mod = moduleDirs().find((m) => m.num === MODULE);
if (!mod) fail(`no module ${MODULE} in the course`);
/** Copy the reference of module `num` (default 10), its cards and the levels into `dir`. Returns `dir`. */
export function stagePlay(lang, dir, num = MODULE) {
const mod = moduleDirs().find((m) => m.num === num);
if (!mod) fail(`no module ${num} in the course`);
const ref = join(mod.path, "reference", lang);
if (!existsSync(ref)) fail(`no ${lang} reference in ${mod.path}`);
if (dir) {
@@ -61,7 +61,7 @@ export function stagePlay(lang, dir) {
dir = mkdtempSync(join(tmpdir(), `spellgrid-${lang}-`));
}
cpSync(ref, dir, { recursive: true });
for (const cards of cardDirs(MODULE)) cpSync(cards, join(dir, "cards"), { recursive: true });
for (const cards of cardDirs(num)) cpSync(cards, join(dir, "cards"), { recursive: true });
if (existsSync(join(mod.path, "levels"))) cpSync(join(mod.path, "levels"), join(dir, "levels"), { recursive: true });
writeFileSync(join(dir, MARKER), `${lang}\n`);
return dir;
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env node
// Search a SpellGrid level (Module 10) for solutions, by brute force, with the course's
// TypeScript reference engine. Used in Module 11 to check that a level has the solution
// TypeScript reference engine. Used in Module 13 to check that a level has the solution
// its designer meant and no other.
//
// node tools/solve-spellgrid.mjs [level] [--dir <dir>] [--max N] [--naive] [--never-rule "X IS Y"] [--all]
+3
View File
@@ -15,6 +15,9 @@ if (track !== "rust" && track !== "ts") fail(`bad track in .course.json: ${track
const mods = moduleDirs();
const mod = mods.find((m) => m.num === num);
if (!mod) fail(`no module ${num} in ${courseRoot()}`);
if (!testFiles(mod.path, "rust").length && !testFiles(mod.path, "ts").length) {
fail(`Module ${num} has no tests to start: it is made of reading and exercises. Open ${join(mod.path, "lesson.md")}.`);
}
if (num !== "01") {
const prevNum = String(Number(num) - 1).padStart(2, "0");
+114
View File
@@ -0,0 +1,114 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { toolsDir } from "./helpers.mjs";
import { validate } from "../schema.mjs";
import { cardDirs } from "../lib.mjs";
import { generateCase, loadVocab, parseArgs, rng, unifiedDiff, verdict } from "../diff-tracks.mjs";
const course = join(toolsDir, "..", "course");
const levelSchema = JSON.parse(readFileSync(join(course, "schema", "level.schema.json"), "utf8"));
// The Module 10 vocabulary: its words.json and every card its workspace has.
const vocab = loadVocab(join(course, "10-spellgrid", "reference", "ts", "src", "content", "words.json"), cardDirs("10"));
const SEEDS = Array.from({ length: 200 }, (_, i) => i + 1);
test("diff_tracks_parses_args", () => {
assert.deepEqual(parseArgs([]), { seeds: 100, seed: null, module: null, dir: null, keep: false });
assert.deepEqual(parseArgs(["--seeds", "500", "--module", "10", "--dir", "x", "--keep"]), {
seeds: 500, seed: null, module: "10", dir: "x", keep: true,
});
assert.equal(parseArgs(["--seed", "42"]).seed, 42);
});
test("diff_tracks_rng_is_seeded", () => {
const draw = (seed) => { const r = rng(seed); return [r(), r(), r()]; };
assert.deepEqual(draw(7), draw(7));
assert.notDeepEqual(draw(7), draw(8));
for (const x of draw(123)) assert.ok(x >= 0 && x < 1);
});
test("diff_tracks_cases_are_deterministic_and_vary", () => {
assert.deepEqual(generateCase(17, vocab), generateCase(17, vocab));
const distinct = new Set(SEEDS.map((s) => JSON.stringify(generateCase(s, vocab))));
assert.equal(distinct.size, SEEDS.length);
});
test("diff_tracks_levels_are_valid", () => {
const nouns = new Set(vocab.words.filter((w) => w.role === "noun").map((w) => w.name));
const words = new Set([...vocab.words.map((w) => w.name), "IS", "AND", "NOT"]);
const placeable = new Set(vocab.cards.filter((c) => c.type === "minion" && c.noun).map((c) => c.name));
const holdable = new Set(vocab.cards.filter((c) => c.type === "spell" || (c.type === "minion" && c.noun)).map((c) => c.name));
for (const seed of SEEDS) {
const { level } = generateCase(seed, vocab);
assert.deepEqual(validate(levelSchema, level), [], `seed ${seed}`);
assert.ok(level.width >= 4 && level.width <= 8 && level.height >= 4 && level.height <= 8, `seed ${seed} size`);
for (const o of [...level.objects, ...level.cards]) {
assert.ok(o.x < level.width && o.y < level.height, `seed ${seed}: (${o.x}, ${o.y}) off the grid`);
}
for (const { kind } of level.objects) {
const ok = kind.startsWith("text:") ? words.has(kind.slice(5)) : nouns.has(kind.toUpperCase());
assert.ok(ok, `seed ${seed}: kind ${kind}`);
}
for (const { card } of level.cards) assert.ok(placeable.has(card), `seed ${seed}: card ${card}`);
for (const card of level.hand) assert.ok(holdable.has(card), `seed ${seed}: hand ${card}`);
}
});
test("diff_tracks_scripts_follow_the_cli_grammar", () => {
const names = vocab.cards.map((c) => c.name);
const answers = new Set([...names, "top", "bottom"]);
let casts = 0;
let moves = 0;
for (const seed of SEEDS) {
const { script } = generateCase(seed, vocab);
assert.ok(script.length >= 5 && script.length <= 25, `seed ${seed}: ${script.length} commands`);
for (const line of script) {
const cast = /^c (.+?)(?: (\d+) (\d+))?$/.exec(line);
const card = cast && cast[1].replace(/ \(Assembled\)$/, "");
const ok = /^[wasd]$/.test(line) || /^\d+$/.test(line) || answers.has(line) || (cast && names.includes(card));
assert.ok(ok, `seed ${seed}: ${JSON.stringify(line)}`);
if (cast) casts++;
if (/^[wasd]$/.test(line)) moves++;
}
}
assert.ok(casts > 100 && moves > 500, `casts ${casts}, moves ${moves}`);
});
test("diff_tracks_levels_form_sentences", () => {
// Biased toward whole sentences: most levels have at least one NOUN IS ... in a row or column.
const withSentence = SEEDS.filter((seed) => {
const { level } = generateCase(seed, vocab);
const at = new Map(level.objects.filter((o) => o.kind.startsWith("text:")).map((o) => [`${o.x},${o.y}`, o.kind.slice(5)]));
return [...at].some(([key, word]) => {
const [x, y] = key.split(",").map(Number);
return word !== "IS" && (at.get(`${x + 1},${y}`) === "IS" && at.has(`${x + 2},${y}`) ||
at.get(`${x},${y + 1}`) === "IS" && at.has(`${x},${y + 2}`));
});
});
assert.ok(withSentence.length > SEEDS.length * 0.8, `${withSentence.length} of ${SEEDS.length}`);
});
test("diff_tracks_verdict", () => {
const ok = (out, code = 1) => ({ out, err: "", code, signal: null });
assert.equal(verdict(ok("a\n"), ok("a\n")), "agree");
// The same error printed by both is agreement; so is the same exit code for a lost game.
assert.equal(verdict(ok("x is not in the hand\n"), ok("x is not in the hand\n")), "agree");
assert.equal(verdict(ok("a\n", 0), ok("b\n", 0)), "outputs differ");
const panic = { out: "a\n", err: "thread 'main' panicked at src/cli.rs:1:1", code: 101, signal: null };
const thrown = { out: "a\n", err: "TypeError: boom", code: 1, signal: null };
const killed = { out: "", err: "", code: null, signal: "SIGKILL" };
assert.equal(verdict(panic, ok("a\n")), "the Rust CLI crashed");
assert.equal(verdict(ok("a\n"), thrown), "the TypeScript CLI crashed");
assert.equal(verdict(killed, thrown), "both crashed");
});
test("diff_tracks_unified_diff", () => {
const a = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"].join("\n");
const b = ["1", "2", "3", "4", "5", "six", "7", "8", "9", "10", "11"].join("\n");
assert.equal(unifiedDiff(a, b), [
"--- rust", "+++ ts",
"@@ -3,8 +3,9 @@", " 3", " 4", " 5", "-6", "+six", " 7", " 8", " 9", " 10", "+11",
].join("\n"));
assert.equal(unifiedDiff("same", "same"), "--- rust\n+++ ts");
});
+12
View File
@@ -98,3 +98,15 @@ test("start_module_00_not_startable", () => {
assert.notEqual(r.code, 0);
assert.match(r.out, /naive-engine/);
});
test("start_module_explains_a_module_with_no_tests", () => {
// A module made only of reading and exercises (Module 13): no tests for either track.
const course = copyFixture("course");
mkdirSync(join(course, "06-no-tests"));
writeFileSync(join(course, "06-no-tests", "lesson.md"), "# Module 6\n");
const ws = copyFixture("ws-start");
const r = run("start-module.mjs", ["06", ws], { COURSE_ROOT: course });
assert.equal(r.code, 1);
assert.match(r.out, /Module 06 has no tests to start/);
assert.doesNotMatch(r.out, /finish module/);
});