diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 00d019a..3dd532f 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -59,6 +59,41 @@ jobs: node tools/coupling-audit.mjs "$d" done + diff-tracks: + name: Differential fuzzing of the Rust and TS references (300 seeds) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + - uses: dtolnay/rust-toolchain@1.85.1 + # The highest module with a SpellGrid CLI; any divergence prints the seed, the level and a diff. + - run: node tools/diff-tracks.mjs --seeds 300 + + mutation-check: + name: Mutation check (tools/mutants.json against the highest TS reference) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + - run: node tools/mutation-check.mjs + + verify-levels: + name: Verify the SpellGrid level proofs (solver) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + - run: node tools/verify-levels.mjs + site: name: Site build runs-on: ubuntu-latest diff --git a/README.md b/README.md index 69854d4..0e286e6 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Every module follows the same beats: Observe, Predict, Spec, Build, Name it, Exp ## Two paths -- **Designer path**: read the lessons and scenarios, predict outcomes, and reason about the design. +- **Designer path**: read the lessons and scenarios, predict outcomes, and reason about the design. After Module 10, the [designer capstone](course/10-spellgrid/designer-capstone.md) has you design three SpellGrid cards and a word as data, predict what they do, and run your predictions with `tools/run-scenario.mjs`, on the course's reference engine. - **Builder path**: also make the tests pass in your own engine. ## Mentor path @@ -42,7 +42,7 @@ node tools/start-module.mjs 01 ~/my-engine `start-module` copies a module's tests for your track (from your workspace's `.course.json`) and refuses to start a module if you have not yet taken the previous one. Module 00 uses the `naive-engine/` project and is not started this way, and Module 13 has no tests to start. -Other tools: `tools/coupling-audit.mjs ` (engine code must not name specific cards or nouns), `tools/expansion-check.mjs ` (an Expansion Drop must change content only), `tools/verify-references.mjs [NN]` (reference solutions pass the tests; `--expansion` also checks every Expansion Drop solution), `tools/play-spellgrid.mjs [l1..l10]` (play the Module 10 capstone, SpellGrid, on the command line with the reference engine), `tools/solve-spellgrid.mjs [l1..l10|level.json]` (search a SpellGrid level for its shortest solution by brute force; `--never-rule "X IS Y"` asks whether it can be solved without a given rule; see Module 13), `tools/verify-levels.mjs [level ...]` (checks every claim in `course/10-spellgrid/levels/proofs.json` with the solver: each level's shortest solution, and that it can't be solved without its idea; `npm run verify:levels`), `tools/diff-tracks.mjs [--seeds N] [--seed S] [--module NN] [--dir D] [--keep]` (differential fuzzer: plays random SpellGrid levels and scripts on the Rust and the TypeScript reference CLIs and fails on the first seed where they print different text or one of them crashes; `npm run diff:tracks`). +Other tools: `tools/coupling-audit.mjs ` (engine code must not name specific cards or nouns), `tools/expansion-check.mjs ` (an Expansion Drop must change content only), `tools/verify-references.mjs [NN]` (reference solutions pass the tests; `--expansion` also checks every Expansion Drop solution), `tools/play-spellgrid.mjs [l1..l10]` (play the Module 10 capstone, SpellGrid, on the command line with the reference engine), `tools/solve-spellgrid.mjs [l1..l10|level.json]` (search a SpellGrid level for its shortest solution by brute force; `--never-rule "X IS Y"` asks whether it can be solved without a given rule; see Module 13), `tools/verify-levels.mjs [level ...]` (checks every claim in `course/10-spellgrid/levels/proofs.json` with the solver: each level's shortest solution, and that it can't be solved without the rule its proof forbids; `npm run verify:levels`), `tools/diff-tracks.mjs [--seeds N] [--seed S] [--module NN] [--dir D] [--keep]` (differential fuzzer: plays random SpellGrid levels and scripts on the Rust and the TypeScript reference CLIs and fails on the first seed where they print different text or one of them crashes; `npm run diff:tracks`), `tools/mutation-check.mjs [name ...]` (applies each mutant in `tools/mutants.json`, a one-line change to an ordering decision, to the highest module's TypeScript reference, runs every module's tests, and fails if a mutant survives or no longer applies; see Module 13; `npm run mutation:check`), `tools/run-scenario.mjs ` (plays a designer's scenario, new cards, words and seam words as data with a level and inputs, on the Module 10 TypeScript reference and prints the board, rules, minions, hand, heroes' Health and log tree after each input; `npm run run:scenario`). ## Licence diff --git a/course/10-spellgrid/designer-capstone.md b/course/10-spellgrid/designer-capstone.md new file mode 100644 index 0000000..3bda01d --- /dev/null +++ b/course/10-spellgrid/designer-capstone.md @@ -0,0 +1,113 @@ +# SpellGrid designer capstone + +The modules ask you to build an engine. This capstone asks you to *use* one, as a designer would: add content, predict what it does, and find out where your predictions were wrong. It needs no engine of your own. The course's TypeScript reference for [Module 10](lesson.md) does the work, and `tools/run-scenario.mjs` runs it on your data. You do need Node, as for the course's other tools. + +## The task + +1. **Design three cards and one word**, within the vocabulary SpellGrid already has. A card can use any trigger, effect and aura of the card format (`course/schema/card.schema.json`); its noun, if it has one, must be a word the grid knows: one it has already, such as `ROCK` or `SKULL`, or your new word. The word is a noun or a property. If it crosses the seam, say what it names: a noun can name a trigger (`"DEATHRATTLE": "deathrattle"`), and a property can multiply one (`"DOUBLE": { "times": 2, "stacking": "max" }`). +2. **Write them as data**: three card files and the word's entries, exactly as an [Expansion Drop](lesson.md#expansion-drop) would add them. No code. +3. **Write five predictions as scenarios.** A level that uses your cards and word, and the inputs to play on it. For each prediction, write down *before you run anything* what you expect after its input: the rules, each minion's Health, the heroes' Health, what the log shows. +4. **Run them** with `node tools/run-scenario.mjs `. +5. **Explain each surprise** in terms of a pattern the course named. Where you were right, say which pattern you used to get there. + +What a designer can say is exactly what an Expansion Drop can. The tool never touches the engine. If an idea needs a new kind of trigger or effect, that is an engine change: write it down as a finding, and design around it. + +## The scenario file + +A scenario is one JSON object: + +```json +{ + "cards": [ { "name": "Ash Totem", "type": "minion", "...": "card format" } ], + "words": [ { "name": "WOUND", "role": "noun" } ], + "seams": { "triggerWords": { "WOUND": "selfDamaged" } }, + "level": { "width": 6, "height": 6, "objects": [], "cards": [], "hand": [] }, + "inputs": ["c Scorch", "w"] +} +``` + +- `cards`, `words` and `seams` are optional. The cards are added to the course's cards, so a card may not reuse a course card's name. The words are added to `words.json`, and the seam words to `seams.json`. +- `level` is a level in the course's level format (`course/schema/level.schema.json`), as in `course/10-spellgrid/levels/`. Its `cards` stand on the grid as minions; its `hand` is the cards you can play. +- `inputs` are the commands the SpellGrid command line reads ([Play it](lesson.md#play-it)): `w`, `a`, `s`, `d` for one turn, `c [x y]` to cast a spell (at the minion on that tile, if given) or play a minion onto an empty tile. If a card asks a question (Graft), the next input is the answer. + +The tool checks the whole file before it runs anything, and lists every problem: a card that doesn't match the card format (with the field that is wrong), a word that clashes with an existing one, a seam word that isn't a word. Then it prints the board, the rules, the minions, the hand and both heroes' Health at the start, and again after each input, with that input's Part I log as an indented tree. The tree is the same as `logTree()` from Module 2: each trigger's consequences are indented under it. + +You can put all five predictions in one scenario, one input or a few each, or write a scenario per prediction. One scenario is less typing; separate ones keep a surprise in one prediction from changing the board for the next. + +## Worked example + +The files are in [`designer-example/`](designer-example/): `cards/`, `words.json`, `seams.json`, `scenario.json` (which holds all of them, as the tool reads them) and `output.txt`, the tool's output. `npm run test:tools` runs the scenario again and fails if `output.txt` is out of date. + +### The design + +- **Word: `WOUND`**, a noun that names the trigger "whenever this minion takes damage" (`selfDamaged`). `WOUND IS DOUBLE` on the grid makes every such trigger activate twice. This is the Module 10 Expansion Drop again, with a different trigger. +- **Ash Totem**, a 0/6 minion whose noun is `ROCK`: "Whenever this minion takes damage, deal 1 damage to the enemy hero." +- **Thorn Idol**, a 0/3 minion whose noun is `SKULL`: "Whenever this minion takes damage, deal 1 damage to all other minions. Deathrattle: write `WIN` on this tile." +- **Scorch**, a spell: "Deal 1 damage to all minions." + +The level: `BABA IS YOU` along the top; `WOUND IS` with a gap, and `DOUBLE` below the gap; Baba below `DOUBLE`. The totem stands to the right. At the bottom, `BABA IS` ends on the idol's tile. The hand is three Scorches. + +```text +BABA IS YOU . . . +WOUND IS . . . . +. . DOUBLE . . . +. . baba . rock . +. . . . . . +BABA IS skull . . . +``` + +### The predictions + +Written before the first run. The inputs are `c Scorch`, `w`, `c Scorch`, `c Scorch`, `d`. + +1. **`c Scorch`.** Scorch hits the totem and the idol. The totem's trigger hits the enemy hero once. The idol's trigger hits the totem, which triggers again. Enemy hero 28, totem 4, idol 2. +2. **`w`.** Baba pushes `DOUBLE` up into the gap. `WOUND IS DOUBLE` is in force. Nothing else changes. +3. **`c Scorch`.** The same as prediction 1, doubled: the enemy hero loses 4, to 24. Totem 2, idol 1. +4. **`c Scorch`.** The totem and the idol both reach 0 at once. The totem dies first, so the idol's trigger finds nothing to hit: the enemy hero loses 2. The idol's Deathrattle writes `WIN` on its tile, `BABA IS WIN` is spelled, and I win. +5. **`d`.** If I haven't won already, this move wins. + +### What the tool printed + +All of it is in [`designer-example/output.txt`](designer-example/output.txt). Predictions 1, 2 and 5 held. After input 1: + +```text +Minions: Ash Totem (4,3) 4 Health; Thorn Idol (2,5) 2 Health +Hand: Scorch, Scorch +Heroes: 30 Health (yours), 28 Health (enemy) +``` + +Prediction 3 was wrong: the enemy hero went from 28 to 22, not 24, and the totem has 1 Health, not 2. The log of input 3 shows why. The idol's trigger activates twice, and *each* of its hits on the totem sets off the totem's trigger twice: + +```text +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +``` + +Prediction 4 was wrong twice. The enemy hero lost 6 again, to 16, and the level was not won after the cast: the board shows `BABA IS WIN`, and the output goes straight on to the next input. The `d` that followed won it. + +### The surprises, explained + +- **Prediction 1 held because of the action queue, resolved depth-first** (Module 2). Scorch's two Damage Events queue the totem's trigger and the idol's, in order of play. The idol's hit on the totem is a new event inside the idol's trigger, so the totem's second trigger resolves there, one level down in the tree, before anything else. +- **Prediction 3: the interceptor counts every activation** (Module 4). I treated `WOUND IS DOUBLE` as "twice the result". It is a rule about each trigger: the one routine that counts activations is asked every time a damage trigger starts, including the ones nested inside another trigger. The idol's trigger runs twice; each run damages the totem; each of those damages starts the totem's trigger, which runs twice. Two levels of doubling multiply: 2 from the Scorch on the totem, plus 2 × 2 from the idol, is 6. +- **Prediction 4, first half: the deferred state-based check** (Module 3). I had the totem leave play when it reached 0. Nothing leaves play in the middle of a Sequence. The totem is mortally wounded, still in play and still triggering, until the Death Creation Step at the end of the outermost Phase. So the idol's hits land on a totem at 0 and then at -1, and each one still reaches the hero, twice. Only then do both die (`Destroyed{id: 10}`, `Destroyed{id: 11}`, then `DeathPhase{round: 1}` and the idol's Deathrattle). +- **Prediction 4, second half: the phase pipeline** (Module 9). `BABA IS WIN` is a rule as soon as the Sequence ends: the integration seam (Module 10) writes the dead idol's text while its object is still on the tile, then removes the dead, then reparses. But winning is not a rule being in force. It is the win check, one phase of a turn, and casting a spell is not a turn. The next move runs the phases, and the win phase finds Baba is `YOU` and `WIN`. + +What the example teaches as design: a damage trigger that damages other minions is an amplifier, and a multiplier on such a trigger is multiplied by every level of the chain. A designer who wants `WOUND IS DOUBLE` to be a modest bonus should give the idol a target that doesn't trigger back (the enemy hero), or put a cap on the board as well (`WOUND IS NOT DOUBLE`, Module 7). + +## Rubric + +Mark each line 0, 1 or 2. + +| | 0 | 1 | 2 | +|---|---|---|---| +| **Data** | The tool rejects the files, or a card needs an engine change. | The files run, but a card or the word plays no part in any prediction. | Three cards and one word, each used in at least one prediction, and no engine change. | +| **Predictions** | Written after running, or too vague to be wrong ("something happens"). | Some predictions state numbers or rules; others don't. | Each prediction states what can be checked in the output: Health, rules, the shape of the log, the win. | +| **Surprises** | Wrong predictions are corrected but not explained. | Each surprise is explained, but in terms of this level only ("the totem triggered again"). | Each surprise, and each prediction that held, is explained by a named pattern, with the module it comes from. | +| **Reach** | All five predictions test one mechanism. | Two mechanisms. | At least three: for example a Part I order, a seam, and a Part II phase or rule. | +| **Design conclusion** | None. | A remark about the level. | What you would change in the cards or the word, and why, as in the last paragraph of the example. | + +A full mark is 10. A prediction that turned out right is worth as much as one that turned out wrong, as long as it was precise enough that it could have been wrong. diff --git a/course/10-spellgrid/designer-example/cards/ash-totem.json b/course/10-spellgrid/designer-example/cards/ash-totem.json new file mode 100644 index 0000000..d233170 --- /dev/null +++ b/course/10-spellgrid/designer-example/cards/ash-totem.json @@ -0,0 +1,14 @@ +{ + "name": "Ash Totem", + "type": "minion", + "cost": 2, + "attack": 0, + "health": 6, + "noun": "ROCK", + "split": 0, + "text": [ + { "kind": "trigger", "on": "selfDamaged" }, + { "kind": "effect", "effect": { "type": "dealDamage", "target": "enemyHero", "amount": 1 } } + ], + "game": "spellgrid" +} diff --git a/course/10-spellgrid/designer-example/cards/scorch.json b/course/10-spellgrid/designer-example/cards/scorch.json new file mode 100644 index 0000000..dd50a7e --- /dev/null +++ b/course/10-spellgrid/designer-example/cards/scorch.json @@ -0,0 +1,10 @@ +{ + "name": "Scorch", + "type": "spell", + "cost": 1, + "split": 0, + "text": [ + { "kind": "effect", "effect": { "type": "dealDamage", "target": "allMinions", "amount": 1 } } + ], + "game": "spellgrid" +} diff --git a/course/10-spellgrid/designer-example/cards/thorn-idol.json b/course/10-spellgrid/designer-example/cards/thorn-idol.json new file mode 100644 index 0000000..fcf7c0a --- /dev/null +++ b/course/10-spellgrid/designer-example/cards/thorn-idol.json @@ -0,0 +1,16 @@ +{ + "name": "Thorn Idol", + "type": "minion", + "cost": 3, + "attack": 0, + "health": 3, + "noun": "SKULL", + "split": 2, + "text": [ + { "kind": "trigger", "on": "selfDamaged" }, + { "kind": "effect", "effect": { "type": "dealDamage", "target": "allOtherMinions", "amount": 1 } }, + { "kind": "trigger", "on": "deathrattle" }, + { "kind": "effect", "effect": { "type": "spawnText", "word": "WIN", "at": "self" } } + ], + "game": "spellgrid" +} diff --git a/course/10-spellgrid/designer-example/output.txt b/course/10-spellgrid/designer-example/output.txt new file mode 100644 index 0000000..697b0f6 --- /dev/null +++ b/course/10-spellgrid/designer-example/output.txt @@ -0,0 +1,126 @@ +== start +BAB IS YOU . . . +WOU IS . . . . + . . DOU . . . + . . bab . roc . + . . . . . . +BAB IS sku . . . +Rules: BABA IS YOU, TEXT IS PUSH +Minions: Ash Totem (4,3) 6 Health; Thorn Idol (2,5) 3 Health +Hand: Scorch, Scorch, Scorch +Heroes: 30 Health (yours), 30 Health (enemy) + +> c Scorch +Log: +SpellCast{card: "Scorch"} +Damaged{target: 10, amount: 1} +Damaged{target: 11, amount: 1} +Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +BAB IS YOU . . . +WOU IS . . . . + . . DOU . . . + . . bab . roc . + . . . . . . +BAB IS sku . . . +Rules: BABA IS YOU, TEXT IS PUSH +Minions: Ash Totem (4,3) 4 Health; Thorn Idol (2,5) 2 Health +Hand: Scorch, Scorch +Heroes: 30 Health (yours), 28 Health (enemy) + +> w +BAB IS YOU . . . +WOU IS DOU . . . + . . bab . . . + . . . . roc . + . . . . . . +BAB IS sku . . . +Rules: BABA IS YOU, TEXT IS PUSH, WOUND IS DOUBLE +Minions: Ash Totem (4,3) 4 Health; Thorn Idol (2,5) 2 Health +Hand: Scorch, Scorch +Heroes: 30 Health (yours), 28 Health (enemy) + +> c Scorch +Log: +SpellCast{card: "Scorch"} +Damaged{target: 10, amount: 1} +Damaged{target: 11, amount: 1} +Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +BAB IS YOU . . . +WOU IS DOU . . . + . . bab . . . + . . . . roc . + . . . . . . +BAB IS sku . . . +Rules: BABA IS YOU, TEXT IS PUSH, WOUND IS DOUBLE +Minions: Ash Totem (4,3) 1 Health; Thorn Idol (2,5) 1 Health +Hand: Scorch +Heroes: 30 Health (yours), 22 Health (enemy) + +> c Scorch +Log: +SpellCast{card: "Scorch"} +Damaged{target: 10, amount: 1} +Damaged{target: 11, amount: 1} +Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Triggered{source: 11, trigger: "damaged"} + Damaged{target: 10, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} + Triggered{source: 10, trigger: "damaged"} + Damaged{target: 13, amount: 1} +Destroyed{id: 10} +Destroyed{id: 11} +DeathPhase{round: 1} +Triggered{source: 11, trigger: "deathrattle"} +BAB IS YOU . . . +WOU IS DOU . . . + . . bab . . . + . . . . . . + . . . . . . +BAB IS WIN . . . +Rules: BABA IS WIN, BABA IS YOU, TEXT IS PUSH, WOUND IS DOUBLE +Minions: none +Hand: empty +Heroes: 30 Health (yours), 16 Health (enemy) + +> d +BAB IS YOU . . . +WOU IS DOU . . . + . . . bab . . + . . . . . . + . . . . . . +BAB IS WIN . . . +Rules: BABA IS WIN, BABA IS YOU, TEXT IS PUSH, WOUND IS DOUBLE +Minions: none +Hand: empty +Heroes: 30 Health (yours), 16 Health (enemy) +You win! diff --git a/course/10-spellgrid/designer-example/scenario.json b/course/10-spellgrid/designer-example/scenario.json new file mode 100644 index 0000000..3919339 --- /dev/null +++ b/course/10-spellgrid/designer-example/scenario.json @@ -0,0 +1,167 @@ +{ + "cards": [ + { + "name": "Ash Totem", + "type": "minion", + "cost": 2, + "attack": 0, + "health": 6, + "noun": "ROCK", + "split": 0, + "text": [ + { + "kind": "trigger", + "on": "selfDamaged" + }, + { + "kind": "effect", + "effect": { + "type": "dealDamage", + "target": "enemyHero", + "amount": 1 + } + } + ], + "game": "spellgrid" + }, + { + "name": "Thorn Idol", + "type": "minion", + "cost": 3, + "attack": 0, + "health": 3, + "noun": "SKULL", + "split": 2, + "text": [ + { + "kind": "trigger", + "on": "selfDamaged" + }, + { + "kind": "effect", + "effect": { + "type": "dealDamage", + "target": "allOtherMinions", + "amount": 1 + } + }, + { + "kind": "trigger", + "on": "deathrattle" + }, + { + "kind": "effect", + "effect": { + "type": "spawnText", + "word": "WIN", + "at": "self" + } + } + ], + "game": "spellgrid" + }, + { + "name": "Scorch", + "type": "spell", + "cost": 1, + "split": 0, + "text": [ + { + "kind": "effect", + "effect": { + "type": "dealDamage", + "target": "allMinions", + "amount": 1 + } + } + ], + "game": "spellgrid" + } + ], + "words": [ + { + "name": "WOUND", + "role": "noun" + } + ], + "seams": { + "triggerWords": { + "WOUND": "selfDamaged" + } + }, + "level": { + "width": 6, + "height": 6, + "objects": [ + { + "kind": "text:BABA", + "x": 0, + "y": 0 + }, + { + "kind": "text:IS", + "x": 1, + "y": 0 + }, + { + "kind": "text:YOU", + "x": 2, + "y": 0 + }, + { + "kind": "text:WOUND", + "x": 0, + "y": 1 + }, + { + "kind": "text:IS", + "x": 1, + "y": 1 + }, + { + "kind": "text:DOUBLE", + "x": 2, + "y": 2 + }, + { + "kind": "baba", + "x": 2, + "y": 3 + }, + { + "kind": "text:BABA", + "x": 0, + "y": 5 + }, + { + "kind": "text:IS", + "x": 1, + "y": 5 + } + ], + "cards": [ + { + "card": "Ash Totem", + "x": 4, + "y": 3 + }, + { + "card": "Thorn Idol", + "x": 2, + "y": 5 + } + ], + "hand": [ + "Scorch", + "Scorch", + "Scorch" + ] + }, + "inputs": [ + "c Scorch", + "w", + "c Scorch", + "c Scorch", + "d" + ] +} diff --git a/course/10-spellgrid/designer-example/seams.json b/course/10-spellgrid/designer-example/seams.json new file mode 100644 index 0000000..8a00db1 --- /dev/null +++ b/course/10-spellgrid/designer-example/seams.json @@ -0,0 +1 @@ +{ "triggerWords": { "WOUND": "selfDamaged" } } diff --git a/course/10-spellgrid/designer-example/words.json b/course/10-spellgrid/designer-example/words.json new file mode 100644 index 0000000..768633b --- /dev/null +++ b/course/10-spellgrid/designer-example/words.json @@ -0,0 +1 @@ +[{ "name": "WOUND", "role": "noun" }] diff --git a/course/10-spellgrid/lesson.md b/course/10-spellgrid/lesson.md index f8bc431..0868ead 100644 --- a/course/10-spellgrid/lesson.md +++ b/course/10-spellgrid/lesson.md @@ -473,6 +473,8 @@ If the check fails, fix your engine (a normal commit, outside the drop), then re +The [designer capstone](designer-capstone.md) asks for a larger drop of the same kind, from the designer's side: three cards and one word as data, five predictions about what they do, run with `tools/run-scenario.mjs` on this module's reference, and each surprise explained by a pattern the course named. It needs no engine of your own, and it has a worked example. + ## Stress it Optional. Each question pushes the design past what this module's tests check. Predict first, then open the discussion. @@ -536,9 +538,9 @@ What it reports for the shipped levels: - **Level 2** has no solution shorter than 4 inputs, and among some 18,600 positions within 4 inputs exactly one wins: the intended graft, played and pushed into the lava. - **Level 3** is solved in 6 inputs, and under `--naive` the search exhausts all 11 reachable positions with no win. "The knight dies" is not only what the one scripted test shows; no other line of play saves the naive engine. -- **Levels 4 to 10** were designed against the solver: each was changed until its shortest solution was the intended one and the search, with the level's idea forbidden, ran out of positions. The idea is forbidden as a rule the player has to form: `DEATHRATTLE IS DOUBLE` (level 4), `KEG IS NOT KEG` (5), `LAVA IS HOT` (6), `SHARD IS OPEN` (7), `SHARD IS WIN` (8), `SKULL IS SINK` (9), `KNIGHT IS YOU` (10). +- **Levels 4 to 10** were designed against the solver: each was changed until its shortest solution was the intended one and the search, with one rule forbidden, ran out of positions. The forbidden rule is one the player has to form: `DEATHRATTLE IS DOUBLE` (level 4), `KEG IS NOT KEG` (5), `LAVA IS HOT` (6), `SHARD IS OPEN` (7), `SHARD IS WIN` (8), `SKULL IS SINK` (9), `KNIGHT IS YOU` (10). For levels 4, 5, 7 and 10 that rule is the level's idea. For levels 6, 8 and 9 it is only one ingredient of it: the search proves that the lava has to be `HOT` (6), that `SHARD IS WIN` has to be finished (8), and that the skulls have to be `SINK` (9). It doesn't prove the rest of the idea: that the keg melts before the shard's `DEFEAT` is checked, that the shard comes from Graft's top-half swap, that the two rocks move as one chain. -Each of these claims is written down in `course/10-spellgrid/levels/proofs.json`, one entry per level (`{ "level": "l4-crossed-out", "shortest": 7, "neverRule": "DEATHRATTLE IS DOUBLE" }`, or `"naiveUnsolvable": true` for level 3), and `node tools/verify-levels.mjs` (`npm run verify:levels`) checks every one with the solver. A forbidden rule is the closest a rule filter gets to "the idea": for level 6 it proves the keg has to melt, and the map does the rest, because the only push that sends the keg into the lava puts the knight on the shard. +Each of these claims is written down in `course/10-spellgrid/levels/proofs.json`, one entry per level (`{ "level": "l4-crossed-out", "shortest": 7, "neverRule": "DEATHRATTLE IS DOUBLE" }`, or `"naiveUnsolvable": true` for level 3), and `node tools/verify-levels.mjs` (`npm run verify:levels`) checks every one with the solver. A forbidden rule is the closest a rule filter gets to "the idea". For level 6 the map does the rest, because the only push that sends the keg into the lava puts the knight on the shard. For levels 8 and 9 the rest is shown, not proved: each level's test plays one line that does it, and checks on the way that it happened. An unintended solution is called a *cook*, and a level with one doesn't teach what it was built to teach. [Module 13](../13-break-your-engine/lesson.md) explains how the solver works and what it can't tell you. Then design an eleventh level, and try to cook it before the tool does. diff --git a/course/13-break-your-engine/lesson.md b/course/13-break-your-engine/lesson.md index f52f428..ac4d8b7 100644 --- a/course/13-break-your-engine/lesson.md +++ b/course/13-break-your-engine/lesson.md @@ -1,6 +1,6 @@ # Module 13: Break your own engine -The course is finished at Module 10. This module is about what to do with an engine once its tests pass, which is where most of an engine's life is spent. +The course's engines are finished at Module 12. This module is about what to do with an engine once its tests pass, which is where most of an engine's life is spent. Every module so far gave you tests and asked you to make them green. That teaches design, and it hides a question: how do you know the *tests* are any good? This course is a useful case study, because it got the answer wrong. Its first complete version had reference engines in two languages, every test green, an automated check that both languages ran the same tests, and three bugs in the engines that no test could see. All three were found by the methods in this module, and none by reading the code more carefully. @@ -24,7 +24,7 @@ Notice what the three bugs have in common. None is in a hard algorithm. Each is Write your answers down before you open the discussion. -1. Your engine passes about seventy tests. Pick any `if` in it and flip the condition. What do you think the chance is that a test fails? +1. Your engine passes about a hundred and ten tests. Pick any `if` in it and flip the condition. What do you think the chance is that a test fails? 2. You want to test the parser on a board nobody designed, with random words on random tiles. You can't know what the right rules are. What can you check? 3. Level 3 of SpellGrid has a scripted solution in the tests. What does that test prove about the level, and what doesn't it prove? 4. Can your engine loop forever? Can a level? @@ -42,24 +42,33 @@ Write your answers down before you open the discussion. Change the program slightly, on purpose, and run the tests. If they still pass, they don't check the thing you changed. The changed program is a *mutant*; a mutant that passes is said to survive. Mutation testing "involves making small changes to the program being tested", and a test "detects, and therefore rejects, a mutant upon test failure" ([Mutation testing, Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)). -General-purpose tools generate thousands of mutants by flipping operators. You don't need one to start, because this course's engines have their most important decisions written down as *lists*, and a list has obvious mutants: swap two neighbours, drop an entry. These were run against the TypeScript reference with the tests the course first shipped: +General-purpose tools generate thousands of mutants by flipping operators. You don't need one to start, because this course's engines have their most important decisions written down as *lists*, and a list has obvious mutants: swap two neighbours, drop an entry. -| Mutant | Tests that failed | +Eight such changes were first made by hand to the TypeScript reference, with the tests the course first shipped. Five of them left every test green: no parse between conversion and hazards, hazards before conversion, no last parse, the hazards in reverse order, and auras read live from the board instead of from a snapshot (no test in Modules 1–9 failed). The engines were right in each case; the tests would not have noticed if they hadn't been. Each survivor then got a test with its name on it (`test_09_conversion_reparses_before_hazards`, `test_09_conversion_before_hazards`, `test_09_last_parse_matches_board`, the two `test_09_hazard_order_…` tests, `test_05_own_aura_misses_own_battlecry`). + +The list is now data, and a tool runs it. `tools/mutants.json` holds one entry per mutant: a file of the reference, a line to find, what to replace it with, and the tests that must fail. `node tools/mutation-check.mjs` (`npm run mutation:check`) stages the TypeScript reference of Module 12, the highest module with a reference, with the tests of every module up to it, and checks that the unmutated reference passes. Then, for each mutant, it makes the replacement, runs the tests and puts the file back. It fails if any test the mutant names still passes, if a test file didn't run, or if the line to find is no longer in the reference exactly once, which is how a list that was rewritten makes its mutant stale instead of silently harmless. It takes about ten seconds, and it is a CI job. Its output today: + +| Mutant | Tests that fail (Module 12 reference, tests of Modules 1–12) | |---|---| -| End of Phase: Aura Update before the Death Creation Step | `test_04_simultaneous_death_not_doubled`, `test_04_aura_timing_order`, `test_10_level_3_solution` | -| Turn: no parse between movement and conversion | `test_09_mid_turn_conversion` | -| Turn: win check before hazards | `test_09_defeat_beats_win` | -| **Turn: no parse between conversion and hazards** | **none** | -| **Turn: hazards before conversion** | **none** | -| **Turn: no last parse** | **none** | -| **Hazards in reverse order** | **none** | -| **Auras read live from the board, no snapshot** | **none in Modules 1–9** | +| End of Phase: Aura Update before the Death Creation Step | `test_04_aura_timing_order`, `test_04_simultaneous_death_not_doubled`, `test_10_level_3_naive_would_differ`, `test_10_level_3_solution` | +| Turn: no parse between movement and conversion | `test_09_mid_turn_conversion`, `test_10_level_5_solution` | +| Turn: win check before hazards | `test_09_defeat_beats_win`, `test_10_level_2_solution`, and the reference's own command-line test `cli_wins_level_2_with_choices_and_play` | +| Turn: no parse between conversion and hazards | `test_09_conversion_reparses_before_hazards` | +| Turn: hazards before conversion | `test_09_conversion_before_hazards`, `test_09_conversion_reparses_before_hazards`, `test_09_undo_restores_everything` | +| Turn: no last parse | `test_09_last_parse_matches_board` | +| Hazards in reverse order | `test_09_hazard_order_melt_before_defeat`, `test_09_hazard_order_sink_before_melt`, `test_10_level_6_solution`, `test_11_condition_rechecked_between_hazards` | +| Hazards: DEFEAT before HOT/MELT | `test_09_hazard_order_melt_before_defeat`, `test_10_level_6_solution` | +| Auras read live from the board, no snapshot | `test_05_own_aura_misses_own_battlecry`, `test_10_level_3_naive_would_differ` | +| Damage: Armor before redirect | `test_04_redirect_before_armor` | +| Conditions decided at parse time | `test_11_condition_rechecked_between_hazards` | +| Combat: no defender priority | `test_12_defender_triggers_first` | +| SpellGrid: a summoned minion without a noun enters the grid | `test_10_summon_without_noun_stays_off_grid` | -Five survivors. The engines were right in each case; the tests would not have noticed if they hadn't been. Each survivor now has a test with its name on it (`test_09_conversion_reparses_before_hazards`, `test_09_conversion_before_hazards`, `test_09_last_parse_matches_board`, the two `test_09_hazard_order_…` tests, `test_05_own_aura_misses_own_battlecry`), and the whole table has no empty row. +No row is empty. The first eight are the hand-made table. The last five are the decisions added since: the order of the damage steps ([Module 4](../04-interceptors/lesson.md)), conditions checked when a rule is used and not when it is parsed ([Module 11](../11-conditions/lesson.md)), the defender's triggers first ([Module 12](../12-combat/lesson.md)), and the noun check in SpellGrid's fourth seam ([Module 10](../10-spellgrid/lesson.md)). Several rows have one killer. Delete that test and the decision is unpinned again; the tool would say so. -Killing a mutant is a design exercise. You have to find the smallest situation in which the two orders give different results, and sometimes you learn that there isn't one. With one property per object, the order of SINK and HOT/MELT can't be observed at all ([Module 9](../09-turn-pipeline/lesson.md), Stress it question 2). The live-aura mutant survived because three separate mechanisms guard the same behaviour ([Module 4](../04-interceptors/lesson.md), Stress it questions 2 and 3). A surviving mutant means one of three things: a missing test, a decision that isn't observable yet, or redundant machinery. All three are worth knowing. +Killing a mutant is a design exercise. You have to find the smallest situation in which the two orders give different results, and sometimes you learn that there isn't one. With one property per object, the order of SINK and HOT/MELT can't be observed at all ([Module 9](../09-turn-pipeline/lesson.md), Stress it question 2), until a condition lets one hazard change what another hazard's victim is ([Module 11](../11-conditions/lesson.md), Stress it question 1). The live-aura mutant survived because three separate mechanisms guard the same behaviour ([Module 4](../04-interceptors/lesson.md), Stress it questions 2 and 3). A surviving mutant means one of three things: a missing test, a decision that isn't observable yet, or redundant machinery. All three are worth knowing. -**Try it.** List every ordered list in your engine: the turn, the hazards, the movement responses, the end-of-Phase steps, the order in which `react` walks events and listeners, the order `YOU` objects move. For each, swap two neighbours and run everything. Keep a table like the one above. For each survivor, write the test or write down why none can exist. +**Try it.** List every ordered list in your engine: the turn, the hazards, the movement responses, the end-of-Phase steps, the order in which `react` walks events and listeners, the order `YOU` objects move. For each, swap two neighbours and run everything. Keep a table like the one above. For each survivor, write the test or write down why none can exist. `tools/mutation-check.mjs` runs only on the course's reference, but the idea is a short script for any engine: a list of find-and-replace pairs, and a loop that applies one, runs the tests and puts the file back. ## Invariants and random inputs @@ -93,21 +102,23 @@ Some relations you might expect are false, and working out why is as useful as a ## Two engines, one specification -This repository has a Rust reference and a TypeScript reference for every module. They pass identical tests. That says they agree on the seventy-odd situations somebody wrote down. It says nothing about any other situation, and two engines written from one specification are the best oracle there is for the rest: generate a random level and a random input sequence, run both, compare what they print. Any difference is a bug in at least one of them or a hole in the specification, and you didn't need to know the right answer to find it. +This repository has a Rust reference and a TypeScript reference for every module. They pass identical tests. That says they agree on the hundred-odd situations somebody wrote down. It says nothing about any other situation, and two engines written from one specification are the best oracle there is for the rest: generate a random level and a random input sequence, run both, compare what they print. Any difference is a bug in at least one of them or a hole in the specification, and you didn't need to know the right answer to find it. This is **differential testing**. It is also how [Module 6](../06-parser/lesson.md) suggests you protect an optimisation: keep the slow version and compare. If you did both tracks of the course, you own two engines; if you did one, the reference in your language is the second. The command-line front ends from Module 10 (`tools/play-spellgrid.mjs`) read inputs from standard input and print the board, the rules, the minions and the hand after each one, so two engines can be compared as text. -`node tools/diff-tracks.mjs` does this for the two references. For each seed it makes a random SpellGrid level and a random script, runs both command lines, and prints a diff for the first seed on which they disagree. Its first run of 500 seeds found nothing. The references did differ, though: in how each decides that an Assembled card already exists, and in how each command line reads a signed number or quotes a card name. The fuzzer could reach none of these. The first needs a card registered again in the middle of a game, and the others need input outside the command grammar it generates. A differential tester finds only what its generator can reach. All three are now fixed, and `test_05_assemble_never_rewrites_an_earlier_card` and the reference command-line tests pin them. Widening the generator is an open exercise, and a good one. +`node tools/diff-tracks.mjs` does this for the two references (`npm run diff:tracks`). It takes the highest module whose references have the SpellGrid command line, Module 12 today. For each seed it makes a random SpellGrid level from that module's words and cards, and a random script of 5 to 25 commands. It pipes the script to both command lines and, for the first seed on which they print different text, shows the seed, the level, the script and a diff. A crash on one side counts as a difference. `--seeds N` sets how many seeds (100 by default), `--seed S` replays one case, and a CI job runs 300. + +When it was written, on Module 10, runs of 500, 1,000 and 5,000 seeds found nothing. The references did differ, though, in three places, all found by reading the two tracks side by side. The TypeScript reference decided that an Assembled card already existed by comparing only its text, so assembling again after a card's cost changed rewrote the earlier card. The Rust command line accepted a signed number such as `-1` as a coordinate, where the TypeScript one read it as part of the card name. And the two quoted a card name differently in an error message. The fuzzer could reach none of these. The first needs a card registered again in the middle of a game, which the command line can't do, and the others need input outside the command grammar the generator uses. A differential tester finds only what its generator can reach. All three are fixed, and `test_05_assemble_never_rewrites_an_earlier_card` and the reference command-line tests `cli_numbers_are_plain_digits` and `cli_prints_card_names_as_typed` pin them. The generator is where to improve it: it seldom puts an Assembled minion on the grid, and it never steps outside the grammar. Widening it is a good exercise. ## State, undo and replay -[Module 9](../09-turn-pipeline/lesson.md)'s Stress it asks you to build Undo and find out what a snapshot has to contain. The general lesson is in two parts. +[Module 9](../09-turn-pipeline/lesson.md)'s Stress it asks you to build Undo and find out what a snapshot has to contain. The reference has one for Part II: `undo()` on a puzzle level puts back every object's id, kind and position in age order, the id counter and `won`, and recompiles the rules instead of saving them. `test_09_undo_restores_everything` is a round trip: a turn that moves, converts and destroys, then undo, then the same turn again, with every id the same as the first time. SpellGrid has no undo. Its harness function fails with "undo is not supported in SpellGrid", because Part I has state the board snapshot doesn't cover: Health, hands and decks, the aura snapshot, the order-of-play counter, the Assembled cards. The general lesson is in two parts. **Derived data is not state.** The rule table, the aura snapshot at the start of a Sequence, a minion's position on the grid as seen by the card engine: all of these are computed from something else. If you can recompute it, don't save it, don't restore it, and above all don't let anything edit it in place. Both parts of the course were built around this: "The previous rules are removed entirely" (Module 6); auras are re-read at an Aura Update (Module 4). Martin Fowler's description of state held as a log of events makes the same cut from the other side: "Event Sourcing ensures that all changes to application state are stored as a sequence of events", and from them "we can also use the event log to reconstruct past states" ([*Event Sourcing*](https://martinfowler.com/eaaDev/EventSourcing.html)). **The real state is larger than it looks.** Ids, the id counter, the order-of-play counter, the age order of objects on a tile, `won`, the contents of each deck in order, the chooser's future answers. None of these appear when you draw the board, and every one of them can change what happens next. The honest way to find the list is the round-trip test: snapshot, restore into a fresh engine, play both forward with the same random inputs, and compare. -An engine that is deterministic gets a second kind of undo for nothing: record the inputs, replay all but the last. It needs no snapshot and can't forget a field. The solver in the next section is built entirely on it. Robert Nystrom's [*Command*](https://gameprogrammingpatterns.com/command.html) chapter is the short introduction to both. +An engine that is deterministic gets a second kind of undo for nothing: record the inputs, replay all but the last. It needs no snapshot and can't forget a field, which is why it works for SpellGrid, where the snapshot doesn't. The solver in the next section is built entirely on it. Robert Nystrom's [*Command*](https://gameprogrammingpatterns.com/command.html) chapter is the short introduction to both. ## Solving your own levels @@ -127,7 +138,7 @@ How it works: - **A key for "the same position".** Two input sequences that lead to the same tiles, minions, Health, hand and hero Health are one position, and only the first is explored. This is the delicate part, for the reason the last section gave: the key has to include everything that can change the future. The solver adds the Assemble choices made so far, because two Assembled cards can share a name and differ in text. Each tile's objects go into the key oldest first (`stackAt`, from Module 8), because age breaks ties: which `YOU` object moves first, which `SHUT` partner an `OPEN` key takes. The key still doesn't record age between objects on different tiles, or the order in which minions were played, so in a level where one of those breaks a tie it could merge two positions that differ. Keep that in mind when your level has two `YOU` objects, or two minions that die together. - **A filter.** `--never-rule` discards positions in which a given rule is in force. That turns "is the level solvable?" into "is it solvable *without the idea*?", which is the question a designer cares about. -What it found for the shipped levels is in [Module 10](../10-spellgrid/lesson.md)'s Stress it: each is solvable in the scripted number of inputs, Level 1 has no solution that avoids `DEATHRATTLE IS DOUBLE`, and Level 3 has no solution at all on the naive engine. Those are stronger statements than any test in Module 10 makes, and they hold for every line of play, not for one script. Levels 4 to 10 were designed with the solver as the adversary, each around one idea that it proves the level can't do without. `course/10-spellgrid/levels/proofs.json` writes down all ten claims, and `node tools/verify-levels.mjs` checks them again: run it after any change to the engine or the levels. +What it found for the shipped levels is in [Module 10](../10-spellgrid/lesson.md)'s Stress it: each is solvable in the scripted number of inputs, Level 1 has no solution that avoids `DEATHRATTLE IS DOUBLE`, and Level 3 has no solution at all on the naive engine. Those are stronger statements than any test in Module 10 makes, and they hold for every line of play, not for one script. Levels 4 to 10 were designed with the solver as the adversary, each around one idea, and for each the solver proves the level can't be solved without one rule the player has to form. For levels 4, 5, 7 and 10 that rule is the idea; for levels 6, 8 and 9 it is one ingredient of it, and Module 10 says what the rest rests on. `course/10-spellgrid/levels/proofs.json` writes down all ten claims, and `node tools/verify-levels.mjs` checks them again, in about two and a half minutes: run it after any change to the engine or the levels. CI runs it too. An unintended solution is called a *cook*. Puzzle designers have always hunted cooks by hand; a solver hunts them while you sleep. The method generalises: state what the level is *for* as a predicate on positions, forbid it, and search. @@ -155,7 +166,7 @@ The pattern is the **oracle problem**, and the techniques in this module are way | Oracle | Knows | Module 13 section | |---|---|---| -| An expected value written by a person | the exact answer, for one input | (Modules 1–10) | +| An expected value written by a person | the exact answer, for one input | (Modules 1–12) | | An invariant | a fact about every answer | Invariants | | A metamorphic relation | how two answers relate | Metamorphic relations | | A second implementation | the same answer, by other means | Two engines | diff --git a/mentor/SKILL.md b/mentor/SKILL.md index 2aa592c..ab2fc91 100644 --- a/mentor/SKILL.md +++ b/mentor/SKILL.md @@ -26,6 +26,17 @@ You teach; the learner builds. The curriculum lives in the course repository (`< - `node /tools/coupling-audit.mjs ` - `node /tools/expansion-check.mjs ` +## The modules + +`/course/` holds them in order; `start-module` knows 01 to 12. + +- 00 The Rewrite Trap (the naive engine; see Module 0 below). +- Part I, the card game: 01 Handles and order of play, 02 The action queue, depth-first, 03 Deferred death, 04 Interceptors, 05 Behavior as data and Assemble. +- Part II, the rules puzzle: 06 Grid, lexer and parser, 07 Resolving rules, 08 Push chains, 09 The turn pipeline. +- Part III: 10 SpellGrid, which joins the two engines. Its folder also holds the designer capstone (`designer-capstone.md`; see Tone). +- Extensions, taken after 10: 11 Conditions (Part II), 12 Combat (Part I). Each has tests, a reference and a drop like the modules before it. +- 13 Break your own engine: optional, no tests, no drop (see Stress it). + ## The beats Read the module's `lesson.md` and `scenarios.md` before starting it. Teach the lesson's H2 beats in order: Observe, Predict, Spec, Build, Name it, Expansion Drop, Stress it, Further reading. Track the current one in `beat`. @@ -73,4 +84,4 @@ Branch for no `.course.json`: ask for `courseRoot` (verify `course/catalog.json` ## Tone -Short turns, one question at a time. Praise specific reasoning, not effort. If the learner wants the Designer path only (no building), skip Build and the audits, keep Predict, Spec, Name it, and discuss the Expansion Drop on paper. +Short turns, one question at a time. Praise specific reasoning, not effort. If the learner wants the Designer path only (no building), skip Build and the audits, keep Predict, Spec, Name it, and discuss the Expansion Drop on paper. After Module 10, offer the designer capstone (`/course/10-spellgrid/designer-capstone.md`): the learner designs cards and a word as data, writes predictions, and runs them with `node /tools/run-scenario.mjs `. Rule 1 holds here too: the scenario file is theirs to write. Ask for each prediction before they run it, and for each surprise ask which named pattern explains it before you offer one. Do not show its worked example before they have written their own predictions. diff --git a/package.json b/package.json index 350e7dd..2f20a47 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,8 @@ "play:spellgrid": "node tools/play-spellgrid.mjs", "solve:spellgrid": "node tools/solve-spellgrid.mjs", "diff:tracks": "node tools/diff-tracks.mjs", - "verify:levels": "node tools/verify-levels.mjs" + "verify:levels": "node tools/verify-levels.mjs", + "mutation:check": "node tools/mutation-check.mjs", + "run:scenario": "node tools/run-scenario.mjs" } } diff --git a/tools/mutants.json b/tools/mutants.json new file mode 100644 index 0000000..e9f490d --- /dev/null +++ b/tools/mutants.json @@ -0,0 +1,131 @@ +[ + { + "name": "End of Phase: Aura Update before the Death Creation Step", + "file": "src/engine/queue.ts", + "find": "export const END_OF_PHASE: readonly EndOfPhaseStep[] = [\"deathCreation\", \"auraUpdate\"];", + "replace": "export const END_OF_PHASE: readonly EndOfPhaseStep[] = [\"auraUpdate\", \"deathCreation\"];", + "expect": [ + "test_04_simultaneous_death_not_doubled", + "test_04_aura_timing_order", + "test_10_level_3_solution", + "test_10_level_3_naive_would_differ" + ] + }, + { + "name": "Turn: no parse between movement and conversion", + "file": "src/engine/puzzle/turn.ts", + "find": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "replace": "export const TURN: readonly Phase[] = [\"movement\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "expect": [ + "test_09_mid_turn_conversion", + "test_10_level_5_solution" + ] + }, + { + "name": "Turn: win check before hazards", + "file": "src/engine/puzzle/turn.ts", + "find": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "replace": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"win\", \"hazards\", \"parse\"];", + "expect": [ + "test_09_defeat_beats_win", + "test_10_level_2_solution" + ] + }, + { + "name": "Turn: no parse between conversion and hazards", + "file": "src/engine/puzzle/turn.ts", + "find": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "replace": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"hazards\", \"win\", \"parse\"];", + "expect": [ + "test_09_conversion_reparses_before_hazards" + ] + }, + { + "name": "Turn: hazards before conversion", + "file": "src/engine/puzzle/turn.ts", + "find": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "replace": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"hazards\", \"parse\", \"conversion\", \"win\", \"parse\"];", + "expect": [ + "test_09_conversion_before_hazards", + "test_09_conversion_reparses_before_hazards", + "test_09_undo_restores_everything" + ] + }, + { + "name": "Turn: no last parse", + "file": "src/engine/puzzle/turn.ts", + "find": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\", \"parse\"];", + "replace": "export const TURN: readonly Phase[] = [\"movement\", \"parse\", \"conversion\", \"parse\", \"hazards\", \"win\"];", + "expect": [ + "test_09_last_parse_matches_board" + ] + }, + { + "name": "Hazards in reverse order", + "file": "src/content/hazards.ts", + "find": "return [sink, hotMelt, defeat];", + "replace": "return [defeat, hotMelt, sink];", + "expect": [ + "test_09_hazard_order_melt_before_defeat", + "test_09_hazard_order_sink_before_melt", + "test_10_level_6_solution", + "test_11_condition_rechecked_between_hazards" + ] + }, + { + "name": "Hazards: DEFEAT before HOT/MELT", + "file": "src/content/hazards.ts", + "find": "return [sink, hotMelt, defeat];", + "replace": "return [sink, defeat, hotMelt];", + "expect": [ + "test_09_hazard_order_melt_before_defeat", + "test_10_level_6_solution" + ] + }, + { + "name": "Auras read live from the board, no snapshot", + "file": "src/engine/world.ts", + "find": " activeAuras(): readonly ActiveAura[] {\n return this.auras;\n }", + "replace": " activeAuras(): readonly ActiveAura[] {\n // MUTANT: every minion in play now, and the rule auras, read at each use.\n return [\n ...this.minionsInPlay().flatMap((id) => this.card(this.cardName(id)).auras.map((aura) => ({ owner: this.ownerOf(id), source: id, aura }))),\n ...this.rules.flatMap((aura) => ([0, 1] as const).map((owner) => ({ owner, aura }))),\n ];\n }", + "expect": [ + "test_05_own_aura_misses_own_battlecry", + "test_10_level_3_naive_would_differ" + ] + }, + { + "name": "Damage: Armor before redirect", + "file": "src/engine/queue.ts", + "find": "export const DAMAGE_STEPS: readonly DamageStep[] = [\"redirect\", \"armor\"];", + "replace": "export const DAMAGE_STEPS: readonly DamageStep[] = [\"armor\", \"redirect\"];", + "expect": [ + "test_04_redirect_before_armor" + ] + }, + { + "name": "Conditions decided at parse time", + "file": "src/engine/puzzle/level.ts", + "find": " this.compiled = parse(this.board, this.lexicon);\n this.active = resolve(this.compiled);\n }", + "replace": " this.compiled = parse(this.board, this.lexicon);\n this.active = resolve(this.compiled);\n // MUTANT: decide every condition now, for every object, and answer from that table until the next parse.\n const self = this as unknown as { live?: ConditionChecks; conditions: ConditionChecks };\n const live = (self.live ??= self.conditions);\n const decided = new Map();\n const key = (word: string, id: ObjectId, nouns: readonly string[]) => `${word} ${nouns.join(\" \")} ${id}`;\n for (const o of this.board.objects()) {\n for (const r of this.active) for (const c of r.conditions) decided.set(key(c.word, o.id, c.nouns), live[c.word](this, o.id, c.nouns));\n }\n self.conditions = Object.fromEntries(\n Object.keys(live).map((word) => [word, (_: Level, id: ObjectId, nouns: readonly string[]) => decided.get(key(word, id, nouns)) ?? false]),\n );\n }", + "expect": [ + "test_11_condition_rechecked_between_hazards" + ] + }, + { + "name": "Combat: no defender priority", + "file": "src/engine/queue.ts", + "find": " queue.sort((a, b) => b[4] - a[4]);\n", + "replace": " // MUTANT: no sort; the Queue keeps the order it was filled in.\n", + "expect": [ + "test_12_defender_triggers_first" + ] + }, + { + "name": "SpellGrid: a summoned minion without a noun enters the grid", + "file": "src/engine/spellgrid.ts", + "find": " if (this.ids.has(event.id) || this.world.card(card).def.noun === undefined) continue;\n", + "replace": " if (this.ids.has(event.id)) continue;\n", + "expect": [ + "test_10_summon_without_noun_stays_off_grid" + ] + } +] diff --git a/tools/mutation-check.mjs b/tools/mutation-check.mjs new file mode 100644 index 0000000..57df944 --- /dev/null +++ b/tools/mutation-check.mjs @@ -0,0 +1,194 @@ +#!/usr/bin/env node +// Mutation check: are the course's ordering decisions pinned by tests? +// +// node tools/mutation-check.mjs [name ...] [--mutants ] [--module NN] +// npm run mutation:check +// +// Reads tools/mutants.json (or --mutants), a list of mutants +// { "name": "Hazards in reverse order", "file": "src/content/hazards.ts", +// "find": "return [sink, hotMelt, defeat];", "replace": "return [defeat, hotMelt, sink];", +// "expect": ["test_09_hazard_order_melt_before_defeat"] } +// and checks each one against the TypeScript reference of the highest module that has one +// (or --module NN). The reference is staged once with the tests of every module up to it, as +// verify-references does, and its dependencies are installed. The unmutated reference must pass. +// Then, for each mutant (or only the ones named), the tool replaces `find` in `file` with +// `replace` (`find` must occur exactly once), runs the tests, and restores the file. A mutant +// is killed when every test in `expect` fails and the run is otherwise sane: every test of the +// unmutated run ran again (no test file failed to load). Prints one line per mutant with the +// tests that failed; exits 1 if any mutant survives, any `find` is stale, or any run is not sane. +// +// MUTATION_CMD replaces the test command, and then nothing is installed (the tool's tests use it). +// Whatever runs must write a vitest JSON report to `.mutation-report.json` in the staged directory. +import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { fail, isMain, moduleDirs } from "./lib.mjs"; +import { planVerification, stage } from "./verify-references.mjs"; + +const REPORT = ".mutation-report.json"; +const DEFAULT_CMD = `npx vitest run --reporter=json --outputFile=${REPORT}`; +const KEYS = ["name", "file", "find", "replace", "expect"]; + +export function parseArgs(argv) { + const opts = { names: [], mutants: null, module: null }; + for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--mutants") opts.mutants = argv[++i]; + else if (argv[i] === "--module") opts.module = String(argv[++i]).padStart(2, "0"); + else if (argv[i].startsWith("--")) fail(`mutation-check: unknown option ${argv[i]}\nusage: mutation-check.mjs [name ...] [--mutants ] [--module NN]`); + else opts.names.push(argv[i]); + } + return opts; +} + +/** What is wrong with a mutant entry's shape, or [] if nothing. */ +export function mutantProblems(m) { + if (typeof m !== "object" || m === null || Array.isArray(m)) return ["a mutant must be an object"]; + const problems = []; + for (const key of Object.keys(m)) if (!KEYS.includes(key)) problems.push(`unknown key "${key}"`); + for (const key of ["name", "file", "find"]) { + if (typeof m[key] !== "string" || m[key] === "") problems.push(`${key} must be a non-empty string`); + } + if (typeof m.replace !== "string") problems.push("replace must be a string"); + else if (m.replace === m.find) problems.push("replace is the same as find: nothing would change"); + if (!Array.isArray(m.expect) || m.expect.length === 0 || !m.expect.every((t) => typeof t === "string" && /^test_\w+$/.test(t))) { + problems.push("expect must list at least one test name (test_...)"); + } + return problems; +} + +/** `text` with the mutant applied. Throws if `find` does not occur exactly once (a stale mutant). */ +export function applyMutant(text, mutant) { + const count = text.split(mutant.find).length - 1; + if (count !== 1) { + throw new Error(`stale: \`find\` occurs ${count} times in ${mutant.file}, not once`); + } + return text.replace(mutant.find, () => mutant.replace); +} + +/** + * Read a vitest JSON report: the names of the tests that passed and failed, and the test files + * that failed without a failing test (a file that did not load, or an error outside a test). + */ +export function readReport(json) { + const passed = new Set(); + const failed = new Set(); + const broken = []; + for (const file of json.testResults ?? []) { + const results = file.assertionResults ?? []; + for (const t of results) { + if (t.status === "passed") passed.add(t.title); + else if (t.status === "failed") failed.add(t.title); + } + if (file.status === "failed" && !results.some((t) => t.status === "failed")) { + broken.push(`${file.name}: ${String(file.message ?? "").split("\n")[0]}`); + } + } + return { passed, failed, broken }; +} + +/** + * The verdict on one mutant run, given the unmutated run (`baseline`) and this one (`run`), both + * from readReport: { killed, sane, failed: [names], problems: [text] }. + */ +export function judge(mutant, baseline, run) { + const problems = []; + const unknown = mutant.expect.filter((t) => !baseline.passed.has(t)); + if (unknown.length) problems.push(`expects tests the reference doesn't have: ${unknown.join(", ")}`); + for (const b of run.broken) problems.push(`a test file did not run: ${b}`); + const missing = [...baseline.passed].filter((t) => !run.passed.has(t) && !run.failed.has(t)); + if (missing.length && run.broken.length === 0) problems.push(`${missing.length} tests did not run, e.g. ${missing.slice(0, 3).join(", ")}`); + const sane = problems.length === 0; + const survivors = mutant.expect.filter((t) => !run.failed.has(t)); + return { killed: sane && survivors.length === 0, sane, survivors, failed: [...run.failed].sort(), problems }; +} + +/** Run the tests in `dir`; returns readReport of what they wrote, or throws. */ +function runTests(dir, cmd) { + rmSync(join(dir, REPORT), { force: true }); + const r = spawnSync(cmd, { cwd: dir, shell: true, encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }); + const report = join(dir, REPORT); + if (!existsSync(report)) throw new Error(`the tests wrote no report (exit ${r.status}):\n${(r.stdout || "") + (r.stderr || "")}`); + return readReport(JSON.parse(readFileSync(report, "utf8"))); +} + +/** The plan entry (as verify-references makes it) for module `num`, or the highest with a TS reference. */ +export function targetModule(num) { + const plans = planVerification(moduleDirs(), 99).filter((p) => p.langs.includes("ts")); + const plan = num === null ? plans.at(-1) : plans.find((p) => p.num === num); + if (!plan) fail(num === null ? "mutation-check: no module has a TypeScript reference" : `mutation-check: module ${num} has no TypeScript reference`); + return plan; +} + +function main() { + const opts = parseArgs(process.argv.slice(2)); + const file = opts.mutants ?? fileURLToPath(new URL("./mutants.json", import.meta.url)); + let mutants = JSON.parse(readFileSync(file, "utf8")); + if (!Array.isArray(mutants)) fail(`${file}: expected a list of mutants`); + const shapes = mutants.flatMap((m, i) => mutantProblems(m).map((p) => `mutant ${i + 1} (${m?.name ?? "no name"}): ${p}`)); + if (shapes.length) fail(shapes.join("\n")); + if (opts.names.length) { + mutants = mutants.filter((m) => opts.names.includes(m.name)); + if (mutants.length === 0) fail(`no mutant named ${opts.names.join(", ")} in ${file}`); + } + + const plan = targetModule(opts.module); + const dir = stage(plan, "ts"); + const cmd = process.env.MUTATION_CMD || DEFAULT_CMD; + let bad = 0; + try { + if (existsSync(join(dir, "package.json")) && !process.env.MUTATION_CMD) { + const r = spawnSync("npm", ["install", "--no-audit", "--no-fund", "--loglevel=error"], { cwd: dir, encoding: "utf8" }); + if (r.status !== 0) throw new Error(`npm install failed:\n${r.stdout}${r.stderr}`); + } + console.log(`Module ${plan.num} TypeScript reference, tests of Modules ${plan.testModules[0].num}-${plan.num}`); + const baseline = runTests(dir, cmd); + if (baseline.failed.size || baseline.broken.length || baseline.passed.size === 0) { + throw new Error(`the unmutated reference does not pass: ${[...baseline.failed, ...baseline.broken].join(", ") || "no tests ran"}`); + } + console.log(`unmutated: ${baseline.passed.size} tests pass`); + for (const mutant of mutants) { + const path = join(dir, mutant.file); + if (!existsSync(path)) { + bad++; + console.log(`STALE ${mutant.name}: no file ${mutant.file}`); + continue; + } + const original = readFileSync(path, "utf8"); + let mutated; + try { + mutated = applyMutant(original, mutant); + } catch (e) { + bad++; + console.log(`STALE ${mutant.name}: ${e.message}`); + continue; + } + writeFileSync(path, mutated); + let verdict; + try { + verdict = judge(mutant, baseline, runTests(dir, cmd)); + } catch (e) { + verdict = { killed: false, sane: false, survivors: [], failed: [], problems: [e.message] }; + } finally { + writeFileSync(path, original); + } + const failed = verdict.failed.join(", ") || "none"; + if (verdict.killed) console.log(`killed ${mutant.name}: ${failed}`); + else { + bad++; + if (!verdict.sane) console.log(`INSANE ${mutant.name}: ${verdict.problems.join("; ")}`); + else console.log(`SURVIVED ${mutant.name}: ${verdict.survivors.join(", ")} passed; failed: ${failed}`); + } + } + } catch (e) { + console.error(`mutation-check: ${e.message}`); + process.exitCode = 1; + return; + } finally { + rmSync(dir, { recursive: true, force: true }); + } + console.log(bad === 0 ? `${mutants.length} mutants killed` : `${bad} of ${mutants.length} mutants not killed`); + process.exit(bad === 0 ? 0 : 1); +} + +if (isMain(import.meta.url)) main(); diff --git a/tools/run-scenario.mjs b/tools/run-scenario.mjs new file mode 100644 index 0000000..63bb4a8 --- /dev/null +++ b/tools/run-scenario.mjs @@ -0,0 +1,178 @@ +#!/usr/bin/env node +// Run a designer's scenario on SpellGrid (Module 10): new cards, words and seam words as data, +// a level, and the inputs to play on it. +// +// node tools/run-scenario.mjs +// npm run run:scenario -- course/10-spellgrid/designer-example/scenario.json +// +// A scenario is a JSON object +// { "cards"?: [card, ...], "words"?: [{ "name": "WOUND", "role": "noun" }, ...], +// "seams"?: { "triggerWords"?: { "WOUND": "selfDamaged" }, "multiplierWords"?: { ... } }, +// "level": level, "inputs": ["d", "c Whirlwind", ...] } +// with cards in the course's card format (course/schema/card.schema.json), a level in the +// level format (course/schema/level.schema.json), and inputs as the SpellGrid command line reads +// them (`w a s d`, `c [x y]`, and an answer to each Assemble question). The scenario is +// checked first, and every problem is listed. Then the Module 10 TypeScript reference is staged +// with its cards, as play-spellgrid does, the scenario's cards are added to cards/, its words and +// seam words are merged into src/content/words.json and seams.json, and tools/scenario/run.ts +// plays the inputs there. It prints, at the start and after each input, the Part I log of that +// input as an indented tree, the board, the rules, the minions, the hand and both heroes' Health. +// No engine file is touched: a scenario can say exactly what an Expansion Drop can. +import { copyFileSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { basename, join } from "node:path"; +import { courseRoot, fail, isMain } from "./lib.mjs"; +import { stagePlay } from "./play-spellgrid.mjs"; +import { validate } from "./schema.mjs"; + +const MODULE = "10"; +const KEYS = ["cards", "words", "seams", "level", "inputs"]; +const ROLES = ["noun", "property"]; + +const schema = (name) => JSON.parse(readFileSync(join(courseRoot(), "schema", `${name}.schema.json`), "utf8")); + +/** The trigger kinds a card or a seam word may name, from the card schema. */ +const triggers = (cardSchema) => cardSchema.properties.text.items.oneOf.find((b) => b.properties.on).properties.on.enum; + +/** + * Everything wrong with a scenario, as readable lines (empty if nothing). `ctx` is what the staged + * reference already has: `words` (its words.json) and `cardNames` (the names of its cards). + */ +export function scenarioProblems(s, ctx) { + if (typeof s !== "object" || s === null || Array.isArray(s)) return ["the scenario must be a JSON object"]; + const cardSchema = schema("card"); + const problems = []; + for (const key of Object.keys(s)) if (!KEYS.includes(key)) problems.push(`unknown key "${key}" (a scenario has ${KEYS.join(", ")})`); + if (!("level" in s)) problems.push('missing "level"'); + else problems.push(...validate(schema("level"), s.level, "level")); + if (!Array.isArray(s.inputs) || !s.inputs.every((i) => typeof i === "string")) problems.push("inputs: expected a list of command lines, such as [\"d\", \"c Whirlwind\"]"); + + const cards = s.cards ?? []; + if (!Array.isArray(cards)) problems.push("cards: expected a list of cards"); + else { + cards.forEach((card, i) => { + const at = `cards[${i}]${typeof card?.name === "string" ? ` (${JSON.stringify(card.name)})` : ""}`; + problems.push(...validate(cardSchema, card, "$").map((e) => `${at}${e.slice(1)}`)); + if (ctx.cardNames.includes(card?.name)) problems.push(`${at}: the course already has a card named ${JSON.stringify(card.name)}; give yours another name`); + if (cards.findIndex((c) => c?.name === card?.name) < i) problems.push(`${at}: two scenario cards are named ${JSON.stringify(card.name)}`); + }); + } + + const words = s.words ?? []; + const before = problems.length; + if (!Array.isArray(words)) problems.push("words: expected a list of words"); + else { + words.forEach((w, i) => { + const at = `words[${i}]`; + if (typeof w !== "object" || w === null) return problems.push(`${at}: expected { "name": ..., "role": ... }`); + for (const key of Object.keys(w)) if (!["name", "role", "selects"].includes(key)) problems.push(`${at}: unknown key "${key}"`); + if (typeof w.name !== "string" || !/^[A-Z]+$/.test(w.name)) problems.push(`${at}.name: ${JSON.stringify(w.name)} must be capital letters, such as "WOUND"`); + if (!ROLES.includes(w.role)) problems.push(`${at}.role: must be "noun" or "property" (IS, AND and NOT belong to the engine)`); + if ("selects" in w && w.role !== "noun") problems.push(`${at}.selects: only a noun selects`); + }); + } + let merged = ctx.words; + if (problems.length === before) { + try { + merged = mergeWords(ctx.words, words); + } catch (e) { + problems.push(`words: ${e.message}`); + } + } + const roleOf = (name) => merged.find((w) => w.name === name)?.role; + + const seams = s.seams ?? {}; + if (typeof seams !== "object" || seams === null || Array.isArray(seams)) problems.push("seams: expected an object"); + else { + for (const key of Object.keys(seams)) if (!["triggerWords", "multiplierWords"].includes(key)) problems.push(`seams: unknown key "${key}"`); + const kinds = triggers(cardSchema); + for (const [word, on] of Object.entries(seams.triggerWords ?? {})) { + const at = `seams.triggerWords.${word}`; + if (roleOf(word) !== "noun") problems.push(`${at}: ${word} is not a noun; add it to "words" with role "noun"`); + if (!kinds.includes(on)) problems.push(`${at}: ${JSON.stringify(on)} is not a trigger (one of ${kinds.join(", ")})`); + } + const multiplier = cardSchema.properties.text.items.oneOf.find((b) => b.properties.modifier).properties.modifier.oneOf[0]; + for (const [word, m] of Object.entries(seams.multiplierWords ?? {})) { + const at = `seams.multiplierWords.${word}`; + if (roleOf(word) !== "property") problems.push(`${at}: ${word} is not a property; add it to "words" with role "property"`); + const shape = { type: "object", required: ["times", "stacking"], properties: { times: multiplier.properties.times, stacking: multiplier.properties.stacking } }; + problems.push(...validate(shape, m, at)); + } + } + return problems; +} + +/** `base` words with `added` appended; a word already there with the same role is skipped. Throws on a clash. */ +export function mergeWords(base, added) { + const out = [...base]; + for (const w of added) { + const old = out.find((o) => o.name === w.name); + if (old === undefined) out.push(w); + else if (old.role !== w.role || old.selects !== w.selects) throw new Error(`${w.name} is already a ${old.role}${old.selects ? ` that selects ${old.selects}` : ""}`); + } + return out; +} + +/** `base` seams with `added` merged in; a word that already crosses the seam must keep its meaning. Throws on a clash. */ +export function mergeSeams(base, added) { + const out = { triggerWords: { ...base.triggerWords }, multiplierWords: { ...base.multiplierWords } }; + for (const part of ["triggerWords", "multiplierWords"]) { + for (const [word, meaning] of Object.entries(added[part] ?? {})) { + const old = out[part][word]; + if (old !== undefined && JSON.stringify(old) !== JSON.stringify(meaning)) throw new Error(`${word} already names ${JSON.stringify(old)}`); + out[part][word] = meaning; + } + } + return out; +} + +/** A file name for a card: its name in lower case, words joined by "-". */ +const fileName = (name) => `${name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}.json`; + +/** The names of the cards in a staged reference's cards/. */ +const cardNames = (dir) => + readdirSync(join(dir, "cards")).filter((f) => f.endsWith(".json")).map((f) => JSON.parse(readFileSync(join(dir, "cards", f), "utf8")).name); + +function main() { + const [file, ...rest] = process.argv.slice(2); + if (file === undefined || rest.length > 0) fail("usage: run-scenario.mjs "); + let s; + try { + s = JSON.parse(readFileSync(file, "utf8")); + } catch (e) { + fail(existsSync(file) ? `${file} is not JSON: ${e.message}` : `no file ${file}`); + } + const dir = stagePlay("ts", null, MODULE); + const content = join(dir, "src", "content"); + let code = 1; + try { + const words = JSON.parse(readFileSync(join(content, "words.json"), "utf8")); + const problems = scenarioProblems(s, { words, cardNames: cardNames(dir) }); + if (problems.length > 0) { + console.error(`${basename(file)} is not a valid scenario:\n${problems.map((p) => ` - ${p}`).join("\n")}`); + return; + } + for (const card of s.cards ?? []) writeFileSync(join(dir, "cards", fileName(card.name)), `${JSON.stringify(card, null, 2)}\n`); + writeFileSync(join(content, "words.json"), `${JSON.stringify(mergeWords(words, s.words ?? []), null, 2)}\n`); + const seams = JSON.parse(readFileSync(join(content, "seams.json"), "utf8")); + writeFileSync(join(content, "seams.json"), `${JSON.stringify(mergeSeams(seams, s.seams ?? {}), null, 2)}\n`); + mkdirSync(join(dir, "scenario")); + writeFileSync(join(dir, "scenario", "level.json"), JSON.stringify(s.level)); + writeFileSync(join(dir, "scenario", "inputs.json"), JSON.stringify(s.inputs)); + copyFileSync(new URL("./scenario/run.ts", import.meta.url), join(dir, "run.ts")); + if (!existsSync(join(dir, "node_modules"))) { + const r = spawnSync("npm", ["install", "--no-audit", "--no-fund", "--loglevel=error"], { cwd: dir, encoding: "utf8" }); + if (r.status !== 0) { + console.error(`npm install failed:\n${r.stdout}${r.stderr}`); + return; + } + } + const r = spawnSync(join(dir, "node_modules", ".bin", "tsx"), ["run.ts", "scenario/level.json", "scenario/inputs.json"], { cwd: dir, stdio: "inherit" }); + code = r.status ?? 1; + } finally { + rmSync(dir, { recursive: true, force: true }); + process.exitCode = code; + } +} + +if (isMain(import.meta.url)) main(); diff --git a/tools/scenario/run.ts b/tools/scenario/run.ts new file mode 100644 index 0000000..cb6654f --- /dev/null +++ b/tools/scenario/run.ts @@ -0,0 +1,88 @@ +// Play a designer's scenario on a staged SpellGrid reference (tools/run-scenario.mjs copies +// this file into the staged directory and runs it there with tsx). +// +// npx tsx run.ts +// +// The inputs are command-line commands, as `src/cli.ts` reads them: `w a s d`, `c [x y]`, +// and the answers to Assemble's questions, one input each. After the start and after each +// input it prints the input, the Part I log of that input as an indented tree, the board, the +// rules, the minions, the hand and both heroes' Health. No rules live here: it calls the +// harness, as the command line does, and draws with the command line's `render`. + +import { readFileSync } from "node:fs"; +import { render } from "./src/cli"; +import { newSpellGrid, type LevelDef } from "./src/harness"; + +const DIRS = { w: "up", a: "left", s: "down", d: "right" } as const; +/** A coordinate or a chooser's number, as the command line reads one. */ +const NUMBER = /^\d{1,9}$/; + +const level = JSON.parse(readFileSync(process.argv[2], "utf8")) as LevelDef; +const inputs = JSON.parse(readFileSync(process.argv[3], "utf8")) as string[]; +let next = 0; +const out = (s: string) => process.stdout.write(`${s}\n`); + +/** The next input, echoed after its prompt; undefined when there are none left. */ +function read(prompt: string): string | undefined { + const line = inputs[next]; + if (line === undefined) return undefined; + next++; + out(`${prompt}${line}`); + return line; +} + +/** The level, or a readable error: an unknown card, a card with no noun on the grid, ... */ +function start() { + try { + return newSpellGrid(level, []); + } catch (e) { + process.stderr.write(`The level does not start: ${e instanceof Error ? e.message : String(e)}\n`); + process.exit(1); + } +} + +const g = start(); +g.setChooser((kind, options) => { + out(`Choose the ${kind}:\n${options.map((o, i) => ` ${i + 1}) ${o}`).join("\n")}`); + const answer = read("? ")?.trim() ?? ""; + if (options.includes(answer)) return answer; + return (NUMBER.test(answer) ? options[Number(answer) - 1] : undefined) ?? options[0]; +}); + +const show = () => out(`${render(g, level)}\nHeroes: ${g.health(g.hero(0))} Health (yours), ${g.health(g.hero(1))} Health (enemy)`); + +out("== start"); +show(); +for (;;) { + if (g.won() || g.lost()) { + out(g.won() ? "You win!" : "Nothing is YOU. You lose."); + const left = inputs.length - next; + if (left > 0) out(`${left} input${left === 1 ? "" : "s"} not run: the level is over.`); + break; + } + out(""); + const line = read("> "); + if (line === undefined) { + out("(end of inputs)"); + break; + } + const [command, ...args] = line.trim().split(/\s+/); + g.clearLog(); + try { + if (command in DIRS) g.step(DIRS[command as keyof typeof DIRS]); + else if (command === "c" && args.length > 0) { + const n = args.length; + const at = n >= 3 && NUMBER.test(args[n - 2]) && NUMBER.test(args[n - 1]) ? ([Number(args[n - 2]), Number(args[n - 1])] as const) : undefined; + const card = (at === undefined ? args : args.slice(0, n - 2)).join(" "); + const target = at === undefined ? null : g.minionAt(...at); + if (at === undefined) g.cast(card); + else if (target !== null) g.cast(card, target); + else g.play(card, ...at); + } else out("Commands: w a s d (move), c [x y] (cast or play)"); + } catch (e) { + out(`Error: ${e instanceof Error ? e.message : String(e)}`); + } + const tree = g.logTree(); + if (tree !== "") out(`Log:\n${tree.trimEnd()}`); + show(); +} diff --git a/tools/schema.mjs b/tools/schema.mjs index f2389f6..f533e8d 100644 --- a/tools/schema.mjs +++ b/tools/schema.mjs @@ -1,5 +1,6 @@ // Minimal JSON-schema subset validator (node built-ins only). -// Supports: type, required, enum, const, properties, items, minimum, oneOf, anyOf. +// Supports: type, required, enum, const, properties, items, minimum, oneOf, anyOf. A value that +// matches no oneOf branch is reported against the branch its const fields pick, if exactly one. const typeOk = (t, v) => t === "array" ? Array.isArray(v) : t === "null" ? v === null @@ -24,7 +25,17 @@ export function validate(schema, value, path = "$") { if (schema.anyOf && !schema.anyOf.some((s) => validate(s, value, path).length === 0)) errs.push(`${path}: matches no anyOf branch`); if (schema.oneOf) { const n = schema.oneOf.filter((s) => validate(s, value, path).length === 0).length; - if (n !== 1) errs.push(`${path}: matches ${n} oneOf branches (need 1)`); + // No branch matches: if the value's `const` fields pick out one branch (a clause's "kind", + // an effect's "type"), say what is wrong with it; that is what the author meant. + const meant = n === 0 ? schema.oneOf.filter((s) => discriminates(s, value)) : []; + if (meant.length === 1) errs.push(...validate(meant[0], value, path)); + else if (n !== 1) errs.push(`${path}: matches ${n} oneOf branches (need 1)`); } return errs; } + +/** Does `value` have every `const` field of the object schema `s`, and does `s` have one? */ +function discriminates(s, value) { + const consts = Object.entries(s.properties || {}).filter(([, p]) => "const" in p); + return typeOk("object", value) && consts.length > 0 && consts.every(([k, p]) => value[k] === p.const); +} diff --git a/tools/test/fixtures/course-mutation/01-m01/reference/ts/run-tests.mjs b/tools/test/fixtures/course-mutation/01-m01/reference/ts/run-tests.mjs new file mode 100644 index 0000000..5694119 --- /dev/null +++ b/tools/test/fixtures/course-mutation/01-m01/reference/ts/run-tests.mjs @@ -0,0 +1,17 @@ +// A stand-in for vitest (MUTATION_CMD="node run-tests.mjs"): runs two tests against +// src/order.mjs and writes a report in the shape of vitest's JSON reporter. +import { writeFileSync } from "node:fs"; + +const tests = { + test_01_first_comes_first: (o) => o.ORDER[0] === "first", + test_01_two_steps: (o) => o.ORDER.length === 2, +}; +let file; +try { + const o = await import("./src/order.mjs"); + const assertionResults = Object.entries(tests).map(([title, t]) => ({ title, status: t(o) ? "passed" : "failed" })); + file = { name: "order", status: assertionResults.every((t) => t.status === "passed") ? "passed" : "failed", assertionResults }; +} catch (e) { + file = { name: "order", status: "failed", message: e.message, assertionResults: [] }; +} +writeFileSync(".mutation-report.json", JSON.stringify({ testResults: [file] })); diff --git a/tools/test/fixtures/course-mutation/01-m01/reference/ts/src/order.mjs b/tools/test/fixtures/course-mutation/01-m01/reference/ts/src/order.mjs new file mode 100644 index 0000000..0b45be7 --- /dev/null +++ b/tools/test/fixtures/course-mutation/01-m01/reference/ts/src/order.mjs @@ -0,0 +1,2 @@ +// The fixture's one ordering decision, as a list. +export const ORDER = ["first", "second"]; diff --git a/tools/test/mutation-check.test.mjs b/tools/test/mutation-check.test.mjs new file mode 100644 index 0000000..0f43400 --- /dev/null +++ b/tools/test/mutation-check.test.mjs @@ -0,0 +1,106 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fixtures, run, toolsDir } from "./helpers.mjs"; +import { applyMutant, judge, mutantProblems, parseArgs, readReport, targetModule } from "../mutation-check.mjs"; + +// A course with one module whose TypeScript reference has one list (src/order.mjs) and a +// stand-in test runner (run-tests.mjs) that writes a vitest-shaped report. +const env = { COURSE_ROOT: join(fixtures, "course-mutation"), MUTATION_CMD: "node run-tests.mjs" }; +const SWAP = { + name: "Order swapped", + file: "src/order.mjs", + find: 'export const ORDER = ["first", "second"];', + replace: 'export const ORDER = ["second", "first"];', + expect: ["test_01_first_comes_first"], +}; + +/** Run the tool on the fixture course with these mutants. */ +function check(mutants, ...args) { + const file = join(mkdtempSync(join(tmpdir(), "sdc-mut-")), "mutants.json"); + writeFileSync(file, JSON.stringify(mutants)); + return run("mutation-check.mjs", ["--mutants", file, ...args], env); +} + +test("mutation_check_parses_args", () => { + assert.deepEqual(parseArgs([]), { names: [], mutants: null, module: null }); + assert.deepEqual(parseArgs(["A", "--mutants", "m.json", "--module", "9", "B"]), { names: ["A", "B"], mutants: "m.json", module: "09" }); +}); + +test("mutation_check_kills_a_fixture_mutant", () => { + const r = check([SWAP]); + assert.equal(r.code, 0, r.out); + assert.match(r.out, /unmutated: 2 tests pass/); + assert.match(r.out, /killed {3}Order swapped: test_01_first_comes_first\n/); + assert.match(r.out, /1 mutants killed/); + // The file is restored: the staged copy is gone, and the course's copy was never touched. + assert.match(readFileSync(join(env.COURSE_ROOT, "01-m01", "reference", "ts", "src", "order.mjs"), "utf8"), /\["first", "second"\]/); +}); + +test("mutation_check_fails_on_a_surviving_mutant", () => { + // Renaming the second step changes nothing the fixture's tests look at. + const survivor = { ...SWAP, name: "Second renamed", replace: 'export const ORDER = ["first", "later"];' }; + const r = check([SWAP, survivor]); + assert.equal(r.code, 1, r.out); + assert.match(r.out, /killed {3}Order swapped/); + assert.match(r.out, /SURVIVED Second renamed: test_01_first_comes_first passed; failed: none/); + assert.match(r.out, /1 of 2 mutants not killed/); +}); + +test("mutation_check_fails_on_a_stale_find", () => { + const stale = { ...SWAP, name: "Stale", find: 'export const ORDER = ["first", "third"];' }; + const r = check([stale]); + assert.equal(r.code, 1, r.out); + assert.match(r.out, /STALE {4}Stale: stale: `find` occurs 0 times in src\/order\.mjs, not once/); + const missing = check([{ ...SWAP, name: "Gone", file: "src/gone.mjs" }]); + assert.equal(missing.code, 1); + assert.match(missing.out, /STALE {4}Gone: no file src\/gone\.mjs/); +}); + +test("mutation_check_fails_on_a_run_that_is_not_sane", () => { + // A mutant that breaks the module kills every test, but by not loading: that isn't a kill. + const r = check([{ ...SWAP, name: "Broken", replace: "export const ORDER = [" }]); + assert.equal(r.code, 1, r.out); + assert.match(r.out, /INSANE {3}Broken: a test file did not run: order: /); +}); + +test("mutation_check_runs_only_the_named_mutants", () => { + const r = check([SWAP, { ...SWAP, name: "Stale", find: "nowhere" }], "Order swapped"); + assert.equal(r.code, 0, r.out); + assert.doesNotMatch(r.out, /Stale/); +}); + +test("mutation_check_applies_exactly_one_replacement", () => { + assert.equal(applyMutant("a [x] b", { file: "f", find: "[x]", replace: "$&y" }), "a $&y b"); + assert.throws(() => applyMutant("x x", { file: "f", find: "x", replace: "y" }), /occurs 2 times in f/); +}); + +test("mutation_check_judges_from_the_report", () => { + const report = (tests, extra = {}) => ({ testResults: [{ name: "t", status: "failed", assertionResults: tests.map(([title, status]) => ({ title, status })), ...extra }] }); + const baseline = readReport(report([["test_a", "passed"], ["test_b", "passed"]])); + const m = { expect: ["test_a"] }; + assert.equal(judge(m, baseline, readReport(report([["test_a", "failed"], ["test_b", "passed"]]))).killed, true); + assert.deepEqual(judge(m, baseline, readReport(report([["test_a", "passed"], ["test_b", "failed"]]))).survivors, ["test_a"]); + const lost = judge(m, baseline, readReport(report([["test_a", "failed"]]))); + assert.equal(lost.sane, false); + assert.match(lost.problems[0], /1 tests did not run, e\.g\. test_b/); + assert.match(judge({ expect: ["test_zz"] }, baseline, baseline).problems[0], /doesn't have: test_zz/); +}); + +test("mutation_check_rejects_malformed_mutants", () => { + assert.deepEqual(mutantProblems(SWAP), []); + assert.ok(mutantProblems({ ...SWAP, expect: [] }).some((p) => /expect/.test(p))); + assert.ok(mutantProblems({ ...SWAP, replace: SWAP.find }).some((p) => /nothing would change/.test(p))); + assert.ok(mutantProblems({ ...SWAP, why: "x" }).some((p) => /unknown key "why"/.test(p))); +}); + +test("mutation_check_course_mutants_are_well_formed", () => { + const mutants = JSON.parse(readFileSync(join(toolsDir, "mutants.json"), "utf8")); + for (const m of mutants) assert.deepEqual(mutantProblems(m), [], m.name); + assert.equal(new Set(mutants.map((m) => m.name)).size, mutants.length, "names are unique"); + // Each mutant's find occurs once in the highest module's TypeScript reference (the full run checks it too). + const ref = join(targetModule(null).path, "reference", "ts"); + for (const m of mutants) assert.doesNotThrow(() => applyMutant(readFileSync(join(ref, m.file), "utf8"), m), m.name); +}); diff --git a/tools/test/run-scenario.test.mjs b/tools/test/run-scenario.test.mjs new file mode 100644 index 0000000..3d92fa6 --- /dev/null +++ b/tools/test/run-scenario.test.mjs @@ -0,0 +1,113 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { run, toolsDir } from "./helpers.mjs"; +import { mergeSeams, mergeWords, scenarioProblems } from "../run-scenario.mjs"; + +const course = join(toolsDir, "..", "course"); +const example = join(course, "10-spellgrid", "designer-example"); + +/** Write `scenario` to a temp file and run the tool on it. */ +function scenario(s) { + const file = join(mkdtempSync(join(tmpdir(), "sdc-scn-")), "scenario.json"); + writeFileSync(file, typeof s === "string" ? s : JSON.stringify(s)); + return run("run-scenario.mjs", [file]); +} + +const text = (word, x, y) => ({ kind: `text:${word}`, x, y }); +// BABA IS YOU on row 0, WOUND IS DOUBLE on row 1, Baba at (0,3). +const LEVEL = { + width: 5, + height: 4, + objects: [text("BABA", 0, 0), text("IS", 1, 0), text("YOU", 2, 0), text("WOUND", 0, 1), text("IS", 1, 1), text("DOUBLE", 2, 1), { kind: "baba", x: 0, y: 3 }], + cards: [], + hand: [], +}; +const TOTEM = { + name: "Test Totem", + type: "minion", + cost: 1, + attack: 0, + health: 3, + noun: "ROCK", + split: 0, + text: [{ kind: "trigger", on: "selfDamaged" }, { kind: "effect", effect: { type: "dealDamage", target: "enemyHero", amount: 1 } }], +}; + +test("run_scenario_reports_schema_errors_readably", () => { + const bad = { ...TOTEM, text: [TOTEM.text[0], { kind: "effect", effect: { type: "dealDamage", target: "everyone", amount: 1 } }] }; + const r = scenario({ cards: [bad], level: LEVEL, inputs: [] }); + assert.equal(r.code, 1, r.out); + assert.match(r.out, /is not a valid scenario:\n/); + assert.match(r.out, /cards\[0\] \("Test Totem"\)\.text\[1\]\.effect\.target: not in enum \["self",/); + assert.doesNotMatch(r.out, /oneOf/); + + const shape = scenario({ level: { width: 3 }, inputs: "d", seams: { triggerWords: { HURT: "whenever" } } }); + assert.equal(shape.code, 1); + assert.match(shape.out, /level: missing "height"/); + assert.match(shape.out, /inputs: expected a list of command lines/); + assert.match(shape.out, /seams\.triggerWords\.HURT: "whenever" is not a trigger/); + + const json = scenario("{ not json"); + assert.equal(json.code, 1); + assert.match(json.out, /is not JSON: /); +}); + +test("run_scenario_merges_words_into_the_rules", () => { + const r = scenario({ + words: [{ name: "WOUND", role: "noun" }], + seams: { triggerWords: { WOUND: "selfDamaged" } }, + level: LEVEL, + inputs: [], + }); + assert.equal(r.code, 0, r.out); + assert.match(r.out, /^Rules: BABA IS YOU, TEXT IS PUSH, WOUND IS DOUBLE$/m); + // Without the word, the text spells nothing. + const without = scenario({ level: LEVEL, inputs: [] }); + assert.match(without.out, /^Rules: BABA IS YOU, TEXT IS PUSH$/m); +}); + +test("run_scenario_prints_the_log_as_a_tree", () => { + const level = { ...LEVEL, cards: [{ card: "Test Totem", x: 3, y: 3 }], hand: ["Whirlwind"] }; + const r = scenario({ cards: [TOTEM], words: [{ name: "WOUND", role: "noun" }], seams: { triggerWords: { WOUND: "selfDamaged" } }, level, inputs: ["c Whirlwind", "w"] }); + assert.equal(r.code, 0, r.out); + // WOUND IS DOUBLE: the totem's damage trigger activates twice, each with its Damage one level down. + assert.match(r.out, /\n> c Whirlwind\nLog:\nSpellCast\{card: "Whirlwind"\}\nDamaged\{target: (\d+), amount: 1\}\nTriggered\{source: \1, trigger: "damaged"\}\n {2}Damaged\{target: \d+, amount: 1\}\nTriggered\{source: \1, trigger: "damaged"\}\n {2}Damaged/); + assert.match(r.out, /Minions: Test Totem \(3,3\) 2 Health\nHand: empty\nHeroes: 30 Health \(yours\), 28 Health \(enemy\)\n/); + assert.match(r.out, /\n> w\n/); + assert.match(r.out, /\(end of inputs\)\n$/); +}); + +test("run_scenario_validates_words_and_seams", () => { + const base = [{ name: "BABA", role: "noun" }, { name: "DOUBLE", role: "property" }]; + assert.deepEqual(mergeWords(base, [{ name: "WOUND", role: "noun" }, { name: "BABA", role: "noun" }]).map((w) => w.name), ["BABA", "DOUBLE", "WOUND"]); + assert.throws(() => mergeWords(base, [{ name: "BABA", role: "property" }]), /BABA is already a noun/); + const seams = { triggerWords: { DEATHRATTLE: "deathrattle" }, multiplierWords: { DOUBLE: { times: 2, stacking: "max" } } }; + assert.deepEqual(mergeSeams(seams, { triggerWords: { WOUND: "selfDamaged" } }).triggerWords, { DEATHRATTLE: "deathrattle", WOUND: "selfDamaged" }); + assert.throws(() => mergeSeams(seams, { triggerWords: { DEATHRATTLE: "battlecry" } }), /DEATHRATTLE already names "deathrattle"/); + + const ctx = { words: base, cardNames: ["Whirlwind"] }; + const problems = (s) => scenarioProblems({ level: LEVEL, inputs: [], ...s }, ctx); + assert.deepEqual(problems({}), []); + assert.match(problems({ words: [{ name: "wound", role: "noun" }] }).join("\n"), /words\[0\]\.name: "wound" must be capital letters/); + assert.match(problems({ words: [{ name: "AND", role: "operator" }] }).join("\n"), /words\[0\]\.role: must be "noun" or "property"/); + assert.match(problems({ seams: { triggerWords: { HURT: "selfDamaged" } } }).join("\n"), /seams\.triggerWords\.HURT: HURT is not a noun/); + assert.match(problems({ seams: { multiplierWords: { DOUBLE: { times: 1, stacking: "max" } } } }).join("\n"), /seams\.multiplierWords\.DOUBLE\.times: below minimum 2/); + assert.match(problems({ cards: [{ ...TOTEM, name: "Whirlwind" }] }).join("\n"), /cards\[0\] \("Whirlwind"\): the course already has a card named "Whirlwind"/); + assert.match(problems({ extra: 1 }).join("\n"), /unknown key "extra"/); +}); + +test("run_scenario_designer_example_output_is_current", () => { + // The capstone's worked example shows the tool's real output; this keeps it from going stale. + const r = run("run-scenario.mjs", [join(example, "scenario.json")]); + assert.equal(r.code, 0, r.out); + assert.equal(r.out, readFileSync(join(example, "output.txt"), "utf8")); + // The scenario carries the example's data files: the three cards, the word and its seam. + const json = (f) => JSON.parse(readFileSync(join(example, f), "utf8")); + const s = json("scenario.json"); + assert.deepEqual(s.cards, ["ash-totem", "thorn-idol", "scorch"].map((c) => json(`cards/${c}.json`))); + assert.deepEqual(s.words, json("words.json")); + assert.deepEqual(s.seams, json("seams.json")); +}); diff --git a/tools/test/schema.test.mjs b/tools/test/schema.test.mjs index 510cf1f..df35d52 100644 --- a/tools/test/schema.test.mjs +++ b/tools/test/schema.test.mjs @@ -166,3 +166,13 @@ test("spellgrid_cards_are_module_local", () => { assert.notEqual(json(join(shared, f)).game, "spellgrid", `${f} belongs in course/10-spellgrid/cards`); } }); + +test("schema_oneof_error_names_the_meant_branch", () => { + const card = (effect) => ({ name: "X", type: "spell", cost: 1, split: 0, text: [{ kind: "effect", effect }] }); + // The clause's kind and the effect's type pick one branch each: the error is about that branch. + assert.deepEqual(validate(cardSchema, card({ type: "dealDamage", target: "everyone", amount: 1 })), [ + '$.text[0].effect.target: not in enum ["self","allMinions","allOtherMinions","allOtherCharacters","enemyHero","chosen"]', + ]); + // An unknown effect type picks no branch: the general message stays. + assert.deepEqual(validate(cardSchema, card({ type: "heal", amount: 1 })), ["$.text[0].effect: matches 0 oneOf branches (need 1)"]); +});