systemic-game-design-course/course/13-break-your-engine/lesson.md

Module 13: Break your own engine

The course is finished at Module 10. This module is about what to do with an engine once its tests pass, which is where most of an engine's life is spent.

Every module so far gave you tests and asked you to make them green. That teaches design, and it hides a question: how do you know the tests are any good? This course is a useful case study, because it got the answer wrong. Its first complete version had reference engines in two languages, every test green, an automated check that both languages ran the same tests, and three bugs in the engines that no test could see. All three were found by the methods in this module, and none by reading the code more carefully.

There is no Spec or Build here, no reference solution and no Expansion Drop. start-module doesn't know this module. Each section is a technique, what it found in this course, and something to try on your own engine. The sections are independent; take them in any order.

Observe

Three bugs, all in the reference engines, all with every test passing.

A Deathrattle summon in the wrong place. The Death Creation Step remembered each dying minion's board index. If a minion further left died in the same step, the index was stale and the token appeared one place too far right (Module 3, Predict question 5). The lesson that introduced the bug sits two modules after the lesson that explains why indexes go stale.

A negation that lost to a group. ALL IS ROCK with BABA IS NOT ROCK turned Baba into a rock. Properties were settled per object; conversions were settled per rule (Module 9, Predict question 10). The lesson text for Module 7 described this exact mistake as a way for learners to fail the Expansion Drop.

An object that moved twice. A key that was YOU and PULL, standing behind Baba, was dragged one tile by Baba's move and then took its own step (Module 8, Expansion Drop). The lesson told learners to "make sure a YOU object that is also PUSH doesn't move twice"; the reference did that with an ordering trick that only worked for PUSH.

And one that wasn't a bug but could have become one at any time: six different changes to the order in which the engines do things (two phases of the turn swapped, a parse dropped, the hazards reversed, the aura snapshot replaced by a live scan) left every test green.

Notice what the three bugs have in common. None is in a hard algorithm. Each is the meeting of two features that were each tested alone: simultaneous death and summon position; group nouns and conversion; pull and several YOU objects. This course is about systems whose parts combine freely, and the price of that is that the number of combinations grows much faster than anyone can write tests by hand.

Predict

Write your answers down before you open the discussion.

1. Your engine passes about seventy tests. Pick any if in it and flip the condition. What do you think the chance is that a test fails?
2. You want to test the parser on a board nobody designed, with random words on random tiles. You can't know what the right rules are. What can you check?
3. Level 3 of SpellGrid has a scripted solution in the tests. What does that test prove about the level, and what doesn't it prove?
4. Can your engine loop forever? Can a level?

Discussion

1. Lower than you'd like, and you can measure it. That is mutation testing, below.
2. You can check that two related boards give related answers (the same board shifted, or with x and y swapped), and you can check facts that hold for every board. Those are metamorphic relations and invariants.
3. It proves that the level is solvable and that one particular input sequence wins. It doesn't prove that the solution needs the idea the level was built around, or that there is no shorter way. A search does.
4. The engine: before Module 5, yes, with the right two cards (Modules 2 and 3). Module 5's nesting guard and Death Phase limit stop both of those, but not every card: a damage trigger that summons a new copy of its own minion gets a fresh (minion, trigger) pair every time, so the guard never fires. A level: there is no general way to tell for the real games, which is a theorem, and the last section says what follows from it in practice.

Mutation testing: test the tests

Change the program slightly, on purpose, and run the tests. If they still pass, they don't check the thing you changed. The changed program is a mutant; a mutant that passes is said to survive. Mutation testing "involves making small changes to the program being tested", and a test "detects, and therefore rejects, a mutant upon test failure" (Mutation testing, Wikipedia).

General-purpose tools generate thousands of mutants by flipping operators. You don't need one to start, because this course's engines have their most important decisions written down as lists, and a list has obvious mutants: swap two neighbours, drop an entry. These were run against the TypeScript reference with the tests the course first shipped:

Mutant	Tests that failed
End of Phase: Aura Update before the Death Creation Step	test_04_simultaneous_death_not_doubled, test_04_aura_timing_order, test_10_level_3_solution
Turn: no parse between movement and conversion	test_09_mid_turn_conversion
Turn: win check before hazards	test_09_defeat_beats_win
Turn: no parse between conversion and hazards	none
Turn: hazards before conversion	none
Turn: no last parse	none
Hazards in reverse order	none
Auras read live from the board, no snapshot	none in Modules 1–9

Five survivors. The engines were right in each case; the tests would not have noticed if they hadn't been. Each survivor now has a test with its name on it (test_09_conversion_reparses_before_hazards, test_09_conversion_before_hazards, test_09_last_parse_matches_board, the two test_09_hazard_order_… tests, test_05_own_aura_misses_own_battlecry), and the whole table has no empty row.

Killing a mutant is a design exercise. You have to find the smallest situation in which the two orders give different results, and sometimes you learn that there isn't one. With one property per object, the order of SINK and HOT/MELT can't be observed at all (Module 9, Stress it question 2). The live-aura mutant survived because three separate mechanisms guard the same behaviour (Module 4, Stress it questions 2 and 3). A surviving mutant means one of three things: a missing test, a decision that isn't observable yet, or redundant machinery. All three are worth knowing.

Try it. List every ordered list in your engine: the turn, the hazards, the movement responses, the end-of-Phase steps, the order in which react walks events and listeners, the order YOU objects move. For each, swap two neighbours and run everything. Keep a table like the one above. For each survivor, write the test or write down why none can exist.

Invariants and random inputs

A hand-written test checks one path. An invariant is a fact that must hold after every operation on every path, and checking it needs no knowledge of what the right answer is. Module 1's Stress it gave three for the world of cards. Some more that should hold in an engine built like the reference:

• Every minion on a board has more than 0 Health and is not pending destroy, whenever a Sequence is not running. (During one, mortally wounded minions are in play. An invariant has a scope, and finding the scope is half the work.)
• The log never has a Triggered whose source has never been summoned.
• activeRules() is a subset of rules().
• rules() equals what a fresh level built from the same objects would compute. This one says "the rule table is a function of the board", which is the thesis of Part II, as one line of test.
• After step, no object is off the level, and no destroyed id has a position.

To use them, generate operations at random from a seed you choose, so that a failure can be replayed, and check every invariant after each one. A library is optional. fast-check for TypeScript and proptest for Rust add one thing a hand-rolled loop lacks and you will want eventually: shrinking, the automatic search for a smaller failing input. The idea comes from QuickCheck, a library "designed to assist in software testing by generating test cases for test suites – an approach known as property testing" (QuickCheck, Wikipedia).

The summon-position bug is the kind this finds. A generator that summons three minions, damages two at random and casts Whirlwind will produce the failing arrangement within a few dozen tries, provided some property mentions position. This one would do: "a Deathrattle token stands between the same two surviving neighbours as the minion that summoned it".

Try it. Write the fourth invariant above as a test: twenty random place and step calls on a random level, and after each, compare rules() with those of a level rebuilt from scratch. If you added a "reparse only when text changed" flag in Module 6, this is the test that keeps it safe.

Metamorphic relations

When you can't say what the output should be, say how two outputs must relate. A metamorphic relation is a necessary property "of the intended functionality" that "must involve multiple executions of the software" (Metamorphic testing, Wikipedia). The course's engines have several, and each one encodes a claim a lesson made in prose:

Relation	The claim it tests
rules(board) == rules(board shifted by (dx, dy))	The grammar mentions no absolute position (Module 6).
rules(board) == rules(board with x and y swapped)	Rows and columns are parsed by the same rule (Module 6).
resolve(shuffle(rules)) is a shuffle of resolve(rules)	Where a sentence sits can't matter (Module 7).
resolve(resolve(rules)) == resolve(rules)	The resolution pass finishes the job in one go (Module 7).
A game replayed from its recorded inputs and choices gives the same log	The chooser is the only way randomness gets in (Module 5).

Some relations you might expect are false, and working out why is as useful as a passing test. "Adding a textless minion to the enemy board changes nothing but its own events" fails as soon as a Test Echo is in play, because a trigger's condition can be about any minion. Mirroring a board left to right changes the rules, because sentences have a direction. "Press right, then left" is not the identity (Module 8, Stress it question 4). Dealing 1 damage twice in one Sequence is not the same as in two Sequences, because death is checked between Sequences and not inside one (Module 3). Each false relation marks a place where the system has a direction, a memory or a checkpoint.

Two engines, one specification

This repository has a Rust reference and a TypeScript reference for every module. They pass identical tests. That says they agree on the seventy-odd situations somebody wrote down. It says nothing about any other situation, and two engines written from one specification are the best oracle there is for the rest: generate a random level and a random input sequence, run both, compare what they print. Any difference is a bug in at least one of them or a hole in the specification, and you didn't need to know the right answer to find it.

This is differential testing. It is also how Module 6 suggests you protect an optimisation: keep the slow version and compare. If you did both tracks of the course, you own two engines; if you did one, the reference in your language is the second. The command-line front ends from Module 10 (tools/play-spellgrid.mjs) read inputs from standard input and print the board, the rules, the minions and the hand after each one, so two engines can be compared as text.

node tools/diff-tracks.mjs does this for the two references. For each seed it makes a random SpellGrid level and a random script, runs both command lines, and prints a diff for the first seed on which they disagree. Its first run of 500 seeds found nothing. The references did differ, though: in how each decides that an Assembled card already exists, and in how each command line reads a signed number or quotes a card name. The fuzzer could reach none of these. The first needs a card registered again in the middle of a game, and the others need input outside the command grammar it generates. A differential tester finds only what its generator can reach. All three are now fixed, and test_05_assemble_never_rewrites_an_earlier_card and the reference command-line tests pin them. Widening the generator is an open exercise, and a good one.

State, undo and replay

Module 9's Stress it asks you to build Undo and find out what a snapshot has to contain. The general lesson is in two parts.

Derived data is not state. The rule table, the aura snapshot at the start of a Sequence, a minion's position on the grid as seen by the card engine: all of these are computed from something else. If you can recompute it, don't save it, don't restore it, and above all don't let anything edit it in place. Both parts of the course were built around this: "The previous rules are removed entirely" (Module 6); auras are re-read at an Aura Update (Module 4). Martin Fowler's description of state held as a log of events makes the same cut from the other side: "Event Sourcing ensures that all changes to application state are stored as a sequence of events", and from them "we can also use the event log to reconstruct past states" (Event Sourcing).

The real state is larger than it looks. Ids, the id counter, the order-of-play counter, the age order of objects on a tile, won, the contents of each deck in order, the chooser's future answers. None of these appear when you draw the board, and every one of them can change what happens next. The honest way to find the list is the round-trip test: snapshot, restore into a fresh engine, play both forward with the same random inputs, and compare.

An engine that is deterministic gets a second kind of undo for nothing: record the inputs, replay all but the last. It needs no snapshot and can't forget a field. The solver in the next section is built entirely on it. Robert Nystrom's Command chapter is the short introduction to both.

Solving your own levels

A SpellGrid level has a few inputs per position (four directions, plus a card or two), no hidden information once the chooser's answers are enumerated, and no randomness. So the positions reachable from the start form a graph that can be explored exhaustively. The course's solver is about 150 lines, in tools/solver/solve.ts:

node tools/solve-spellgrid.mjs l3                 # shortest solution
node tools/solve-spellgrid.mjs l3 --naive         # the same level on the engine with the wrong aura timing
node tools/solve-spellgrid.mjs l1 --max 40 --never-rule "DEATHRATTLE IS DOUBLE"
node tools/solve-spellgrid.mjs path/to/your-level.json --all

How it works:

• Breadth-first over inputs, so the first solution found is a shortest one.
• Replay, not copy. Each position is reached by starting a fresh level and applying its inputs. The solver never clones engine state, so it needs nothing from the engine except determinism and the harness you already wrote. It is slow per position and it cannot be wrong about what a position is.
• A key for "the same position". Two input sequences that lead to the same tiles, minions, Health, hand and hero Health are one position, and only the first is explored. This is the delicate part, for the reason the last section gave: the key has to include everything that can change the future. The solver adds the Assemble choices made so far, because two Assembled cards can share a name and differ in text. It does not include object age, so in a level where age breaks a tie it could merge two positions that differ. The three shipped levels have no such tie. Yours might.
• A filter. --never-rule discards positions in which a given rule is in force. That turns "is the level solvable?" into "is it solvable without the idea?", which is the question a designer cares about.

What it found for the shipped levels is in Module 10's Stress it: each is solvable in the scripted number of inputs, Level 1 has no solution that avoids DEATHRATTLE IS DOUBLE, and Level 3 has no solution at all on the naive engine. Those are stronger statements than any test in Module 10 makes, and they hold for every line of play, not for one script.

An unintended solution is called a cook. Puzzle designers have always hunted cooks by hand; a solver hunts them while you sleep. The method generalises: state what the level is for as a predicate on positions, forbid it, and search.

Try it. Write a fourth SpellGrid level around one idea from the course (say, a NOT that has to be broken before a Deathrattle can double). Run the solver with --all and read every solution it prints. Then run it with the idea forbidden.

What search can't tell you

The solver finishes on small levels. It is worth knowing how far that goes.

• Sokoban, which is Module 8 with one property and no text: "deciding whether any given Sokoban puzzle is solvable is a problem known to be NP-hard and PSPACE-complete" (Sokoban, Wikipedia). The cost of exhaustive search grows explosively with the size of the level.
• Baba Is You, with its full rule set: "We establish the undecidability of 2019 puzzle game Baba is You through a reduction from the Post correspondence problem." (Jonathan Geller, Baba is You is Undecidable, 2022). The result is about a version of the game without a fixed level size; no program can decide solvability for all of those.
• Magic: The Gathering, the card game Modules 3, 4 and 7 borrowed rules from: "optimal play in real-world Magic is at least as hard as the Halting Problem" (Churchill, Biderman and Herrick, Magic: The Gathering is Turing Complete, 2019).

These are what you should expect from the systems this course teaches. Rules that are data, that combine freely and that can modify one another are, give or take some bookkeeping, a programming language. The player's "what happens if I do this?" becomes the programmer's "what does this program do?", and that question has no general shortcut.

Three practical consequences:

• Bound everything. A cap on Death Phase rounds, on trigger nesting, on solver depth. A cap that fails loudly is a design decision; a hang is not. Magic's rule 104.4b turns the unbounded case into an outcome: a game in "a 'loop' of mandatory actions ... with no way to stop ... is a draw" (Comprehensive Rules).
• Search small, argue large. Exhaustive search for the levels you ship; arguments like Module 3's decreasing quantity for properties of the whole card set.
• Expect surprises and keep them cheap. You will not foresee every interaction. The engine's job is to make each one a matter of data and one generic routine, so that fixing it doesn't mean rewriting. That is where Module 0 started.

Name it

The pattern is the oracle problem, and the techniques in this module are ways round it. A test needs something that knows the right answer. Hand-written expected values are one such oracle, the most precise and the most expensive. The others trade precision for coverage:

Oracle	Knows	Module 13 section
An expected value written by a person	the exact answer, for one input	(Modules 1–10)
An invariant	a fact about every answer	Invariants
A metamorphic relation	how two answers relate	Metamorphic relations
A second implementation	the same answer, by other means	Two engines
The tests themselves, against a changed program	whether anyone is checking	Mutation testing
Exhaustive search	every reachable position	Solving your own levels

Systemic games need the cheaper oracles more than most software, for the reason given under Observe: their behaviour is the product of their parts, and hand-written tests grow as the sum.

Further reading

Testing:

• Mutation testing and Metamorphic testing, Wikipedia. Short, with the standard references. Read the second for the phrase "test oracle problem".
• fast-check (TypeScript) and the proptest book (Rust). Property-based testing libraries for the two tracks. Proptest's book has a chapter on state machine testing, which is the "random operations, check invariants" loop with shrinking.
• Robert Nystrom, Command (Game Programming Patterns), for undo and replay. Martin Fowler, Event Sourcing, for state as a log of events.

The wider field of systemic design, which this course approached from the rules-engine side only:

• Mark Brown, The Rise of the Systemic Game (Game Maker's Toolkit, video). The designer's definition of the word in the course's title, with examples from games whose systems are physical (fire, AI, stealth) where this course's are textual. Watch it and ask which of the course's patterns each example would need.
• Hidemaro Fujibayashi, Satoru Takizawa and Takuhiro Dohta, Change and Constant: Breaking Conventions with 'The Legend of Zelda: Breath of the Wild' (GDC 2017). The talk on a game whose objects interact through a small set of shared properties instead of pairwise scripts. Compare that with Module 9's hazards, which ask "what shares this tile and has this property?" and never "is this a skull?".
• PuzzleScript and its documentation, especially Rules 101 and Execution Order. A complete language for games like Part II's, where the designer writes the rules as rewrite patterns. Rebuilding a Module 8 level in it takes an evening and shows which of your engine's decisions were forced.
• HearthSim's python-hearthstone and python-hslog. The real game's vocabulary of tags and its log format, as reverse-engineered by the community. After building Part I you can read a Power.log and recognise most of what is in it.