systemic-game-design-course/course/06-parser/lesson.md

Module 6: Grid, lexer and parser

This is the first module of Part II. In Part I the rules lived inside the cards: Harvest Golem's Deathrattle was a clause on Harvest Golem, and your engine's job was to find the right clause at the right moment and run it. Part II turns that around. In Baba Is You the objects on the board have no behaviour of their own. A rock is just a rock until some text on the board says ROCK IS PUSH: "When three blocks in a row make a sentence, like "ROCK IS PUSH," it creates a rule that immediately takes effect in-game", and "If you push one of the words in a ruleset out of line, that effect instantly ceases" (Game Developer). Apart from a few base rules that hold with no text, TEXT IS PUSH among them (Rule), every rule comes from words lying on the grid, and the words can be pushed around, so the rules change while you play. The engine's job is now to compile the rules from the board, and to compile them again every time the board changes.

This module builds the compiler's front end: a grid that holds objects and text, a lexer that turns text into words with grammatical roles, and a parser that turns words into rules. Nothing moves yet (that is Module 8), and no rule does anything yet (Modules 7 and 9). By the end, rules() reads any board you build.

You need your Module 5 workspace with all Module 1–5 tests passing, under git. Part I's engine stays where it is, and its tests keep running: Part II is a second engine next to the first, in the same project. Part III will plug them together.

Observe

All quotes in this module come from sources recorded in the course's research notes. The Baba Is You Wiki is an unofficial community wiki, and the course treats it as the best available description of the game, not as Hempuli's own word.

How a sentence reads, from the wiki's Rule page:

Rules read from left to right in a row, or from top to bottom in a column.

A noun can take a property or another noun (Category:Nouns):

A noun can be used as a NOUN RULE PROPERTY statement e.g. BABA IS YOU to give it a property or as a NOUN IS NOUN statement eg. WALL IS WATER to turn an object into another object.

Sentences can cross. The Esolangs wiki says "multiple rules are able to cross and overlap if the parsing rules allow for a valid statement", so one IS can sit in a row sentence and a column sentence at once.

AND joins words on either side of IS. The AND page calls it "mechanically equivalent to each rule being defined individually", and gives this table row:

BABA AND KEKE IS PUSH AND STOP = BABA IS PUSH, BABA IS STOP, KEKE IS PUSH, KEKE IS STOP

Text can share a tile. The Rule page: "If more than one parsable object is stacked, the game will attempt to produce all possible rules." The game's designer, Arvi "Hempuli" Teikari, told Game Developer that reworking the rule system for stacked words "turned out to be one of the larger undertakings over the game's development, because the stacking adds the need to go through all the possible permutations a rule could have" (Designing Baba Is You's delightfully innovative rule-writing system).

Some rules have no text at all. Again from Rule:

The following rules are enabled by default, even though there isn't any text saying so. TEXT IS PUSH, LEVEL IS STOP, CURSOR IS SELECT. These rules can still be detected by FEELING and MIMIC and negated by NOT and BROKEN. These rules are not listed in the pause menu.

That is why you can push text around although no sentence says so. TEXT is a noun about text itself (TEXT).

And when the rules are worked out. The Rule page: "Rules parse when any parsable object changes position or is destroyed, or an object is transformed to text. Between parsings, active rules are simply stored." Teikari describes the same thing from the inside (same Game Developer article): "The game only updates the rules if it needs to", and when it does, "The previous rules are removed entirely, and words that could potentially start rule sentences are detected on the level."

Read that last quote twice. The game doesn't edit its rule list when a word moves. It throws the list away and reads the board again.

Predict

Write your answers down before you open the discussion, for example in notes/06.md in your workspace. Coordinates are (x, y), with x growing right and y growing down.

1. Text BABA (0,1), IS (1,1), YOU (2,1), FLAG (1,0), WIN (1,2). Draw it. Which rules does it spell?
2. One row: BABA AND ROCK IS PUSH AND STOP. How many rules? Which?
3. Text BABA and text ROCK both on (0,0), then IS (1,0), YOU (2,0). Which rules?
4. One row: YOU IS FLAG. Which rules? And a level with no text at all?
5. Your Module 5 engine stored each card's behaviour on the card. Where would you store "Baba is YOU" here: on each Baba object, on the noun BABA, somewhere else? What has to happen when the YOU text is pushed out of line?
6. Your engine has to know that IS is special. Does it have to know that BABA is a noun?

Discussion

1. BABA IS YOU across the middle row and FLAG IS WIN down the middle column, sharing the IS. Plus TEXT IS PUSH, which is always there.

2. Four: BABA IS PUSH, BABA IS STOP, ROCK IS PUSH, ROCK IS STOP. AND distributes over both sides: two subjects times two complements.

3. BABA IS YOU and ROCK IS YOU. Each stacked word gets its turn as the start of the sentence: "all possible rules".

4. None from the text: a sentence must start with a noun, and YOU is a property. The parser never reads right to left, so it doesn't find FLAG IS YOU either. With no text at all, the rules are exactly TEXT IS PUSH.

5. Nowhere permanent. If "Baba is YOU" were stored on each Baba, pushing YOU away would have to find every Baba and take it back, and a Baba created later would have to look up what it missed. The rule belongs to a table derived from the board. Objects ask the table. When the board changes, the table is rebuilt: the game "removes the previous rules entirely".

6. It shouldn't. IS, AND, NOT are grammar, and the engine must know them to parse at all. BABA, ROCK and FLAG are vocabulary: the parser only needs to know each one's role (noun or property), and that can come from data. That is the Zero-Coupling Law of Part I, applied to words: the engine doesn't name nouns.

Spec

From the course repository, copy the module's tests into your workspace:

node tools/start-module.mjs 06 ~/my-engine

The level format

A level is data, in the format of course/schema/level.schema.json: a size and a list of objects.

{
  "width": 3,
  "height": 3,
  "objects": [
    { "kind": "text:BABA", "x": 0, "y": 1 },
    { "kind": "text:IS", "x": 1, "y": 1 },
    { "kind": "text:YOU", "x": 2, "y": 1 },
    { "kind": "baba", "x": 1, "y": 2 }
  ]
}

• A kind is either an object, named by its noun in lower case ("baba"), or a piece of text carrying one word ("text:BABA"). Text objects are objects like any other; they just happen to be readable.
• x grows right and y grows down. Any number of objects may share a tile.
• The tests build most levels with place rather than from files.

The harness

Part II has its own harness object, Baba, next to Part I's Game:

TypeScript	Rust
newBaba(level: LevelDef): Baba	new_baba(level: LevelDef) -> Baba (free function)
place(kind, x, y): Id	place(&mut self, kind: &str, x: i32, y: i32) -> Id
rules(): string[]	rules(&self) -> Vec<String>

• newBaba builds the level and parses its text straight away.
• place adds one object and returns its id.
• rules() returns every distinct rule in force as a normalised string ("BABA IS YOU"), sorted. It always includes "TEXT IS PUSH". A rule spelled twice on the board appears once. In the real game a stacked noun can make a conversion rule parse more than once, and that duplicates objects (the wiki's Advanced rulebook), so copies are not nothing; but rules() is a set, and no test counts copies.
• In Rust, LevelDef and LevelObject are structs in (or re-exported from) src/harness.rs, and LevelDef must implement Default: the tests write LevelDef { width: 3, height: 3, ..Default::default() }. The exact shape is in course/harness.md.

The tests

The full statements are in course/06-parser/scenarios.md.

• test_06_horizontal_rule: a level file with BABA IS YOU on the top row, a baba object, and YOU IS FLAG on the bottom row. Only BABA IS YOU (and TEXT IS PUSH).
• test_06_crossword_shared_is: Predict question 1, built one word at a time. Before WIN arrives, FLAG above IS is half a sentence; after, the column spells FLAG IS WIN. The rules are recompiled when the board changes.
• test_06_and_distributes: BABA AND ROCK IS PUSH AND STOP, four rules.
• test_06_stacked_text_all_combinations: BABA and ROCK stacked, then IS YOU. Both rules.
• test_06_default_text_is_push: a level with one baba and no text. Exactly TEXT IS PUSH.

The crossword test:

#[test]
fn test_06_crossword_shared_is() {
    let mut b = new_baba(empty_level(3, 3));
    write(
        &mut b,
        &[("BABA", 0, 1), ("IS", 1, 1), ("YOU", 2, 1), ("FLAG", 1, 0)],
    );

    assert_eq!(b.rules(), vec!["BABA IS YOU", "TEXT IS PUSH"]);

    b.place("text:WIN", 1, 2);

    assert_eq!(
        b.rules(),
        vec!["BABA IS YOU", "FLAG IS WIN", "TEXT IS PUSH"]
    );
}

test("test_06_crossword_shared_is", () => {
  const b = newBaba(emptyLevel(3, 3));
  write(b, [
    ["BABA", 0, 1],
    ["IS", 1, 1],
    ["YOU", 2, 1],
    ["FLAG", 1, 0],
  ]);

  expect(b.rules()).toEqual(["BABA IS YOU", "TEXT IS PUSH"]);

  b.place("text:WIN", 1, 2);

  expect(b.rules()).toEqual(["BABA IS YOU", "FLAG IS WIN", "TEXT IS PUSH"]);
});

write places one text:<WORD> object per entry; empty_level/emptyLevel is a level with no objects.

Run cargo test or npm test. The Module 6 tests fail to compile (Rust) or fail (TypeScript): there is no newBaba. Modules 1–5 still pass. That is RED.

Build

Make the five tests pass, and keep Modules 1–5 green. Where code goes:

• A second engine, beside the first. Put Part II in its own directory under src/engine/. It shares nothing with Part I's world yet. Pick its name with care: the coupling audit reads every spelling of every noun in the catalog, so a module called baba names BABA. The reference calls it puzzle.
• Grammar in the engine, vocabulary in content. The engine may name the operators IS, AND and NOT, the meta-noun TEXT, and properties such as PUSH (the base rule needs it). It must not name a noun. The nouns, and the properties the parser accepts, are a word list in src/content/: data, one entry per word with its role. The reference uses a JSON file, src/content/words.json, the same file in both tracks.
• Three stages. The grid stores objects by id with a tile each. The lexer reads the text objects and, with the word list, turns them into words with a role, grouped by tile. Text whose word isn't in the list is left out, so it spells nothing. The parser turns words into rules.
• Compile, don't patch. Store the rule list next to the board. Every place (and, from Module 8, every move) throws it away and parses the whole board again. Levels are small; a full reparse is cheap and can't drift out of date.
• src/harness stays thin: newBaba builds the level with your content's words, place forwards, rules formats, sorts and removes duplicates.

The grammar for this module:

sentence   = subjects IS complements
subjects   = NOUN { AND NOUN }
complements = (NOUN | PROPERTY) { AND (NOUN | PROPERTY) }

read left to right in a row and top to bottom in a column. NOT is a known operator but not part of the grammar yet (Module 7). Where a sentence needs a noun or a property, a NOT stops it, as an unknown word does: BABA IS NOT YOU spells nothing, and in BABA IS YOU AND NOT WIN the run of complements ends at the NOT. In front of a subject, though, nothing looks at it: NOT BABA IS YOU reads as BABA IS YOU, which is wrong. Module 7 makes the parser refuse that sentence.

Hint 1 — nudge

Where does a sentence begin? You could try every noun on the board as a start and read right and down. That works, but BABA AND ROCK IS PUSH then reads twice: once from BABA and once from ROCK. Every sentence has exactly one IS in it. What if you start there and read outwards?

Hint 2 — concept

• Start at every IS, twice: once in its row, once in its column. Each IS is in at most one row sentence and one column sentence, so nothing is read twice.
• Read a run outwards. Before the IS, walking backwards, collect the nouns on the next tile; if the tile after them holds an AND, keep going from the tile after that. After the IS, the same, forwards, with nouns and properties. If the first tile has nothing that fits, the run is empty and the sentence spells nothing.
• Stacking is free. A tile is a list of words. "The nouns on this tile" is a list too, and every one of them goes into the run. Two stacked subjects and one complement are two rules, with no special case.
• One rule per pair. Subjects × complements. That is AND distributing.
• Base rules first. Start the list with TEXT IS PUSH, then add what the text spells.
• Keep the text ids. A rule can remember the three text objects that spelled it. Two copies of BABA IS YOU spelled by different text are two rules; the same text found from two directions can't happen when you start from IS.

Hint 3 — pseudo-code

lex(board, words):
    tiles = map (x, y) -> list of {id, word, role}
    for each text object o, oldest first:
        role = words.role(o.word)          // IS AND NOT: operator, TEXT: noun, else from content
        if role is known: tiles[o.x, o.y].push({o.id, o.word, role})

parse(board, words):
    tiles = lex(board, words)
    rules = [TEXT IS PUSH]
    for each IS token t at (x, y):
        for (dx, dy) in [(1, 0), (0, 1)]:
            subjects    = run(tiles, (x - dx, y - dy), (-dx, -dy), noun)
            complements = run(tiles, (x + dx, y + dy), (dx, dy), noun or property)
            for s in subjects, c in complements:
                rules.push({s.word, c.word, text: [s.id, t.id, c.id]})
    return rules

run(tiles, at, step, fits):
    found = [w in tiles[at] where fits(w)]
    if found is empty: return []
    if tiles[at + step] has an AND:
        found += run(tiles, at + 2 * step, step, fits)
    return found

When your tests pass, compare with the reference engine in course/06-parser/reference/rust/src/engine/puzzle/ or course/06-parser/reference/ts/src/engine/puzzle/: grid (the level format and the board), word (lexicon and lexer), parse and level (the board plus its compiled rules). The words are in src/content/words.json.

Decisions in the reference that the tests don't force:

• rules() deduplicates; the engine doesn't. The engine's list keeps a rule once per set of text objects that spells it, because stacked text really does produce duplicate rules (see Spec). The harness turns that list into a sorted set of strings.
• Unknown words spell nothing. Text whose word isn't in the lexicon is skipped by the lexer, so it breaks any sentence it stands in. That is what makes the Expansion Drop below a data change.
• Content can't redefine grammar. The lexicon starts from content's word list and then sets IS, AND, NOT and TEXT itself.
• A kind is its noun in lower case. "baba" is a thing of noun BABA. The engine never checks that the noun exists: an object of an unknown noun is just an object no rule can name.
• Off the board is an error. place outside the level's width and height panics (Rust) or throws (TypeScript).
• Reparse on every change, including placing a non-text object. The real game only reparses when text moves (Observe), which gives the same rules; the reference trades a little speed for one fewer condition.

How the real game differs

• The real game has more grammar: conditions such as ON, NEAR and FACING, which the wiki's Order of Operations page shows reparsing between movement stages ("conditions (e.g. IT ON BABA IS YOU AND MOVE) reparse in between"), verbs other than IS, such as HAS and WRITE (same page), and NOT (Module 7). The course's grammar is NOUN [AND NOUN]* IS (NOUN|PROPERTY) [AND (NOUN|PROPERTY)]*.
• Where two sentences overlap on an AND, the real game does something subtler: "Overlapping rules with AND in the overlapped part does not apply AND to the right/downward rule" (AND). The course's parser applies AND in both, and no test covers the case.
• The real game has two more base rules, LEVEL IS STOP and CURSOR IS SELECT (Rule). The course has no level object and no cursor.
• Base rules "are not listed in the pause menu" (same page). The course's rules() lists TEXT IS PUSH, because Module 7 negates it and the tests need to see it.
• The real game reparses only when text has moved, been destroyed or been created (same page). The reference reparses on every change to the board.

Name it

Two names, one old and one for this course.

The old one is from compilers: a lexer and a parser. The lexer turns a stream of characters into tokens with a category; the parser turns tokens into structure according to a grammar. Here the "characters" are text objects on a 2D grid, the categories are noun, property and operator, and the structure is a list of rules. The split pays off at once: the parser never looks at an object, only at tokens with a role, and the lexer is the only place that consults the word list. Adding a word touches neither.

The new one is the theme of Part II: rules are compiled from world state. In Part I, a rule was something a card had. Here a rule is something the board says, and the engine's rule table is a function of the board: change the board, recompute the table. Teikari's description of the real game is exactly this: "The previous rules are removed entirely, and words that could potentially start rule sentences are detected on the level" (Game Developer). Nothing in the engine edits a rule in place, so nothing can forget to.

The heart of the reference parser, one IS in two directions:

for (&(x, y), tokens) in &tiles {
    for is in tokens.iter().filter(|t| t.word == IS) {
        for (dx, dy) in [(1, 0), (0, 1)] {
            let subjects = run(&tiles, (x - dx, y - dy), (-dx, -dy), is_subject);
            let complements = run(&tiles, (x + dx, y + dy), (dx, dy), is_complement);
            for s in &subjects {
                for c in &complements {
                    rules.push(Rule {
                        subject: s.word.clone(),
                        complement: c.word.clone(),
                        text: vec![s.id, is.id, c.id],
                    });
                }
            }
        }
    }
}

for (const o of board.objects()) {
  if (o.kind.type !== "text" || o.kind.word !== IS) continue;
  for (const [dx, dy] of [
    [1, 0],
    [0, 1],
  ] as const) {
    const subjects = run(tiles, o.x - dx, o.y - dy, [-dx, -dy], isSubject);
    const complements = run(tiles, o.x + dx, o.y + dy, [dx, dy], isComplement);
    for (const s of subjects) {
      for (const c of complements) {
        rules.push({ subject: s.word, complement: c.word, text: [s.id, o.id, c.id] });
      }
    }
  }
}

The same shape appears in baba-is-auto, a C++ simulator of a subset of the game. Game::ParseRules in Sources/baba-is-auto/Games/Game.cpp (around line 611, commit 2305a3d) starts with m_ruleManager.ClearRules(); and then scans every cell horizontally and vertically; its ARCHITECTURE.md says "Rules are rebuilt after every move, so moving text can immediately change the active rule set." The rest of the simulator asks RuleManager (AddRule, GetRules, HasProperty) rather than the board. That is the split you just built: derive a table from the world, then let every system query the table.

One warning about baba-is-auto: it is a good example of rules as data, and not an authority on the real game. Its own rule-language.md says "all text is pushable without an explicit rule", where the real game has TEXT IS PUSH as a base rule that NOT can switch off, and its NOT support is partial ("Negated subjects, predicates, condition targets, repeated NOT, and nested forms are unsupported"), so it has no IS NOT. Module 7 needs both.

Run the coupling audit again: node tools/coupling-audit.mjs ~/my-engine should print coupling-audit: ok. Your Part II engine knows IS, AND, NOT, TEXT and PUSH. BABA lives only in your word list.

Expansion Drop

A new word arrives: HOT. The HOT page: "HOT is a property that causes the object to destroy any same-float MELT objects it contacts." What HOT does waits for the turn in Module 9. For now your parser only has to read it.

The constraint, as always: do not edit anything under src/engine/. The harness doesn't change either. Only your word list should.

1. Commit your workspace and note the commit: git add -A && git commit -m "module 6 done".
2. Copy the expansion test for your track from course/06-parser/expansion/tests/rust/test_06x.rs or course/06-parser/expansion/tests/ts/test_06x.test.ts into your workspace's tests/. Run it and watch it fail.
3. Add HOT to your word list.
4. When the tests pass, run the check from the course repository:

node tools/expansion-check.mjs ~/my-engine HEAD

The test, test_06x_hot_word, places LAVA IS HOT AND PUSH and expects LAVA IS HOT and LAVA IS PUSH. Before the drop the parser doesn't know HOT: the sentence stops right after IS, and not even LAVA IS PUSH survives, because PUSH is only reached through HOT AND. The full scenario is in course/06-parser/expansion/README.md.

Discussion: what this drop measures

If your nouns and properties are data, HOT is one line:

{ "name": "HOT", "role": "property" }

That is the whole reference answer, in course/06-parser/expansion/solution/words.json (the Module 6 word list with that line added), the same file for both tracks.

Shapes that fail this drop:

• Properties as an enum in the engine. If Property::You | Property::Push | ... lives under src/engine/, HOT is a new variant there. A typed enum feels safer, but every new word becomes an engine change, and the course's catalog alone lists fourteen properties.
• The parser checks words, not roles. If the complement test is word == "YOU" || word == "PUSH" || ..., the list of properties is in the parser. Move it to the lexer's input, and let the parser ask only "noun or property?".
• Unknown words accepted as anything. If your lexer treats every unknown word as a property, the test passes before the drop, and so does BABA IS BANANA. The drop's first run should fail.

If the check fails, fix your engine (a normal commit, outside the drop), then redo the drop from step 1.

Further reading

• The Baba Is You Wiki's Rule and AND pages: reading directions, stacking, base rules, when the game reparses, and AND's edge cases. Read the AND table against your parser. Then TEXT and PUSH for TEXT IS PUSH as a base rule.
• John Harris, Designing Baba Is You's delightfully innovative rule-writing system, Game Developer, 8 May 2019: Teikari on clearing and rebuilding the rules, on stacked text, and on the final pass that adjusts the rule list (which is Module 7).
• baba-is-auto, Game::ParseRules in Sources/baba-is-auto/Games/Game.cpp (around line 611) and RuleManager in Sources/baba-is-auto/Rules/RuleManager.cpp: compare its scan with your parser and its HasProperty with what Module 7 asks of you. Read it as an example of rules as data, not as an authority on the game: it hard-codes text as pushable and has no IS NOT (rule-language.md).