# Module 6: Grid, lexer and parser
This is the first module of Part II. In Part I the rules lived inside the cards: Harvest Golem's Deathrattle was a clause on Harvest Golem, and your engine's job was to find the right clause at the right moment and run it. Part II turns that around. In Baba Is You the objects on the board have no behaviour at all. A rock is just a rock until some text on the board says `ROCK IS PUSH`. Every rule comes from words lying on the grid, and the words can be pushed around, so the rules change while you play. The engine's job is now to *compile* the rules from the board, and to compile them again every time the board changes.
This module builds the compiler's front end: a grid that holds objects and text, a lexer that turns text into words with grammatical roles, and a parser that turns words into rules. Nothing moves yet (that is Module 8), and no rule does anything yet (Modules 7 and 9). By the end, `rules()` reads any board you build.
You need your Module 5 workspace with all Module 1–5 tests passing, under git. Part I's engine stays where it is, and its tests keep running: Part II is a second engine next to the first, in the same project. Part III will plug them together.
## Observe
All quotes in this module come from sources recorded in the course's research notes. The Baba Is You Wiki is an unofficial community wiki, and the course treats it as the best available description of the game, not as Hempuli's own word.
How a sentence reads, from the wiki's [Rule](https://babaiswiki.fandom.com/wiki/Rule) page:
> Rules read from left to right in a row, or from top to bottom in a column.
A noun can take a property or another noun ([Category:Nouns](https://babaiswiki.fandom.com/wiki/Category:Nouns)):
> A noun can be used as a NOUN RULE PROPERTY statement e.g. BABA IS YOU to give it a property or as a NOUN IS NOUN statement eg. WALL IS WATER to turn an object into another object.
Sentences can cross. The [Esolangs wiki](https://esolangs.org/wiki/Baba_Is_You) says "multiple rules are able to cross and overlap if the parsing rules allow for a valid statement", so one `IS` can sit in a row sentence and a column sentence at once.
`AND` joins words on either side of `IS`. The [AND](https://babaiswiki.fandom.com/wiki/AND) page calls it "mechanically equivalent to each rule being defined individually", and gives this table row:
> BABA AND KEKE IS PUSH AND STOP = BABA IS PUSH, BABA IS STOP, KEKE IS PUSH, KEKE IS STOP
Text can share a tile. The [Rule](https://babaiswiki.fandom.com/wiki/Rule) page: "If more than one parsable object is stacked, the game will attempt to produce all possible rules." The game's designer, Arvi "Hempuli" Teikari, told Game Developer that this was one of the hard parts: "the stacking adds the need to go through all the possible permutations a rule could have" ([Designing Baba Is You's delightfully innovative rule-writing system](https://www.gamedeveloper.com/design/designing-i-baba-is-you-i-s-delightfully-innovative-rule-writing-system)).
Some rules have no text at all. Again from [Rule](https://babaiswiki.fandom.com/wiki/Rule):
> The following rules are enabled by default, even though there isn't any text saying so. TEXT IS PUSH, LEVEL IS STOP, CURSOR IS SELECT. These rules can still be detected by FEELING and MIMIC and negated by NOT and BROKEN. These rules are not listed in the pause menu.
That is why you can push text around although no sentence says so. `TEXT` is a noun about text itself ([TEXT](https://babaiswiki.fandom.com/wiki/TEXT)).
And when the rules are worked out. The [Rule](https://babaiswiki.fandom.com/wiki/Rule) page: "Rules parse when any parsable object changes position or is destroyed, or an object is transformed to text. Between parsings, active rules are simply stored." Teikari describes the same thing from the inside (same Game Developer article): "The game only updates the rules if it needs to", and when it does, "The previous rules are removed entirely, and words that could potentially start rule sentences are detected on the level."
Read that last quote twice. The game doesn't edit its rule list when a word moves. It throws the list away and reads the board again.
## Predict
Write your answers down before you open the discussion, for example in `notes/06.md` in your workspace. Coordinates are (x, y), with x growing right and y growing down.
1. Text `BABA` (0,1), `IS` (1,1), `YOU` (2,1), `FLAG` (1,0), `WIN` (1,2). Draw it. Which rules does it spell?
2. One row: `BABA AND ROCK IS PUSH AND STOP`. How many rules? Which?
3. Text `BABA` and text `ROCK` both on (0,0), then `IS` (1,0), `YOU` (2,0). Which rules?
4. One row: `YOU IS FLAG`. Which rules? And a level with no text at all?
5. Your Module 5 engine stored each card's behaviour on the card. Where would you store "Baba is YOU" here: on each Baba object, on the noun `BABA`, somewhere else? What has to happen when the `YOU` text is pushed out of line?
6. Your engine has to know that `IS` is special. Does it have to know that `BABA` is a noun?
Discussion
1. `BABA IS YOU` across the middle row and `FLAG IS WIN` down the middle column, sharing the `IS`. Plus `TEXT IS PUSH`, which is always there.
2. Four: `BABA IS PUSH`, `BABA IS STOP`, `ROCK IS PUSH`, `ROCK IS STOP`. AND distributes over both sides: two subjects times two complements.
3. `BABA IS YOU` and `ROCK IS YOU`. Each stacked word gets its turn as the start of the sentence: "all possible rules".
4. None from the text: a sentence must start with a noun, and `YOU` is a property. The parser never reads right to left, so it doesn't find `FLAG IS YOU` either. With no text at all, the rules are exactly `TEXT IS PUSH`.
5. Nowhere permanent. If "Baba is YOU" were stored on each Baba, pushing `YOU` away would have to find every Baba and take it back, and a Baba created later would have to look up what it missed. The rule belongs to a table derived from the board. Objects ask the table. When the board changes, the table is rebuilt: the game "removes the previous rules entirely".
6. It shouldn't. `IS`, `AND`, `NOT` are grammar, and the engine must know them to parse at all. `BABA`, `ROCK` and `FLAG` are vocabulary: the parser only needs to know each one's *role* (noun or property), and that can come from data. That is the Zero-Coupling Law of Part I, applied to words: the engine doesn't name nouns.
## Spec
From the course repository, copy the module's tests into your workspace:
```sh
node tools/start-module.mjs 06 ~/my-engine
```
### The level format
A level is data, in the format of `course/schema/level.schema.json`: a size and a list of objects.
```json
{
"width": 3,
"height": 3,
"objects": [
{ "kind": "text:BABA", "x": 0, "y": 1 },
{ "kind": "text:IS", "x": 1, "y": 1 },
{ "kind": "text:YOU", "x": 2, "y": 1 },
{ "kind": "baba", "x": 1, "y": 2 }
]
}
```
- A kind is either an object, named by its noun in lower case (`"baba"`), or a piece of text carrying one word (`"text:BABA"`). Text objects are objects like any other; they just happen to be readable.
- `x` grows right and `y` grows down. Any number of objects may share a tile.
- The tests build most levels with `place` rather than from files.
### The harness
Part II has its own harness object, `Baba`, next to Part I's `Game`:
| TypeScript | Rust |
|---|---|
| `newBaba(level: LevelDef): Baba` | `new_baba(level: LevelDef) -> Baba` (free function) |
| `place(kind, x, y): Id` | `place(&mut self, kind: &str, x: i32, y: i32) -> Id` |
| `rules(): string[]` | `rules(&self) -> Vec` |
- `newBaba` builds the level and parses its text straight away.
- `place` adds one object and returns its id.
- `rules()` returns every distinct rule in force as a normalised string (`"BABA IS YOU"`), sorted. It always includes `"TEXT IS PUSH"`. A rule spelled twice on the board appears once. In the real game a stacked noun can make a conversion rule parse more than once, and that duplicates objects (the wiki's [Advanced rulebook](https://babaiswiki.fandom.com/wiki/Advanced_rulebook)), so copies are not nothing; but `rules()` is a set, and no test counts copies.
- In Rust, `LevelDef` and `LevelObject` are structs in (or re-exported from) `src/harness.rs`, and `LevelDef` must implement `Default`: the tests write `LevelDef { width: 3, height: 3, ..Default::default() }`. The exact shape is in `course/harness.md`.
### The tests
The full statements are in `course/06-parser/scenarios.md`.
- `test_06_horizontal_rule`: a level file with `BABA IS YOU` on the top row, a `baba` object, and `YOU IS FLAG` on the bottom row. Only `BABA IS YOU` (and `TEXT IS PUSH`).
- `test_06_crossword_shared_is`: Predict question 1, built one word at a time. Before `WIN` arrives, `FLAG` above `IS` is half a sentence; after, the column spells `FLAG IS WIN`. The rules are recompiled when the board changes.
- `test_06_and_distributes`: `BABA AND ROCK IS PUSH AND STOP`, four rules.
- `test_06_stacked_text_all_combinations`: `BABA` and `ROCK` stacked, then `IS YOU`. Both rules.
- `test_06_default_text_is_push`: a level with one `baba` and no text. Exactly `TEXT IS PUSH`.
The crossword test:
```rust
#[test]
fn test_06_crossword_shared_is() {
let mut b = new_baba(empty_level(3, 3));
write(
&mut b,
&[("BABA", 0, 1), ("IS", 1, 1), ("YOU", 2, 1), ("FLAG", 1, 0)],
);
assert_eq!(b.rules(), vec!["BABA IS YOU", "TEXT IS PUSH"]);
b.place("text:WIN", 1, 2);
assert_eq!(
b.rules(),
vec!["BABA IS YOU", "FLAG IS WIN", "TEXT IS PUSH"]
);
}
```
```ts
test("test_06_crossword_shared_is", () => {
const b = newBaba(emptyLevel(3, 3));
write(b, [
["BABA", 0, 1],
["IS", 1, 1],
["YOU", 2, 1],
["FLAG", 1, 0],
]);
expect(b.rules()).toEqual(["BABA IS YOU", "TEXT IS PUSH"]);
b.place("text:WIN", 1, 2);
expect(b.rules()).toEqual(["BABA IS YOU", "FLAG IS WIN", "TEXT IS PUSH"]);
});
```
`write` places one `text:` object per entry; `empty_level`/`emptyLevel` is a level with no objects.
Run `cargo test` or `npm test`. The Module 6 tests fail to compile (Rust) or fail (TypeScript): there is no `newBaba`. Modules 1–5 still pass. That is RED.
## Build
Make the five tests pass, and keep Modules 1–5 green. Where code goes:
- **A second engine, beside the first.** Put Part II in its own directory under `src/engine/`. It shares nothing with Part I's world yet. Pick its name with care: the coupling audit reads every spelling of every noun in the catalog, so a module called `baba` names `BABA`. The reference calls it `puzzle`.
- **Grammar in the engine, vocabulary in content.** The engine may name the operators `IS`, `AND` and `NOT`, the meta-noun `TEXT`, and properties such as `PUSH` (the base rule needs it). It must not name a noun. The nouns, and the properties the parser accepts, are a word list in `src/content/`: data, one entry per word with its role. The reference uses a JSON file, `src/content/words.json`, the same file in both tracks.
- **Three stages.** The *grid* stores objects by id with a tile each. The *lexer* reads the text objects and, with the word list, turns them into words with a role, grouped by tile. Text whose word isn't in the list is left out, so it spells nothing. The *parser* turns words into rules.
- **Compile, don't patch.** Store the rule list next to the board. Every `place` (and, from Module 8, every move) throws it away and parses the whole board again. Levels are small; a full reparse is cheap and can't drift out of date.
- `src/harness` stays thin: `newBaba` builds the level with your content's words, `place` forwards, `rules` formats, sorts and removes duplicates.
The grammar for this module:
```text
sentence = subjects IS complements
subjects = NOUN { AND NOUN }
complements = (NOUN | PROPERTY) { AND (NOUN | PROPERTY) }
```
read left to right in a row and top to bottom in a column. `NOT` is a known operator but not part of the grammar yet (Module 7). Where a sentence needs a noun or a property, a `NOT` stops it, as an unknown word does: `BABA IS NOT YOU` spells nothing, and in `BABA IS YOU AND NOT WIN` the run of complements ends at the `NOT`. In front of a subject, though, nothing looks at it: `NOT BABA IS YOU` reads as `BABA IS YOU`, which is wrong. Module 7 makes the parser refuse that sentence.
Hint 1 — nudge
Where does a sentence *begin*? You could try every noun on the board as a start and read right and down. That works, but `BABA AND ROCK IS PUSH` then reads twice: once from `BABA` and once from `ROCK`. Every sentence has exactly one `IS` in it. What if you start there and read outwards?
Hint 2 — concept
- **Start at every `IS`, twice**: once in its row, once in its column. Each `IS` is in at most one row sentence and one column sentence, so nothing is read twice.
- **Read a run outwards.** Before the `IS`, walking backwards, collect the nouns on the next tile; if the tile after them holds an `AND`, keep going from the tile after that. After the `IS`, the same, forwards, with nouns and properties. If the first tile has nothing that fits, the run is empty and the sentence spells nothing.
- **Stacking is free.** A tile is a list of words. "The nouns on this tile" is a list too, and every one of them goes into the run. Two stacked subjects and one complement are two rules, with no special case.
- **One rule per pair.** Subjects × complements. That is AND distributing.
- **Base rules first.** Start the list with `TEXT IS PUSH`, then add what the text spells.
- **Keep the text ids.** A rule can remember the three text objects that spelled it. Two copies of `BABA IS YOU` spelled by different text are two rules; the same text found from two directions can't happen when you start from `IS`.
Hint 3 — pseudo-code
```text
lex(board, words):
tiles = map (x, y) -> list of {id, word, role}
for each text object o, oldest first:
role = words.role(o.word) // IS AND NOT: operator, TEXT: noun, else from content
if role is known: tiles[o.x, o.y].push({o.id, o.word, role})
parse(board, words):
tiles = lex(board, words)
rules = [TEXT IS PUSH]
for each IS token t at (x, y):
for (dx, dy) in [(1, 0), (0, 1)]:
subjects = run(tiles, (x - dx, y - dy), (-dx, -dy), noun)
complements = run(tiles, (x + dx, y + dy), (dx, dy), noun or property)
for s in subjects, c in complements:
rules.push({s.word, c.word, text: [s.id, t.id, c.id]})
return rules
run(tiles, at, step, fits):
found = [w in tiles[at] where fits(w)]
if found is empty: return []
if tiles[at + step] has an AND:
found += run(tiles, at + 2 * step, step, fits)
return found
```
When your tests pass, compare with the reference engine in `course/06-parser/reference/rust/src/engine/puzzle/` or `course/06-parser/reference/ts/src/engine/puzzle/`: `grid` (the level format and the board), `word` (lexicon and lexer), `parse` and `level` (the board plus its compiled rules). The words are in `src/content/words.json`.
Decisions in the reference that the tests don't force:
- **`rules()` deduplicates; the engine doesn't.** The engine's list keeps a rule once per set of text objects that spells it, because stacked text really does produce duplicate rules (see Spec). The harness turns that list into a sorted set of strings.
- **Unknown words spell nothing.** Text whose word isn't in the lexicon is skipped by the lexer, so it breaks any sentence it stands in. That is what makes the Expansion Drop below a data change.
- **Content can't redefine grammar.** The lexicon starts from content's word list and then sets `IS`, `AND`, `NOT` and `TEXT` itself.
- **A kind is its noun in lower case.** `"baba"` is a thing of noun `BABA`. The engine never checks that the noun exists: an object of an unknown noun is just an object no rule can name.
- **Off the board is an error.** `place` outside the level's width and height panics (Rust) or throws (TypeScript).
- **Reparse on every change**, including placing a non-text object. The real game only reparses when text moves (Observe), which gives the same rules; the reference trades a little speed for one fewer condition.
> **How the real game differs**
> - The real game has more grammar: conditions such as `ON`, `NEAR` and `FACING`, which the wiki's [Order of Operations](https://babaiswiki.fandom.com/wiki/Order_of_Operations) page shows reparsing between movement stages ("conditions (e.g. IT ON BABA IS YOU AND MOVE) reparse in between"), verbs other than `IS`, such as `HAS` and `WRITE` (same page), and `NOT` (Module 7). The course's grammar is `NOUN [AND NOUN]* IS (NOUN|PROPERTY) [AND (NOUN|PROPERTY)]*`.
> - Where two sentences overlap on an `AND`, the real game does something subtler: "Overlapping rules with AND in the overlapped part does not apply AND to the right/downward rule" ([AND](https://babaiswiki.fandom.com/wiki/AND)). The course's parser applies AND in both, and no test covers the case.
> - The real game has two more base rules, `LEVEL IS STOP` and `CURSOR IS SELECT` ([Rule](https://babaiswiki.fandom.com/wiki/Rule)). The course has no level object and no cursor.
> - Base rules "are not listed in the pause menu" (same page). The course's `rules()` lists `TEXT IS PUSH`, because Module 7 negates it and the tests need to see it.
> - The real game reparses only when text has moved, been destroyed or been created (same page). The reference reparses on every change to the board.
## Name it
Two names, one old and one for this course.
The old one is from compilers: a **lexer** and a **parser**. The lexer turns a stream of characters into tokens with a category; the parser turns tokens into structure according to a grammar. Here the "characters" are text objects on a 2D grid, the categories are noun, property and operator, and the structure is a list of rules. The split pays off at once: the parser never looks at an object, only at tokens with a role, and the lexer is the only place that consults the word list. Adding a word touches neither.
The new one is the theme of Part II: **rules are compiled from world state.** In Part I, a rule was something a card *had*. Here a rule is something the board *says*, and the engine's rule table is a function of the board: change the board, recompute the table. Teikari's description of the real game is exactly this: "The previous rules are removed entirely, and words that could potentially start rule sentences are detected on the level" ([Game Developer](https://www.gamedeveloper.com/design/designing-i-baba-is-you-i-s-delightfully-innovative-rule-writing-system)). Nothing in the engine edits a rule in place, so nothing can forget to.
The heart of the reference parser, one `IS` in two directions:
```rust
for (&(x, y), tokens) in &tiles {
for is in tokens.iter().filter(|t| t.word == IS) {
for (dx, dy) in [(1, 0), (0, 1)] {
let subjects = run(&tiles, (x - dx, y - dy), (-dx, -dy), is_subject);
let complements = run(&tiles, (x + dx, y + dy), (dx, dy), is_complement);
for s in &subjects {
for c in &complements {
rules.push(Rule {
subject: s.word.clone(),
complement: c.word.clone(),
text: vec![s.id, is.id, c.id],
});
}
}
}
}
}
```
```ts
for (const o of board.objects()) {
if (o.kind.type !== "text" || o.kind.word !== IS) continue;
for (const [dx, dy] of [
[1, 0],
[0, 1],
] as const) {
const subjects = run(tiles, o.x - dx, o.y - dy, [-dx, -dy], isSubject);
const complements = run(tiles, o.x + dx, o.y + dy, [dx, dy], isComplement);
for (const s of subjects) {
for (const c of complements) {
rules.push({ subject: s.word, complement: c.word, text: [s.id, o.id, c.id] });
}
}
}
}
```
The same shape appears in [baba-is-auto](https://github.com/utilForever/baba-is-auto), a C++ simulator of a subset of the game. `Game::ParseRules` in [`Sources/baba-is-auto/Games/Game.cpp`](https://github.com/utilForever/baba-is-auto/blob/2305a3d/Sources/baba-is-auto/Games/Game.cpp) (around line 611, commit `2305a3d`) starts with `m_ruleManager.ClearRules();` and then scans every cell horizontally and vertically; its [`ARCHITECTURE.md`](https://github.com/utilForever/baba-is-auto/blob/main/ARCHITECTURE.md) says "Rules are rebuilt after every move, so moving text can immediately change the active rule set." The rest of the simulator asks `RuleManager` (`AddRule`, `GetRules`, `HasProperty`) rather than the board. That is the split you just built: derive a table from the world, then let every system query the table.
One warning about baba-is-auto: it is a good example of rules as data, and **not an authority** on the real game. Its own [`rule-language.md`](https://github.com/utilForever/baba-is-auto/blob/main/Documents/rule-language.md) says "all text is pushable without an explicit rule", where the real game has `TEXT IS PUSH` as a base rule that `NOT` can switch off, and its `NOT` support is partial ("Negated subjects, predicates, condition targets, repeated `NOT`, and nested forms are unsupported"), so it has no `IS NOT`. Module 7 needs both.
**Run the coupling audit** again: `node tools/coupling-audit.mjs ~/my-engine` should print `coupling-audit: ok`. Your Part II engine knows `IS`, `AND`, `NOT`, `TEXT` and `PUSH`. `BABA` lives only in your word list.
## Expansion Drop
A new word arrives: `HOT`. The [HOT](https://babaiswiki.fandom.com/wiki/HOT) page: "HOT is a property that causes the object to destroy any same-float MELT objects it contacts." What HOT *does* waits for the turn in Module 9. For now your parser only has to read it.
The constraint, as always: **do not edit anything under `src/engine/`**. The harness doesn't change either. Only your word list should.
1. Commit your workspace and note the commit: `git add -A && git commit -m "module 6 done"`.
2. Copy the expansion test for your track from `course/06-parser/expansion/tests/rust/test_06x.rs` or `course/06-parser/expansion/tests/ts/test_06x.test.ts` into your workspace's `tests/`. Run it and watch it fail.
3. Add `HOT` to your word list.
4. When the tests pass, run the check from the course repository:
```sh
node tools/expansion-check.mjs ~/my-engine HEAD
```
The test, `test_06x_hot_word`, places `LAVA IS HOT AND PUSH` and expects `LAVA IS HOT` and `LAVA IS PUSH`. Before the drop the parser doesn't know `HOT`: the sentence stops right after `IS`, and not even `LAVA IS PUSH` survives, because `PUSH` is only reached through `HOT AND`. The full scenario is in `course/06-parser/expansion/README.md`.
Discussion: what this drop measures
If your nouns and properties are data, `HOT` is one line:
```json
{ "name": "HOT", "role": "property" }
```
That is the whole reference answer, in `course/06-parser/expansion/solution/words.json` (the Module 6 word list with that line added), the same file for both tracks.
Shapes that fail this drop:
- **Properties as an enum in the engine.** If `Property::You | Property::Push | ...` lives under `src/engine/`, HOT is a new variant there. A typed enum feels safer, but every new word becomes an engine change, and the course's catalog alone lists fourteen properties.
- **The parser checks words, not roles.** If the complement test is `word == "YOU" || word == "PUSH" || ...`, the list of properties is in the parser. Move it to the lexer's input, and let the parser ask only "noun or property?".
- **Unknown words accepted as anything.** If your lexer treats every unknown word as a property, the test passes before the drop, and so does `BABA IS BANANA`. The drop's first run should fail.
If the check fails, fix your engine (a normal commit, outside the drop), then redo the drop from step 1.
## Further reading
- The Baba Is You Wiki's [Rule](https://babaiswiki.fandom.com/wiki/Rule) and [AND](https://babaiswiki.fandom.com/wiki/AND) pages: reading directions, stacking, base rules, when the game reparses, and AND's edge cases. Read the AND table against your parser. Then [TEXT](https://babaiswiki.fandom.com/wiki/TEXT) and [PUSH](https://babaiswiki.fandom.com/wiki/PUSH) for `TEXT IS PUSH` as a base rule.
- John Harris, [*Designing Baba Is You's delightfully innovative rule-writing system*](https://www.gamedeveloper.com/design/designing-i-baba-is-you-i-s-delightfully-innovative-rule-writing-system), Game Developer, 8 May 2019: Teikari on clearing and rebuilding the rules, on stacked text, and on the final pass that adjusts the rule list (which is Module 7).
- baba-is-auto, `Game::ParseRules` in [`Sources/baba-is-auto/Games/Game.cpp`](https://github.com/utilForever/baba-is-auto/blob/2305a3d/Sources/baba-is-auto/Games/Game.cpp) (around line 611) and `RuleManager` in [`Sources/baba-is-auto/Rules/RuleManager.cpp`](https://github.com/utilForever/baba-is-auto/blob/2305a3d/Sources/baba-is-auto/Rules/RuleManager.cpp): compare its scan with your parser and its `HasProperty` with what Module 7 asks of you. Read it as an example of rules as data, not as an authority on the game: it hard-codes text as pushable and has no `IS NOT` ([`rule-language.md`](https://github.com/utilForever/baba-is-auto/blob/main/Documents/rule-language.md)).