feat(database): store gameplay replays in compact form

- Add compact storage package: codec, records, hydrate, replay blobs, cache.
- Running games write game_turns and snake_states keyed by integer ids.
- Encode moves as 1-byte codes and bodies as head plus step codes.
- Store food and hazards only when they changed since the previous turn.
- Strip reasoning fields that duplicate stored values and restore on read.
- Pack finished games into one zstd blob per game in game_replays.
- Merge late turns into packed blobs during the stale-game sweep.
- Extract SQLite and PostgreSQL schemas into dedicated modules.
- Add read/import game bundle and list_game_ids to the backend template.
- Add legacy readers and converter for the old turns/snake_turns layout.
- Add migrate_gameplay_to_compact.py with in-place or cross-DB migration.
- Verify every migrated game and keep legacy rows on failure.
- Support --since, --minimum-quality and guarded --drop-legacy.
- Add just recipe migrate-gameplay-postgres.
- Rewrite merge script to copy game by game via backend bundles.
- Move benchmark state loading into benchmark_states for both backends.
- Update PostgreSQL cleanup script to the new replay tables.
- Drop sqlite-zstd build stages from the Dockerfile.
- Remove old migration scripts, normalized_turn and their tests.
- Add tests for compact storage, schema, legacy migration and PostgreSQL.
- Document the compact layout and migration workflow in the README.

README.md

@@ -107,8 +107,9 @@ python scripts/benchmark_snakes_from_db.py \
   --samples 100
 ```
 
-The benchmark opens SQLite read-only and reports mean, median, p95, and maximum
-move latency. Increase `--samples` for a broader but slower comparison.
+The benchmark samples every `--stride`-th stored turn from the newest games
+(SQLite is opened read-only; a `postgresql://` DSN works too) and reports mean,
+median, p95, and maximum move latency. Increase `--samples` for a broader but slower comparison.
 
 Run the deterministic CI-friendly arena benchmark without a gameplay database:
 ```sh
@@ -135,39 +136,52 @@ both snake servers with gameplay persistence disabled, adds the engine identity
 header required by the API, and shuts them down when finished.
 
 ## Compact gameplay database
-New gameplay turns use normalized storage: the turn row stores food, hazards,
-move, and thinking data once; snake identity is stored once per game in
-`game_snakes`; and changing snake state/body data lives in `snake_turns`. Replay
-loading rebuilds the normal Battlesnake board payload.
+Gameplay storage keeps game results in `games` and snake identity (name,
+customizations) once per game in `game_snakes`. Replay data is stored compactly:
 
-Create and verify a separate compact copy of an existing SQLite database. By
-default, replay-heavy rows are retained for games rated `medium` or `high` by
+- **Running games** write one `game_turns` row per turn and one `snake_states`
+  row per snake and turn, keyed by small integer ids (`games.gid`,
+  `game_snakes.gsid`) instead of repeating text ids. Moves are 1-byte codes,
+  latency is an integer, bodies are a head position plus one step code per
+  segment, and food/hazards are only stored when they changed since the previous
+  turn. Reasoning fields that equal stored values (`turn`, `move`, `head`,
+  `health`, `length`, `food_count`, `board_size`, `snakes`) are stripped and
+  restored exactly on read.
+- **Finished games** are packed into one zstd-compressed blob per game in
+  `game_replays`, and their rows are deleted. Turns arriving after a game was
+  packed are merged into the blob by the stale-game sweep.
+
+Replay loading rebuilds the normal Battlesnake board payload, so the dashboard
+is unchanged. Head position, length, and inferred moves are derived from the
+stored bodies.
+
+### Migrate the legacy layout
+Databases written by older versions store replays in `turns`/`snake_turns`.
+Migrate them in place (SQLite path or `postgresql://` DSN). Every game is read
+back and compared after writing; failed games are reported and keep their
+legacy rows:
+```sh
+python scripts/migrate_gameplay_to_compact.py --source /path/to/gameplay.sqlite3
+python scripts/migrate_gameplay_to_compact.py --source /path/to/gameplay.sqlite3 --drop-legacy
+```
+
+`--destination` writes to a different database instead, for example from
+SQLite into PostgreSQL. `--minimum-quality medium` re-rates replays by
 structural completeness, valid moves, thinking coverage, game length, move
-diversity, opponent data, and terminal outcome. All game-result rows remain
-stored, so historical win/loss rates stay persistent when low-quality replay
-data is removed.
-```sh
-python scripts/migrate_gameplay_database.py \
-  --source /path/to/gameplay.sqlite3 \
-  --destination /path/to/gameplay.compact.sqlite3 \
-  --minimum-quality medium
-```
+diversity, opponent data, and terminal outcome, and keeps only replays of that
+tier or better. All game-result rows remain stored, so historical win/loss
+rates stay persistent. `--drop-legacy` only drops the old tables when every
+legacy replay exists in compact form.
 
-After reviewing the compact copy, `--replace` renames the original to a
-timestamped backup and puts the verified compact database at the original path.
-Stop all writers before using it:
-```sh
-python scripts/migrate_gameplay_database.py \
-  --source /path/to/gameplay.sqlite3 \
-  --replace
-```
-
-The migration never modifies the source in place. It verifies row counts and
-runs SQLite's `integrity_check` before any replacement.
+For a live PostgreSQL database: run the migration while the old version keeps
+recording, deploy the new version, run the migration again with
+`--since <start time of the first run>` to pick up games recorded meanwhile,
+then run `--drop-legacy`. A PostgreSQL server configured with a SQLite
+`GAMEPLAY_DB_PATH` imports that legacy file automatically on startup.
 
 ### Record new games while cleanup runs
 Point the running server at a temporary delta database while the old database
-is being compacted. After stopping the writer and flushing the delta database,
+is being migrated or cleaned. After stopping the writer and flushing the delta database,
 merge it into the cleaned copy:
 ```sh
 python scripts/merge_gameplay_databases.py \
@@ -177,8 +191,9 @@ python scripts/merge_gameplay_databases.py \
   --minimum-quality medium
 ```
 
-The merger keeps all game results, quality-rates delta games, regenerates
-numeric turn IDs, and verifies row counts, foreign keys, and database integrity.
+The merger copies game by game, keeps all game results, quality-rates delta
+games, assigns fresh integer ids, and verifies game, snake, turn, and snake-state
+counts.
 Identical game IDs are skipped; conflicting duplicates abort the merge. After
 reviewing the result, `--replace-base` backs up and replaces the cleaned base.
 Stop the delta writer before the final merge and file swap.