feat(database): store gameplay replays in compact form
Build and Push Docker Container / build-and-push (push) Successful in 3m52s

- Add compact storage package: codec, records, hydrate, replay blobs, cache.
- Running games write game_turns and snake_states keyed by integer ids.
- Encode moves as 1-byte codes and bodies as head plus step codes.
- Store food and hazards only when they changed since the previous turn.
- Strip reasoning fields that duplicate stored values and restore on read.
- Pack finished games into one zstd blob per game in game_replays.
- Merge late turns into packed blobs during the stale-game sweep.
- Extract SQLite and PostgreSQL schemas into dedicated modules.
- Add read/import game bundle and list_game_ids to the backend template.
- Add legacy readers and converter for the old turns/snake_turns layout.
- Add migrate_gameplay_to_compact.py with in-place or cross-DB migration.
- Verify every migrated game and keep legacy rows on failure.
- Support --since, --minimum-quality and guarded --drop-legacy.
- Add just recipe migrate-gameplay-postgres.
- Rewrite merge script to copy game by game via backend bundles.
- Move benchmark state loading into benchmark_states for both backends.
- Update PostgreSQL cleanup script to the new replay tables.
- Drop sqlite-zstd build stages from the Dockerfile.
- Remove old migration scripts, normalized_turn and their tests.
- Add tests for compact storage, schema, legacy migration and PostgreSQL.
- Document the compact layout and migration workflow in the README.
This commit is contained in:
2026-11-05 00:02:56 +01:00
parent 4403978537
commit 39a7a53873
41 changed files with 2779 additions and 2493 deletions
+11
View File
@@ -90,6 +90,17 @@ cleanup-gameplay-postgres older_than_days="30" execute="false" vacuum="false":
if [ "{{vacuum}}" = "true" ]; then args+=(--vacuum); fi
PYTHONPATH="{{justfile_directory()}}" uv run python "{{justfile_directory()}}/scripts/cleanup_postgresql_gameplay.py" "${args[@]}"
# Migrate the legacy turns/snake_turns layout to compact storage (in place)
migrate-gameplay-postgres since="" drop_legacy="false":
#!/usr/bin/env bash
set -euo pipefail
: "${GAMEPLAY_DB_PG_DSN:?Set GAMEPLAY_DB_PG_DSN in .env or the environment}"
args=(--source "$GAMEPLAY_DB_PG_DSN")
if [ -n "{{since}}" ]; then args+=(--since "{{since}}"); fi
if [ "{{drop_legacy}}" = "true" ]; then args+=(--drop-legacy); fi
PYTHONPATH="{{justfile_directory()}}" uv run python "{{justfile_directory()}}/scripts/migrate_gameplay_to_compact.py" "${args[@]}"
bench-snake-tournament games="20" gametype="standard" map="standard" output="":
#!/usr/bin/env bash
set -euo pipefail