feat(database): store gameplay replays in compact form
Build and Push Docker Container / build-and-push (push) Successful in 3m52s

- Add compact storage package: codec, records, hydrate, replay blobs, cache.
- Running games write game_turns and snake_states keyed by integer ids.
- Encode moves as 1-byte codes and bodies as head plus step codes.
- Store food and hazards only when they changed since the previous turn.
- Strip reasoning fields that duplicate stored values and restore on read.
- Pack finished games into one zstd blob per game in game_replays.
- Merge late turns into packed blobs during the stale-game sweep.
- Extract SQLite and PostgreSQL schemas into dedicated modules.
- Add read/import game bundle and list_game_ids to the backend template.
- Add legacy readers and converter for the old turns/snake_turns layout.
- Add migrate_gameplay_to_compact.py with in-place or cross-DB migration.
- Verify every migrated game and keep legacy rows on failure.
- Support --since, --minimum-quality and guarded --drop-legacy.
- Add just recipe migrate-gameplay-postgres.
- Rewrite merge script to copy game by game via backend bundles.
- Move benchmark state loading into benchmark_states for both backends.
- Update PostgreSQL cleanup script to the new replay tables.
- Drop sqlite-zstd build stages from the Dockerfile.
- Remove old migration scripts, normalized_turn and their tests.
- Add tests for compact storage, schema, legacy migration and PostgreSQL.
- Document the compact layout and migration workflow in the README.
This commit is contained in:
2026-11-05 00:02:56 +01:00
parent 4403978537
commit 39a7a53873
41 changed files with 2779 additions and 2493 deletions
-32
View File
@@ -1,37 +1,5 @@
# Stage 1 — compile sqlite-zstd + SQLite 3.49.1
FROM ghcr.io/astral-sh/uv:python3.15-trixie-slim AS sqlite-zstd-builder
RUN apt-get update && apt-get install -y --no-install-recommends \
curl unzip git libzstd-dev pkg-config build-essential ca-certificates && \
rm -rf /var/lib/apt/lists/*
# Compile SQLite 3.49.1 as a shared library so the runtime image can use it
RUN curl -fsSL https://www.sqlite.org/2025/sqlite-amalgamation-3490100.zip -o /tmp/sqlite.zip && \
unzip /tmp/sqlite.zip -d /tmp/ && \
cd /tmp/sqlite-amalgamation-3490100 && \
gcc -O2 -shared -fPIC -o /usr/local/lib/libsqlite3.so.0 sqlite3.c -ldl -lpthread
# Install Rust and compile sqlite-zstd (uses bundled SQLite 3.49.1 — no patching needed)
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal
ENV PATH="/root/.cargo/bin:${PATH}"
RUN git clone --depth=1 --branch v0.3.5 https://github.com/phiresky/sqlite-zstd.git /tmp/sqlite-zstd && \
cd /tmp/sqlite-zstd && \
cargo build --release --features build_extension && \
cp target/release/libsqlite_zstd.so /usr/local/lib/libsqlite_zstd.so
# Stage 2 — runtime image
FROM ghcr.io/astral-sh/uv:python3.15-trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends libzstd1 && \
rm -rf /var/lib/apt/lists/*
# Replace system SQLite 3.46.1 with 3.49.1 so it matches the extension
COPY --from=sqlite-zstd-builder /usr/local/lib/libsqlite3.so.0 /usr/local/lib/libsqlite3.so.0
RUN echo "/usr/local/lib" > /etc/ld.so.conf.d/local.conf && ldconfig
COPY --from=sqlite-zstd-builder /usr/local/lib/libsqlite_zstd.so /usr/local/lib/libsqlite_zstd.so
# Install app
COPY . /app
WORKDIR /app