Nadeem Sajjad – Lead Software Engineer

Overview

Note: this is a personal evals/systems experiment, not a primary product. Public chess LLM benchmarks — including Kaggle's Game Arena — already exist; this project isn't positioned to compete with them.

Public chess benchmarks lead with win rate, which hides what actually matters: whether a model can reliably follow the rules it's given, whether its "efficiency" claims hold up once it's doing real work instead of answering a benchmark question, and — under tournament pressure across many runs — which model actually performs best rather than just who won the most individual games.

Chess Arena runs LLMs against each other in full games and knockout tournaments, enforces every rule itself so model output is never trusted blindly, and logs each move with legality, retries, tokens, and cost — producing data a results dashboard can render directly, with zero code changes required to add a new model.

Key Features

LLM vs. LLM matches — full rules enforcement: legality, check/checkmate, castling, en passant, 50-move rule, threefold repetition.

Human vs. model play — synchronous, in-browser games with the same rules and prompt template as the benchmark path.

Single-elimination tournaments — 4-model knockout brackets with configurable best-of-K per match.

Multi-tournament aggregation — run the same bracket 10+ times and roll up championships, average placement, and a composite performance score.

Illegal-move handling — retries up to a hard cap, then an auto-played legal move flagged as "forced" in the log.

Per-move telemetry — legality, retry count, and input/thinking/output tokens logged for every ply.

Live match viewing — the frontend polls the moves table every few seconds while a match is in progress, no websockets needed.

Model-agnostic by config — adding a new model is a config entry, never a code change.

System Architecture

Two independently deployed services share one database rather than one monolith. The Python runner plays LLM-vs-LLM matches and tournaments start-to-finish with no idle time — exactly what a Cloud Run Job is priced for. Human-vs-model play is the opposite: it waits on a person, so it runs synchronously inside a Next.js API route instead of ever being dispatched as a billable job.

Both services connect directly to the same DigitalOcean-managed database. The runner writes moves incrementally as a game is played, which is what makes live viewing possible — the frontend just polls for new rows on the current match.

Chess Arena system architecture: Next.js frontend and Python runner sharing one database — Two Services, One Shared Database

How a Game Works

The board is served to each model as a piece-location matrix — no PGN, no move history — plus turn, castling rights, en passant target, repetition count, and halfmove clock. That last pair matters: since the model gets no move history, it has no way to know it's stalling or repeating unless it's told the current-state facts directly.

Every proposed move is validated against python-chess's legal move list before it's applied — model output is always treated as untrusted input, never trusted on its own claim of legality. An illegal move gets re-prompted, up to a configurable retry cap; if it's still illegal after that, the engine auto-plays a random legal move and flags that ply as forced, so it never silently corrupts the results.

Games end only through real conditions — checkmate, stalemate, the 50-move rule, threefold repetition, or a hard move cap — no engine-based resignation in v1. The shared prompt template is identical across every model and explicitly instructs decisive play, pairing that incentive with the repetition/halfmove signals so a model can actually act on "don't stall."

Tournament Mode

V1 is fixed at a 4-model bracket — two semifinals into a final — with each matchup configurable as best-of-K games to dampen the real move-to-move variance a single LLM game has. Color assignment alternates by bracket position across repeated runs so no model is structurally favored.

Running the same bracket many times over answers a better question than a single knockout can: who actually performs best, not who got the friendlier draw. Each model's composite score leans heavily on average placement (about 90% of the score) with light efficiency penalties for retries and cost — enough to break ties between similarly-placed models without overriding placement outright.

Chess Arena 4-model knockout tournament bracket — 4-Model Knockout — Semifinals to Champion

Challenges & Solutions

Challenge

A model could theoretically claim an illegal move as legal, silently corrupting match integrity.

Solution

Legality is never sourced from the model's own claim — every proposed move is checked against python-chess's actual legal move list, with illegal attempts logged and retried before falling back to a flagged forced move.

Challenge

Human-vs-model play needed the same rules and prompt template as the benchmark path, but couldn't bill for the time a person spends thinking.

Solution

Ran it synchronously inside a Next.js API route instead of dispatching it as a Cloud Run Job — one request/response per move, so nothing sits around billing while a human decides.

Challenge

A single knockout tournament is noisy — one bad bracket draw shouldn't define whether a model is actually good.

Solution

Built a multi-tournament batch runner that repeats the same bracket across N runs and aggregates championships, average placement, and a composite score, so the tournament layer reflects consistency, not one lucky run.

Challenge

Giving the model full move history would help it avoid repetition, but the design intentionally serves only the current board matrix.

Solution

Added explicit repetition-count and halfmove-clock fields to the served position, paired with a prompt instruction toward decisive play — giving the model awareness of stalling without handing it a move log.

Outcome

Chess Arena turns "which model is best at chess" into a question with an actual evidentiary trail — every move, retry, token, and dollar logged, every rule enforced independently of the model, and every ranking backed by repeated runs instead of a single noisy bracket.

The engine is in active development against the architecture above: a Python runner for LLM-vs-LLM batches and tournaments, a Next.js dashboard for live viewing and human play, and a shared database that keeps both in sync without any real-time infrastructure.