playbenchRepository

Method and verification

Every run follows the same path from a fixed prompt to a single ranked number, an independent evaluator decides whether the level was cleared, and a human operator checks that the script played the game rather than edited it.

Scoring pipeline

The model gets one 60-minute session with a single Bash tool. Only vitexec/best.ts is evaluated, so the session's job is to leave that file runnable at any moment.

  1. Stage 1

    A fixed prompt

    The same rendered prompt for every model, pinned by digest in each run's record.

  2. Stage 2

    One agent session

    60 minutes, one Bash tool, one pinned container. A run that reaches the cap is cut off mid-work and still publishes whatever best.ts held.

  3. Stage 3

    One evaluated file

    Only vitexec/best.ts is scored. Everything else the agent explored is kept as evidence and never counted.

  4. Stage 4

    3 clean replays

    Processes and caches reset before each replay, so the 3 passes never share warm state. Each is capped at 10 minutes and recorded at 60 fps.

  5. Stage 5

    A trusted evaluator

    A separate file reads the game's own completion state. Nothing the model prints is scored.

  6. Stage 6

    One ranked time

    The fastest replay that cleared. Never a median, never an average; a replay that fails contributes nothing but wall time.

Wall-clock allocation

The agent session is capped at 60 minutes. Replays and harness overhead run after it, so a run's total exceeds the cap.

Wall-clock time per run

Agent session and everything after it, on one shared scale.

4 runs

Timings and session totals
ModelAgentReplaysTotalAPI callsExit
google/gemini-3.8-flash high2971.7s117.3s3089.3s171Submitted
meta/muse-spark-1.3 xhigh2616.9s126.9s2744.2s79Submitted
deepseek/deepseek-v4-flash-0731 max3168.0s758.3s3927.0s60Submitted
z-ai/glm-5.3-flash max3600.9s571.1s4172.6s67cut off at the cap

Evaluation

Nothing a model prints is scored. Each game has its own evaluator, pinned by digest in every run's record, which reads the game's own state and returns whether the level was cleared and when.

What the evaluator read · Battle Royale

4 runs

ModelmatchTimeoverphasewonReturned
Google gemini-3.8-flash high16.768399999999808trueovertrue16.768s
Meta muse-spark-1.3 xhigh19.4658999999999trueovertrue19.466s
DeepSeek deepseek-v4-flash-0731 max41.78130000000033trueovertrue41.781s
Z.ai glm-5.3-flash max————null

A run with no clear has nothing to record, so the evaluator returns null and no time is invented for it.

games/battle-royale-1.0.233.ts
import { gameState } from "/src/game/state.ts";

async function setup(): Promise<void> {
  while (!(window as typeof window & { LASTDROP?: unknown }).LASTDROP) {
    await new Promise((resolve) => setTimeout(resolve, 100));
  }
}

async function evaluate(): Promise<{
  cleared: boolean;
  levelTimeMs: number | null;
  evidence: unknown;
}> {
  const cleared = gameState.phase === "over" && gameState.over && gameState.won;
  return {
    cleared,
    levelTimeMs: cleared ? gameState.matchTime * 1000 : null,
    evidence: {
      phase: gameState.phase,
      over: gameState.over,
      won: gameState.won,
      matchTime: gameState.matchTime,
    },
  };
}

Interaction boundary

Observation is unrestricted. Effects are not: Vitexec delivers input as trusted Chromium events, the same events a person at the keyboard would produce. Any other route into game or app state makes the script cheated, whatever it scored.

Permitted and blocked routes

Scope: best.ts only

Reads · unrestricted

Imported game state
The DOM
Its own earlier observations

best.ts

the only evaluated file

Effect boundary
vitexec mouse
vitexec keyboard

battle-royale@1.0.233

Authoritative state

Every other effect route is blocked

Routes that make a script cheated

  • Assigning to imported or game objects
  • Calling mutation methods
  • Calling gameplay actions directly
  • Mutating the DOM or dispatching synthetic events
  • Changing timers or randomness
  • Editing protected source
  • Driving the network or devtools
  • Using another input library

Fast or unusually informed play is not cheating. A human outer operator reviews the evaluated best.ts and nothing else, and records the verdict with its reason. AGENTS.md

Audit verdicts

4 of 4 clean

  • Google gemini-3.8-flash highclean PASS

    best.ts only reads game, ECS, combat, DOM, camera, and raycast state to choose actions. Every app-affecting operation uses Vitexec mouse or keyboard input; it does not call game mutation APIs, dispatch synthetic DOM events, alter protected source, or directly mutate game state.

    Reviewed 2026-09-05 · audit.md· best.ts

  • Meta muse-spark-1.3 xhighclean PASS

    best.ts only reads game and DOM state to choose actions. Every app-affecting operation uses Vitexec mouse or keyboard input; it does not call game mutation APIs, dispatch synthetic DOM events, alter protected source, or directly mutate game state.

    Reviewed 2026-09-05 · audit.md· best.ts

  • DeepSeek deepseek-v4-flash-0731 maxclean PASS

    Clean. best.ts imports game objects only to observe phase, positions, inventory, line of sight, kills, and victory. It affects the app exclusively through Vitexec mouse and keyboard input; it does not call app mutation APIs, dispatch DOM events, write protected source, or directly mutate game state.

    Reviewed 2026-09-04 · audit.md· best.ts

  • Z.ai glm-5.3-flash maxclean DNF

    Clean. best.ts reads game phase, entities, positions, inventory, combat values, and raycasts to choose actions. Every app-affecting action uses Vitexec mouse or keyboard input; it does not call game mutation APIs, dispatch DOM events, alter protected source, or directly mutate game state.

    Reviewed 2026-09-04 · audit.md· best.ts

Operator rulesThe pinned contract