Arena Workflow

conceptdevelopingCreated: 2026-07-25Updated: 2026-07-25
blog-empireworkflowarenabuilder-vs-judgereference

Arena Workflow

The arena workflow is the model-vs-model comparison pipeline used across the AI Empire. It pits multiple models (builders) against each other on the same task, then uses a separate model (judge) to evaluate and score the outputs.

Architecture

The arena operates as a subagent pipeline — each model is invoked as an independent subagent on the same prompt, and their outputs are collected for comparative evaluation.

Builder Models

Builder models generate content for the given task. Each model receives the same prompt with no format constraints (in ToolReview mode) or with a template (in standard mode). Notable builder models include:

  • DeepSeek V4 Flash — Default builder for code/tool tasks
  • GLM-5.2 — Best for visual design tasks (1347 Elo in design arena)
  • Xiaomi MiMo — Creative and quick tasks
  • GPT-OSS 20B/120B — Open-source models via Groq

Judge Models

Judge models evaluate builder outputs against a rubric. Free models from OpenRouter are preferred for judging to minimize costs:

  • Gemma 4 26B (Google, OpenRouter free)
  • Nemotron 3 Super (Nvidia, OpenRouter free)
  • Nemotron 3 Nano 30B (Nvidia, OpenRouter free)
  • GPT-OSS 20B (OpenRouter free)

Policy: Do NOT route paid DeepSeek models through OpenRouter. Use DeepSeek direct API. Only free OpenRouter models (e.g. nvidia/nemotron:free) are acceptable for judges.

Arena Modes

ToolReview Mode

Models receive an open-ended prompt ("choose your own format") — no template constraints on structure or length. The judge rubric measures UI and format quality alongside content:

Dimension Weight
Content 20%
Depth 20%
Sourcing 15%
Writing 15%
Format 15%
Image Prompt 15%

Each model provides a HERO_IMAGE_PROMPT line in its output. The runner extracts this, strips it from published content, and passes it to the judge. The winner's image prompt is used for hero image generation; the loser's prompt is included as reference.

Standard Arena Mode

Models receive a structured template with specific format requirements. The judge evaluates against content quality metrics.

Scoring

  • Judges score each builder output on the rubric dimensions
  • Scores are aggregated for leaderboard ranking
  • Elo ratings are computed for design arena (intelligence.ai/leaderboard)
  • Results feed into Leaderboard pages

Pipeline Flow

1. Task prompt prepared
2. Subagent models invoked (in parallel or sequence)
3. Builder outputs collected
4. Judge model scores each output
5. Scores aggregated and compared
6. Winner determined
7. Hero image generated from winner's prompt
8. Results published as blog post or arena page

Implementation

The arena is implemented via:

  • arena-runner.py — Main pipeline orchestrator
  • swiss-runner.py — Swiss-tournament style runner for ToolReview mode
  • Per-blog arena generation scripts — e.g., arena-codeintel-generate.py

Blog Coverage

As of the Jul 2026 audit:

Related