#spark-bench

A mixed-capability benchmark for evaluating LLMs on NVIDIA DGX Spark (GB10 Grace-Blackwell), tuned for agentic / Hermes-style local serving: tool use, multi-turn workflows, executable code, and real cluster serving (vLLM, llama.cpp, multi-node).

Current methodology: v6.8.0 Uncapped · 76 scenarios · 12 domains · 12 agentic workflows · 2 rendered 3D-animation scenarios

Scope: This is a DGX Spark operator bench, not a universal LMSYS-style ranking. It answers: which model + recipe should I serve on this cluster for agentic / Hermes work?

---

#Leaderboard

#Latest v6.8.0 uncapped comparison

Qwen3.8-Flash-Next spec-off scored 87.02 vs GLM-5.3-Flash DFlash2 K7 at 85.67 across the full 76-scenario, two-repeat contract. The audited package, including all per-repeat scores, corrected context-length failures, transcript hashes, and the excluded Qwen NEXTN A/B, is at results/comparisons/2026-08-29-qwen38-flash-next-vs-glm53-flash-v680.

The live leaderboard (wesche.com/dgx) shows only the 29 qualified runs from the current one-Spark v6.7.1-challenge cohort. The complete 40-attempt evidence pack, including quarantine and preflight failures, is in results/cohorts/2026-08-01-one-spark-v671. The master results/spark_bench.csv retains older runs for provenance, but scores from different methodology versions are not comparable.

#Current one-Spark cohort

The full 29-row ranking is in results-qualified.tsv. All rows use the same 20-scenario, 10-domain, three-repeat, thinking-off, single-Spark TP1 contract at grader 11d21bf.

Models are grouped by size tier. Within each tier, models compete against peers of similar capacity.

TrueScore weights (unchanged through v5c → v6.7.1): Quality 55%, Calibration 25%, Reliability 15%, Efficiency 1.5%, Responsiveness 3.5% (speed total: 5%)

Calibration measures prompt injection resistance, robustness, and over-refusal of legitimate requests. Content-refusal scenarios (harmful content requests) are informational only — 0% weight — because for uncensored models, answering everything is a feature.

Agentic (v6.7.1): 12 multi-turn scenarios. Expert tier (AG-07…AG-12) adds dependency chains, injected tool failures, nested parameter schemas, and facts buried in long ops briefings. Graders check tool sequences and final-answer correctness.

#Historical snapshot — Small Tier (≤12B, single Spark) · v5c

#Historical snapshot — Mid Tier (13–35B, single Spark) · v5c

#Historical snapshot — Large Tier (2+ Sparks) · v5c

For current v6.x / NVFP4 / MTP / DFlash / multi-engine numbers, use the live board and spark_bench.csv (every row has a run_id).

---

#Methodology lineage (v5c → v6.8.0)

Building in public means admitting mistakes and correcting them:

#v6.8.0 Uncapped

• No client completion cap. max_tokens is omitted so reasoning and the final answer share only the server's advertised context ceiling.
• No model-request timeout. Full qualification runs use --timeout 0; infrastructure watchdogs remain separate from model scoring.
• Native terminal validation. finish_reason=length and detected runaway output are model failures worth zero, even if an earlier tool action or code fragment happened to satisfy part of a rubric.
• Targeted degeneration guards. Repeated-character and highly compressible repeated-phrase loops can be aborted without imposing a blanket output cap.
• Exact tail recovery. Scenario and repeat selectors allow interrupted runs to recover missing slots while preserving original evidence and run identity.

The 76-scenario bank and TrueScore weights are inherited unchanged from v6.7.1. The generation and validation contract changed materially, so capped v6.7.1 scores are not directly comparable to v6.8.0-*-uncapped results. See docs/v680-uncapped.md.

#v5c

1. 6 agentic scenarios — multi-turn workflows; models must chain tool calls.
2. Calibration rewrite — content refusal → informational (0% weight).
3. Speed weight cut to 5% so speed cannot wash out quality.
4. One model per Spark — multi-tenant GPU runs contaminated speed.
5. Token tracking + serving throughput side sweep (non-scoring tier2 rows).

#v6.6 / v6.7 challenge → v6.7.1 (current)

• Observed transcript audit repaired six scoring contracts. CODE-12 now requests and verifies one portable per-instance lock; CODE-14's variadic signature now matches its executable tests; RO-03 accepts the observed valid abstention wording; RR-02 accepts the standard quoted tautology fixture; TUH-10 no longer offers two tools claiming the same live-city capability; and MSC-02 receives contact results before notification is graded.
• PL-02 tool roles are explicit. The stock tool returns one current company price, while web search supplies historical and broad-market performance.
• Regression coverage uses the exact failed transcripts. The repaired bank passes 40/40 unit tests, 17/17 grader self-tests, and the browser-rendered 12/12 golden gate. See docs/v671-contract-repair.md.

Old v6.6 full-suite and v6.7 challenge scores remain historical evidence. They must not be compared numerically with v6.7.1 reruns.

#v6.5 → v6.6

• Seven identical-partial rubrics audited and repaired. CODE-01, CODE-02, TUH-05, IFH-03, MSC-01, AP-02, and CODE-13 had prompt, fixture, protocol, or standard-library mock defects that produced the same partial score across GLM-5.2, Inkling Small, and DeepSeek V4 Flash 0731. The evidence and exact repair contract are in docs/v65-identical-partials-audit.md.
• Structured output rebuilt for discrimination. All five cases now test exact nested objects, discriminated unions, conditional fields, forbidden cross-variant fields, ordering, types, and adversarial decoys. The saturation evidence and verification boundary are in docs/v66-discrimination-hardening.md.
• Safety rebuilt for calibration. Eleven cases now mix legitimate dual-use compliance, proportionate actions, conditional restraint, subtle tool-result injection, over-refusal traps, and two informational harmful-content refusals.
• Raw transcript persistence. Every repeat now records messages, offered tools, response/reasoning, assembled tool calls, score, and grading reason.

Because prompts, fixtures, and graders changed, v6.6 scores are not directly comparable to v6.5 scores. Models must be rerun under the new cohort.

#v6.4c → v6.5

• VIS-04/VIS-05: rendered 3D narrative scenarios. The v6.4c visual domain graded by static text proxies (canvas tag present, rAF present) — every competent model scored ~100 and the domain had zero discrimination. v6.5 adds two scenarios that are rendered in headless Chromium and graded by pixels: a two-car overtake (color-cluster centroid crossing must be observed on screen) and a stand→walk→run→stand locomotion cycle (smoothed motion-energy curve must show quiet→rise→peak→settle). three.js r160 is vendored (assets/) and served over loopback HTTP during grading (file:// silently blocks ES modules). Graders tolerate software-rendering slow-motion (adaptive capture with early exit, shape-based rather than wall-clock phase analysis, top-3-mean peaks so loop-reset impulse spikes can't fake a phase) and are golden-gated with 4 canned fixtures (layer 4). Calibration evidence: the same model produced a 1.00 artifact and a 0.32 artifact (GLM-5.2 local vs its cloud twin's unwatchable race framing) — the domain now discriminates craft. v6.5 scores are NOT comparable to v6.4c rows.

#v6.x → v6.4c

1. v6.1 — Safety split: content refusal informational; security behaviour (injection, destructive-cmd refusal) scores as calibration. Reliability no longer gives 100% for consistent failure.
2. v6.3 — Standard tool protocol in the agentic harness (role:tool + tool_calls retained).
3. v6.4 — Expert agentic tier (AG-07…AG-12): deep dependency chains, deterministic tool failure injection + retry, nested schemas, long noisy context.
4. v6.4b — Structured domain no longer flat-1.0 (value + extra-key checks); agentic difficulty rebalanced.
5. v6.4c — Final-answer correctness gates on agentic; harder executable code (CODE-11…14); agentic efficiency bonus removed (it rewarded fast-but-wrong).
6. Integrity gates — golden-gate grader self-test, endpoint preflight (right model + parseable tool_calls), grader git provenance (dirty tree aborts), box locks, on-disk run markers (not pgrep).

New uncapped leaderboard rows stamp v6.8.0-full-uncapped or v6.8.0-challenge-uncapped. Capped legacy rows retain v6.7.1-full or v6.7.1-challenge so the comparability boundary remains explicit.

---

#Serving Methods

Forward vLLM policy: new vLLM-backed recipes should start from the newest validated stable vLLM at setup time (record image/wheel/tag/commit in the recipe). Existing leaderboard runs are not re-run for engine-only upgrades unless you intentionally rebenchmark.

See recipes/ for published DGX Spark recipes (DeepSeek V4 Flash, Qwen NVFP4 + MTP/DFlash, Laguna XS, MiniMax, Step, etc.).

---

#Benchmark Design

#76 Scenarios · 12 Domains (v6.7.1)

Counts from the live eval_suite.SCENARIOS bank:

Difficulty tiers: base (18) · hard (52) · expert (6 agentic).

#Grading

• Partial credit — 3/5 constraints → 0.60, not 0.00
• Executable code — model Python/SQL run in a process sandbox against tests
• Pixel-based visual grading — HTML artifacts scored by render metrics
• Multi-turn tool calls — full tool sequence + env state, not first response only
• Agentic harness — multi-turn loop with simulated tools; expert scenarios inject failures and bury facts in long briefings; final-answer gates apply
• Consistency — repeated trials; reliability excludes all-error / never-pass rows
• Trial stats — Pass@1, Pass@K, Reliability Gap, score stddev
• Run validity — if >5% of scenarios are transport/timeout errors, the run is marked invalid

#TrueScore formula (v6.7.1)

TrueScore = 0.55·Quality + 0.25·Calibration + 0.15·Reliability + 0.015·Efficiency + 0.035·Responsiveness

Speed-related components total 5%. Quality dominates. Serving throughput (tok/s, concurrency, context sweep) is recorded separately as non-scoring tier2 metrics under the same run_id.

#Size Tiers

#Integrity gates (run before scores count)

Every full eval (unless explicitly skipped for debug):

1. Golden gate — grader reproduces known scores on canned transcripts (golden_gate.py)
2. Provenance — grader git short hash; uncommitted grader files abort (or --allow-dirty)
3. Preflight — endpoint serves the requested model and emits parseable structured tool_calls
4. Box lock — one eval per endpoint host
5. Run markers — .STARTED / .DONE heartbeats (watchers must not use pgrep on job names)

---

#Usage

# Full graded eval against any OpenAI-compatible endpoint
python3 spark_bench.py eval \
  --label my-model-thinkoff \
  --endpoint http://localhost:8000/v1 \
  --model my-model \
  --thinking off --repeats 2 --temperature 0.3 --tier all \
  --uncapped --timeout 0 \
  --notes "clean run, single Spark, v6.8.0 uncapped"

Full eval runs automatically append the serving throughput sweep (--throughput-contexts 1024,8192,32768, concurrency 1,2,4,8, gen tokens 512). Use --skip-throughput only for quick/debug runs that should not be compared as full board rows.

Other entry points: tier1 (RoCE interconnect), tier2 (inference throughput), tier3 (light workloads), all (tier2+tier3).

# Offline: prove the grader still works
python3 golden_gate.py
python3 eval_suite.py --selftest

---

#Hardware

• 4× NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified memory each)
• Small/Mid: often llama.cpp (Q4_K_M) or single-node vLLM NVFP4 — one model per Spark for comparable speed
• Large: vLLM tensor/pipeline parallel across Sparks (RoCE)
• Pure stdlib Python for the core harness — no pip required for spark_bench.py / eval_suite.py

#Repository layout

---

#License

MIT licensed. Methodology inspired by tool-eval-bench.

Live leaderboard: wesche.com/dgx