DeepSeek Harness Benchmarks: Official Scores, Minimal Mode, and Caveats

Every official agent score DeepSeek publishes for its models is a measurement of the pair — model plus harness — not the model alone. The July 31, 2026 changelog states that all Code Agent benchmarks for V4-Flash-0731 were run using the DeepSeek Harness minimal mode, with max effort, topp=0.95, and temperature=1.0^[1].

That is the whole point of the Agent = Model + Harness formula: agentic capability lives in the combination. When DeepSeek reports a score, the harness is part of the result, and swapping the harness can change the number even with the same model^[2].

Here is the complete official table from the V4-Flash-0731 changelog: nine benchmarks, two of which are internal test sets^[1][3]:

The headline is the DeepSWE jump from 7.3 to 54.4 (+645%) on the same model after re-post-training. Two rows (DSBench-FullStack, DSBench-Hard) are DeepSeek's own sets and cannot be reproduced externally; the other seven are public benchmarks but every score is vendor-reported^[1][3].

Minimal mode is a two-tool coding agent: a persistent bash shell and a str_replace_editor for file edits — nothing else^[1][4].

The reasoning is comparability. A full agent environment (web search, skills, subagents) would make benchmark runs hard to attribute: did the model solve the task, or did the harness's extra tools? Minimal mode isolates the model's coding ability behind the thinnest possible tool layer, which is exactly what you want when the benchmark is about the model^[1][4].

It is also the reproducible configuration for third parties: install dsh, switch to the minimal profile, point it at any OpenAI-compatible model, and run a benchmark suite under the same two tools^[4].

Because the official scores are vendor-reported, the community immediately asked how much the harness contributes — the 'benchmark maxxing' question^[5].

The sharpest challenge came from the DeepSWE thread. DeepSWE is a harder, from-scratch benchmark with behavioral verifiers, and its author ran every model through one shared harness. But the fairness pilot only covered Claude, GPT, and Gemini — DeepSeek was not in it. One top comment summarized the worry: 'A bottom-of-board score for the one model nobody validated the harness against reads as a harness result, not a capability result.'

Independent numbers add context. A separate DeepSWE audit by yage.ai in June 2026 scored V4-Pro at just 8% pass@1, versus 70% for GPT-5.5 and 54% for Opus 4.7. Compare that with the 54.4 the official Harness reports for Flash 0731 on the same benchmark, and the gap is the whole debate in miniature^[5].

With the harness now public, third parties can finally re-run the suite — which is the single biggest change since July 31. Until an independent lab publishes a reproduction, treat every official agent number as directional^[5].

Now that dsh is open source, the minimal-mode rig is runnable^[1][4]:

For a fair model-vs-model comparison, keep the harness constant (minimal mode, same tools, same config) and vary only the model — the methodology recommended by third-party reviewers^[6].

# 1. install
npx @deepseek-ai/dsh web        # or build from source (pnpm)

# 2. point at the model you want to benchmark
#    Settings → Models → add provider (any OpenAI-compatible endpoint)

# 3. run the minimal profile headless on a task
dsh --profile headless "solve this coding task and run the tests"

Related material^[1][7]:

• DeepSeek V4 Pro benchmarks — the GA 0813 numbers, including Terminal-Bench 87.9 and Cybergym
• V4 Flash benchmarks — full fine print on the +645% figure
• Minimal mode explained — the two-tool rig in detail
• V4 Pro vs GPT-5.5 — independent comparison context