[Eval] MiniMax-M3.1-Flash in WorkBuddy: several-fold time/token cost vs. other models on the same task, with no meaningful quality gain (English version of #518) #520

Note: This is the English version of the original Chinese report in this same repository: #518
The content is identical; it is duplicated here so that English-speaking maintainers and the model team can review the same evidence. Please treat #518 and this issue as a single report.

Summary

While completing a simple front-end visualization task inside the WorkBuddy harness, MiniMax-M3.1-Flash exhibited a pronounced over-deliberation + revisionism pattern:

• It consumed several times the wall-clock time and tokens of the other models evaluated on the identical task.
• The final output quality was not meaningfully better than those other models.
• Critically, the extra time and tokens produced no quality gain at all.

This is not a capability shortfall. It is a case where the model converted its own main advantage (small footprint, fast turnaround) into a disadvantage. Had M3.1-Flash delivered a 200–300 line file in a single pass within 5–10 minutes, its cost-effectiveness on this class of task would have been far better than what actually happened. We ask the team to take this pattern seriously and address it in post-training and calibration.

Evaluation setup

• Model: MiniMax-M3.1-Flash (highest thinking mode in WorkBuddy)
• Harness: WorkBuddy (Windows desktop). Other models in the same comparison ran on other harnesses.
• Inference path: standard agent loop; tools used were Read / Write / Edit / Bash.
• User prompt (verbatim):

创建一个带有二维 SVG 动画的 HTML 页面，动画内容是一只鹈鹕正在骑自行车。直接给出完整的代码；请你一次性写好之后，不要运行测试。 本次任务严禁你上网搜索任何信息。 你把这个鹈鹕骑车的文件弄好之后， 请你把他放到E:\DshApp 里面自己弄一个新的文件夹取个名字然后再把你的鹈鹕骑车的文件放进去。

(Translation: Create an HTML page with a 2D SVG animation of a pelican riding a bicycle. Just give the complete code; once you have written it in one go, do not run tests. You must not search the web for any information for this task. After you have finished the pelican-bike file, put it inside E:\DshApp in a new folder of your own choosing, and place your pelican-bike file inside it.)

• Task difficulty: low. A single self-contained HTML file with inline SVG. The minimum viable implementation is roughly 150 lines of pure CSS (rotating wheels, swinging legs, scrolling background).
• Explicit constraints given by the user: (1) write it in one go; (2) do not run tests; (3) do not search the web.
• Control group — other models evaluated on the same prompt, same user, same output directory:

Important observation: Gemini 4 argon produced by far the largest file (950 lines / 43.8 KB) yet wrote it in a single pass and finished fastest. So large output volume is not the problem — writing once and stopping is. This run landed in the worst quadrant: medium-sized output plus repeated rework.

Failure modes

1. The same file was written/modified 6 times, and several of those edits were pure self-reversal

Actual write sequence (recoverable from the tool-call log):

2. Hard evidence: constant flip-flop — the direct fingerprint of revisionism

Complete change history of the same constants across those Edit calls:

HIP_NEAR / HIP_FAR :  (355,220)/(340,212)
                  →  (352,232)/(338,224)
                  →  (356,226)/(344,230)      ← changed 3 times, a few pixels each time

THIGH / SHIN      :  100 / 90
                  →  88 / 80
                  →  100 / 88                  ← changed 2 times; the 3rd change reverted to the original value

These were not bug fixes — they were indecision. A genuine bug fix converges; these diverged. Note that the third change to THIGH / SHIN restored the original 100/88, which is pure churn with zero effect.

3. Scope inflation

None of the following were requested by the user. All were self-added by the model:

• Two-bone IK gait solving (thigh/shin lengths plus per-frame trigonometry for the knee joint)
• Chainring/wheel gear-ratio coupling (GEAR = 2.4)
• Deriving a physically consistent ground speed from wheel circumference × wheel rate
• Multi-layer parallax scrolling (clouds 46 s / hills 22 s)
• Seamless infinite tiling plus a "period must divide evenly" constraint
• SMIL blinking / wing flapping / pulsing sun
• Pause control plus a three-speed variant selector

Every added layer creates another set of coordinates that must stay mutually consistent, so rework after the first draft becomes inevitable. The 30 minutes were produced by complexity the model invented itself — the task itself is not hard.

4. Termination-criterion failure (a structural issue that "trying harder" cannot fix)

For a visual artifact whose rendering the model cannot directly observe, the effective stopping condition degrades into "I can no longer find a defect in my internal model of the output." That condition is almost never satisfied — one can always imagine another "the saddle might be hidden behind the body." Behavior therefore slides from finding correctness to finding defects, and defect-hunting is unbounded.

The key point: when the user explicitly said do not run tests, the correct response is to narrow the scope, write once, and deliver immediately (moving the uncertainty cost onto "do less"). Instead the model translated "no testing" into "then I will reason through more rounds", which turned into 25 minutes of blind review. This is an instruction-following deviation, not a capability limit.

5. Net result

• v1 (16:33, 27.8 KB) was already a functionally complete, directly deliverable artifact.
• The contribution of iterations 2–6 to "does it actually run / is it correct" is zero.
• The only net gain was a handful of pixel-level cosmetic tweaks.
• The cost was ~25 minutes plus several times the tokens.
• Before final delivery, no rendering verification was performed at all, so there is no way to confirm that the "improvements" even took effect.

In short: the several-fold increase in time and token consumption over the other models did not make the result meaningfully better than the other models.

Expected vs. actual

Expected — for a low-complexity task carrying the instruction "write it in one go, do not run tests", M3.1-Flash should:

1. Adopt the simplest implementation that satisfies the requirement (pure CSS keyframes are sufficient; no IK, no physics coupling);
2. Produce the file in a single Write, with no post-hoc pixel-level self-polishing;
3. When explicitly told not to run tests, manage uncertainty by shrinking scope, not by adding more reasoning rounds.

Actual — scope ballooned to 552 lines / 27.7 KB; the file was written and edited 6 times; multiple zero-value flip-flop reversals occurred; total elapsed time was roughly 30 minutes; and the final quality showed no meaningful advantage over the 200–300 line versions other models delivered in 5–10 minutes.

Suggested directions for calibration / post-training

1. Termination criteria. For scenarios where the output is not self-observable, impose a harder stop condition — e.g. a bounded number of self-review rounds, or a rule that a modification requires a concretely statable defect — to suppress unbounded self-review.
2. Separate "fixing a bug" from "adjusting a preference." The model currently treats "I think this could be better" as sufficient grounds to edit. Ideally a modification must be anchored to a specific, articulable defect.
3. Honour the "write it in one go" instruction. When that instruction is present, it should trigger a scope reduction (smaller output, drop self-added features) rather than "write it again more carefully".
4. Over-design and token efficiency. Treat "output volume more than 2× the other models" as a detectable over-design signal that triggers self-downgrading.

Additional verifiable details

• Reproduced artifact (archived from this run): pelican-bike.html, 552 lines / 27.8 KB, md5 c7bafe132e93a4ef0b114e571aafe7c7
• Platform: Windows 10 x64 (10.0.19045), WorkBuddy desktop
• No private code or credentials were involved in the task. This report contains no keys, account details, or sensitive paths.