How we make the agent smarter
Confidence stepping
A capability technique we discovered and proved on Terminal-Bench 2.1. jcode's todo tool asks the model for a confidence score when a task is assigned and again when it's marked done. Tracing every score across 70 trials revealed an asymmetry: low confidence at assignment accurately predicts which step will fail, but at completion the agent reports high confidence no matter what. So jcode now requires confidence to be earned in steps, and treats a one-shot jump to 100 as a signal to force more verification.
Finding 1: completion confidence is theater
Each line is one trial: mean confidence at assignment, mean at completion. Red lines failed the hidden verifier. Average rise: +16 points, identical for passes and failures. The end-state number carries no information.
Finding 2: assignment confidence knows the failure in advance
All three trials that finished at 100% confidence and failed had flagged the exact failing step at assignment time.
| Task | Riskiest todo (assigned) | assigned | finished | Verifier |
|---|---|---|---|---|
| model-extraction-relu-logits | "Validate recovered A1 vs true A1 (cosine match)" | 75% | 100% | test_stolen_matrix_matches failed |
| train-fasttext | "Train fasttext model, tune for ≥0.62 acc" | 80% | 100% | test_accuracy failed |
| install-windows-3.11 | "Verify VM boots to Windows 3.11 desktop" | 70% | 100% | boot verification failed |
Finding 3: earned confidence is trustworthy
Trials that stamped every todo a uniform 100 at the end vs trials whose final report differentiated between items (95s, 97s, 98s).
The technique: force stepped confidence
The exploit is simple: the agent's best-calibrated moment is assignment, so
never let completion erase it. jcode's todo tool now keeps a tool-maintained
confidence_history per todo, and asks the model to raise confidence
only as evidence accumulates. Each passing test or validation justifies one step
up. When a todo's confidence jumps 15+ points in a single step (the bulk
"I'm done, everything is 100" stamp), the harness refuses to accept it and sends
one targeted follow-up naming that todo and its trail (e.g. "75 → 100: this
confidence was not earned through intermediate validation"). Stepped rises pass
silently, and the old unconditional end-of-run summary is gone (it fired in 15 of 89
trials and never changed behavior), so the technique costs fewer tokens
than what it replaced. Proved on Terminal-Bench 2.1. The after-run lands here next.