fix: weight:0 un-gates criteria and renders as informational

[uipreliga]

Summary

Makes weight: 0 mean what the docs say it means, end to end.

1. weight: 0 un-gates a criterion

Previously a weight: 0 criterion contributed nothing to the weighted score but could still single-handedly flip a task to FAILURE — the docs told authors to also set pass_threshold: 0, which was easy to miss.

• New BaseSuccessCriterion.is_gating (weight > 0.0) is the single source of truth for "does a low score here fail the task".
• EvaluationResult.all_criteria_passed and the coder-eval evaluate exit code now gate over is_gating criteria only. A weight: 0 miss no longer flips final_status or the exit code.
• A validator rejects weight: 0 combined with stop_when or suite_thresholds — arming a non-gating criterion for a pass/fail gate (early-stop gate, or the across-row suite gate that drives the run exit code) is incoherent and is now caught at load time.

2. Display parity (second commit)

final_status and both exit codes were correct after (1), but every display surface still labeled a below-threshold informational criterion as failed — a header-vs-list contradiction where the task reads SUCCESS while one of its rows reads FAIL. Fixed with one carried marker rather than four re-derivations:

• CriterionResult.gating (default True) mirrors is_gating, stamped by SuccessChecker alongside pass_threshold on all three construction paths. Defaulting True means task.json files written before this field read back as gating.
• Checker log line says "below threshold (informational)" instead of FAILED.
• HTML report tags the row "informational — not gated (weight: 0)"; the (n/m passed) header counts gating criteria only and appends "+ k informational".
• Suite failure-reason sampling skips non-gating criteria — an informational miss must not explain why a row failed.
• evalboard: DTO carries gating + passThreshold; non-gating criteria render a gray INFO pill.

Note

One change in that commit is not about weight: 0. The evalboard criterion pill computed passed = c.score === 1, ignoring pass_threshold entirely — so any fractional criterion scoring e.g. 0.95 against the default 0.9 threshold rendered FAIL. It now compares against the criterion's own threshold. This affects how every criterion renders in every run, including tasks with no weight: 0 anywhere. Called out explicitly because it is outside the scope reviewers approved.

3. Doc drift

• Dropped the non-existent rephrase prompt mutation from the catalog.
• Corrected the stale "max_memory_mb NOT enforced" claim (it maps to docker run --memory; max_cpus/max_pids → --cpus/--pids-limit).
• reports.py failure-reason sampling uses pass_threshold, not score < 1.0.

Test plan

• make verify green — 3442 passed, 2 skipped, 90.91% coverage, ruff + pyright + custom CE lint clean. tsc --noEmit clean on evalboard.
• New regression tests: weight: 0 un-gating semantics, the stop_when / suite_thresholds rejections, a coder-eval evaluate exit-code test (with fixture) proving an informational miss doesn't fail the run, and for the display work — gating stamped from the criterion, default-True read-back, the HTML header/row labeling, and suite sampling omitting an informational miss (verified failing without the fix).

Split out of this PR, per review

• Configurable retry policy → feat: configurable retry policy as a top-level retry: root #44, per @akshaylive. Also moved out of run_limits to a top-level retry: root there, per the same review.
• evalboard OSS-edition change → dropped entirely, per @bai-uipath: /trends and /watchlist clear portability but not product fit — mature_skipped has no producer in src/, and dominantFailureTags needs the review index, which the OSS core doesn't write.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 59s —— View job

---

Code Review In Progress

Review Tasks

• Read review guidelines from .github/code_review.md
• Read project conventions from CLAUDE.md
• Get full PR diff
• Review each changed file for context
• Perform cross-file consistency checks
• Analyze design decisions
• Check for missing elements
• Provide structured feedback

Starting review...

[akshaylive]

The weight 0 fix LGTM. The configurable retry policy should ideally be a separate PR. Additionally the retry policy configuration shouldn't be in "run_limits" (retry policy is not really a limit, so a separate top level field is better). I haven't looked too deep into the impl but would this clear the working directory between retries?

[bai-uipath]

The evalboard change moves /trends and /watchlist into the OSS edition on the grounds that they "derive purely from run artifacts." That conflates two different bars: "runs without internal config" (portability — true) vs. "produces something meaningful for a generic adopter" (product fit — not really). Portability is cleared; product fit is the unaddressed concern, and it's the one that matters for an OSS-credibility play. Reasons these shouldn't ship OSS as-is:

• They assume an operating model OSS users won't have. The value prop — "last 10 runs," regression (recent-vs-prev pass rate), streaks, volatility sparklines, "never passed across the window" — assumes a recurring scheduled run of a stable suite (the UiPath nightly model). An adopter doing a handful of ad-hoc experiment runs gets degenerate output. Not hardcoded — assumed.
• Graceful ≠ useful. Several panels go blank without internal enrichment: dominantFailureTags needs a review index (readRunReviewIndex, not an OSS-core artifact), the maturity column needs matureSkipped (not produced by OSS core — always false in a bare clone), component-SHA attribution needs CI wiring. A new adopter lands on a half-empty Watchlist and reads it as broken — worse than not shipping it.

[uipreliga]

@akshaylive thanks — all three points taken.

1. Separate PR. Done. This PR is now the weight: 0 fix alone (plus the small doc/report/evalboard corrections); the retry work moved to #44, stacked on this branch. Rebased onto latest main while I was in there.

2. retry shouldn't live under run_limits. Agreed, and moved to a top-level retry: block in #44. Your framing is the right one: run_limits holds caps that abort a run, while a retry policy is how hard the run tries to survive a transient error. Worth noting run_limits had already drifted in this direction (count_cached_input, count_cache_creation, stop_early, expected_turns aren't caps either) — which I took as an argument for not adding a fifth exception. It's now a first-class 4th -D root next to agent/run_limits/sandbox, so -D retry.max_retries=0 merges through the same resolve_root engine at every config layer.

3. "Would this clear the working directory between retries?" No, it does not — good catch, and it's a real pre-existing gap rather than something the retry knob introduced:

• Retry wraps three seams in orchestrator.py: sandbox setup, agent start, and _communicate_with_retry.
• On a communicate retry the sandbox is untouched — files the crashed attempt already wrote survive into the retried attempt, and the same agent/SDK session is reused, so the re-sent prompt lands in a conversation that already contains the failed trajectory.
• The crashed partial TurnRecord is also drained onto the result (_on_attempt_failure), so trajectory-observing criteria — command_executed, commands_efficiency, skill_triggered — see both attempts. commands_efficiency in particular scores worse purely because a retry happened.

I chose to document this rather than add a reset seam: clearing the sandbox mid-task would discard work the agent had already done and budget already spent, and preserving partials is deliberate (crash forensics). #44 adds a retry section to TASK_DEFINITION_GUIDE.md spelling out the in-place semantics, with retry.max_retries: 0 as the escape hatch when you want a clean single-attempt trajectory. If you'd rather have an explicit reset option, I'm happy to add it — say the word and I'll do it in #44.

[uipreliga]

@bai-uipath you're right, and I've dropped the evalboard change from this PR entirely rather than argue it.

The justification in my PR description — "they derive purely from run artifacts" — was true and answered the wrong question. Portability was the bar I checked; product fit is the bar that matters here, and I didn't check it. Your two specifics hold up when I trace them:

• matureSkipped has no producer in src/. grep -rn mature_skipped src/ returns nothing — the Python core never emits mature_skipped, so lib/trends.ts falls through to ?? false on every task in a bare clone. The maturity column is structurally dead OSS-side, not just sparse.
• dominantFailureTags needs the review index. app/watchlist/page.tsx → loadRecentRuns (lib/overview.ts) → readRunReviewIndex (lib/reviews.ts), and review.json is produced by the coder-eval-review skill, not by core. A plain adopter gets an empty tag list.

So a new adopter lands on a Watchlist with a dead column and an empty failure-tag panel, and reasonably concludes the page is broken. That's worse than not shipping it, exactly as you said.

Dropping it also fixes a packaging problem @akshaylive flagged on the retry policy — this PR is now the weight: 0 fix alone, which is what it claims to be.

If we do want these OSS-side later, the prerequisite is core actually emitting the fields the panels read (mature_skipped at minimum, and some core-produced equivalent of the review index), which is its own piece of work rather than a nav-link change. Happy to open an issue for that if you think it's worth tracking.

[akshaylive]

🚢 Please update the PR description before merging (squash-merge picks up the PR description into the commit message by default)

[uipreliga]

Heads-up: a second commit landed on this branch after your reviews, so the approvals here predate it. b268875 — "render weight:0 criteria as informational on every display surface."

Why it belongs in this PR: the un-gating fix was only half done. final_status and both exit codes ignored weight: 0 criteria, but every display surface still labeled a below-threshold informational criterion as FAILED — so a task could read SUCCESS in its header while one of its criteria read FAIL right underneath. Same issue, so same PR. It carries a CriterionResult.gating marker (mirroring is_gating, default True so older task.json files read back as gating) out to the checker log, the HTML report, the suite failure-reason sampling, and the evalboard pill.

The part that is NOT about weight: 0, and the reason I'm flagging rather than quietly pushing: the evalboard criterion pill computed

const passed = c.score === 1;

which ignores pass_threshold entirely — so any fractional criterion scoring 0.95 against the default 0.9 threshold has been rendering FAIL on the task page. It now compares against the criterion's own threshold. That changes how every criterion renders in every run, including tasks with no weight: 0 anywhere, and it's outside what "the weight 0 fix LGTM" covered.

I fixed it in place because leaving a threshold-ignoring pill three lines from a threshold-aware one seemed worse than the scope creep — but if you'd rather keep this PR strictly to what you approved, say so and I'll pull those four lines into their own PR.

make verify green: 3442 passed, 2 skipped, 90.91% coverage; tsc --noEmit clean on evalboard. PR description updated to match what's actually on the branch.