feat(review): graceful fallback when org model policy blocks the selected model + clear review errors - #106
Conversation
|
Droid finished @factory-nizar's task —— View job The model-policy fallback approach is solid and well-covered by tests, but the base-action stdout parsing and per-attempt state handling can make fallback detection and surfaced error messages unreliable across retries. Tightening those up will make the new behavior robust under real stream and process conditions. |
6f9635b to
f7804f8
Compare
…he selected model When an org's model policy disallows the model chosen for review / security / fill flows (e.g. the review_depth presets gpt-5.2 or kimi-k2.6), runs previously died with an opaque 403 and the tracking comment only said 'Droid encountered an error'. - Pre-flight: prepare steps now fetch the org model policy via /api/organization/managed-settings and omit --model (falling back to the org default model) when the selected model is not allowed, surfacing a notice in the tracking comment. Fails open on any lookup error. - Reactive: base-action detects the model-policy 403 in the exec result and retries once without --model / --reasoning-effort. - Error surfacing: base-action now exposes the final result error as an error_message output, and update-comment-link renders it in the tracking comment with an actionable hint for model-policy 403s. Drive-bys: fix pre-existing duplicate type import in src/core/review/artifacts/write.ts (broke tsc) and prettier formatting in gitlab/examples/README.md (broke format:check). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
f7804f8 to
52c8513
Compare
| let currentDroidArgs = config.droidArgs; | ||
| let modelArgsStripped = false; | ||
| type ResultEvent = { is_error?: boolean; result?: string }; | ||
| let lastResultEvent: ResultEvent | null = null; |
There was a problem hiding this comment.
[P1] Reset last result event between retries
lastResultEvent persists across retry attempts, so if an earlier attempt emits a type=result error and a later attempt exits before producing its own result event, the retry logic can miss a model-policy fallback decision and can publish an error_message from a previous attempt instead of the final failure.
| parsed !== null && | ||
| parsed.type === "result" | ||
| ) { | ||
| lastResultEvent = { |
There was a problem hiding this comment.
[P1] Buffer stdout to parse stream-json safely
The new result-event capture relies on text.split("\n") and JSON.parse(line) per stdout chunk, but Node stream chunks can split a single JSON line across chunks. When that happens parsing fails, output is suppressed in non-full-output mode, and lastResultEvent never updates, which breaks model-policy fallback detection and final error_message reporting.
| bodyContent = bodyContent.replace(/\n?\[View job run\]\([^\)]+\)/g, ""); | ||
| bodyContent = bodyContent.replace(/\n?\[View branch\]\([^\)]+\)/g, ""); | ||
|
|
||
| // Remove stale model-policy notices from previous runs |
There was a problem hiding this comment.
[P2] Strip a leading separator when re-rendering comments
updateCommentBody removes the previous completion header but leaves the prior --- separator at the start of bodyContent, then appends a new separator before reattaching bodyContent. On retries or subsequent updates this produces duplicated separators at the top of the preserved content; consider stripping a leading separator during the cleanup pass (near the new stale-notice stripping) to keep updates idempotent.
… tracking comment A failed automatic review kept 'Droid is reviewing code and running a security check…' below the error header, which made the failure comment read as still in progress. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…error dumps An unrecognized --model value now retries with the organization's default model (same as a policy-blocked model) instead of failing the run. When a model error still surfaces, the CLI's full available-models dump is condensed to the meaningful lines and the comment hints link to https://docs.factory.ai/models. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Motivation
A customer (Becoming You Labs) tightened their org model allowlist while their review workflows still used the
review_depthpreset defaults (gpt-5.2deep /kimi-k2.6shallow). Review runs died with an opaque403 This model is not available due to your organization's security settings, and the tracking comment only said "Droid encountered an error" with no reason (Pylon #13165).Changes
1. Policy-aware model fallback
src/utils/model-policy.ts): prepare steps fetch the org model policy fromGET /api/organization/managed-settingsusing theFACTORY_API_KEYand, when the selected model is disallowed, omit--model/--reasoning-effortsodroid execuses the org's default model. A notice is surfaced in the tracking comment. The check fails open: any lookup error preserves today's behavior.review/sub-action), security review/scan (security_model), PR description fill (fill_model), and the GitLab review prepare entrypoint.base-action/src/run-droid.ts): if a run still fails with the model-policy 403 (policy changed mid-run, custom models, etc), the retry loop strips--model/--reasoning-effortonce and retries with the org default.2. Clear error surfacing
base-actioncaptures the finalstream-jsonresult event and exposes its error text as anerror_messagestep output (plus a one-lineDroid Exec failed: ...log even in sanitized output mode).update-comment-link.ts/comment-logic.tsrender that error in the tracking comment, with an actionable hint when it is a model-policy 403 ("setreview_modelto an approved model"), and render the fallback notice on success so model switches are never silent.Drive-by fixes (pre-existing CI breakage on main)
src/core/review/artifacts/write.ts(failedtsc --noEmit).gitlab/examples/README.md(failedformat:check).Validation
bun test: 484 pass / 0 fail (includes newtest/model-policy.test.ts,base-action/test/model-policy-error.test.ts, and new comment-logic cases)bun run typecheckclean in root andbase-actionbun run format:checkclean