Skip to content

feat(review): graceful fallback when org model policy blocks the selected model + clear review errors - #106

Merged
factory-nizar merged 3 commits into
devfrom
nizar/model-policy-fallback
Jul 31, 2026
Merged

feat(review): graceful fallback when org model policy blocks the selected model + clear review errors#106
factory-nizar merged 3 commits into
devfrom
nizar/model-policy-fallback

Conversation

@factory-nizar

Copy link
Copy Markdown
Contributor

Motivation

A customer (Becoming You Labs) tightened their org model allowlist while their review workflows still used the review_depth preset defaults (gpt-5.2 deep / kimi-k2.6 shallow). Review runs died with an opaque 403 This model is not available due to your organization's security settings, and the tracking comment only said "Droid encountered an error" with no reason (Pylon #13165).

Changes

1. Policy-aware model fallback

  • Pre-flight (src/utils/model-policy.ts): prepare steps fetch the org model policy from GET /api/organization/managed-settings using the FACTORY_API_KEY and, when the selected model is disallowed, omit --model/--reasoning-effort so droid exec uses the org's default model. A notice is surfaced in the tracking comment. The check fails open: any lookup error preserves today's behavior.
  • Wired into all model-selecting flows: review candidates + validator (tag mode and the review/ sub-action), security review/scan (security_model), PR description fill (fill_model), and the GitLab review prepare entrypoint.
  • Reactive (base-action/src/run-droid.ts): if a run still fails with the model-policy 403 (policy changed mid-run, custom models, etc), the retry loop strips --model/--reasoning-effort once and retries with the org default.

2. Clear error surfacing

  • base-action captures the final stream-json result event and exposes its error text as an error_message step output (plus a one-line Droid Exec failed: ... log even in sanitized output mode).
  • update-comment-link.ts / comment-logic.ts render that error in the tracking comment, with an actionable hint when it is a model-policy 403 ("set review_model to an approved model"), and render the fallback notice on success so model switches are never silent.

Drive-by fixes (pre-existing CI breakage on main)

  • Duplicate type import in src/core/review/artifacts/write.ts (failed tsc --noEmit).
  • Prettier formatting in gitlab/examples/README.md (failed format:check).

Validation

  • bun test: 484 pass / 0 fail (includes new test/model-policy.test.ts, base-action/test/model-policy-error.test.ts, and new comment-logic cases)
  • bun run typecheck clean in root and base-action
  • bun run format:check clean

@factory-droid

factory-droid Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Droid finished @factory-nizar's task —— View job


Security Review

The model-policy fallback approach is solid and well-covered by tests, but the base-action stdout parsing and per-attempt state handling can make fallback detection and surfaced error messages unreliable across retries. Tightening those up will make the new behavior robust under real stream and process conditions.

@factory-nizar
factory-nizar changed the base branch from main to dev July 31, 2026 00:41
@factory-nizar
factory-nizar force-pushed the nizar/model-policy-fallback branch from 6f9635b to f7804f8 Compare July 31, 2026 00:44
…he selected model

When an org's model policy disallows the model chosen for review /
security / fill flows (e.g. the review_depth presets gpt-5.2 or
kimi-k2.6), runs previously died with an opaque 403 and the tracking
comment only said 'Droid encountered an error'.

- Pre-flight: prepare steps now fetch the org model policy via
  /api/organization/managed-settings and omit --model (falling back to
  the org default model) when the selected model is not allowed,
  surfacing a notice in the tracking comment. Fails open on any lookup
  error.
- Reactive: base-action detects the model-policy 403 in the exec result
  and retries once without --model / --reasoning-effort.
- Error surfacing: base-action now exposes the final result error as an
  error_message output, and update-comment-link renders it in the
  tracking comment with an actionable hint for model-policy 403s.

Drive-bys: fix pre-existing duplicate type import in
src/core/review/artifacts/write.ts (broke tsc) and prettier formatting
in gitlab/examples/README.md (broke format:check).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@factory-nizar
factory-nizar force-pushed the nizar/model-policy-fallback branch from f7804f8 to 52c8513 Compare July 31, 2026 00:54
let currentDroidArgs = config.droidArgs;
let modelArgsStripped = false;
type ResultEvent = { is_error?: boolean; result?: string };
let lastResultEvent: ResultEvent | null = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reset last result event between retries

lastResultEvent persists across retry attempts, so if an earlier attempt emits a type=result error and a later attempt exits before producing its own result event, the retry logic can miss a model-policy fallback decision and can publish an error_message from a previous attempt instead of the final failure.

parsed !== null &&
parsed.type === "result"
) {
lastResultEvent = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Buffer stdout to parse stream-json safely

The new result-event capture relies on text.split("\n") and JSON.parse(line) per stdout chunk, but Node stream chunks can split a single JSON line across chunks. When that happens parsing fails, output is suppressed in non-full-output mode, and lastResultEvent never updates, which breaks model-policy fallback detection and final error_message reporting.

bodyContent = bodyContent.replace(/\n?\[View job run\]\([^\)]+\)/g, "");
bodyContent = bodyContent.replace(/\n?\[View branch\]\([^\)]+\)/g, "");

// Remove stale model-policy notices from previous runs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Strip a leading separator when re-rendering comments

updateCommentBody removes the previous completion header but leaves the prior --- separator at the start of bodyContent, then appends a new separator before reattaching bodyContent. On retries or subsequent updates this produces duplicated separators at the top of the preserved content; consider stripping a leading separator during the cleanup pass (near the new stale-notice stripping) to keep updates idempotent.

factory-nizar and others added 2 commits July 30, 2026 18:01
… tracking comment

A failed automatic review kept 'Droid is reviewing code and running a
security check…' below the error header, which made the failure comment
read as still in progress.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…error dumps

An unrecognized --model value now retries with the organization's
default model (same as a policy-blocked model) instead of failing the
run. When a model error still surfaces, the CLI's full available-models
dump is condensed to the meaningful lines and the comment hints link to
https://docs.factory.ai/models.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@factory-nizar
factory-nizar merged commit 4028e0a into dev Jul 31, 2026
1 check passed
@factory-nizar
factory-nizar deleted the nizar/model-policy-fallback branch July 31, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants