feat: Implement optimization code paths and functionality for initial release#140
Merged
Merged
Conversation
…ype, remove required context_choices argument and default to anon
**Requirements**
- [x] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [x] I have validated my changes against all supported platform
versions
**Describe the solution you've provided**
We added first class support for some fields on the UI --
- Latency optimization
- Token optimization
- Auto commit toggle
This PR pulls them into the SDK. token/latency optimization are using
their previous paths for now in this PR. Rather than the regex approach
we just use the flags from the API now. The latency/cost optimization
paths will be updated in a subsequent PR.
**Describe alternatives you've considered**
The initial implementation of these two code paths for optimizations
were kind of hacky to begin with (just using a dictionary to look up
words that might mean they want to do it). This was the intended
solution.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes when latency/cost gates and judge templates apply (explicit
flags vs inferred text) and alters config-judge loading and auto-commit
gating, which can shift optimization outcomes for existing runs.
>
> **Overview**
> Wires **LaunchDarkly agent optimization API** fields into the Python
SDK: `latencyOptimization`, `tokenOptimization`, and `autoCommit` on
remote configs, plus optional **`variation_key`** on
`OptimizationOptions` / `GroundTruthOptimizationOptions` to start from a
specific AI config variation (REST fetch; requires API key and
`project_key`).
>
> **Latency and token behavior** no longer infer goals from
acceptance-statement keyword regexes. Gates, judge prompt augmentations,
variation prompts, and model-pricing warnings now key off
**`latency_optimization`** and **`token_optimization`** booleans (from
options or API). When unset/false, those paths stay off.
>
> **Config judges** resolve via raw flag **`variation()`** and local
`{{key}}` interpolation (including `message_history` /
`response_to_evaluate`) instead of `LDAIClient.judge_config`.
System-only judge templates get an auto-built user turn.
>
> **`optimize_from_config`** maps the new API fields into built options;
**auto-commit** runs only when both the fetched config’s `autoCommit`
and caller options allow it. Tests drop regex helpers and cover the new
flags, judge path, and `variation_key` validation.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
509240f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
**Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [x] I have validated my changes against all supported platform versions **Describe the solution you've provided** Moves the cost and latency optimization process to happen as a post-process pass rather than attempting to optimize for everything in each loop. This helps reduce the amount of noise the LLM is dealing with in a single loop. Flow is now optimize for quality -> validate with additional samples -> optimize for meta (latency, cost). **Describe alternatives you've considered** The ultimate goal here is to move to distinct scorers/criteria that can be ranked. For now, this is a better solution than the all-in-one passes we were doing previously which could regress. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes when optimizations pass/fail, which model/parameters are committed, and callback timing—behavioral regressions are possible despite extensive test updates. > > **Overview** > **Cost and latency are no longer mixed into the main optimization loop.** Phase 1 only chases judge/validation quality; duration and cost gates are removed from standard turns, validation, and ground-truth samples. When latency or token optimization is enabled and Phase 1 succeeds, **`_run_cost_latency_phase`** runs with instructions frozen, reuses the winner’s input/variables, evaluates each distinct `model_choices` entry, applies latency/cost gates there, and picks the best passing candidate via normalized duration + cost vs baseline. > > **Prompting and variation generation split by phase:** `build_new_variation_prompt` no longer takes cost/latency flags; Phase 2 uses new **`build_token_latency_variation_prompt`** (content lock, model/param-only changes). LLM instruction edits in Phase 2 are reverted if they drift from the frozen winner. Judge prompts inject latency/cost guidance only while **`_in_cost_latency_phase`**. > > **Run lifecycle and API surface:** **`on_passing_result`** fires once with the true final context (Phase 2 winner or Phase 1 fallback); **`_handle_success`** can suppress that callback during intermediate success. Every agent turn adds a **`_meta`** score entry for raw latency/cost telemetry. **`auto_commit`** now persists **`parameters`** on the created variation. Tests were updated so Phase 1 success no longer depends on duration gates. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4eb0bb0. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…optimization package (#162) **Requirements** - [x] I have added test coverage for new or changed functionality - [x] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [x] I have validated my changes against all supported platform versions **Describe the solution you've provided** Adds the ability to specify a specific variation when setting up your configuration within LaunchDarkly, or specify a specific variation key in the `from_options` method as long as the API key is present. This allows a user to optimize against a specific variation rather than the default. **Describe alternatives you've considered** This is a relatively straightforward feature request. Allowing users to specify a specific variation cuts down on toil of managing your agents in the UI (don't need to change targeting or create a dummy config). **Additional context** This is purely additive. Functionality for configs not using the variation key, or that don't have it set, is unchanged -- it will continue using the default pulled via the SDK. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes which agent instructions/model/tools seed an optimization run; wrong variation_key or API failures abort the run, but default behavior when unset is unchanged. > > **Overview** > Adds **`variation_key`** support so optimization can start from a named LaunchDarkly AI config variation instead of the SDK’s context-evaluated default. > > When **`variation_key`** is set (on **`OptimizationOptions`** / **`GroundTruthOptimizationOptions`**, or **`variationKey`** on the remote agent optimization config), **`_get_agent_config`** loads that variation through a new **`LDApiClient.get_ai_config_variation`** beta REST call and uses its **instructions**, **tools**, and **model** (`modelConfigKey` plus optional **parameters**). An optional reused **`api_client`** avoids extra clients in **`optimize_from_config`**. Missing variations surface **`LDApiError`** with no SDK fallback. > > **`optimize_from_options`**, ground-truth options, and **`optimize_from_config`** forward the key and enforce **API key** + **`project_key`** when it is set. Tests cover the API client, agent config wiring, and entry points. > > **Note:** **`dataclasses.py`** currently declares **`variation_key` twice** on both options types (duplicate field definitions in the diff)—worth cleaning before merge. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit bc2508e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
…n-dict JSON - Store the context selected at the start of each optimization run (`self._ld_context`) and use it in `_evaluate_config_judge` instead of always falling back to `context_choices[0]`. When multiple contexts are supplied, config-type judges now evaluate the same LaunchDarkly context that was used for the agent turn, so scores reflect the variation that was actually being optimized. - Guard `extract_json_from_response` against non-dict JSON: the direct `json.loads` fast-path now checks `isinstance(parsed, dict)` before returning. Previously a model response that was valid JSON but not an object (e.g. an array or bare string) would pass through and cause a `TypeError` inside `validate_variation_response`. Co-authored-by: Cursor <cursoragent@cursor.com>
…g lookup _find_model_config previously only compared against the catalog `id` (e.g. "gpt-4o"). When variation_key is used, ModelConfig.name is set from the variation's modelConfigKey, which the API returns in the `key` format (e.g. "OpenAI.gpt-4o"). The lookup now checks both fields, so cost/latency gate comparisons work correctly for runs started from a named variation. Co-authored-by: Cursor <cursoragent@cursor.com>
**Requirements** - [ ] I have added test coverage for new or changed functionality - [ ] I have followed the repository's [pull request submission guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests) - [ ] I have validated my changes against all supported platform versions **Related issues** https://launchdarkly.atlassian.net/browse/AIC-2980 **Describe the solution you've provided** Fixes a bug where ground truth optimizations that had only a single iteration + optimization loop would fail to ever mark the final item as completed. > 1 iterations weren't affected by this bug. Additionally moves the API call for the first result forward so that failure is evident more immediately. **Describe alternatives you've considered** bug fix <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes how optimization results are persisted and how Phase 2 marks non-winner iterations, which affects run completion and auto-commit; limited to the optimization client, not auth or data stores. > > **Overview** > Fixes ground-truth / config-driven runs that never show as **completed** when Phase 2 (cost/latency) or sparse iterations leave the backend on the wrong result record. > > On API persistence, **`success` patches now update `_last_optimization_result_id`** so auto-commit’s `createdVariationKey` lands on the winning iteration instead of the last POSTed Phase 2 row (which could stay RUNNING and dominate run status). > > **Phase 2** behavior changes: model evaluation order is built from **`model_choices` without re-listing the Phase 1 winner** (fallback to the winner only when there are no alternatives); latency/cost gates show **“evaluating” placeholders** during generation; agent turns use a **120s timeout**; and after a successful Phase 2, **non-winning iterations are PATCHed as run-level PASSED (`success`)** rather than `failure`, with the true winner updated last so internal winner state and callbacks stay correct. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit eba89fc. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
- Unpack (opts, summary_fn) tuple from _build_options_from_config in all
test _build() helpers and direct call sites that were using the old
single-return form
- Update model-name assertions to expect provider-prefix-stripped names
(e.g. 'gpt-4o-mini' instead of 'OpenAI.gpt-4o-mini')
- Validate that variable_choices is non-empty in OptimizationOptions.__post_init__
- Populate Phase 2 baseline before _run_cost_latency_phase when quality
passes on the very first iteration (both standard and GT paths)
- Record baseline from optimize_context after validation failure, not from
the failing last_ctx
- Merge variation-level tools into _current_parameters so _extract_agent_tools
can find them during agent turns
- Fix _interpolate regex to support hyphenated keys ({{user-id}} style)
- Add test classes: TestOptimizationOptionsValidation, TestInternalInterpolate,
TestPhase2BaselineSet, TestVariationToolsMergedIntoCurrentParameters
Co-authored-by: Cursor <cursoragent@cursor.com>
…503/529) Add _is_transient_error and _invoke_with_retry helpers that detect and retry on recoverable HTTP errors (Anthropic 529 Overloaded, 429 Rate Limit, 503 Service Unavailable) with exponential back-off. Apply them to all four LLM call-sites: agent evaluation, variation generation, and both judge evaluation paths. Works without importing any provider SDK — detects transient status via status_code/status/http_status attributes and a class-name keyword scan. Adds TestIsTransientError and TestInvokeWithRetry unit tests. Co-authored-by: Cursor <cursoragent@cursor.com>
- Pass expected_response from the last GT sample into _run_cost_latency_phase and forward it to _execute_agent_turn so Phase 2 quality judges can score against the ground-truth labeled answer (same context used in Phase 1) - Fix __version__ constant to match pyproject.toml (0.0.0 -> 0.1.0) - Add TestPhase2GroundTruthExpectedResponse test class to verify both fixes Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e8ec8d1. Configure here.
The persistence summary warning was intentionally removed per user request. This cleans up the now-dead counter variables (_post_failures, _patch_failures, _post_successes, _patch_successes) and simplifies _build_options_from_config to return the options directly rather than a tuple, since there is no longer a summary callback to return. Individual API failures continue to be logged at WARNING level at the point they occur in the LD API client. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
Related issues
This PR encapsulates all previous changes in the chain of optimization PRs that were broken up into smaller pieces. Consolidating here so that we can have a single commit/release of the package. The PRs were independently reviewed and approved.
Describe the solution you've provided
See:
#116
#117
#119
#122
#127
#128
#130
#135
#139
Note
High Risk
Large new surface area touching LaunchDarkly REST API writes (results, variations), API key handling, and complex multi-phase optimization state; regressions could affect live optimization runs or publish unintended variations when auto-commit is enabled.
Overview
This PR delivers the initial release of the optimization package under the renamed PyPI distribution
launchdarkly-ai-optimizerand import pathldai_optimizer, removing the oldldai_optimizationplaceholder (ApiAgentOptimizationClient).OptimizationClientis the new public entry point. Callers supplyhandle_agent_call/ optionalhandle_judge_callso all LLM traffic stays in app code; the library runs the loop: agent turns, parallel judges (LaunchDarkly config judges or inline acceptance statements), optional validation on extra random samples, LLM-driven variation generation when attempts fail, and optional Phase 2 cost/latency tuning after a quality win.Three entry modes are added:
optimize_from_options(local options, optionalauto_commit),optimize_from_ground_truth_options(all labeled samples must pass per attempt), andoptimize_from_config(loads agent optimization config from the REST API, streams iteration results via POST/PATCH, preflight write check, default auto-commit). Supporting pieces includeLDApiClient, option/dataclass types, prompts/utilities (retries, cost estimation, slug keys), expanded README, andPROVENANCE.mdfor wheel attestation verification.Reviewed by Cursor Bugbot for commit e8ec8d1. Bugbot is set up for automated code reviews on this repo. Configure here.