Skip to content

BaseIntelligence/deepagent

BranchesTags

Repository files navigation

DeepAgent

Manufacture hard, Docker-verifiable SWE tasks as DeepAgent / Harbor pack trees.

Install · Env · CLI · Product N=9 · Hugging Face · Scoreboard · Architecture

Python License HF

Overview

DeepAgent builds agent-facing Harbor pack trees from multi-file hard Real-PR tasks. Each certified pack carries a real public repository URL, an immutable base commit, multi-file gold solution, held-out verifier tests, Docker oracle dual-truth (solution reward = 1, null reward = 0), and agent isolation.

This repository is a single installable package (deepagent) at git root. There is no nested second package and no legacy forge product path.

Surface Path / ref Role
Current product datasets/prod_hard_deepswe_med DeepSWE-median hard Real-PR, N=9 (M27 floors + M28 max 2 packs/repo, unique_repos=7)
HF stable pin BaseIntelligence/deepagent main Current product on Hub (N=9)
HF automation mirror BaseIntelligence/deepagent test CI/dev write target; also N=9
Scoreboard datasets/panel_prod_hard_deepswe_med_m28 Grok 4.5 + Kimi 2.7-code dual-model matrix
Product detail datasets/prod_hard_deepswe_med/PRODUCT_README.md Floors, p50 stats, diversity notes

Primary console entrypoint: deepagent (generate / upload / pull / eval / oracle / version).

Note: An older synthetic forge (swe-forge / Composer path) was removed from this monorepo. The only product story is DeepAgent real_pr Harbor A→Z. A thin internal compat entry swe-factory may still resolve to factory helpers; new work should use deepagent only.

Install

From a clean checkout of this repository root:

cd /path/to/deepagent          # git root (this directory)
python3 -m venv .venv
.venv/bin/pip install -U pip
.venv/bin/pip install -e ".[dev]"
cp .env.example .env           # set placeholders; never commit .env
.venv/bin/deepagent --help

Python ≥ 3.12. Package name deepagent exposes:

Entry Module Role
deepagent swe_factory.deepagent_cli:app Primary product CLI
swe-factory swe_factory.cli:app Optional internal compat only

Environment (.env)

Copy .env.example. Placeholders only — never commit secrets.

Variable Purpose
HF_TOKEN Hugging Face push/pull for BaseIntelligence/deepagent
OPENROUTER_API_KEY Live panel / eval model calls
OPENROUTER_BASE_URL Default https://openrouter.ai/api/v1
FACTORY_PANEL_MODELS Historical default includes k2.6; current median panel uses grok-4.5 + kimi-k2.7-code via explicit --model
FACTORY_BUDGET_USD Hard cap (default 600)
GITHUB_TOKEN / GH_TOKEN Live Real-PR mine. Prefer export GITHUB_TOKEN="$(gh auth token)" after gh auth login
OXYLABS_PROXY_URL SOCKS residential proxy for GitHub REST rate-limit relief
ALL_PROXY / HTTPS_PROXY Optional proxy chain so git/HTTPS clients share the same egress

GitHub 429 mitigation: authenticated REST (GITHUB_TOKEN / gh auth) plus optional SOCKS via OXYLABS_PROXY_URL (and/or ALL_PROXY / HTTPS_PROXY) is the primary anti-429 path. The Oxylabs realtime Web Scraper API is optional and not required for GitHub REST/Search mining.

Secrets never appear in logs, CLI output, or shipped datasets.

Current product (authoritative)

Field Value
Product root datasets/prod_hard_deepswe_med
N 9 certified packs
unique_repos 7
max packs / repo 2 (M28 diversity)
HF stable pin BaseIntelligence/deepagent revision main (N=9)
HF automation mirror same repo revision test (also N=9; CI/dev default write target)
Primary CLI deepagent (generate / upload / pull / eval / oracle)
Scoreboard datasets/panel_prod_hard_deepswe_med_m28
Default eval models x-ai/grok-4.5 + moonshotai/kimi-k2.7-code

M27 DeepSWE-median hardness floors

Every keep must pass:

  • Multi-file: source files ≥ 4, or hybrid files ≥ 3 + gold added ≥ 500 + hunks ≥ 14
  • source hunks ≥ 14
  • gold added lines ≥ 400
  • F2P nodes ≥ 5
  • HarborDocker dual-truth (sol = 1, null = 0) + prompt–verifier alignment
  • live-mined source_track=real_pr only (no fixture pad, no hybrid motors)
  • M28 diversity: max 2 packs per upstream repo

See datasets/prod_hard_deepswe_med/PRODUCT_README.md, coverage_stats.json, and median_stats.json.

Pack IDs (N=9)

realpr-click-3442 · realpr-itemadapter-101 · realpr-oauthlib-889 · realpr-packaging-1120 · realpr-packaging-1267 · realpr-rich-3930 · realpr-werkzeug-2637 · realpr-werkzeug-3116 · realpr-wtforms-923

Primary CLI (deepagent)

Commands (current product paths)

# Live-mine hard real_pr under M27 floors → prod_hard_deepswe_med
deepagent generate \
  --target 10 --min-packs 5 --max-packs 15 \
  --out datasets/prod_hard_deepswe_med \
  --live-mine --oracle docker --panel offline --pier scripted

# HarborDocker dual-truth on one pack (sol=1 / null=0)
deepagent oracle --pack-dir datasets/prod_hard_deepswe_med/tasks/realpr-click-3442

# Push pack trees + manifest to HF stable pin (main) and/or automation mirror (test)
deepagent upload \
  --src datasets/prod_hard_deepswe_med \
  --repo-id BaseIntelligence/deepagent \
  --revision main
deepagent upload \
  --src datasets/prod_hard_deepswe_med \
  --repo-id BaseIntelligence/deepagent \
  --revision test

# Pull packs from HF stable pin (main)
deepagent pull \
  --repo-id BaseIntelligence/deepagent \
  --revision main \
  --out datasets/hf_pull_main

# Pier mini-swe + Harbor dual-model eval (current median models)
deepagent eval \
  --product-root datasets/prod_hard_deepswe_med \
  --max-packs 9 --k 1 --n-concurrent 5 \
  --hard-stop-usd 600 \
  --model x-ai/grok-4.5 \
  --model moonshotai/kimi-k2.7-code \
  --out datasets/panel_prod_hard_deepswe_med_m28

deepagent version
deepagent --help
Command What it does
generate Live mine hard real_pr; product out datasets/prod_hard_deepswe_med; Docker oracle only; refuses fixture pad
oracle HarborDocker dual-truth cert on one pack dir (solution reward 1, null 0; refuse fake)
upload Validate local pack root, push trees + pack_manifest to HF. Prefer main for the stable pin; test for automation
pull Download pack trees from HF revision (main or test) into a local out dir
eval Pier + mini-swe-agent + HarborDocker model eval (n_concurrent 1..5, hard-stop $600, fidelity pier_miniswe_harbor). Current median panel: grok-4.5 + kimi-k2.7-code
version Package version identity

Operator A→Z loop

  1. Install root package (pip install -e ".[dev]") and fill .env.
  2. Generate live-mined hard real_pr packs into datasets/prod_hard_deepswe_med.
  3. Oracle-cert keep packs (Docker sol=1 / null=0).
  4. Upload to HF BaseIntelligence/deepagent revision main (stable) and keep test in sync.
  5. Pull (optional) a clean Hub mirror for offline eval.
  6. Eval Pier mini-swe + Harbor under the $600 hard-stop with Grok 4.5 + Kimi 2.7-code.

Scoreboard (M28 diversified panel)

Durable dual-model matrix on the current product (observational ranking only; dual-solve rate is the hardness quality gate ≤ 0.30):

Model pass@1 (k=1)
x-ai/grok-4.5 3/9 ≈ 0.33
moonshotai/kimi-k2.7-code 1/9 ≈ 0.11
dual_solve rate ≈ 0.11 (1/9)

Evidence:

Scoreboard is leaderboard / ranking only. Dual-model success does not auto-drop hardness packs (intrinsic policy).

Hugging Face (BaseIntelligence/deepagent)

Ref Content
main (stable product pin) Current N=9 packs from datasets/prod_hard_deepswe_med
test (automation / dev mirror) Same N=9 product; default write target for CI and live automation

Operator loop: generate → oracle → upload (prefer revision main for the stable public pin; keep test in sync) → pull → eval. Do not embed tokens in CLI flags; use HF_TOKEN env / .env only.

What you get in a pack

Each pack under tasks/<task_id>/:

task.toml                 # schema 1.1, repository_url, base_commit_hash
instruction.md
pre_artifacts.sh
environment/Dockerfile    # agent image @ base SHA; offline runtime
tests/
  Dockerfile
  test.sh
  grader.py
  config.json             # fail_to_pass / pass_to_pass node ids
  test.patch              # held-out verifier tests
solution/
  solution.patch          # multi-file product sources only
  solve.sh

Corpus-level artifacts at the pack root:

Artifact Role
pack_manifest.json Certified pack index + band metadata
PRODUCT_README.md Product N, floors, diversity summary
PROVENANCE.md License, upstream URL, base SHA, language per keep
coverage_stats.json / median_stats.json Structural p50 / repo diversity
report.md Language mix, funnel, spend, honesty notes (when present)
oracle_evidence.json / evidence/docker/ Docker sol/null dual-truth index

Architecture

flowchart LR
  Allow[Permissive allowlist] --> Mine[Live mine real_pr]
  Mine --> Env[Envbuild at base SHA]
  Env --> Label[Dual-run F2P / P2P]
  Label --> Oracle[Docker oracle sol=1 null=0]
  Oracle --> Export[Harbor pack export]
  Export --> Product[datasets/prod_hard_deepswe_med N=9]
  Product --> HF[HF BaseIntelligence/deepagent @main]
  Product --> HFtest[HF @test automation mirror]
  HF --> Eval[deepagent eval Pier mini-swe Harbor]
  Eval --> Board[panel_prod_hard_deepswe_med_m28]
Loading

Git is the authority for commits and patches. Live GitHub REST prefers GITHUB_TOKEN + optional SOCKS (OXYLABS_PROXY_URL). Realtime Oxylabs page scrape remains optional.

Honesty floors (brief)

  • Product N counts live-mined real_pr only. Fixture shortlists never pad current product N.
  • Docker oracle only on the certified path (HarborDockerVerifier); fake backends are refused.
  • Dual-truth required: solution reward = 1, null reward = 0.
  • Multi-file gold from a merged public PR; hard floors follow the DeepSWE-median band (files≥4 OR hybrid 3+500+14; hunks≥14; gold added≥400; F2P≥5) plus M28 max 2 packs/repo.
  • Panel / eval spend stops under the hard budget (default $600); never invent panel spend. Current median eval models: grok-4.5 + kimi-k2.7-code.
  • Secrets (HF / OpenRouter / GitHub / proxy passwords) stay in env / .env only — never in help examples, logs, or uploaded trees.
  • Under-supply and offline modes (panel offline, pier scripted when not live) must be stated honestly in ship reports.

Tests & CI

Local non-integration gate (matches CI quality job):

.venv/bin/ruff format --check src tests
.venv/bin/ruff check src tests
.venv/bin/mypy src
.venv/bin/python -m pytest tests -q -p no:cacheprovider -p no:httpbin -p no:cov \
  -m "not integration" -n 2 --maxprocesses=2 --tb=line

Focused suites:

.venv/bin/python -m pytest tests/test_deepagent_cli.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_hf_packs.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_ship_deepagent.py -q -p no:cacheprovider -n 0
.venv/bin/python -m pytest tests/test_package_layout.py -q -p no:cacheprovider -n 0

Prefer pytest -n 0 (or ≤ -n 2); never pytest -n auto on shared hosts.

How it works

  1. Live-mine multi-file Real-PR candidates (deepagent generate --live-mine).
  2. Envbuild agent images pinned at a real base SHA with offline runtime.
  3. Label fail-to-pass and pass-to-pass node ids via dual-run suites.
  4. Oracle-cert with Docker: solution reward 1, null reward 0; refuse fake.
  5. Export Harbor v1.1 tree with held-out tests and isolation-clean agent view.
  6. Upload / pull HF (BaseIntelligence/deepagent @main stable pin; @test automation mirror).
  7. Eval Pier mini-swe + Harbor under hard-stop budget (fidelity pier_miniswe_harbor) with Grok 4.5 + Kimi 2.7-code for the current median panel.

Documentation

Audience Guide Contents
Operators this README install, CLI, HF, current product N=9
Product consumers datasets/prod_hard_deepswe_med/PRODUCT_README.md floors, diversity, pack list
Scoreboard datasets/panel_prod_hard_deepswe_med_m28/SUMMARY.md Grok≈0.33 / Kimi2.7≈0.11 / dual≈0.11
Implementers docs/architecture.md pipeline stages and gates
Hardness policy docs/PRODUCT_HARDNESS.md DeepSWE-median floors, intrinsic, opt-out
M32 keep/drop docs/M32_KEEP_DROP.md monorepo cleanup boundaries

Repository layout

pyproject.toml                 # package name = deepagent
.env.example
src/swe_factory/
  deepagent_cli.py             # primary deepagent entrypoint
  cli.py                       # optional swe-factory compat entrypoint
  export/hf_packs.py           # HF upload / pull
  panel/eval_deepagent.py
  pipeline/ship_deepagent.py
  harbor/                      # pack export + docker oracle cert
  producers/                   # mine / labeling
  sources/                     # allowlist, git mine, SOCKS proxy
datasets/
  prod_hard_deepswe_med/       # CURRENT product: median + diversity N=9
  panel_prod_hard_deepswe_med_m28/  # authoritative Grok/Kimi2.7 scoreboard
fixtures/                      # unit shortlist only (never product N)
tests/
docs/
  architecture.md
  PRODUCT_HARDNESS.md
  M32_KEEP_DROP.md
assets/banner.png
.github/workflows/             # root package lint + unit gate

Historical bulk trees (test_n10, soft prod_hard_keep, older product archives, live materials workdirs) are not tracked in git after the M32 cleanup. Product truth is only prod_hard_deepswe_med N=9 + the M28 panel (+ HF mirrors).

Limits and non-goals

  • Current product N is prod_hard_deepswe_med / HF main (stable pin; test is the automation mirror at the same N=9).
  • Not a browser UI; CLI + Docker + Pier only.
  • Copyleft / unknown licenses are fail-closed and never appear in PROVENANCE.
  • On constrained hosts, prefer serial Docker cert and lower n_concurrent eval.
  • No Composer / Cursor / synthetic-forge product path.

Spend

Metric Value
Cap $600 (FACTORY_BUDGET_USD / eval hard-stop)
Default eval concurrency 1 (allowlist 1..5)
Current median eval models grok-4.5 · kimi-k2.7-code
M28 panel spend (durable) ≈ $23.64 under cap
Eval fidelity pier_miniswe_harbor

Source of truth: panel ledger_summary.json.

License

MIT (see LICENSE and pyproject.toml).

About

[🤖] DeepAgent is a Base tool for generating real-code software engineering benchmarks, exporting reproducible task workspaces, and evaluating agent patches with deterministic fail-to-pass tests.

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages