feat: add REST API server and Knowledge Workbench web UI

[jidechao]

Summary

Adds a production-ready REST API (FastAPI) and a bundled Knowledge Workbench web UI to OpenKB, so a knowledge base can be built, queried, and maintained entirely from the browser, Postman, or any HTTP client — no CLI required.

The two are co-served from one origin: openkb-api mounts the built SPA at / while exposing the JSON/SSE API under /api/v1.

REST API (openkb/api.py, openkb/watch_service.py)

• GET /kbs + POST /init — list and create knowledge bases (resolvable by short name via OPENKB_KB_ROOT)
• POST /add (multipart, SSE) — upload + compile documents with per-file progress
• POST /query (SSE) — one-shot question, streamed answer
• POST /chat + POST /chat/sessions{,/load,/delete} (SSE) — multi-turn chat with persisted, resumable sessions
• POST /list, /status — inventory and index stats
• POST /lint (optional fix), POST /remove (SSE), POST /recompile (SSE)
• POST /watch/{start,stop,status} + GET /watch/events (SSE) — background file watcher
• Bearer-token auth; a shared iter_agent_response_events layer means the CLI and API emit identical event streams
• A Postman collection is included at openkb-postman.json

Knowledge Workbench (frontend/ → built web/)

A React + Vite single-page app (dark, three-pane):

• Overview — stat cards (indexed/concepts/summaries/reports), clickable concept chips, recent docs, activity
• Documents — drag-and-drop multi-file upload with per-file SSE progress, hash table, delete with confirmation
• Query / Chat — streamed answers, live retrieval & reasoning inspector timeline, GFM Markdown rendering (react-markdown + remark-gfm + rehype-highlight), multi-turn sessions with history/resume/delete
• Maintenance — lint (optional auto-fix), recompile (all/single doc), file-watcher toggle
• Responsive: panes collapse to a single column on narrow screens

initialize_kb inherits the operator's project-root config.yaml and LLM credentials (.env, with OPENKB_* server vars filtered out), so a KB created from the UI runs out of the box.

Build / packaging

• web/ holds the pre-built bundle and is mounted by the API; frontend/ source is included and node_modules/dist are gitignored
• New entry point: openkb-api (also python -m openkb.api)

Test plan

• pytest tests/test_api.py tests/test_api_watch.py tests/test_watch_service.py tests/test_remove.py — 126 passed
• npm run build produces web/index.html + web/assets/*
• Manual: KB switch, upload → SSE progress → table update, streamed query with tool_call timeline, chat create/resume/delete, lint/recompile/watch

Notes

• This is a new capability layered on top of existing wiki/agent internals; core compile/query agents are unchanged in behavior — a shared SSE event helper was factored out so CLI and REST emit the same stream.
• Happy to split this into separate PRs (API vs. Web UI) if that's easier to review, or adjust to fit the project's conventions.

[KylinMountain]

@jidechao
Thanks, this is a big and useful addition! A few things before I review in depth:

1. Could you add a few screenshots / a short GIF of the Workbench? Hard to review the UI from the bundled JS alone.

2. Some changes look unintentional — please revert:

   • config.yaml.example: defaults were flipped to model: openai/deepseek-v4-flash / language: zh (looks like a local dev value). Please restore gpt-5.4 / en.
   • openkb/cli.py: a UTF-8 BOM got added to line 1 — please strip it.

3. Same question for the committed web/ build output — better to ship frontend/ source + a build step than to vendor the bundle.

I'll follow up with implementation feedback after.

[jidechao]

@jidechao Thanks, this is a big and useful addition! A few things before I review in depth:

1. Could you add a few screenshots / a short GIF of the Workbench? Hard to review the UI from the bundled JS alone.

2. Some changes look unintentional — please revert:

   • config.yaml.example: defaults were flipped to model: openai/deepseek-v4-flash / language: zh (looks like a local dev value). Please restore gpt-5.4 / en.
   • openkb/cli.py: a UTF-8 BOM got added to line 1 — please strip it.

3. Same question for the committed web/ build output — better to ship frontend/ source + a build step than to vendor the bundle.

I'll follow up with implementation feedback after.

[jidechao]

@KylinMountain Thanks for the review. All three actionable points are addressed in the latest push (3d3e086):

1. Screenshots / GIF — pending. I'll capture and attach them once I rebuild and run the Workbench locally (Overview / Documents / Query·Chat / Maintenance). Will follow up in this thread.

2. Reverted unintentional changes:

   • config.yaml.example: restored model: gpt-5.4 / language: en (the openai/deepseek-v4-flash / zh flip was a local dev value). git diff against base is now clean.
   • BOM: stripped the stray UTF-8 BOM from openkb/cli.py line 1. While auditing, found the same BOM in tests/test_api.py (same batch of accidental edits) and removed it too.

3. Unvendored web/: deleted the committed build bundle, added web/ to .gitignore, and documented the build step in the README quick-start. frontend/ source stays tracked; npm run build regenerates web/. The API's _mount_web_ui already no-ops when web/ is absent, so openkb-api falls back to API-only with no breakage.

Verification: pytest tests/test_api.py tests/test_api_watch.py tests/test_watch_service.py tests/test_remove.py → 126 passed; npm run build regenerates web/ cleanly.

Standing by for the implementation-layer feedback.

[KylinMountain]

Review — REST API + Knowledge Workbench

Nice, largely-additive feature — CLI behavior is unchanged and the shared SSE helper is only consumed by the new API paths. The high-severity issues cluster in three areas, flagged inline.

Must-fix (correctness / data):

• Mutation endpoints (recompile / lint --fix / init) run sync, lock-holding work directly on the event loop; remove/add already use run_in_threadpool, these don't. Because kb_ingest_lock's reentrancy bookkeeping is threading.local, concurrent same-KB requests on the one event-loop thread bypass mutual exclusion → KB corruption (different-KB requests instead stall the loop). [api.py:321/541/581]
• Watcher stop() can orphan an in-flight compile and a restart then double-ingests raw/. [watch_service.py:204]
• _save_query_answer: empty/colliding exploration slug silently loses data (acute for CJK questions), drops the CLI's ghost-link stripping, and writes the question into YAML unescaped. [api.py:801]

Should-fix: leaked/never-terminating /watch/events SSE [api.py:1145]; frontend doesn't abort the stream on unmount/KB-switch [useSSEStream.js]; CORS * + credentials [api.py:699]; token sent to a user-supplied origin + non-constant-time compare [sse.js:50, api.py:772]; per-KB LLM key ignored due to override=False [api.py:321].

Verified clean (so you don't re-chase): KB short-names go through validate_kb_name (fullmatch on [A-Za-z0-9_-]+) → no path traversal; upload filenames via Path(...).name; the .env inheritance filter correctly excludes OPENKB_* server vars while keeping LLM_API_KEY; _sse always emits single-line data:; CLI query/chat streaming is unchanged by the refactor.

Minor / cleanup: duplicated KB-dir check (api.py:730 & 787 → extract _is_kb_dir); redundant _setup_llm_key wrapper; dead model.dict() Pydantic-v1 branch.

Thanks for the thorough test plan and the offer to split the PR — splitting API vs Web UI would make these land more incrementally.

[KylinMountain]

Also, I suggest setting the default webpage language to English, but with a language switcher available—that might work better. @jidechao

[KylinMountain]

Follow-up: structure & docs organization

A few organizational notes beyond the inline correctness comments — all about keeping the surface lean and single-sourced.

1. api.py re-implements logic the CLI already owns

The API redoes core operations instead of sharing one source of truth, so the two paths will drift (some already have):

• _save_query_answer (api.py:801) diverges from query --save — it drops the ghost-wikilink stripping the CLI applies, and its slug collapses to empty for CJK / punctuation-only questions (flagged inline). Extract a shared save_exploration(kb_dir, question, answer) used by both CLI and API.
• KB-dir check is inlined twice (api.py:730 and :787) — extract _is_kb_dir(path).
• _setup_llm_key (api.py:795) is just a thin wrapper over the CLI's.

Net: let the API be a thin layer over the same core helpers the CLI calls, so behavior can't fork.

(watcher.py + watch_service.py are fine as-is — low-level debounced watch vs. per-KB service registry is a reasonable split.)

2. openkb-postman.json (844 lines) duplicates the OpenAPI schema

FastAPI already serves an interactive schema at /docs and /openapi.json. A hand-maintained Postman collection is a second API surface that will drift from the routes. Suggest dropping it and pointing users at /docs (the OpenAPI URL can be imported into Postman directly if someone wants a collection).

3. README is carrying a full API reference (+463 lines)

The bulk of the README addition is (a) the Workbench setup + per-feature tour and (b) a full REST endpoint reference (Auth, SSE, per-endpoint HTTP request/response). That inlines an API manual into the top-level README, which pulls against the recent direction of keeping the README to "what features exist + Quick Start + command tables" and pushing deep usage into examples/.

Suggested shape:

• README: a short (~3–5 line) subsection each for the Web UI and the REST API — one-line positioning, one start command, and a link.
• Move the endpoint reference + Workbench tour into examples/rest-api/README.md, matching the existing examples/<case>/ convention (examples/slides/, examples/visualize/, …). Note that docs/ is gitignored, so the published docs in this repo live under examples/ — that's the natural home for the API reference, not docs/.

Happy to help wire up examples/rest-api/ if that's useful.

[jidechao]

@KylinMountain Thanks for the thorough review — all of it is addressed in d7e82b8. Replying point by point below.

Review summary

Must-fix

1. Concurrent same-KB mutation (api.py) — Added a per-KB asyncio.Lock (_kb_mutation_lock) that serializes lint --fix and recompile before they enter kb_ingest_lock. Since kb_ingest_lock tracks reentrancy in threading.local, two same-KB requests on the one event-loop thread were mis-counted as re-entrant and bypassed mutual exclusion; the asyncio lock closes that window. init now runs initialize_kb + register_kb_alias in run_in_threadpool (its own thread → its own threading.local), matching add/remove.

2. Watcher stop() orphan + double-ingest (watch_service.py) — stop() now signals the worker and observer, joins both (5s timeout), and if the worker is still alive mid-compile it leaves the state in place with a watcher_draining marker so start() refuses to spawn a duplicate. The worker itself only ingests debounced file events (no initial full-scan of raw/), and _add_for_api dedupes by hash, so a restart can't re-ingest existing files.

3. _save_query_answer (api.py) — Extracted a shared save_exploration(kb_dir, question, answer) in cli.py that both the CLI query --save and REST /query?save call. It strips ghost wikilinks, generates a CJK-safe slug (hash fallback when the question collapses to an empty slug), uniquifies on collision, and escapes the question for YAML frontmatter.

Should-fix

4. Leaked /watch/events SSE (api.py) — _stream_watch_events now has a default timeout cap (OPENKB_WATCH_SSE_TIMEOUT, 300s), checks await request.is_disconnected(), and terminates with done on watcher stop / max_events / timeout.

5. Stream not aborted on unmount/KB-switch (useSSEStream.js) — Added an AbortController; start() aborts any prior stream before opening a new one, and a cleanup effect aborts on unmount so no callbacks fire after the view is gone.

6. CORS * + credentials (api.py) — allow_credentials is forced off when origins is a wildcard, while still allowing unauthenticated cross-origin GETs.

7. Token compare + cross-origin base (api.py, sse.js) — Token check now uses hmac.compare_digest. The SPA validates the configured API base via checkBaseSafety() before every request and warns when it's cross-origin and non-HTTPS (same-origin/empty base is always safe).

8. Per-KB LLM key ignored via override=False (api.py) — Introduced LlmCredentialBundle (reads the KB .env via dotenv_values without polluting os.environ) and a per-request LitellmModel instance in RunConfig, so concurrent requests for different KBs can't read each other's key/headers/timeout. A shared resolve_per_request_overrides keeps litellm.* override semantics identical to the CLI path.

   Note: the bundle isolation surfaced one regression — build_run_config_from_bundle prefixed the model with litellm/ (an Agent-layer convention), but LitellmModel passes model verbatim to litellm.acompletion, so REST query/chat failed with litellm/openai/<model>. Fixed by passing model=model directly, with a regression test (test_build_run_config_from_bundle).

Minor/cleanup

9. api.py:730 & 787 duplicated KB-dir check → extracted _is_kb_dir.
10. Redundant _setup_llm_key wrapper → moved _setup_llm_key to cli.py, API imports it.
11. Dead model.dict() (pydantic v1) → model_dump().

Follow-up: language

Default is now English with a zh/en toggle (icon button in the sidebar, persisted to localStorage via i18n.jsx). index.html lang/<title> also default to English.

Follow-up: structure & docs

• API re-implementing CLI logic: covered by the shared save_exploration above and the _init_kb_for_api thin wrapper. The API now layers over the same core helpers the CLI calls.
• openkb-postman.json: dropped — FastAPI serves the interactive schema at /docs and /openapi.json (importable into Postman directly).
• README: slimmed to short subsections for the Web UI and REST API with links; the full endpoint reference and Workbench tour moved to examples/rest-api/README.md, matching the examples/<case>/ convention.

Verification

pytest (query/api/api_watch/watch_service/per_request_overrides/save_exploration) → 118 passed. npm run build regenerates web/ cleanly. Manual end-to-end: KB switch, upload + SSE progress, streamed query/chat, lint/recompile/watch.

Screenshots from the first round are in the thread above; happy to capture fresh ones against the updated UI if useful.

[KylinMountain]

@jidechao there's conflicts you need to resolve, thanks.