feat(i18n): automated translation pipeline with machine review gate#623
feat(i18n): automated translation pipeline with machine review gate#623Timur Tukaev (tym83) wants to merge 18 commits into
Conversation
In-tree localization engine for cozystack.io (hack/i18n/), building on the existing source_digest freshness convention (hack/check-i18n.sh): - worklist.py: diff detector (missing/stale pages via source_digest) - translate.py: Claude Opus translator with glossary, per-language style guides, protected code/shortcodes/URLs, SEO front-matter transcreation - ahrefs_keywords.py: per-locale SEO keyword maps (optional, degrades gracefully without AHREFS_API_KEY) - i18n-translate.yml: nightly + dispatch; PR then auto-merge (publish-then-review) - config.yaml: languages, model routing (all Opus), scope, blog cutoff (last ~2 months), publish mode Add Spanish (es) and Portuguese-BR (pt-br): hugo.yaml + production mounts, i18n/es.toml + i18n/pt-br.toml (key parity verified). hreflang alternates + x-default in head-end.html. CODEOWNERS exempts content/<lang>/ so the pipeline auto-merges translations while code stays gated. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
… OAuth, daily runner Rework the pipeline per maintainer decisions: - Auth: OAuth subscription (Max), not a metered API key — bare Anthropic() client resolves the logged-in credential; ANTHROPIC_API_KEY warns. - Machine review gate on EVERY page before publish: translate -> back-translate + meaning-drift compare -> two native reviewers (technical editor for fluency, Cozystack maintainer for technical correctness) -> revise-if-needed, bounded by review.max_rounds. Pages written atomically only after passing. - Daily-until-limit: run stops cleanly on 429 and resumes next day; adds hack/i18n/run-daily.sh (commit + daily PR + auto-merge). - New prompts: back-translate(-compare), review-editor, review-maintainer, revise. config.yaml gains auth/rate_limit/back_translation/review sections and a translation_review front-matter stamp. - Workflow switched off metered API key to optional CLAUDE_CODE_OAUTH_TOKEN; local daily runner is the primary path. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
The base anthropic SDK cannot use a Claude subscription (metered API only), so the model-call layer now uses the Claude Agent SDK (claude-agent-sdk), which reads CLAUDE_CODE_OAUTH_TOKEN from `claude setup-token`. - translate.py: call() runs a single-turn, tool-less Agent SDK query (allowed_tools=[], max_turns=1) via asyncio; hard-fails if ANTHROPIC_API_KEY is set (it would shadow the subscription); warns if CLAUDE_CODE_OAUTH_TOKEN is missing; rate-limit detection maps to a clean daily stop. - run-daily.sh: require CLAUDE_CODE_OAUTH_TOKEN, forbid ANTHROPIC_API_KEY, install claude-agent-sdk. - workflow: CLAUDE_CODE_OAUTH_TOKEN secret + install claude-agent-sdk and the claude-code CLI. - config.yaml/README: document the Agent SDK subscription path + setup. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Verified on a real machine: the Agent SDK authenticates off an existing `claude` CLI login — CLAUDE_CODE_OAUTH_TOKEN is only needed headless/CI. So: - run-daily.sh no longer hard-fails without the token; it requires only that ANTHROPIC_API_KEY is unset (the money guard) and that some subscription credential exists (claude login or token). - run-daily.sh bootstraps a venv (.venv-i18n) and runs the pipeline from it — distro Pythons are PEP 668 externally-managed, so plain pip install fails. - translate.py: drop the misleading missing-token warning. - gitignore the venv. Smoke-tested end to end: claude-agent-sdk 0.2.121, single-turn tool-less query against claude-opus-4-8 returns text over the subscription. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Ahrefs API v3 access is an expensive, separately-licensed add-on, and the English source is already SEO-optimized during authoring — the translation inherits that intent. The translate prompt still transcreates title/description to read naturally per locale, so nothing is lost by removing the keyword step. Removes ahrefs_keywords.py, keyword-maps/, the ahrefs config block, the keyword-hint injection in translate.py, the workflow step, and the AHREFS_API_KEY secret. No secrets are required for translation now beyond the subscription credential. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
…injection Adversarial review found the pipeline did not actually work and that its central claim was not enforced by code. Fixes: P0 publish: run-daily.sh used "git diff --quiet", which ignores untracked files — every new translation is untracked, so it reported "nothing to publish" forever while source_digest on disk made the page look fresh the next day (silent permanent no-op). Now uses "git status --porcelain" scoped to content/<lang>, branches from a fresh origin/main (days no longer stack), and honours publish_mode (pr_only was documented but never implemented). P0 gate honesty: _parse_verdict failed OPEN — refusals, prose and truncated JSON all silently passed, making the gate unfalsifiable; and the revise loop stamped pages that never cleared it as auto-reviewed. Now fails CLOSED (unparseable = revise), a missing prompt file hard-fails instead of sending a reviewer out with no instructions, and pages are stamped honestly (auto-reviewed vs auto-reviewed-with-findings) with per-run counters. P0/P1 correctness: reject replies without the ===BODY=== protocol instead of writing the model's preamble as page content; verify every protected placeholder survived (a dropped fence silently deleted a code block); hard-fail on unparseable front matter instead of dropping slug/date/aliases. P1 scope: the docs version is now read from hugo.yaml (latest_version_id) instead of hardcoded v1.4 while latest is v1.5 — 164 of 352 pages were being translated into a noindex'd version. Scope drops 352 -> 188 pages (2096 -> 1128 jobs). fnmatch replaced with pathlib semantics so the config no longer lies about what it matches. P1 security: the workflow interpolated github.event.inputs into the shell, letting anyone with write access exfiltrate the subscription token; inputs now go through env and are quoted. P1 auth: auth is now a config choice (oauth-subscription | api-key) so the project can move from a maintainer's subscription to an org-owned key without code changes. P2: read the whole front matter for source_digest (a long one made pages look permanently stale and re-translate daily); unknown --lang errors instead of silently doing nothing; stable YAML dump; drop dead config/code; stale v1.2 default in head-end.html. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Resolves the governance objection from review: nothing about CODEOWNERS or branch protection changes any more, and no machine output reaches the production site without a maintainer merging it. - CODEOWNERS: reverted to the original single rule. The previous exemption for content/<lang>/ would have removed required review from those paths for ANY author, not just the bot — on a tree where goldmark renders unsafe HTML. - publish_mode: pr_only is now the default and is actually honoured. - Cadence: the runner still runs DAILY (each day's quota is used in full), but the week's output accumulates onto one i18n/week-<ISO week> branch, so maintainers review and merge a single translation PR per week instead of a stream. A new week branches fresh from origin/main. - Docs: README documents the daily-run/weekly-PR split, the honest auto-reviewed vs auto-reviewed-with-findings stamps, and an "Ownership and continuity" section stating the intent to move from a maintainer's subscription to an organization-owned API key. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Second review pass over the pipeline. The findings that mattered were all silent-failure modes: the run would look successful while producing wrong or no output. Nested front matter. The landing page renders its hero and cards from taglines[], benefits[], and features[]; only top-level keys were translated, so regenerating a localized homepage would copy the English hero back over a hand-translated one and drop locale-only keys (seo:, l10n:) that natives had added. Front-matter values are now addressed by path, translated at any depth, and applied onto a deep copy, with target-only keys re-attached. The front-matter wire format moves from "key: value" lines to JSON, which also fixes multi-line descriptions being shredded by the line parser. Fail closed. A "revise" verdict with an empty findings list no longer passes the gate; latest_docs_version raises instead of returning None (which made the scope filter a no-op and pulled in every old docs version); a duplicated placeholder is rejected like a dropped one; a missing front-matter key is rejected rather than shipping a page whose body is translated but whose hero is still English. run-daily.sh no longer stashes in the working tree — it requires a clean dedicated clone and restores the starting branch on exit. Stashing wrote conflict markers into translations and could pop someone else's stash. Also fixes a BrokenPipeError that killed the run before the first page, and stops treating normal check-i18n.sh staleness as a publish blocker. Tests cover the pure functions. One of them corrected a docstring that claimed path globbing narrowed `*.md` to the repo root; it does not, and the config does not want it to. The GitHub workflow is removed: the pipeline runs from a maintainer's clone, and a workflow implied CI-hosted credentials we deliberately do not use. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Localized pages are indexed from day one so readers get the docs now instead of after a native review that may take months. That trade is only honest if the page says what it is, and until now nothing did. Add a translation-banner partial that carries a machine-translation notice and links to the English original. The rule is fail-safe: it shows on any non-English page unless the front matter says `translation_review: ratified`. Keying off the field's presence instead would have exempted every page from the i18n PoC, which is machine output with no such field — precisely the pages the notice exists for. Wired into the same four layouts as version-banner, so docs, blog, and pages are covered; the homepage is deliberately left out, since covering it means overriding the theme's home layout and a warning across a marketing hero costs more than it buys. Strings are localized in all seven locales. Stop declaring es and pt-br in hugo.yaml. Declaring a language with no content does not build nothing: Hugo emits /es/, /es/tags/, /es/categories/, /es/topics/, /es/article_types/ and /es/404.html regardless, all `index, follow` and self-canonical — twelve empty but indexable pages across the two, which is the thin content the rest of this work is careful to avoid. They are commented out in hugo.yaml and config.yaml together, with the enable order documented: translate first, then declare, in the PR that carries the content. The pipeline now writes `l10n: mt`, reusing the site's existing convention for how a page was localized rather than inventing another field. README documents what each of the three front-matter fields means and who reads it. README also now says plainly what the review gate is not: both "native reviewers" are the same model as the translator with different prompts, which measures self-consistency, not native ratification. `auto-reviewed` must not be read as "a human checked this" — only a human sets `ratified`, and only that drops the banner. Adds a rollback section, since a pipeline that publishes to production needs a documented way to stop. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
… scope The preferred-terms list covered de, ru, es, and pt-br. es and pt-br are not enabled; zh-cn and hi are, and had zero entries — the two live languages with nothing keeping their terminology consistent had the coverage the disabled ones got. Adds all eight terms for both. `Tenant` and `tenant` are in do_not_translate and preferred respectively, which reads as a contradiction and is not one: the capitalized term is the API kind a reader has to match against `kind: Tenant` in a manifest, the lower-case one is an ordinary noun that has to be translated or the prose is unreadable. Both the glossary and the translate prompt now say so, and a test asserts any other overlap between the lists is a real contradiction. Excludes oss-health/** — five pages that are front matter plus `layout: oss-health-app`, a dashboard rendered client-side from live English data. Translating them wraps localized chrome around an English dashboard. They also carry the site's only `lede` field, a user-visible string that was not in the translatable-key list and would have shipped in English on every localized copy; dropping these pages is what makes that list true rather than approximately true. Tests now run against the real content tree: that config.yaml and hugo.yaml agree on which languages are enabled, and that no unrecognized user-visible front-matter key has appeared. The second is the one that would have caught `lede`. Scope is now 183 pages x 4 languages = 720 jobs. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
run-daily.sh forwards the same "$@" to both scripts, so `--limit 3` made worklist.py exit with a usage error. The preview was wrapped in `|| true`, so it failed silently — and only when --limit was passed, which is exactly the pilot-run case it exists to preview. An unknown --lang printed "all languages up to date" and exited 0, since the filter simply matched nothing. It is now an error. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
A pilot run produced a page stamped `auto-reviewed-with-findings` and no record anywhere of what the findings were. The stamp told a maintainer that something was wrong and gave them nothing to act on, which makes the distinction between it and `auto-reviewed` close to useless. translate_page now returns the findings still open on the final round. The runner prints them per page and writes a markdown report, which run-daily.sh posts as a PR comment. A comment, not the PR body: the report file holds only the last run, while the PR accumulates a week of daily runs — rewriting the body each day would drop the earlier days' findings. Comments accumulate on their own, so the thread becomes the week's log. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Two pilot runs on the same release blog post: de cleared the gate in 2m54s, ru took 11m24s through the revise loop. Per-page cost is dominated by whether the revise loop runs, not by the page — which puts the 720-job backlog at roughly 35-140 hours of wall clock before daily limits are even considered, and is the concrete reason the backlog needs an organization API key rather than a personal subscription. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Commenting es and pt-br out of the pipeline config stopped them being translated at all, which is not the goal — the goal is that they are not *served* until they have content. The two lists answer different questions and were wrongly kept identical. config.yaml (what gets translated) now covers all six languages. hugo.yaml (what gets built and served) still declares four; es and pt-br are declared in the PR that lands their content. The invariant is one-directional, and the test now says so: translated but not declared is how a language starts; declared but not translated means the site serves something nothing keeps fresh. A second test asserts every declared language actually has content, which is the failure that started this — Hugo emits ~6 indexable pages for a declared language with an empty content tree. Backlog is now 183 pages x 6 languages = 1084 jobs. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
A virtual reviewer on the ru pilot caught this, which is the gate doing exactly
its job: protect() masked each {{< figure >}} shortcode wholesale, so its
caption (rendered as visible text under the image) and alt (screen readers,
SEO) stayed English on every localized page — four English captions in a row on
a translated release post.
protect() now splits a shortcode into protected structure and exposed
visible-text attribute values (caption, alt, title). src, width, delimiters,
and param names stay byte-for-byte; the values translate like ordinary prose.
Shortcodes with no such attribute are still masked wholesale, unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <timur.tukaev@aenix.io>
Product decision: the machine-translation notice belongs where a wrong technical detail is costly — an operator running a translated command — not on the blog, marketing pages, or the homepage hero. Wired into docs/baseof.html only; removed from page/single, blog/baseof, and resources/list. The partial's guard is unchanged and layout-agnostic, so coverage is now purely a question of which layouts call it. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
A pilot on the v1.4 release blog post (63 opaque placeholders, 49 of them inline code) had the model non-deterministically drop five of them. The gate correctly refused to write the page — a silently deleted code block is worse than a retry — but a page that fails this way on every daily run would never publish while re-spending quota each day. translate.py now retries a page a few times within the run on a protocol error, since the loss is non-deterministic and usually clears on a fresh attempt. A page that fails every attempt is skipped as before and stays in the worklist; README documents that a reproducible failure means translate that page by hand. Adds --path to translate one exact page (pilot runs, or re-translating a single page after editing its source). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: tym83 <timur.tukaev@aenix.io>
Two pipeline outputs for reviewers to judge quality on, both of the same source
page (docs/v1.5/getting-started/install-kubernetes.md) so they can be compared
side by side. Both cleared the review gate (translation_review: auto-reviewed);
inline code, the {{% ref %}} shortcode, and front-matter structure are preserved
verbatim, and each carries the machine-translation banner shown on docs pages.
These are real, reviewable output — not fixtures. The English source is
untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: tym83 <timur.tukaev@aenix.io>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces an automated, machine-reviewed translation pipeline for the Cozystack website, adding localization scripts, prompts, style guides, and tests under hack/i18n/, alongside initial support for Spanish and Brazilian Portuguese. It also integrates a machine-translation disclaimer banner into the documentation layout and adds hreflang alternates for SEO. Feedback on these changes highlights a non-standard Hugo API usage (.Sites.Default) in the translation banner, a duplicate entry in .gitignore, and formatting conflicts in the translation and revision prompts where the instructions ask for key: value lines instead of the JSON format expected by the parsing script.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| {{- $src := .Sites.Default.Language.Lang -}} | ||
| {{- if and (ne .Lang $src) (ne .Params.translation_review "ratified") -}} |
There was a problem hiding this comment.
.Sites.Default is not a standard Hugo API and may cause template evaluation errors depending on the Hugo version. The standard and safe way to retrieve the default content language is .Site.DefaultContentLanguage.
| {{- $src := .Sites.Default.Language.Lang -}} | |
| {{- if and (ne .Lang $src) (ne .Params.translation_review "ratified") -}} | |
| {{- $src := .Site.DefaultContentLanguage -}} | |
| {{- if and (ne .Lang $src) (ne .Params.translation_review "ratified") -}} |
| # i18n pipeline venv (bootstrapped by hack/i18n/run-daily.sh) | ||
| .venv-i18n/ | ||
|
|
||
| # i18n pipeline: per-run artifacts, not content | ||
| hack/i18n/last-run-findings.md | ||
| .venv-i18n/ |
There was a problem hiding this comment.
| ===FRONTMATTER=== | ||
| <the translated values, one `key: value` per line, same keys, same order> | ||
| ===BODY=== | ||
| <the translated body> |
There was a problem hiding this comment.
The system prompt instructs the model to output the front matter as key: value lines, but the translation pipeline (translate.py via _FM_PROTOCOL) expects a JSON object. This conflict can confuse the model and lead to protocol parsing errors. Updating the prompt to match the JSON protocol will improve reliability.
| ===FRONTMATTER=== | |
| <the translated values, one `key: value` per line, same keys, same order> | |
| ===BODY=== | |
| <the translated body> | |
| ===FRONTMATTER=== | |
| <the translated values as a JSON object, same keys, same order> | |
| ===BODY=== | |
| <the translated body> |
| ===FRONTMATTER=== | ||
| <corrected key: value lines, same keys, same order> | ||
| ===BODY=== | ||
| <corrected body> |
There was a problem hiding this comment.
Similar to the translation prompt, the revision system prompt instructs the model to output key: value lines for the front matter, which conflicts with the JSON object format expected by _FM_PROTOCOL in translate.py. Updating this to instruct the model to return a JSON object will prevent protocol errors during the revision loop.
| ===FRONTMATTER=== | |
| <corrected key: value lines, same keys, same order> | |
| ===BODY=== | |
| <corrected body> | |
| ===FRONTMATTER=== | |
| <corrected JSON object, same keys, same order> | |
| ===BODY=== | |
| <corrected body> |
Summary
Builds the automated translation pipeline on top of the i18n proof-of-concept in #593. Where #593 established the multi-language site structure and a few hand-checked pages, this turns translation into a repeatable, reviewable process: it discovers what changed in the English source, translates it through a machine review gate, and opens a weekly PR for a maintainer to merge. Nothing reaches production without that merge.
Stacked on #593 — please review and merge that first; this PR targets
poc/i18n-multilangand will retarget tomainonce #593 lands.What
source_digest(sha256 of the English source, the same conventionhack/check-i18n.shalready enforces) drives a worklist of missing/stale pages per language. Scope is the latest docs version only, plus recent blog posts — older docs versions arenoindex, so translating them would spend budget on pages search engines ignore.caption/alttext inside shortcodes is translated (it renders to readers);src/widthare not.translation_review: auto-reviewed; one that runs out of revise rounds with findings still open is stampedauto-reviewed-with-findingsand its findings are posted to the weekly PR so a maintainer can triage them. Only a human setsratified.ratified. The banner is docs-only by design — the blog, marketing pages, and homepage hero do not carry it.i18n/week-<ISO week>branch, so maintainers review and merge a single translation PR per week.publish_mode: pr_only— CODEOWNERS and branch protection are untouched.docs/v1.5/getting-started/install-kubernetes.md) are included so quality can be judged directly.Why
The English docs are the source of truth and change constantly; native localization takes months. This lets translated docs ship and be indexed immediately (publish-then-ratify), with the banner keeping that honest, while native review happens asynchronously and is tracked per page. The review gate is not a substitute for native ratification — both virtual reviewers are the same model as the translator, so the gate measures self-consistency and catches the obvious failures, no more. It is deliberately conservative: fail-closed on unparseable reviewer verdicts, refuses to write a page with a dropped code block, and never publishes without a maintainer merge.
Notes for reviewers
auth: api-key), no code change. See README "Ownership and continuity".hugo.yaml(no content shipped). Declaring a language before its content exists publishes empty indexable pages; a test enforces the ordering.python3 hack/i18n/test_i18n.py.Preview
The included sample pages render under
/ru/docs/v1.5/getting-started/install-kubernetes/and/de/…in the deploy preview, each with the machine-translation banner.