Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/harness-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ jobs:
- name: Reject whitespace errors
run: git diff --check

- name: Check sibling-harness parity manifest
run: python scripts/parity_check.py

generated-profiles:
strategy:
fail-fast: false
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,16 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and

### Added

- Documentation parity with the sibling `claude-python-engineering-harness`:
`docs/EVALUATION.md` (reproducible evaluation guide and acceptance criteria),
`docs/ENTERPRISE_ROLLOUT.md` (ownership, distribution, MCP rollout, and metrics), and pt-BR
translations `docs/LOOPS.pt-BR.md`, `docs/UPGRADING.pt-BR.md`, and
`docs/ENTERPRISE_ROLLOUT.pt-BR.md`, under the language policy now documented in
`CONTRIBUTING.md`.
- `scripts/parity_check.py` with `parity-manifest.json` and `parity-exceptions.json`: a
deterministic check, shared with the sibling `claude-python-engineering-harness`, that fails CI
when a parity-required artifact is missing and is not declared as an intentional divergence.
Wired into CI.
- Phase 0-1 (report-only) Evidence-Gated Engineering Loop foundation, see `docs/LOOPS.md`:
- `.loop/**` and `scripts/loop_*` are now denylisted for agent writes in
`protect_sensitive_files.py` (template and plugin), so an agent cannot silently build
Expand Down
17 changes: 17 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,23 @@ evaluation order requires it.
Prefer changing the canonical template first, then port the equivalent plugin or profile change.
Regression tests verify duplicated security scripts remain identical.

## Language policy

English is the canonical language for all documentation. The following documents must also ship a
`.pt-BR.md` sibling, updated in the same change: `docs/LOOPS.md`, `docs/UPGRADING.md`, and
`docs/ENTERPRISE_ROLLOUT.md`. Other documents may be translated opportunistically, but a stale
translation is worse than none: if you cannot update the pair, say so in the pull request. The
sibling `claude-python-engineering-harness` follows the same policy so both harnesses keep the
same language matrix.

## Sibling-harness parity

This harness and `claude-python-engineering-harness` share a parity manifest
(`parity-manifest.json`, byte-identical in both repositories) checked in CI by
`scripts/parity_check.py`. When adding or removing a parity-relevant artifact, update the manifest
in both repositories in the same change, or declare an intentional divergence with a reason in
this repository's `parity-exceptions.json`.

## Pull requests

- Keep changes focused and link the relevant issue when one exists.
Expand Down
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,11 @@ uv run python scripts/quality_gate.py
```

See `VALIDATION.md` for the latest verification record and `SOURCES.md` for official OpenAI
references used by this port. See `docs/VERSIONING.md` for artifact lifecycles,
`docs/UPGRADING.md` for non-destructive upgrades, `CHANGELOG.md` for release history, and
references used by this port. See `docs/EVALUATION.md` for the reproducible evaluation guide and
acceptance criteria, `docs/VERSIONING.md` for artifact lifecycles, `docs/UPGRADING.md` for
non-destructive upgrades (também em português: `docs/UPGRADING.pt-BR.md`),
`docs/ENTERPRISE_ROLLOUT.md` for rollout guidance, `CHANGELOG.md` for release history, and
`SECURITY.md` for private vulnerability reporting. Community participation is covered by
`CONTRIBUTING.md`, `SUPPORT.md`, and `CODE_OF_CONDUCT.md`.
`CONTRIBUTING.md`, `SUPPORT.md`, and `CODE_OF_CONDUCT.md`. Structural parity with the sibling
`claude-python-engineering-harness` is checked in CI by `scripts/parity_check.py` against
`parity-manifest.json`.
86 changes: 86 additions & 0 deletions docs/ENTERPRISE_ROLLOUT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Enterprise rollout

*[Português](ENTERPRISE_ROLLOUT.pt-BR.md)*

Use the repository scaffold for project-specific policy and the plugin marketplace for reusable
capabilities.

## Recommended ownership

- Platform engineering owns the plugin, hook runtime, approved configuration, and CI baseline.
- Security owns blocked paths, dangerous command patterns, secret patterns, and exception
governance.
- Architecture owns dependency contracts and the standard ADR template.
- Each product team owns its `AGENTS.md`, project configuration, data inventory, and acceptance
criteria.

## Distribution model

1. Publish this repository in a controlled internal Git host.
2. Add it as an internal marketplace
(`codex plugin marketplace add <internal-git-or-path>`).
3. Validate and version the plugin before promotion.
4. Pin or approve versions through your configuration-management baseline where available.
5. Roll out first to a pilot group and inspect denials, false positives, latency, and developer
overrides.
6. Promote only after the generated project and plugin pass the validation checklist.

## Separation of concerns

- Keep project facts and commands in the repository (`AGENTS.md`, `.codex/config.toml`).
- Keep reusable procedures and skills in the plugin.
- Keep mandatory controls in hooks, sandboxing, CI, identity, network, and repository protection.
- Require explicit trust of `.codex/config.toml` and `.codex/hooks.json`; review new or changed
hooks with `/hooks` before trusting them.
- Do not place credentials in plugin settings or MCP configuration.
- Treat MCP servers and external integrations as data egress paths that require an explicit
threat model.

## MCP rollout model

Treat MCP as an integration platform, not a developer convenience toggle. Recommended progression:

1. Inventory the external system, owner, data classes, operations, authentication, and retention.
2. Pilot with a read-only identity and non-production data.
3. Review the server implementation, release process, dependency pinning, prompt-injection
exposure, and network destinations.
4. Publish approved project servers through `[mcp_servers.*]` in `.codex/config.toml`; the
project validator enforces TLS, environment-name credential indirection, exact
ephemeral-runner versions, direct STDIO commands, and bounded timeouts.
5. Keep credentials per user through environment indirection or a credential helper. Never place
secrets in shared configuration.
6. Monitor server and tool use through the platform's native OpenTelemetry export without
collecting full inputs or outputs.
7. Reapprove integrations periodically and revoke unused servers, scopes, and credentials.

For strict regulated environments, prefer a fixed approved server set or disable MCP entirely
until each integration has an approved threat model.

## Change governance

Every harness release should include:

- semantic version;
- release notes and migration notes;
- test evidence for allowed and denied hook cases;
- plugin validation evidence;
- compatibility statement for the supported Codex and Python versions;
- rollback instructions;
- named owner and exception process.

## Metrics

Track adoption and control effectiveness without collecting source code or prompts:

- repositories and developers on each harness version;
- hook denials by category, from each project's local `.codex/logs/hooks-audit.jsonl` (written by
`log_event` in `.codex/hooks/_common.py`; one JSON line per deny/block decision with timestamp,
hook name, category, decision, and tool name - never command text, file contents, or matched
values). Aggregate this file centrally through your existing log pipeline; it is not collected
automatically;
- false-positive and override rates;
- quality-gate duration and failure category;
- time to remediate secrets and vulnerable dependencies;
- percentage of projects with current lock files and documented data handling;
- MCP servers by owner, version, scope, and review expiry;
- OpenTelemetry-derived agent and tool-use metrics, kept to metadata.
89 changes: 89 additions & 0 deletions docs/ENTERPRISE_ROLLOUT.pt-BR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Rollout corporativo

*[English](ENTERPRISE_ROLLOUT.md)*

Use o scaffold do repositório para políticas específicas do projeto e o marketplace de plugins
para capacidades reutilizáveis.

## Ownership recomendado

- Engenharia de plataforma é dona do plugin, do runtime de hooks, da configuração aprovada e da
baseline de CI.
- Segurança é dona dos caminhos bloqueados, padrões de comandos perigosos, padrões de secrets e
da governança de exceções.
- Arquitetura é dona dos contratos de dependência e do template padrão de ADR.
- Cada time de produto é dono do seu `AGENTS.md`, configuração do projeto, inventário de dados e
critérios de aceite.

## Modelo de distribuição

1. Publique este repositório em um Git host interno controlado.
2. Adicione-o como marketplace interno
(`codex plugin marketplace add <git-interno-ou-caminho>`).
3. Valide e versione o plugin antes da promoção.
4. Fixe ou aprove versões pela sua baseline de gestão de configuração, quando disponível.
5. Faça o rollout primeiro para um grupo piloto e inspecione negações, falsos positivos, latência
e overrides de desenvolvedores.
6. Promova apenas depois que o projeto gerado e o plugin passarem no checklist de validação.

## Separação de responsabilidades

- Mantenha fatos e comandos do projeto no repositório (`AGENTS.md`, `.codex/config.toml`).
- Mantenha procedimentos e skills reutilizáveis no plugin.
- Mantenha controles obrigatórios em hooks, sandboxing, CI, identidade, rede e proteção de
repositório.
- Exija trust explícito de `.codex/config.toml` e `.codex/hooks.json`; revise hooks novos ou
alterados com `/hooks` antes de confiar neles.
- Não coloque credenciais em configurações de plugin ou de MCP.
- Trate servidores MCP e integrações externas como caminhos de saída de dados que exigem um
threat model explícito.

## Modelo de rollout de MCP

Trate MCP como uma plataforma de integração, não como uma conveniência de desenvolvedor.
Progressão recomendada:

1. Inventarie o sistema externo, dono, classes de dados, operações, autenticação e retenção.
2. Pilote com uma identidade somente-leitura e dados não produtivos.
3. Revise a implementação do servidor, processo de release, pinning de dependências, exposição a
prompt injection e destinos de rede.
4. Publique servidores aprovados do projeto via `[mcp_servers.*]` em `.codex/config.toml`; o
validador do projeto exige TLS, indireção de credenciais por nome de variável de ambiente,
versões exatas de runners efêmeros, comandos STDIO diretos e timeouts limitados.
5. Mantenha credenciais por usuário via indireção de ambiente ou credential helper. Nunca coloque
secrets em configuração compartilhada.
6. Monitore uso de servidores e ferramentas pelo export nativo de OpenTelemetry da plataforma,
sem coletar entradas ou saídas completas.
7. Reaprove integrações periodicamente e revogue servidores, escopos e credenciais sem uso.

Para ambientes regulados estritos, prefira um conjunto fixo de servidores aprovados ou desabilite
MCP inteiramente até que cada integração tenha um threat model aprovado.

## Governança de mudanças

Cada release do harness deve incluir:

- versão semântica;
- release notes e notas de migração;
- evidência de teste para casos permitidos e negados dos hooks;
- evidência de validação do plugin;
- declaração de compatibilidade com as versões suportadas de Codex e Python;
- instruções de rollback;
- dono nomeado e processo de exceção.

## Métricas

Acompanhe adoção e efetividade dos controles sem coletar código-fonte ou prompts:

- repositórios e desenvolvedores em cada versão do harness;
- negações de hooks por categoria, a partir do `.codex/logs/hooks-audit.jsonl` local de cada
projeto (escrito por `log_event` em `.codex/hooks/_common.py`; uma linha JSON por decisão de
negação/bloqueio com timestamp, nome do hook, categoria, decisão e nome da ferramenta - nunca
texto de comando, conteúdo de arquivo ou valores identificados). Agregue esse arquivo
centralmente pela sua pipeline de logs existente; ele não é coletado automaticamente;
- taxas de falso positivo e de override;
- duração do quality gate e categoria de falha;
- tempo para remediar secrets e dependências vulneráveis;
- percentual de projetos com lock files atualizados e tratamento de dados documentado;
- servidores MCP por dono, versão, escopo e expiração de revisão;
- métricas de uso de agente e ferramentas derivadas de OpenTelemetry, restritas a metadados.
85 changes: 85 additions & 0 deletions docs/EVALUATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Evaluation guide

This guide makes the harness claims reproducible. It separates repository checks from checks run in
freshly generated projects and records the commands, expected artifacts, and acceptance criteria.
It expands the README's five-minute evaluation.

## Five-minute evaluation

From a clean checkout with Python, Git, and uv installed:

```bash
python bootstrap.py --name demo-service --package demo_service \
--target ../demo-service --profile service --git-init --lock
cd ../demo-service
uv sync --frozen --all-groups
uv run python scripts/quality_gate.py
```

The generated project should contain:

```text
demo-service/
├── .agents/skills/ # invocable project skills
├── .codex/ # config.toml, hooks.json, and hook scripts
├── .github/workflows/ # CI invoking the project-owned quality gate
├── docs/ # architecture, privacy, MCP, and observability guidance
├── scripts/ # deterministic quality and policy validators
├── src/demo_service/ # Clean Architecture package roots
├── tests/ # starter regression tests
├── .harness.json # generation metadata and content hashes
├── AGENTS.md
├── Dockerfile
└── pyproject.toml
```

Trust the repository before expecting `.codex/config.toml` and `.codex/hooks.json` to load, and
review new hooks with `/hooks` in Codex. The evaluation needs no credentials.

## Profile comparison

| Capability | service | library | workspace |
|---|---|---|---|
| Installable root package | Yes | Yes | No |
| Runtime container | Yes | No | No |
| Strict Mypy and tests | Yes | Yes | Per member |
| Clean Architecture boundaries | Yes | Yes | Per configured root |
| Observability starter (OpenTelemetry) | Optional | No | Per member |
| Intended use | Deployable backend | Reusable package | Multi-package repository |

## Acceptance criteria

The repository is ready for release when all of the following are true at the release commit:

1. The repository-owned gate (`uv run python scripts/quality_gate.py`) passes, including
regression tests, Ruff, format, Mypy, Bandit, pip-audit, and the governance gate.
2. Fresh `service`, `library`, and `workspace` projects render for every supported Python version.
3. Each generated project passes frozen dependency sync and its own `scripts/quality_gate.py`.
4. The service image builds and runs as a non-root user; non-service profiles emit no Dockerfile.
5. Merge, dry-run, check, conflict numbering, file-mode preservation, and symlink-confinement
regressions pass.
6. Hook and MCP tests cover malformed input, secret scanning, sensitive paths, trust
classification, and prohibited literal credentials.
7. `VALIDATION.md` records the date, Python version, release commit, commands, and observed
results.

## Recording results

Do not update pass counts by hand without running the corresponding command. For a release,
capture:

```bash
git rev-parse HEAD
python3 --version
uv run python scripts/quality_gate.py
git diff --check
```

Then run the generated-profile matrix in CI or reproduce each supported profile and Python version
locally. Link the successful workflow run from `VALIDATION.md`; do not treat a previous run from a
different commit as evidence for the release.

Dependabot checks pinned GitHub Actions and container references that appear directly in workflow
and Docker files. The Python base-image digests in `bootstrap.py` are template data and require a
manual refresh for every supported Python minor. Review upstream release notes, verify each digest
with `docker buildx imagetools inspect`, and require the complete CI matrix before merging updates.
Loading