Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/workflows/publish-npm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Publish @synapt-dev/eval to npm

on:
release:
types: [published]
workflow_dispatch:
# Manual re-trigger — needed because this workflow has no version-mismatch retry path:
# a `release` event fires it exactly once, and if npm's version hadn't moved yet at that
# point (e.g. a Python-only release, or this repo's own version-sync-deferred slices),
# there is no automatic second attempt once the npm version is later bumped.

permissions:
contents: write
id-token: write

jobs:
publish:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ts
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "24"
registry-url: "https://registry.npmjs.org"

- run: npm ci

- run: npm run typecheck

- name: Check if this version is already published
id: check
run: |
VERSION=$(node -p "require('./package.json').version")
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
if npm view "@synapt-dev/eval@$VERSION" version >/dev/null 2>&1; then
echo "already_published=true" >> "$GITHUB_OUTPUT"
echo "@synapt-dev/eval@$VERSION is already published to npm -- skipping publish (this is expected on every release where only the PyPI side bumped, not an error)."
else
echo "already_published=false" >> "$GITHUB_OUTPUT"
fi

- run: npm publish --provenance --access public
if: steps.check.outputs.already_published != 'true'

- name: Generate SBOM
run: npm sbom --omit=dev --sbom-format cyclonedx > sbom.cdx.json

- name: Upload SBOM to release
if: github.event_name == 'release'
env:
GH_TOKEN: ${{ github.token }}
run: gh release upload ${{ github.event.release.tag_name }} sbom.cdx.json --clobber
44 changes: 44 additions & 0 deletions .github/workflows/publish-pypi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Publish synapt-eval to PyPI

on:
release:
types: [published]

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"

- run: pip install build

- run: python -m build

- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: pypi-dist
path: dist/

publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
steps:
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: pypi-dist
path: dist/

- uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
packages-dir: dist/
skip-existing: true
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## v0.1.0 (2026-05-07)

Initial release of @synapt/eval.
Initial release of @synapt-dev/eval.

### Components

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# @synapt/eval
# @synapt-dev/eval

[![PyPI](https://img.shields.io/pypi/v/synapt-eval)](https://pypi.org/project/synapt-eval/)
[![Python](https://img.shields.io/pypi/pyversions/synapt-eval)](https://pypi.org/project/synapt-eval/)
Expand Down
14 changes: 7 additions & 7 deletions docs/ts-migration.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# TypeScript Migration Guide

Migrate from a monolithic eval script to composable `@synapt/eval` adapters.
Migrate from a monolithic eval script to composable `@synapt-dev/eval` adapters.

## Before vs After

Expand All @@ -15,7 +15,7 @@ See the working examples in `ts/examples/migration/`:
Extract your retrieval/generation call into an adapter class:

```typescript
import type { RetrievalAdapter, RetrievalCandidate } from "@synapt/eval";
import type { RetrievalAdapter, RetrievalCandidate } from "@synapt-dev/eval";

class MyRetrievalAdapter implements RetrievalAdapter {
async retrieve(query: string, k = 10): Promise<RetrievalCandidate[]> {
Expand All @@ -30,7 +30,7 @@ class MyRetrievalAdapter implements RetrievalAdapter {
Replace hand-rolled precision/recall with standard primitives:

```typescript
import { precisionAtK, recallAtK } from "@synapt/eval";
import { precisionAtK, recallAtK } from "@synapt-dev/eval";

const p5 = precisionAtK(retrievedIds, expectedIds, 5);
const r10 = recallAtK(retrievedIds, expectedIds, 10);
Expand All @@ -41,7 +41,7 @@ const r10 = recallAtK(retrievedIds, expectedIds, 10);
Replace `if (p5 < 0.5) failures.push(...)` with:

```typescript
import { SuggestionEngine } from "@synapt/eval";
import { SuggestionEngine } from "@synapt-dev/eval";

const engine = SuggestionEngine.withDefaults();
const suggestions = engine.evaluateAll(results);
Expand All @@ -53,7 +53,7 @@ const suggestions = engine.evaluateAll(results);
Replace `console.log` with structured output:

```typescript
import { composeReportCard, generateMarkdown } from "@synapt/eval";
import { composeReportCard, generateMarkdown } from "@synapt-dev/eval";

const card = composeReportCard({ results, suggestions });
console.log(generateMarkdown(card));
Expand All @@ -75,7 +75,7 @@ Write results to JSON and use the GitHub Action:
With adapters, adding generation eval is one new class:

```typescript
import type { GenerationAdapter, GenerationOutput } from "@synapt/eval";
import type { GenerationAdapter, GenerationOutput } from "@synapt-dev/eval";

class MyGenerationAdapter implements GenerationAdapter {
async generate(query: string, context?: unknown[]): Promise<GenerationOutput> {
Expand All @@ -90,7 +90,7 @@ class MyGenerationAdapter implements GenerationAdapter {
Extend the suggestion engine with domain-specific rules:

```typescript
import { SuggestionEngine, suggestionRule, SEVERITY_WARNING } from "@synapt/eval";
import { SuggestionEngine, suggestionRule, SEVERITY_WARNING } from "@synapt-dev/eval";

const latencyRule = suggestionRule({ name: "high_latency" })((result) => {
// Custom rule for your domain
Expand Down
2 changes: 1 addition & 1 deletion ts/examples/migration/after.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* AFTER: Composable eval with @synapt/eval.
* AFTER: Composable eval with @synapt-dev/eval.
*
* Benefits:
* - Adapter pattern decouples eval from backend implementation
Expand Down
2 changes: 1 addition & 1 deletion ts/examples/migration/before.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* BEFORE: Monolithic eval pattern (typical pre-@synapt/eval approach).
* BEFORE: Monolithic eval pattern (typical pre-@synapt-dev/eval approach).
*
* Problems with this pattern:
* - Scoring, assertion, and reporting are interleaved
Expand Down
4 changes: 2 additions & 2 deletions ts/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion ts/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
{
"name": "@synapt/eval",
"name": "@synapt-dev/eval",
"version": "0.1.0",
"description": "Domain-agnostic eval framework for AI applications",
"repository": {
"type": "git",
"url": "https://github.com/synapt-dev/eval.git",
"directory": "ts"
},
"type": "module",
"exports": {
".": "./src/mod.ts"
Expand Down
Loading