Evaluators

Evaluator definitions — the metrics and judges used in evaluation runs.

List Evaluator Catalog Types

List the JSON schema types the evaluator catalog understands.

List Evaluator Catalog Templates

List evaluator templates from the catalog.

Fetch Evaluator Catalog Template

Fetch one evaluator template by key.

List Evaluator Catalog Presets

List presets defined against one evaluator template.

Fetch Evaluator Catalog Preset

Fetch one evaluator preset by template and preset key.

Create Evaluator

Create an evaluator artifact, its first variant, and its initial revision.

Fetch Evaluator

Fetch an evaluator artifact by id.

Edit Evaluator

Edit an evaluator artifact's metadata.

Archive Evaluator

Soft-delete an evaluator artifact.

Unarchive Evaluator

Restore a soft-deleted evaluator artifact.

Query Evaluators

Query evaluator artifacts with filters and pagination.

Create Evaluator Variant

Create a new variant on an existing evaluator.

Fetch Evaluator Variant

Fetch an evaluator variant by id.

Edit Evaluator Variant

Edit a variant's metadata.

Archive Evaluator Variant

Soft-delete an evaluator variant.

Unarchive Evaluator Variant

Restore a soft-deleted evaluator variant.

Query Evaluator Variants

Query evaluator variants with filters, reference scoping, and pagination.

Fork Evaluator Variant

Fork an evaluator variant into a new variant.

Retrieve Evaluator Revision

Retrieve one evaluator revision, either directly or via an environment key.

Deploy Evaluator Revision

Pin an evaluator revision into an environment revision under a key.

Create Evaluator Revision

Create a new revision on an evaluator variant.

Fetch Evaluator Revision

Fetch a specific evaluator revision by id.

Edit Evaluator Revision

Edit a revision's metadata.

Archive Evaluator Revision

Soft-delete an evaluator revision.

Unarchive Evaluator Revision

Restore a soft-deleted evaluator revision.

Query Evaluator Revisions

Query evaluator revisions with filters, reference scoping, and pagination.

Commit Evaluator Revision

Commit a new revision on an evaluator variant.

Log Evaluator Revisions

List the revision log of an evaluator variant.

Resolve Evaluator Revision

Resolve embedded references on an evaluator revision's `data`.

Create Simple Evaluator

Create an evaluator via the simple surface.

List Evaluator Templates

List the legacy built-in evaluator templates.

Fetch Simple Evaluator

Fetch one evaluator via the simple surface.

Edit Simple Evaluator

Edit an evaluator via the simple surface.

Archive Simple Evaluator

Soft-delete an evaluator via the simple surface.

Unarchive Simple Evaluator

Restore a soft-deleted evaluator via the simple surface.

Query Simple Evaluators

Query evaluators via the simple surface with filters and pagination.