feat: add manual judge evaluation (Judge, Evaluator, createJudge) (AIC-2665)

[ctawiah]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Related issues

• Jira: AIC-2665 — Step 5: AIEVALS (Judge, Evaluator & CreateJudge, manual-only)
• Epic: AIC-2629
• Spec: AIEVALS

Stacked on #174 (AIC-2664). Review/merge that first; the diff here is against feat/ai-sdk-tracker.

Describe the solution you've provided

Implements the manual-only evaluation path for AI Configs. v1.0 does not auto-invoke judges on completion/agent calls; the caller drives evaluation: createJudge() → judge.evaluate(...) → track the result yourself.

• Runner SPI + RunnerResult — caller-supplied model invocation. RunnerResult carries content, the run Metrics, and parsed structured output ({score, reasoning}). Provider-specific runners ship post-1.0.
• Judge — sampling is decided before invoking the model; input is formatted as MESSAGE HISTORY:\n{input}\n\nRESPONSE TO EVALUATE:\n{output}; the runner is invoked via the tracker's trackMetricsOf so invocation metrics are recorded; score (0.0–1.0, out-of-range → failure) and reasoning are parsed. The judge returns a JudgeResult but does not call trackJudgeResult — recording is the caller's responsibility. evaluateMessages renders <role>: <content> history and delegates to evaluate. Sampling rate is normalized (NaN/Infinity → 1.0, negative → 0.0, >1 → 1.0).
• Evaluator — runs a set of judges with per-judge fault isolation (a failing/timing-out judge yields a failed JudgeResult; others are preserved in order) and a per-judge timeout so a hung judge can't stall the chain. noop() returns an empty list with no warnings. Thread-safe; uses a short-lived executor per evaluate call.
• LDAIClient.createJudge — fires only $ld:ai:usage:create-judge, resolves the judge config through the internal evaluate path (so no $ld:ai:usage:judge-config event), and returns null if the config is disabled or no runner is supplied.
• README documents the manual-only flow and the auto-attach descope.

Async surface is synchronous, consistent with the rest of this server SDK; concurrency for per-judge timeout is internal to Evaluator.

Tests

• JudgeTest — scoring/metric key, input formatting, zero-sampling skip (runner not invoked), missing metric key, out-of-range score, missing reasoning, runner throw, runner failure metrics, evaluateMessages rendering, sample-rate normalization.
• EvaluatorTest — noop() empty, order preservation, fault isolation, timeout isolation, completion-order independence.
• LDAIClientImplTest — createJudge fires only create-judge (not judge-config), returns a Judge when enabled, null when disabled, null when no runner.

Describe alternatives you've considered

A CompletableFuture-based async API was considered but rejected for consistency with the synchronous server SDK surface. Automatic sample-rate-driven judge auto-attachment and provider runners are intentionally deferred past v1.0 (aligns with the .NET descope).

Additional context

JudgeResult was added in #174 (AIC-2664) and is reused here.

---

Note

Medium Risk
New public API and usage telemetry paths; judge runs depend on caller-supplied runners and correct manual tracking, but no changes to core flag evaluation or auth.

Overview
Adds manual-only AI response evaluation to the server AI SDK: callers use LDAIClient.createJudge() with a custom Runner, run Judge.evaluate() (or evaluateMessages()), and record scores via trackJudgeResult themselves—no auto-attachment on completion/agent calls in v1.0.

New public Runner / RunnerResult SPI for model calls; Judge applies sampling, formats judge input, invokes the runner through the config tracker for invocation metrics, and parses {score, reasoning}. Internal Evaluator runs multiple judges concurrently with per-judge timeouts and fault isolation. createJudge emits only $ld:ai:usage:create-judge (not judge-config) and returns null when disabled or no runner. README documents tracking and the manual judge flow.

^{Reviewed by Cursor Bugbot for commit f6d4a4c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 83a342f. Configure here.}

[cursor]

Sequential waits break judge timeouts

High Severity

Evaluator starts all judges concurrently but awaits each Future in list order with a full perJudgeTimeout on every get. That timeout is measured from each get call, not from when the judge task started, so later judges can run far longer than the configured cap and evaluate can take up to judges.size() × perJudgeTimeout when multiple judges hang.

 

^{Reviewed by Cursor Bugbot for commit 83a342f. Configure here.}

[jsonbailey]

This may come later but we typically use structured outputs and would need to define the output shape for the run.

[jsonbailey]

It might be worth talking about the plans for the experimental features of the SDK if you haven't already. The create judge method should be marked experimental. In node and python we don't accept a runner but build it internally so this is a change in the SDK. Not sure if this is temporary and will be addressed later.