Skip to content

fix(schemas): normalize confidence from 0-100 scale before Pydantic validation#93

Open
Shrotriya-lalit wants to merge 1 commit into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-89-confidence-normalization
Open

fix(schemas): normalize confidence from 0-100 scale before Pydantic validation#93
Shrotriya-lalit wants to merge 1 commit into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-89-confidence-normalization

Conversation

@Shrotriya-lalit

Copy link
Copy Markdown

Problem

LLMFinding and MetaAnalyzerFinding both declare confidence: float = Field(ge=0.0, le=1.0). When a local model (e.g. Ollama-hosted Llama or Mistral) returns confidence as an integer on a 0–100 scale ("confidence": 85), Pydantic raises a ValidationError at the le=1.0 boundary and crashes the meta-analyzer for the entire file — all findings for that file are silently dropped.

Root Cause

The constraint is evaluated after type coercion but before any user logic, so there is no opportunity to rescale at the model level.

Fix

Add a mode="before" @field_validator on confidence in both schemas:

@field_validator("confidence", mode="before")
@classmethod
def _normalize_confidence(cls, v: object) -> float:
    v = float(v)          # raises on non-numeric
    if v > 1.0:
        v = v / 100.0     # rescale 0-100 → 0-1
    return max(0.0, min(1.0, v))

This also clamps negative values and values above 100 (e.g. 110) gracefully instead of crashing.

Tests

  • Confidence 0.85 (already valid) passes through unchanged
  • Confidence 85 (0-100 integer) is rescaled to 0.85
  • Confidence 1001.0, 00.0
  • Confidence -50.0, 1101.0
  • Non-numeric input raises ValueError
  • Both LLMFinding and MetaAnalyzerFinding validated

Closes #89

Checklist

  • make lint passes
  • Tests added for new behaviour
  • DCO sign-off on commit (git commit -s)

…alidation

LLMFinding and MetaAnalyzerFinding both hard-fail with le=1.0 when Ollama
(or other local models) return confidence as an integer on a 0-100 scale.
Add a mode="before" field_validator that: converts to float, divides by 100
if the value exceeds 1.0, then clamps to [0.0, 1.0].  This also handles
negative values and values above 100 gracefully rather than crashing the
meta-analyzer for the entire file.

Closes NVIDIA#89

Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM stage crashes on OpenAI-compatible/local endpoints (Ollama): models return confidence 0-100 but output schema validates 0-1

1 participant