20 May 2026
GenAI eval harness before feature
If the vendor API is the risk, we build the eval harness first and the feature second.
GenAI features fail in production for a predictable reason: the vendor model changed, the prompt drifted, and nobody had a number for “how correct” before the customer noticed.
Where the model is the risk, we build the eval harness first. A frozen set of cases with a stated, reviewed accuracy number; versioned prompts; and the latency/cost measured before anyone calls the feature ready. Retrieval is evaluated separately from generation, so a regression in one does not hide behind the other.
The harness
- Frozen eval set, versioned prompts, latency and cost per case.
- Retrieval accuracy and generation accuracy measured independently.
- Every model bump re-runs the harness before it reaches staging.
Shipping the harness is the feature — the UI that calls it is the easy part.