llm-evals

https://github.com/kigster/llm-evals

Run multi-case evaluation suites against datasets using any model available through OpenRouter or the direct Jev adapter. Write an adapter that calls your application and a Scorer that measures the outcomes you care about. Compare models, or track how prompt and parameter changes affect results. Combine metrics such as accuracy, errors, latency, and cost into weighted scores, then review rankings and case-level results in an HTML report. Integrates with Braintrust Evals and several others. Paid integration is not required.