> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Compare outputs on stable validation samples and understand live, provisional results.

Evaluations answer whether a candidate preserves the project behavior you care about. They use stable validation rows and store one durable result per entry and participant.

> [card] [Run your first evaluation](https://docs.coreweave.com/model-distillation/studio/evaluations-quickstart.md)
>
> Compare a fine-tune with reference outputs using a head-to-head judge.

## Evaluation types

> [accordion] Head to Head
>
> A judge model compares each candidate response with a primary output. Use this for open-ended generation where exact text equality is too strict. Score = `(wins + 0.5 × ties) / completed rows`; 50% means parity with the primary.

> [accordion] Exact Match
>
> Compares canonical assistant outputs deterministically. Use it for structured or discrete answers where equivalent output should be identical.

> [accordion] Categorization
>
> Reads a boolean field and reports precision, recall, F1, and the confusion matrix. Unreadable candidate output hurts both precision and recall; unreadable reference output is skipped.

## Choose participants

Head-to-head evaluations accept original outputs, relabel runs, tuned models, and direct provider models. The first participant is primary. Fine-tunes from the dataset appear near the top and are grouped by base model.

Exact-match and categorization evaluations require a separate reference and accept model or provider candidates.

## Read live results

Results are provisional until all requested rows settle. The UI shows:

* completed rows versus total rows;
* wins, ties, losses, and score for head-to-head participants;
* failed rows and an immediate error-message summary;
* per-row verdicts in the dataset workbench.

Adding a participant or changing settings reconciles only affected work. Unchanged completed cases and reusable model outputs are preserved.

## Retries and failure

Each case receives up to three automatic attempts. The evaluation stops retrying after that bound and can complete with failed rows. Once it has settled, **Retry failed cases** creates a new batch containing only failures; completed results remain intact.

> [!TIP]
>
> Use identical routing parameters when comparing provider and fine-tuned models. A model that is already a project target automatically receives the highest matching project version's parameter overrides.

> [accordion] API: Retry failed cases (POST /evals/\{evalId}/retry)
>
> Wait until the evaluation has settled, then submit a failed-only retry batch. Completed results are preserved.
>
> ```bash theme={"system"}
> curl --request POST \
>   --url "https://forge.coreweave.com/api/distillation/v1/evals/$EVAL_ID/retry" \
>   --header "Authorization: Bearer $WANDB_API_KEY" \
>   --header "Forge-Entity: your-team"
> ```
>
> Use [Evaluations Quick Start](https://docs.coreweave.com/model-distillation/studio/evaluations-quickstart.md) for the create request. See [Retry failed evaluation cases](https://docs.coreweave.com/model-distillation/reference/management/evaluations/retry-failed-evaluation-cases.md) for this operation.
