> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations Quick Start

> Run a head-to-head evaluation between your reference output and one or more candidate models.

Head-to-head evaluation is the best starting point for open-ended tasks where exact text equality would be too strict.

> [!NOTE]
>
> Before you begin, create a dataset with validation rows and add every fine-tune or provider model you want to compare.

**1. Open the dataset's Evaluate tab**

Select **New evaluation**, choose **Head to Head**, and give the evaluation a descriptive name.

**2. Choose the reference**

Select original or relabeled dataset output as the primary participant. Candidates will be scored against this desired behavior.

**3. Add candidate models**

Add fine-tunes from the dataset first, then any base or provider models you want to compare. Fine-tunes are grouped by base model in the selector.

**4. Configure the judge**

Select a capable judge model and write instructions that define what a better answer means for this task. Start with 200 validation rows when enough are available.

**5. Run and inspect**

Start the evaluation. Scores update while rows finish. Open individual rows to read the judge explanation and use the failure indicator to inspect provider or parsing errors.

**6. Decide whether to deploy**

A 50% score means parity with the reference: wins count as 1, ties as 0.5, and losses as 0. Promote only after the completed score and per-row behavior meet your threshold.

Failed cases retry automatically up to three attempts. Once the run settles, **Retry failed cases** requeues only failures and preserves completed results.

> [accordion] API: Create an evaluation (POST /evals)
>
> The first participant is the primary. Replace the candidate model reference with the artifact or provider model you want to evaluate:
>
> ```bash theme={"system"}
> curl --request POST \
>   --url "https://forge.coreweave.com/api/distillation/v1/evals" \
>   --header "Authorization: Bearer $WANDB_API_KEY" \
>   --header "Forge-Entity: your-team" \
>   --header "Content-Type: application/json" \
>   --data "{
>     \"name\": \"ticket-classifier-comparison\",
>     \"dataset_id\": \"$DATASET_ID\",
>     \"spec\": {
>       \"type\": \"h2h_judge\",
>       \"judge_model_ref\": \"openai/gpt-5.6-sol\",
>       \"judge_prompt\": \"Choose the more correct and helpful answer. Return a tie when they are equally good.\"
>     },
>     \"participants\": [
>       {\"kind\": \"original\"},
>       {\"kind\": \"model\", \"model_ref\": \"wandb-inference/your-fine-tuned-model\"}
>     ],
>     \"sample_size\": 200
>   }"
> ```
>
> The response contains the evaluation ID. Poll `GET /evals/{evalId}` until the evaluation settles; see [Create an evaluation](https://docs.coreweave.com/model-distillation/reference/management/evaluations/create-an-evaluation.md).
