> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upload a Dataset

> Create a dataset from examples you already have by writing entries or uploading a JSONL file through the Management API.

If your training examples already exist outside the project, write them into a dataset through the Management API instead of recording traffic through the [inference proxy](https://docs.coreweave.com/model-distillation/proxy/overview.md). The result works with relabeling, fine-tuning, and evaluations like any other dataset.

Uploads are available only through the Management API. The UI shows the resulting dataset, but it doesn't offer a file picker.

This page describes how to create an empty dataset, add rows to it in batches or as one JSONL file, and choose which rows to hold out for evaluation.

## Prerequisites

Before you begin, make sure you have the following:

* A Forge API key for a team that has access to Model Distillation. The examples read it from the `WANDB_API_KEY` environment variable, pass it in the `Authorization` header, and name the entity in the `Forge-Entity` header.
* A project in that team. The project alias appears in every request path. To create one, see the [Quick Start](https://docs.coreweave.com/model-distillation/quickstart.md).
* The tools for the tab you plan to use in [Upload a file](#upload-a-file). You need only one of the following sets:
  * `curl` together with `jq`.
  * Python 3.9 or later with the `requests` package.
  * Node.js 18 or later, with no extra packages.

## How it works

Every upload goes into an existing dataset. You create an empty dataset once, then add rows to it in either of two ways:

* **Write entries** sends up to 5,000 rows in one request and returns the result immediately. Use it for small datasets and for incremental additions.
* **Upload a file** sends one JSONL file of up to 1 GiB and 100,000 rows straight to object storage. Model Distillation validates every line and then commits all rows at once.

Both paths use the same row format and the same rules. A write either commits every row or changes nothing, and each successful write moves the dataset to a new revision. Completed relabels and evaluations on the dataset are then marked stale.

New rows go to the training split unless they say otherwise. To hold rows out for evaluation, set `split` on each row, or move a share of the rows afterward. See [Choose the validation rows](#choose-the-validation-rows).

## Prepare the rows

Each row is an OpenAI Chat Completions request whose last message is the assistant response to train toward. Model Distillation stores the preceding messages, plus any `tools`, `tool_choice`, and `response_format`, as the input, and the final assistant message as the output.

```json theme={"system"}
{"messages":[{"role":"system","content":"Classify the ticket."},{"role":"user","content":"My invoice is wrong."},{"role":"assistant","content":"billing"}],"row_id":"ticket-1042","metadata":{"customer":"acme"}}
```

A row must contain at least two messages and can carry up to 1,000. Message roles can be `system`, `developer`, `user`, `assistant`, or `tool`, and the last message must be from the assistant. `tools` accepts up to 128 function tools and 1 MiB of JSON.

### Optional fields

Rows can carry the following optional fields:

| Field | Purpose |
| - | - |
| `row_id` | Stable identifier within the dataset, up to 512 characters. Upserts match on it. Without it, a digest of the messages and request fields identifies the row, so sending the same example twice is detected. |
| `group_id` | Keeps related rows, such as the turns of one conversation, in the same split. Up to 512 characters. Defaults to the row identity. |
| `split` | `train` or `val`. New rows default to `train`. |
| `metadata` | Filterable attributes, such as the user or customer segment. Up to 64 KiB and 10 levels deep. |
| `provenance` | Where the row came from, such as the source system and trace ID, with the same limits as `metadata`. |

Rows must not contain other top-level fields.

### Append or upsert

Every write uses one of two modes:

* **`append`**, the default, rejects a row whose `row_id` already exists in the dataset.
* **`upsert`** replaces the existing entry. If the new row omits `group_id`, `split`, `metadata`, or `provenance`, the stored value is kept. Relabeled outputs for a replaced entry are discarded.

In either mode, a `row_id` that appears twice in one request or file is rejected, and a write that would place one group in both splits fails with `group_split_conflict`.

## Create an empty dataset

Create the dataset by sending only a name. Without `params`, the request creates an empty dataset that is ready immediately, and the response is `201 Created`:

```bash theme={"system"}
curl --request POST \
  --url "https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Forge-Entity: your-team" \
  --header "Content-Type: application/json" \
  --data '{"name": "Support tickets (uploaded)"}'
```

Keep the `id` from the response. The following examples refer to it as `DATASET_ID`.

## Write entries

To add up to 5,000 rows in one request, send them to the entries endpoint as a JSON array. You can also send the same rows as NDJSON, one row per line, with `Content-Type: application/x-ndjson`:

```bash theme={"system"}
curl --request POST \
  --url "https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/$DATASET_ID/entries?mode=append" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Forge-Entity: your-team" \
  --header "Content-Type: application/x-ndjson" \
  --data-binary @tickets.jsonl
```

The response is `200 OK` with `created`, `updated`, the new `revision`, and `entry_counts` by split. If any row is invalid, the request is rejected and the error names the row by its index. In NDJSON, line N is index N - 1.

To check a batch without changing the dataset, add `dry_run=true`. Model Distillation validates the rows, reports what would change, and rolls the write back.

## Upload a file

For files larger than one request allows, upload the file in four stages: open an upload, send the file with one `PUT`, complete the upload, and poll until the rows are committed. The following tabs show the whole flow as one script. Numbered comments mark the stages, and [How the script works](#how-the-script-works) explains each one.

The scripts use a project alias of `ticket-classifier` and a file named `tickets.jsonl`, and read the dataset ID from the `DATASET_ID` environment variable. Before you run a script, replace the entity, alias, and filename with your own, and set `WANDB_API_KEY` and `DATASET_ID` in your environment. Use a new `idempotency_key` for each distinct upload.

> [tab] curl
>
> ```bash theme={"system"}
> #!/usr/bin/env bash
> # Requires curl and jq.
> set -euo pipefail
>
> API="https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/$DATASET_ID"
> ENTITY="your-team"
> FILE="tickets.jsonl"
> auth=(--header "Authorization: Bearer $WANDB_API_KEY" --header "Forge-Entity: $ENTITY")
>
> # 1. Open the upload.
> size=$(wc -c < "$FILE" | tr -d ' ')
> curl --silent --fail --request POST --url "$API/uploads" "${auth[@]}" \
>   --header "Content-Type: application/json" \
>   --data "$(jq -n --arg name "$FILE" --argjson size "$size" '{
>     idempotency_key: "support-tickets-v1",
>     file: { name: $name, size_bytes: $size },
>     mode: "append"
>   }')" --output upload.json
> upload_id=$(jq -r '.id' upload.json)
> url=$(jq -r '.upload.url' upload.json)
>
> # 2. Send the file. The signed URL carries its own authorization.
> curl --silent --fail --upload-file "$FILE" "$url"
>
> # 3. Complete the upload. This queues validation and the commit.
> curl --silent --fail --request POST --url "$API/uploads/$upload_id/complete" "${auth[@]}" --output /dev/null
>
> # 4. Poll until the upload reaches a final state.
> while true; do
>   curl --silent --fail --url "$API/uploads/$upload_id" "${auth[@]}" --output status.json
>   case "$(jq -r '.state' status.json)" in
>     committed|failed|cancelled|expired) break ;;
>   esac
>   sleep 10
> done
> jq '{state, counters, error}' status.json
> ```

> [tab] Python
>
> ```python theme={"system"}
> # Requires Python 3.9 or later and the requests package.
> import os
> import time
>
> import requests
>
> API = f"https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/{os.environ['DATASET_ID']}"
> HEADERS = {
>     "Authorization": f"Bearer {os.environ['WANDB_API_KEY']}",
>     "Forge-Entity": "your-team",
> }
> FILE = "tickets.jsonl"
>
> # 1. Open the upload.
> response = requests.post(
>     f"{API}/uploads",
>     headers=HEADERS,
>     json={
>         "idempotency_key": "support-tickets-v1",
>         "file": {"name": FILE, "size_bytes": os.path.getsize(FILE)},
>         "mode": "append",
>     },
> )
> response.raise_for_status()
> upload = response.json()
>
> # 2. Send the file. The signed URL carries its own authorization.
> with open(FILE, "rb") as handle:
>     requests.put(upload["upload"]["url"], data=handle).raise_for_status()
>
> # 3. Complete the upload. This queues validation and the commit.
> requests.post(f"{API}/uploads/{upload['id']}/complete", headers=HEADERS).raise_for_status()
>
> # 4. Poll until the upload reaches a final state.
> while True:
>     status = requests.get(f"{API}/uploads/{upload['id']}", headers=HEADERS).json()
>     if status["state"] in ("committed", "failed", "cancelled", "expired"):
>         break
>     time.sleep(10)
>
> print(status["state"], status["counters"], status["error"])
> ```

> [tab] JavaScript
>
> ```javascript theme={"system"}
> // Requires Node.js 18 or later. No packages to install.
> import { readFile } from "node:fs/promises";
>
> const API = `https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/${process.env.DATASET_ID}`;
> const HEADERS = {
>   Authorization: `Bearer ${process.env.WANDB_API_KEY}`,
>   "Forge-Entity": "your-team",
> };
> const FILE = "tickets.jsonl";
>
> async function call(path, init = {}) {
>   const response = await fetch(`${API}${path}`, { ...init, headers: { ...HEADERS, ...init.headers } });
>   if (!response.ok) throw new Error(`${init.method ?? "GET"} ${path} failed: ${await response.text()}`);
>   return response.json();
> }
>
> // 1. Open the upload.
> const body = await readFile(FILE);
> const upload = await call("/uploads", {
>   method: "POST",
>   headers: { "Content-Type": "application/json" },
>   body: JSON.stringify({
>     idempotency_key: "support-tickets-v1",
>     file: { name: FILE, size_bytes: body.byteLength },
>     mode: "append",
>   }),
> });
>
> // 2. Send the file. The signed URL carries its own authorization.
> const put = await fetch(upload.upload.url, { method: "PUT", body });
> if (!put.ok) throw new Error(`Upload failed with ${put.status}`);
>
> // 3. Complete the upload. This queues validation and the commit.
> await call(`/uploads/${upload.id}/complete`, { method: "POST" });
>
> // 4. Poll until the upload reaches a final state.
> let status;
> while (true) {
>   status = await call(`/uploads/${upload.id}`);
>   if (["committed", "failed", "cancelled", "expired"].includes(status.state)) break;
>   await new Promise((resolve) => setTimeout(resolve, 10_000));
> }
> console.log(status.state, status.counters, status.error);
> ```

### How the script works

The numbered comments in each script correspond to the following stages.

**1. Open the upload**

The script sends the filename, its exact byte size, and the write mode. The request also requires an `idempotency_key`, so a retried request returns the existing upload instead of opening a second one. Reusing a key with a different request body returns `409 Conflict` with type `idempotency_conflict`.

The response is `201 Created` with the upload. `upload.url` is a signed URL for the file, and `upload.expires_at` is when it stops working. The following excerpt shows those fields:

```json theme={"system"}
{
  "id": "3f9c2d1e-7b4a-4c58-9e21-0d6f8a1b2c3d",
  "dataset_id": "8a1b2c3d-0d6f-4c58-9e21-3f9c2d1e7b4a",
  "mode": "append",
  "state": "uploading",
  "file": { "name": "tickets.jsonl", "size_bytes": 73400320 },
  "upload": {
    "method": "PUT",
    "url": "https://...",
    "expires_at": "2026-10-06T18:17:11.000Z"
  },
  "expires_at": "2026-10-07T18:02:11.000Z"
}
```

**2. Send the file**

The script sends the whole file in one `PUT` request to `upload.url`. The URL is bound to the declared size, so the body must be exactly `file.size_bytes` long. It carries its own authorization, so the script doesn't add the `Authorization` or `Forge-Entity` headers to this request.

The URL must be used within 15 minutes. If it expires, read the upload again: while the upload is in `uploading`, each read returns a fresh URL.

**3. Complete the upload**

When the file is stored, the script completes the upload. Model Distillation checks that the stored file matches the declared size and queues validation. The response is `202 Accepted`.

If the file hasn't arrived, the response is `409 Conflict` with type `upload_incomplete`. If its size doesn't match, the type is `upload_size_mismatch`, and the stored file is discarded. In both cases, the upload stays in `uploading`, so you can send the file again and complete the upload again. Completing an upload that's already complete returns its current state without error.

**4. Poll until the rows are committed**

The script reads the upload every 10 seconds until it reaches a final state. While validation runs, `validation.validated_lines` and `validation.validated_bytes` report progress.

When `state` is `committed`, `counters` reports `rows`, `created`, `updated`, the new `revision`, and `entry_counts` by split. If `state` is `failed`, see [Read validation results](#read-validation-results).

If relabeling, an evaluation, or a fine-tune is running on the dataset, the commit waits until they finish rather than failing.

### Upload states

An upload moves through the following states:

| State | Meaning |
| - | - |
| `uploading` | The upload accepts the file. |
| `queued`, `validating` | The upload is complete. Model Distillation checks every line, then commits all rows in one transaction. |
| `committed` | The rows are in the dataset. `counters` summarizes the write. |
| `failed` | Validation or the commit failed, and the dataset is unchanged. `error` explains why, and `validation.errors` lists the problems. |
| `cancelled` | You cancelled the upload before it committed. |
| `expired` | The upload stayed in `uploading` for 24 hours without being completed. |

An upload that is still `uploading` 24 hours after creation can no longer be completed. Completing it returns `410 Gone` with type `dataset_upload_expired`, and an hourly job moves it to `expired`. When an upload finishes in any state, its stored file is deleted.

### Limits

Files and uploads must stay within the following limits:

| Limit | Value |
| - | - |
| File size | 1 GiB |
| Rows per file | 100,000 |
| Line size | 10 MiB |
| JSON nesting | 64 levels |
| Uploads in progress per entity | 2 |
| Total `size_bytes` of uploads in progress, per entity | 2 GiB |

An entity is the team or personal account named in the `Forge-Entity` header. An upload counts toward the entity's limits while it is `uploading`, `queued`, or `validating`. A request that would exceed either limit returns `429 Too Many Requests`. To free capacity, cancel an upload you no longer need, or wait for one to finish.

### Read validation results

A `failed` upload reports `error` as a short summary, and `validation` carries the details:

* `error_count` is the total number of problems found.
* `errors` lists up to 100 problems, ordered by line, each with `line`, `code`, and `message`. `line` is `null` for a problem with the whole upload, such as a commit conflict.
* `errors_truncated` is `true` when more problems exist than the list shows.

Line-level codes are `invalid_json`, `invalid_utf8`, `duplicate_json_key`, `json_too_deep`, `row_too_large`, `empty_line`, `duplicate_row_id`, `row_limit_exceeded`, and `validation_failed`, which covers rows that don't match the row format. Upload-level codes are `entry_exists`, when an `append` upload contains a `row_id` that's already in the dataset, and `group_split_conflict`.

A failed upload can't be reopened. Fix the file and open a new upload.

### Cancel an upload

To discard an upload that hasn't committed, delete it. The response is `204 No Content`, the state becomes `cancelled`, and the stored file is deleted. Repeating the request on a cancelled upload returns `204 No Content` again.

```bash theme={"system"}
curl --request DELETE \
  --url "https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/$DATASET_ID/uploads/$UPLOAD_ID" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Forge-Entity: your-team"
```

An upload that already committed returns `409 Conflict` with type `dataset_upload_committed`. To remove its rows, delete the dataset instead. See [Delete a dataset](https://docs.coreweave.com/model-distillation/studio/datasets.md#delete-a-dataset).

## Choose the validation rows

Rows without a `split` value go to training. To hold a share of the dataset out for evaluation after you upload it, move entries to the validation split:

```bash theme={"system"}
curl --request PATCH \
  --url "https://forge.coreweave.com/api/distillation/v1/projects/ticket-classifier/datasets/$DATASET_ID/entries" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Forge-Entity: your-team" \
  --header "Content-Type: application/json" \
  --data '{"split": "val", "fraction": 0.1}'
```

The request selects entries with `filters`, which take the same filters as the entry listing, including `metadata`, and with `row_ids`. Without either, it selects every entry. `fraction` moves only that share of the selected groups. Groups are ranked by a digest of the group ID, so a group is never divided, and a later request can take its share from what remains.

To hold out a whole population, such as one customer, filter on metadata instead. Evaluations then measure the model on users it never saw in training:

```json theme={"system"}
{"split": "val", "filters": [{"field": "metadata", "key": "customer", "operator": "equals", "value": "acme"}]}
```

The response reports how many entries and groups matched, how many changed split, the new `revision`, and `entry_counts`. Add `dry_run=true` to see the result without changing the dataset.

## Errors

Entry and upload requests can return the following errors.

| Status | Type | Cause |
| - | - | - |
| `400 Bad Request` | `validation_error` | The request body isn't valid JSON, or an NDJSON line is empty or isn't valid JSON. |
| `403 Forbidden` | `dataset_uploads_disabled` | File uploads aren't enabled for the entity. |
| `404 Not Found` | `not_found` | The dataset or upload doesn't exist in this project. |
| `409 Conflict` | `dataset_not_ready` | The dataset isn't ready, so its entries can't change. |
| `409 Conflict` | `dataset_busy` | Relabeling, an evaluation, or a fine-tune is running on the dataset. Entry writes return this error; uploads wait instead. |
| `409 Conflict` | `entry_exists` | An `append` write contains a `row_id` that's already in the dataset. Use `mode=upsert` to replace it. |
| `409 Conflict` | `group_split_conflict` | The write would place one group in both splits. |
| `409 Conflict` | `idempotency_conflict` | The upload's `idempotency_key` was already used with a different request body. |
| `409 Conflict` | `upload_incomplete` | You tried to complete the upload before the file arrived. |
| `409 Conflict` | `upload_size_mismatch` | The stored file doesn't match the declared size. |
| `409 Conflict` | `dataset_upload_committed` | You tried to cancel an upload that already committed. |
| `410 Gone` | `dataset_upload_expired` | The upload is older than 24 hours. Open a new one. |
| `422 Unprocessable Entity` | `duplicate_row_id` | A `row_id` appears more than once in the request. |
| `422 Unprocessable Entity` | `validation_error` | A row or the request body doesn't match its schema. |
| `429 Too Many Requests` | `dataset_upload_concurrency_limit` | The entity already has 2 uploads in progress. |
| `429 Too Many Requests` | `dataset_upload_byte_quota` | The new file would push the entity past 2 GiB of declared size across uploads in progress. |
| `503 Service Unavailable` | `uploads_unavailable`, `temporal_unavailable` | Upload storage or the validation queue isn't reachable. Retry later. |

For request and response schemas, see the following pages in the Management API reference:

* [Create a dataset](https://docs.coreweave.com/model-distillation/reference/management/datasets/create-a-dataset.md)
* [Add or replace dataset entries](https://docs.coreweave.com/model-distillation/reference/management/datasets/add-or-replace-dataset-entries.md)
* [Move dataset entries to a split](https://docs.coreweave.com/model-distillation/reference/management/datasets/move-dataset-entries-to-a-split.md)
* [Start a JSONL upload into a dataset](https://docs.coreweave.com/model-distillation/reference/management/datasets/start-a-jsonl-upload-into-a-dataset.md)
* [Get a dataset upload](https://docs.coreweave.com/model-distillation/reference/management/datasets/get-a-dataset-upload.md)
* [Finish a dataset upload](https://docs.coreweave.com/model-distillation/reference/management/datasets/finish-a-dataset-upload.md)
* [Cancel a dataset upload](https://docs.coreweave.com/model-distillation/reference/management/datasets/cancel-a-dataset-upload.md)

## Next steps

> [card] [Relabeling](https://docs.coreweave.com/model-distillation/studio/relabeling.md)
>
> Ask a stronger model to rewrite the assistant responses in the uploaded dataset without changing the original rows.

> [card] [Fine-tuning](https://docs.coreweave.com/model-distillation/studio/fine-tuning.md)
>
> Train a supported base model on the uploaded dataset, using the original outputs or a relabeled output set.
