> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Response caching

> Reuse the exact response for a repeated non-streaming request instead of calling the provider again.

Exact-response caching lets a project-routed request reuse the response the proxy already produced for an identical request. When a request opts in and a matching response exists, the proxy returns that response without contacting the provider. This saves provider spend and latency for workloads that send the same request more than once, such as evaluation suites, regression tests, retries, and development loops.

Caching is off by default and applies per request. A cached response is replayed byte for byte, so a hit returns the same answer even when the request uses sampling parameters such as a nonzero temperature.

## Enable caching for a request

Set the `Forge-Cache-Mode` request header to one of the following modes:

| Value | Read an existing response | Store a successful response |
| - | - | - |
| `readWrite` | Yes | Yes |
| `readOnly` | Yes | No |
| `writeOnly` | No | Yes |

To bypass the cache, omit the header.

> [tab] Python
>
> ```python theme={"system"}
> import os
>
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="https://proxy.training.coreweave.com/v1",
>     api_key=os.environ["WANDB_API_KEY"],
>     project="your-team/ticket-classifier",  # sent as OpenAI-Project
> )
>
> raw = client.chat.completions.with_raw_response.create(
>     model="router",
>     messages=[
>         {"role": "user", "content": "My package arrived damaged. What should I do?"}
>     ],
>     extra_headers={"Forge-Cache-Mode": "readWrite"},
> )
>
> print(raw.headers.get("forge-cache-status"))  # "hit" or "miss"
> response = raw.parse()
> print(response.choices[0].message.content)
> ```

> [tab] JavaScript
>
> ```javascript theme={"system"}
> import OpenAI from "openai";
>
> const client = new OpenAI({
>   baseURL: "https://proxy.training.coreweave.com/v1",
>   apiKey: process.env.WANDB_API_KEY,
>   project: "your-team/ticket-classifier", // sent as OpenAI-Project
> });
>
> const { data: response, response: raw } = await client.chat.completions
>   .create(
>     {
>       model: "router",
>       messages: [
>         { role: "user", content: "My package arrived damaged. What should I do?" },
>       ],
>     },
>     { headers: { "Forge-Cache-Mode": "readWrite" } },
>   )
>   .withResponse();
>
> console.log(raw.headers.get("forge-cache-status")); // "hit" or "miss"
> console.log(response.choices[0].message.content);
> ```

> [tab] cURL
>
> ```bash theme={"system"}
> curl -i https://proxy.training.coreweave.com/v1/chat/completions \
>   -H "Authorization: Bearer $WANDB_API_KEY" \
>   -H "Content-Type: application/json" \
>   -H "OpenAI-Project: your-team/ticket-classifier" \
>   -H "Forge-Cache-Mode: readWrite" \
>   -d '{
>     "model": "router",
>     "messages": [
>       {"role": "user", "content": "My package arrived damaged. What should I do?"}
>     ]
>   }'
> ```

When the request enables cache reads (`readWrite` or `readOnly`), the response includes a `Forge-Cache-Status` header with the value `hit` or `miss`. The response to a `writeOnly` request doesn't include it.

## What makes two requests identical

The proxy looks up a cached response after it resolves the project version and routing revision, and before it selects a provider. For a model override, the lookup happens after the proxy resolves the override's provider. The cache key combines the following values:

* The Weights & Biases entity, project, and project version that the proxy resolves from the `OpenAI-Project` header and `forge.project_version` metadata.
* The [routing revision](https://docs.coreweave.com/model-distillation/studio/routing-and-versions.md#save-to-an-existing-or-new-version) that applies to the version when the request arrives.
* The request path and query string.
* A SHA-256 hash of the request body, serialized with object keys in sorted order and with the following values excluded.

The proxy removes the following values from the body before hashing, so they don't change the key:

* The raw `model` string of a routed request. Its resolved project, version, and routing revision are already in the key, so `"router"`, a blank `model`, and an omitted `model` share entries. Different versions never share entries.
* Metadata keys that start with `forge.` or `wandb.`, including `forge.project_version`.
* The conversation identifier in `gen_ai.conversation.id`.
* `stream: false`, which the proxy treats the same as omitting `stream`.
* An empty `metadata` object, which the proxy treats the same as omitting `metadata`.

The proxy hostname and the order of query parameters don't change the key either.

Each time you save a version's routing, Model Distillation publishes a new routing revision, which changes the key. Existing entries stop matching but remain until they expire. The proxy doesn't hash parameter overrides on a routing target. Changing an override is a routing change like any other, so it also changes the key.

Every other field in the body is part of the hash, regardless of whether the provider uses it. These fields include `messages`, `tools`, `tool_choice`, `response_format`, `temperature`, `top_p`, `max_tokens`, `seed`, `n`, `stop`, `user`, any provider-specific fields, and metadata keys that don't start with `forge.` or `wandb.`, such as `user.id`. The order of keys within an object doesn't matter. The order of items in an array, such as `messages`, does.

Entries are scoped to your Weights & Biases entity and project. An identical request from another entity, or for another project in your entity, never matches your entries.

A cached response records the routing target that produced it. The proxy serves a hit from that target without weighted or sticky routing.

A [model override](https://docs.coreweave.com/model-distillation/proxy/model-routing.md) sends a `provider/model` value in `model` together with the `OpenAI-Project` header. The proxy caches it in the selected project version like routed traffic. Its key also includes the provider, the model reference, and the provider's configuration, so an override never matches routed entries or entries for another model.

## Limits

The cache has the following limits:

* **Non-streaming requests only.** Requests with `stream: true` that set a cache header return `400 Bad Request`.
* **Successful responses only.** The proxy stores only 2xx responses up to 8 MiB.
* **Retention is 7 days.** Entries are deleted 7 days after they're written. Hits don't refresh them.
* **Invalid modes are rejected.** Any `Forge-Cache-Mode` value other than those listed returns `400 Bad Request`.

Cache failures fail open. If the cache can't be read or written, the proxy forwards the request to the provider as usual.

The proxy doesn't coalesce concurrent requests. If identical requests arrive before the proxy stores the first response, each one reaches the provider, and the proxy stores the last response written.

## Traces and Analytics

The proxy still records a cache hit as a project trace. The trace keeps the original response and its token usage, is marked with `cache_hit=true`, and has no provider latency. A request that enables cache reads but is forwarded to the provider is marked with `cache_hit=false`.

Analytics counts the provider spend of a cache hit as zero, because no inference ran, while token totals still include the replayed usage. Dataset creation deduplicates identical inputs, so replayed responses don't add duplicate training rows.

## Browser clients

The proxy allows the `OpenAI-Project` and `Forge-Cache-Mode` request headers in cross-origin requests, and it exposes `Forge-Cache-Status` and `x-proxy-request-id` to browser code.

> [card] [Chat Completions](https://docs.coreweave.com/model-distillation/proxy/chat-completions.md)
>
> See how the proxy builds the provider request, handles streaming, and forwards tools and structured output.
