> ## Documentation Index
> Fetch the complete documentation index at: https://manifest.build/llm-gateway/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Gateway

> Route requests across models, enforce limits, and use fallbacks when a provider fails.

Point your client at the gateway instead of a provider, and every request goes through the gateway. Three things happen there: limits decide whether the request runs at all, routing picks the model, and fallback steps in when that model fails.

## How a request flows

<Steps>
  <Step title="Authentication">
    The `Authorization: Bearer mnfst_...` header resolves to one of your harnesses. A bad or missing key stops here with a [401](/llm-gateway/docs/llm-gateway/docs/errors).
  </Step>

  <Step title="Limits">
    If a hard limit for this harness is already over its threshold, the request is blocked with [M200](/llm-gateway/docs/llm-gateway/docs/errors/M200) before any provider is called. Nothing is spent.
  </Step>

  <Step title="Routing">
    A custom tier matches on a request header, or the request goes to your default tier. Send a real model ID instead of `auto` to skip routing. A matching custom tier header still wins.
  </Step>

  <Step title="Forward">
    The gateway calls the resolved provider with your credentials and streams the response back.
  </Step>

  <Step title="Recovery">
    If the provider returns an error, [Autofix](/llm-gateway/docs/llm-gateway/docs/autofix) may repair and resend the request once. Anything still failing moves down the fallback chain.
  </Step>
</Steps>

Routing runs in-process. There's no extra network call and no added latency.

## Routing

Instead of hard-coding one model into every client, you point your client at the gateway and let it pick. Send `auto` as the model, and the gateway resolves the real model based on the rules you set on the dashboard **Routing** page.

<CardGroup cols={2}>
  <Card title="Default" icon="route" href="#default-tier">
    One model plus up to 5 fallbacks. Every request lands here unless a custom tier matches.
  </Card>

  <Card title="Custom" icon="sliders-horizontal" href="#custom-tiers">
    Match a request header to a tier you define, and route it to its own model.
  </Card>
</CardGroup>

### Default tier

Every harness has a default tier: one model plus up to five fallbacks. You set it on the **Routing** page and can change it anytime without touching your code. Send `auto` as the model, and any request that doesn't match a custom tier goes to your default.

### Custom tiers

Custom tiers route by request header. You create a tier on the dashboard, give it a header key and value, and pin it to a model with its own fallbacks. When an incoming request carries that header, the gateway sends it to that tier's model instead of the default. The header wins even when the body names an explicit model ID.

The header key is yours to choose (lowercase letters, numbers, and hyphens). A few names are reserved and rejected, including `authorization`, `cookie`, and `x-api-key`. Send the header from your client like any other:

```ts theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://app.manifest.build/v1",
  apiKey: "mnfst_YOUR_KEY",
  defaultHeaders: { "x-manifest-tier": "batch" }, // your tier's key and value
});

const response = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this report." }],
});
```

Create as many tiers as you need, each with its own model and parameters. This is handy for isolating an agent's subtasks, A/B testing two models, or letting an orchestration layer decide the tier itself.

### Route a specific model

To skip routing for a single request, send a real model ID instead of `auto`. The gateway forwards it straight to that model's provider, with no tier lookup and no fallbacks. A matching [custom tier](#custom-tiers) header is the exception. The request then runs on the tier's model and the tier's fallbacks. Call [`GET /v1/models`](/llm-gateway/docs/llm-gateway/docs/reference/api#listing-models) to list the model IDs your harness can reach.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://app.manifest.build/v1/chat/completions \
  -H "Authorization: Bearer mnfst_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

The response comes back with `X-Manifest-Tier: direct`, so you can tell a direct call from a routed one. A direct model ID works on all three endpoints: `/v1/chat/completions`, `/v1/responses`, and the Anthropic `/v1/messages`.

The model list is scoped to the harness key. A model may be missing because its provider is not connected yet, or because the provider exists in your workspace but is not enabled for this harness. If you send a model ID that has no connected provider route, the gateway returns [M302: Model not available](/llm-gateway/docs/llm-gateway/docs/errors/M302). Send `auto` to use routing.

## Fallback

When a model fails (provider outage, rate limit, bad request), the gateway retries with a backup model from the same tier. Your agent gets a response instead of an error.

### What triggers a fallback

Any HTTP status code **>= 400** triggers a fallback.

| Status | Example |
| - | - |
| **400** | Bad request |
| **401** | Authentication error |
| **403** | Forbidden |
| **429** | Rate limited |
| **500** | Internal server error |
| **502** | Bad gateway |
| **503** | Service unavailable |
| **529** | Provider overloaded |

A tier also falls back when its pinned model leaves your model list. When the tier has no fallback left, the request returns [M101](/llm-gateway/docs/llm-gateway/docs/errors/M101).

### Configuration

Fallback models are configured **per tier**. Each tier holds up to **5 fallback models**, tried in order.

<Steps>
  <Step title="Open Routing in the dashboard">
    Navigate to **Routing** in the dashboard.
  </Step>

  <Step title="Select a tier">
    Click your Default tier or any custom tier.
  </Step>

  <Step title="Add fallback models">
    Add up to 5 fallback models. Drag to reorder, since models are tried from top to bottom.
  </Step>
</Steps>

<Tip>
  Connect at least two providers. With a single provider, fallback can only switch between that provider's models.
</Tip>

### Hung providers and the per-attempt timeout

A provider that opens a connection but never returns eventually triggers a fallback via the gateway's per-attempt timeout (default 180 seconds), which surfaces as a synthetic `504 Gateway Timeout` and moves to the next model in the chain.

If your upstream client has its own timeout that fires at the same time, the client may disconnect first and the gateway will give up before reaching a healthy fallback. On self-hosted installs, lower [`PROVIDER_TIMEOUT_MS`](/llm-gateway/docs/llm-gateway/docs/reference/environment-variables) strictly below your client's timeout so the chain has room to run inside the client's window.

### Response headers

When a fallback succeeds, the response carries `X-Manifest-Fallback-From` (the primary that failed) and `X-Manifest-Fallback-Index` (its position in the chain) on top of the standard routing headers. When the chain is exhausted, `X-Manifest-Fallback-Exhausted: true` is set, the body carries the provider's own error code (or `fallback_exhausted` when there is none), and the response keeps the primary model's real error status (for example 500 or 529). Full table: [Headers reference](/llm-gateway/docs/llm-gateway/docs/reference/headers).

## Hard limits

A limit rule can block requests as well as email you. When a rule's action includes blocking and the harness is over its threshold for the current period, the gateway returns [M200](/llm-gateway/docs/llm-gateway/docs/errors/M200) as an HTTP `200` response whose assistant message carries the block text, so your agent keeps flowing instead of throwing:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
[🦚 Manifest M200] You hit your cost limit: $12.50 used, $10.00/day allowed. Adjust it here: https://app.manifest.build/...
```

The check runs before any provider is contacted, so a blocked request costs nothing. The block clears on its own at the start of the next period, or immediately if you raise the threshold.

<Note>
  On LLM Gateway Cloud, running out of the Free plan's monthly requests is a
  different thing entirely: that returns [M204](/llm-gateway/docs/llm-gateway/docs/errors/M204) with HTTP `402`
  and isn't something you configured.
</Note>

Rules are defined per harness, with a metric, a threshold, and a period. See [Spend alerts](/llm-gateway/docs/llm-gateway/docs/observability#spend-alerts) for how to create one and how the alerting half works.

## Related

* [CLI](/llm-gateway/docs/llm-gateway/docs/cli) — set up routing and providers from the terminal
* [Autofix](/llm-gateway/docs/llm-gateway/docs/autofix) — repair a failing request before fallback runs
* [Observability](/llm-gateway/docs/llm-gateway/docs/observability) — what the gateway records, and spend alerts
* [API reference](/llm-gateway/docs/llm-gateway/docs/reference/api)
* [Headers reference](/llm-gateway/docs/llm-gateway/docs/reference/headers)
* [Error codes](/llm-gateway/docs/llm-gateway/docs/errors)