> ## Documentation Index
> Fetch the complete documentation index at: https://manifest.build/llm-gateway/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Local models

> Connect a self-hosted LLM Gateway to Ollama, LM Studio, or llama.cpp on your own hardware.

Local model providers run entirely on your own hardware. The gateway detects the running server, fetches the model list, and routes requests to `http://localhost:<port>` like any other provider. No API key, no network egress, no per-token cost.

<Note>
  The built-in Ollama, LM Studio, and llama.cpp connections are self-hosted only. To reach a local model from [LLM Gateway Cloud](https://app.manifest.build), expose the runtime on a public URL or tunnel and add it as a [custom provider](/llm-gateway/docs/llm-gateway/docs/providers/custom-providers).
</Note>

## Supported runtimes

| Runtime                                            | Default port | Install                                                                                        |
| -------------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------- |
| [Ollama](https://ollama.com)                       | `11434`      | [ollama.com/download](https://ollama.com/download)                                             |
| [LM Studio](https://lmstudio.ai)                   | `1234`       | [lmstudio.ai](https://lmstudio.ai)                                                             |
| [llama.cpp](https://github.com/ggml-org/llama.cpp) | `8080`       | [llama.cpp build guide](https://github.com/ggml-org/llama.cpp#obtaining-and-quantizing-models) |

All three speak OpenAI-compatible `/v1/chat/completions` and accept any GGUF model file.

## Start the server

<Tabs>
  <Tab title="Ollama">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
    ollama pull llama3.1:8b   # then:
    ollama serve
    ```
  </Tab>

  <Tab title="LM Studio">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
    lms server start
    ```

    Or open the app: **Developer** tab → **Start server**.
  </Tab>

  <Tab title="llama.cpp">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
    llama-server -m <your-model>.gguf --port 8080
    ```

    Replace `<your-model>.gguf` with the path to a GGUF file on your machine.
  </Tab>
</Tabs>

## Connect to the gateway

<Steps>
  <Step title="Open the Local page">
    In the dashboard sidebar, open **Providers → Local** and click **Connect** on
    your runtime (Ollama, LM Studio, or llama.cpp).
  </Step>

  <Step title="Confirm the server is reachable">
    The gateway probes `http://localhost:<default-port>/v1/models`. If the probe succeeds, every loaded model appears for routing.
  </Step>

  <Step title="Pin a model to a tier">
    Open your default or a custom tier and pick a local model as the primary. You can mix local and cloud models in the same fallback chain.
  </Step>
</Steps>

## Running the gateway in Docker

If you self-host the gateway in Docker, the container can't reach a local server bound to `127.0.0.1` on the host. Two of the three runtimes default to loopback and need an explicit override:

<Tabs>
  <Tab title="LM Studio">
    Either flip the GUI toggle (**LM Studio → ⚙ Developer → Serve on Local Network**) or rebind from the CLI:

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
    lms server start --bind 0.0.0.0 --port 1234 --cors
    ```

    LM Studio remembers the last `--bind`, so this is one-time setup.
  </Tab>

  <Tab title="llama.cpp">
    `llama-server` only listens on `0.0.0.0` if you pass `--host`:

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
    llama-server -m <your-model>.gguf --host 0.0.0.0 --port 8080
    ```
  </Tab>

  <Tab title="Ollama">
    Ollama already binds `0.0.0.0` by default. No change needed.
  </Tab>
</Tabs>

<Note>
  Inside the gateway container, the host is reachable as
  `host.docker.internal`. The gateway sets this automatically when probing local
  providers.
</Note>

## Cost & privacy

| Aspect                | Local                                                           |
| --------------------- | --------------------------------------------------------------- |
| **API cost**          | \$0. The model runs on your hardware.                           |
| **Network egress**    | None. Requests never leave the machine.                         |
| **Cost in dashboard** | Recorded as `0`. Token counts and latency are still tracked.    |
| **Pricing data**      | Not applicable. Local providers are excluded from pricing sync. |

<Tip>
  Mix local and cloud in one chain: set a local model as your default for
  day-to-day calls, and fall back to a cloud model when the local server is offline.
</Tip>