Run a Local LLM Server for Your AI Agent: Step by Step

To run a local LLM server for an agent, install Ollama (simplest) or vLLM (built for throughput on a GPU box), serve one model, point your agent at the local URL, and raise the context window before you test tool calls. You can have a working endpoint in about ten minutes. Getting it to behave like an agent backend takes a bit longer.

Why bother? Prompts and files stay on your hardware, and you stop paying per token while you experiment. The trade is that you now run the server.

1Which server should you start with?

Short version: Ollama on a laptop or workstation, vLLM on a Linux machine with a GPU that several people or agents will hit at once. Our Ollama vs vLLM comparison goes deeper. If you are unsure, start with Ollama. Moving later is easy, because both can speak an OpenAI-style API.

2How do you set up Ollama?

Ollama's README lists the install routes: a shell script on macOS and Linux, a PowerShell command on Windows, a manual installer, or the official ollama/ollama Docker image. Then pull and run a model:

ollama run gemma4

That command is the README's own example. The model library lives at ollama.com/library, so pick something that fits your memory.

By default the server listens on 127.0.0.1, port 11434, and the chat endpoint is /api/chat. Only your own machine can reach it, which is a sensible default. Ollama's FAQ says you change the bind address with the OLLAMA_HOST environment variable. Do that only on a network you control, and put authentication in front, because a model server with no login is an open door to your hardware.

3How do you set up vLLM?

vLLM's quickstart assumes Linux and Python 3.11 to 3.14. For NVIDIA GPUs the documented install is:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

Then serve a model:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server answers on http://localhost:8000 with OpenAI-compatible routes such as /v1/models and /v1/chat/completions. Use --host and --port to change the address, and --api-key (or the VLLM_API_KEY variable) to require a key. The docs show other hardware routes too, including AMD ROCm and Google TPU, and say Apple Silicon goes through the separate vLLM-Metal project.

4How do you connect your agent?

Most agent frameworks accept an OpenAI-style base URL. With the official openai Python client, vLLM's quickstart shows the pattern:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

Swap in your model name and send a chat request. If your framework talks to Ollama directly, use its native endpoint on port 11434 instead.

Keep the agent, the model server and your tools as separate pieces. That way you can change the model without rewriting the agent.

5What settings matter for agents?

Two settings cause most "my local agent is dumb" complaints.

Context length

Agents stuff instructions, tool results and history into one prompt, and a small window truncates it silently. Ollama's context length page says defaults depend on VRAM: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, 256k above that. It recommends at least 64,000 tokens for agents, web search and coding tools, and shows how to set it:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Run ollama ps afterwards. The CONTEXT column shows what was actually allocated, and the PROCESSOR column shows whether the model spilled onto the CPU, which slows everything down.

Tool calling

Ollama's tool calling docs describe the loop an agent runs: send tool definitions, receive a tool_calls entry, run the function, return the result in a tool message, and repeat until a response has no tool calls.

vLLM needs explicit flags. Its tool calling guide requires --enable-auto-tool-choice and a --tool-call-parser that matches your model (for example hermes or llama3_json). Many models also need a tool-aware chat template. The same page warns that smaller models often fail to emit tool calls in the right format, and that with "auto" the arguments can be malformed unless you use strict tools.

6How do you test it before trusting it?

1. Write five real tasks. Use ones from your own work, not a benchmark.
2. Run each three times. Local models vary between runs.
3. Log every tool call. Check names and arguments, not just final answers.
4. Try one task that should fail. A good setup refuses or asks, a bad one invents.
5. Watch memory. If the model spills onto the CPU, shrink the context or the model.

7What about security?

Keep the server on localhost until you need otherwise. If you expose it, require a key, use a firewall rule, and never let the agent run shell commands or write files without a human approval step while you are still testing.

8FAQ

Can I run an AI agent fully offline?

Yes, if the model, the server and your tools are all local. Anything the agent fetches from the web obviously needs a connection.

Is Ollama or vLLM faster?

vLLM is built for serving many requests at once on GPUs. For one person on a laptop, Ollama is simpler and fast enough. Measure on your own task.

How big a model do I need for tool calling?

There is no single answer. vLLM's docs note that smaller models often get the format wrong, so test your actual tools with the model you plan to use.

Do I need a GPU?

Not for Ollama on a small model, but expect slow answers. vLLM's documented install targets GPUs and other accelerators.

Browse more agent and infrastructure tools in the AI Agents & Infrastructure hub or the Woodander product directory. Your next step: serve one small model today and run your five test tasks against it.