GitHub radar
Run your own fast, cheap AI model server
vLLM is an open-source tool that turns a downloaded model into a service answering many users at once, quickly and with low memory overhead.
It's software that runs a large language model on your own server and serves answers to requests — fast, to several users at the same time.
When it helps
Useful if you're deploying your own service on an open model — for privacy or to avoid paying per-word for external APIs. If you just want an AI assistant for daily coding, this isn't it: it doesn't write or fix code itself.
Pros
- Serves many requests at once from a single GPU
- Compatible with OpenAI's API and Anthropic's Messages API — minimal code changes
- Supports 200+ open models from Hugging Face: Llama, Qwen, DeepSeek
- Active project with 2000+ contributors, Apache 2.0 license
Cons
- Needs an NVIDIA or AMD GPU — runs on CPU, but far too slowly
- Doesn't write or fix code itself — just serves an already-trained model
- Needs a server and GPU — adds infrastructure costs
How to set it up — step by step
- 1Open your AI agent (Claude Code, Codex) in your project folder
- 2Send the agent the text from the block below
- 3Give the agent GPU access and, if needed, access to a gated Hugging Face model
- 4Check that the server answers a normal request the same way the OpenAI API does
- 5If it works, point your app at this server instead of an external API
Text for your agent
Copy this and send it to your agent — Claude Code, Codex, any of them:
Deploy a test vLLM server from https://github.com/vllm-project/vllm — install it per the README (uv pip install vllm), bring up a small open model from Hugging Face on a GPU (ask me if you need access to a gated model), and show me that the endpoint responds like a normal OpenAI API call.
Heavy — you need an NVIDIA or AMD GPU; without one it technically starts on a CPU or a laptop, but so slowly that there's little point. This is a tool for running your own server with an open model, not something to use casually on a laptop.
Open on GitHub▌ More finds