Replicate → vLLM
What it saves, what actually moves, what you rebuild — and the thing that catches people.
What it costs, and what it saves
From per-second GPU billing with a platform margin, to the raw cost of a GPU you rent or own. At steady utilisation this is commonly a two-thirds saving; at spiky or low volume it is a loss, because scale-to-zero genuinely costs nothing between requests.
Moves cleanly
The model weights, which are almost always public models you can download yourself. Cog is open source too, so a container built for Replicate runs elsewhere.
You rebuild
The calling code. Replicate's prediction API, webhooks and streaming are its own shape. The mitigating detail: vLLM exposes an OpenAI-compatible server, so if your application already speaks OpenAI anywhere, that part can become a base-URL change.
What Replicate costs you today
No subscription. Pure usage billing by the second of GPU time, with rates that vary by hardware class — roughly $0.000225/sec for an A100 80GB down to about $0.000038/sec for a CPU instance, plus per-token pricing on some hosted language models. A model kept warm for responsiveness bills for the warm time, which is where unexpected invoices come from.
What actually holds you in
Low to moderate, and lower than it looks. The weights are almost always public models you can download yourself, and Cog is open source, so a container built for Replicate runs elsewhere. What you rewrite is the calling code: predictions, webhooks and streaming are Replicate-shaped. Budget days, not weeks — and note that vLLM and TGI both expose an OpenAI-compatible API, so if your code already speaks OpenAI the swap can be a base-URL change.
What you are moving to
vLLM is a high-throughput inference engine born out of UC Berkeley, built around PagedAttention — a memory-management technique borrowed from operating-system paging that lets it batch far more concurrent requests onto the same GPU than naive serving does. It exposes an OpenAI-compatible server, so most existing client code works by changing a base URL. It is the default choice for teams serving one model to real traffic, and the engine underneath a large share of the inference providers you would otherwise pay.
Free and unlimited. You pay only for the hardware you run it on.
vLLM strengths
- Highest throughput per GPU of the mainstream engines
- OpenAI-compatible API — client code often needs no change
- Apache-2.0 with no feature gates or usage limits
- Continuous batching keeps the GPU busy under mixed load
What you give up
- You own the GPU, the drivers and the on-call rota
- No scale-to-zero — an idle GPU still costs whatever it costs
- Setup assumes comfort with CUDA and Python environments
The migration, step by step
- 1Measure your utilisation curve, not your monthly total. Steady load favours self-hosting; spiky load favours Replicate, and the bill alone will not tell you which you have.
- 2Rent one GPU by the hour to prototype — do not buy hardware to test a hypothesis.
- 3`pip install vllm`, then `vllm serve <model>`. It comes up with an OpenAI-compatible endpoint on port 8000.
- 4Point a copy of your client at it and compare output and latency against Replicate on identical prompts.
- 5Load-test at your real concurrency. vLLM's continuous batching means throughput does not degrade the way naive serving does, but you need to find your ceiling before trusting it.
- 6Cut over the steady traffic and keep Replicate for spiky or experimental models. Running both is a legitimate end state.
The gotcha
The cost comparison people run is wrong, because they compare Replicate's bill against the GPU's hourly rate and forget the GPU bills whether or not anyone calls it. If you are serving eight hours a day, you are paying for twenty-four. Compare against your actual utilisation, and if it is below roughly 30% you are probably better off where you are.
When to stay on Replicate
Your traffic is spiky, low or unproven. Scale-to-zero costs nothing between requests and no self-hosted setup can match that. Stay if you serve many different models occasionally rather than one model constantly — that is exactly the shape Replicate's economics reward — and stay while nobody on the team wants to own GPU drivers, CUDA versions and node failures at 3am.
More Layer 3 migrations
Figures verified 2026-08-11 against vendor pricing pages. Prices change and migrations differ by estate — treat the cost delta as a starting model, not a quote. Rankings and recommendations here are merit-only; affiliate income never changes a verdict. See our methodology.