Layer 3 · self-hosting reality check
What it actually takes to self-host BentoML
The docs say 2 GB. In practice you want 4 GB plus model memory. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from Modal alternatives, where BentoML is one of the picks.
Wondering whether you need to at all? Is Modal free? — what the free tier actually allows, and where the wall is.
| RAM — documented minimum | 2 GB |
|---|---|
| RAM — what it really needs | 4 GB plus model memory |
| CPU | 4 vCPU, or a GPU for anything sizeable |
| Disk | Model artifacts plus container images |
| Monthly cost | $40–200/mo depending on hardware, against per-second serverless inference billing |
| Setup time | Half a day |
| How you install it | Python framework — you write a service definition and it builds a container |
| Ongoing maintenance | Moderate. You own the serving infrastructure it produces. |
| Where it stops scaling | Multi-model production serving, which is its genuine strength — many models, many frameworks, one interface. |
The thing that catches people out
It packages models into containers beautifully and does not orchestrate them. People assume BentoML is the whole deployment story, then find they still need Kubernetes, autoscaling and a load balancer underneath. It is the packaging and serving layer, not the platform — decide where the containers will actually run before you commit to it.
When not to self-host BentoML
You serve one language model and nothing else. vLLM does that better with far less scaffolding.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 3 self-hosting guides
Common questions
- How much RAM does BentoML actually need?
- 4 GB plus model memory in practice. The documented minimum is 2 GB, which is the figure at which the process starts rather than the figure at which it works under real use. 4 vCPU, or a GPU for anything sizeable alongside it.
- What does self-hosting BentoML cost per month?
- $40–200/mo depending on hardware, against per-second serverless inference billing This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for moderate. You own the serving infrastructure it produces.
- How long does it take to set up BentoML?
- Half a day, via Python framework — you write a service definition and it builds a container.
- When should I NOT self-host BentoML?
- You serve one language model and nothing else. vLLM does that better with far less scaffolding.
- What is the most common mistake when self-hosting BentoML?
- It packages models into containers beautifully and does not orchestrate them. People assume BentoML is the whole deployment story, then find they still need Kubernetes, autoscaling and a load balancer underneath. It is the packaging and serving layer, not the platform — decide where the containers will actually run before you commit to it.