Layer 3 · self-hosting reality check
What it actually takes to self-host Apache Airflow
The docs say 4 GB. In practice you want 8 GB. Here is the honest version — real requirements, real monthly cost, what you will be maintaining, and the one thing that catches people out.
Usually reached from Astronomer (Astro) alternatives, where Apache Airflow is one of the picks.
| RAM — documented minimum | 4 GB |
|---|---|
| RAM — what it really needs | 8 GB |
| CPU | 4 vCPU |
| Disk | Modest for Airflow itself; task logs accumulate |
| Monthly cost | $50–90/mo self-hosted, against managed Airflow starting well into the hundreds |
| Setup time | A day for a working install, a week for a production-shaped one |
| How you install it | docker compose or Helm — scheduler, webserver, workers, PostgreSQL and a broker |
| Ongoing maintenance | High. It is five components with independent failure modes, and the scheduler in particular needs watching. |
| Where it stops scaling | Thousands of DAGs with the right executor. The metadata database is usually the first real bottleneck. |
The thing that catches people out
DAG files are parsed by the scheduler on a loop — every few seconds, for every DAG. Put a database query, an API call or a heavy import at the top level of a DAG file and you execute it hundreds of times an hour, hammering whatever it touches and slowing scheduling to a crawl. Keep top-level code trivial and do the work inside tasks. This single mistake accounts for most 'Airflow is slow' complaints.
When not to self-host Apache Airflow
Your pipelines are simple and scheduled. Cron plus a script is not embarrassing, and Airflow's operational weight is real. Prefect or Kestra are lighter if you need orchestration without the ecosystem.
Every guide here carries this section. A site that only ever tells you to self-host is selling something — the useful answer is sometimes no.
Other Layer 3 self-hosting guides
Common questions
- How much RAM does Apache Airflow actually need?
- 8 GB in practice. The documented minimum is 4 GB, which is the figure at which the process starts rather than the figure at which it works under real use. 4 vCPU alongside it.
- What does self-hosting Apache Airflow cost per month?
- $50–90/mo self-hosted, against managed Airflow starting well into the hundreds This is commodity VPS pricing and excludes your time, which is the larger cost for most people — budget for high. It is five components with independent failure modes, and the scheduler in particular needs watching.
- How long does it take to set up Apache Airflow?
- A day for a working install, a week for a production-shaped one, via docker compose or Helm — scheduler, webserver, workers, PostgreSQL and a broker.
- When should I NOT self-host Apache Airflow?
- Your pipelines are simple and scheduled. Cron plus a script is not embarrassing, and Airflow's operational weight is real. Prefect or Kestra are lighter if you need orchestration without the ecosystem.
- What is the most common mistake when self-hosting Apache Airflow?
- DAG files are parsed by the scheduler on a loop — every few seconds, for every DAG. Put a database query, an API call or a heavy import at the top level of a DAG file and you execute it hundreds of times an hour, hammering whatever it touches and slowing scheduling to a crawl. Keep top-level code trivial and do the work inside tasks. This single mistake accounts for most 'Airflow is slow' complaints.