Braintrust → Promptfoo
What it saves, what actually moves, what you rebuild — and the thing that catches people.
What it costs, and what it saves
From roughly $249 a month per small team plus trace overage, to free with unlimited seats. The seat count matters more than the headline: everyone who edits a prompt needs to run evals, so the correct licence count is your whole engineering team.
Moves cleanly
Your test cases, which are your own data in your own format. Scoring logic is your code either way.
You rebuild
The hosted dashboard and historical trace comparison. Promptfoo has a local web viewer but nothing that replaces a shared hosted UI a product manager visits. If that matters, this is a downgrade you should make deliberately.
What Braintrust costs you today
Free tier for individuals with limited monthly spans. Paid plans are per-seat — commonly around $249/month for a small team on the pro tier — plus usage-based charges once trace volume grows. Enterprise pricing is negotiated. The per-seat model is what makes it expensive for teams where many engineers touch prompts occasionally.
What actually holds you in
Low. Test cases and scoring functions are your own logic in your own repository, and the SDK integration is thin. The genuinely sticky part is historical trace data, which is worth exporting before you cancel because comparison against past runs is most of the value. Plan an export, not a migration.
What you are moving to
Promptfoo defines evaluations as YAML: your prompts, your test cases, your assertions, versioned in the repository next to the code they test. It runs from the CLI or in CI, compares outputs across models and prompt variants side by side, and fails the build when a change regresses. It also includes red-teaming for prompt injection and jailbreak testing. MIT licensed with no seat limits, which matters because everyone who edits a prompt should be running it.
Free and MIT, unlimited seats. An enterprise tier exists for larger organisations.
Promptfoo strengths
- Evals live in your repo and run in CI — a regression blocks the merge
- Side-by-side model and prompt comparison out of the box
- Includes red-teaming for injection and jailbreak testing
- MIT, no seat limits, nothing leaves your infrastructure by default
What you give up
- YAML configuration gets long on large test suites
- Reporting UI is lighter than a hosted platform's
- Trace history is yours to store and manage
The migration, step by step
- 1Export your test cases and any production traces you have promoted into the eval set. Traces are the sticky part — most of the value is comparison against past runs.
- 2`npm install -g promptfoo` and write `promptfoo.yaml`: providers, prompts, tests, assertions. Start with your ten highest-value cases rather than porting everything.
- 3Add assertions that actually fail. `contains`, `is-json`, `javascript` for custom logic, and `llm-rubric` where a judge model is genuinely needed.
- 4Wire it into CI so a prompt regression blocks the merge. This is the whole point — evaluation you visit is not evaluation, it is a dashboard.
- 5Point the judge model at something cheap, or at a locally served model, so the full suite can run nightly without a token bill.
- 6Add `promptfoo redteam` for prompt-injection and jailbreak coverage, which Braintrust does not include.
The gotcha
LLM-as-judge assertions cost tokens on every run, and a large suite in CI on every commit gets expensive fast — occasionally more expensive than the subscription you left. Split it: cheap deterministic assertions on every pull request, the judged suite nightly or pre-release.
When to stay on Braintrust
The dashboard is doing real work for people who would otherwise never look at evaluation at all. Braintrust's flow from production trace straight into a regression case is genuinely well built, and a UI a non-engineer will open has value a CLI does not. Stay while your seat count is small enough that the pricing is not the dominant line.
More Layer 4 migrations
Figures verified 2026-08-11 against vendor pricing pages. Prices change and migrations differ by estate — treat the cost delta as a starting model, not a quote. Rankings and recommendations here are merit-only; affiliate income never changes a verdict. See our methodology.