Commercial guide - Last reviewed 2026-09-17 - By Ramachandra Vikas Chamarthi
Cost to Host a Private LLM: Self-Hosted vs API
What it costs to host a private LLM in 2026: monthly cost by model size, the operations floor most spreadsheets omit, and where break-even against current API pricing actually sits.
Direct answer for cost to host a private LLM
The short answer
Hosting a private LLM costs about $640 per month for a 7B-class model and $3,180 for a 70B-class model at INT8, plus a $1,500-$2,500 monthly operations floor. Across 900 modeled scenarios, the median break-even against mid-tier APIs is 11-12M tokens per day. At 5 concurrent users, managed APIs usually win once operations are counted.
Stay on APIs at low concurrency. With a $1,500/month operations floor counted, 5 concurrent users can beat mid-tier pricing in only 10.7% of modeled scenarios, and the cheapest tier in 0%.
Self-host when privacy, data residency, latency control, or model control is the requirement. None of those are priced here, and they are often the real reason.
With steady traffic from many concurrent users, self-hosting beats mid-tier API pricing in most scenarios: 98.3% at 100 users, operations included. Against the cheapest tier it still usually loses.
Original study - 900 scenarios - data generated 2026-09-17
We ran 900 private-LLM cost scenarios against current API prices
Across 900 modeled private deployments, self-hosting can beat mid-tier API pricing in 81.8%-84.4% of scenarios and the cheapest API tier in 24.2%. Concurrency decides it, not model size.
Correction, September 17, 2026: the first version of this study, published earlier the same day, reported that self-hosting never beat the cheapest API tier. That model sized one replica per ten users, credited batching at about a third of what we measured, overstated KV-cache memory fourfold, and assumed FP16 only. Checking it against our own benchmark exposed the errors. Every number on this page is from the corrected model, and the conclusion changed with it.
Key findings
- Against mid-tier pricing, self-hosting can win in 84.4% of scenarios versus OpenAI GPT-5.6 Terra and 81.8% versus Anthropic Claude Sonnet 5. "Can win" means a volume exists at which private hosting is cheaper and the selected hardware can actually serve it. The median break-even is 10.5M tokens per day against Terra and 12.4M against Sonnet 5.
- Against the cheapest tier it usually loses. Self-hosting can beat OpenAI GPT-5.6 Luna ($0.2/$1.2 per million tokens) in 24.2% of scenarios, and where it can, the median break-even is 82.2M tokens per day.
- People cost more than GPUs at small scale. Add a $1,500/month operations floor and the share that can beat Terra falls from 84.4% to 67.3%; against Luna, from 24.2% to 8.2%.
- Concurrency decides it. With the operations floor counted, 5 concurrent users can beat Terra in 10.7% of scenarios and Luna in 0%. At 100 concurrent users those become 98.3% and 22%. A GPU that serves 5 people costs the same as one that serves 50.
- Quantization is worth about a fifth of the bill. Median private cost is $1,589/month at INT8 against $2,044 at FP16, 22% lower.
- Buying beats renting. On-prem servers can beat Terra in 91.8% of scenarios and Luna in 34%; rented cloud GPUs in 77.1% and 14.4%.
- Break-even moved up. Measured against the 2024 price sheet this calculator shipped with until this month (GPT-4o, Claude Sonnet at $3/$15), the median break-even volume is now 20% higher.
| Model size | Median private cost / month | Median cost per M tokens at capacity | OpenAI GPT-5.6 Terra | Anthropic Claude Sonnet 5 | Google Gemini 3.7 Flash | OpenAI GPT-5.6 Luna |
|---|---|---|---|---|---|---|
| 7B | $637 | $0.43 | 100% | 100% | 74.4% | 53.3% |
| 13B | $803 | $0.62 | 100% | 97.8% | 72.2% | 47.8% |
| 34B | $1,606 | $0.8 | 86.7% | 83.3% | 65.6% | 30% |
| 70B | $3,178 | $1.61 | 71.1% | 71.1% | 48.9% | 12.2% |
| 120B+ | $3,509 | $1.78 | 81.1% | 81.1% | 46.7% | 7.8% |
How far off is the model? We checked.
The engine's throughput model is checked against our own Llama 3 70B benchmark every time this study runs. It reproduces all 4 measured points within 7.8%, and the run fails if any point drifts past 15%. On cost: our published $0.47 per million tokens assumes one H100 running flat out all month at the best throughput we measured (1,560 tokens/sec). On this calculator's cost basis ($1,450/month for one on-prem H100) that throughput is $0.35 per million flat out, and $0.54 at the 65% duty cycle we assume for interactive work.
| Benchmark point (Llama 3 70B, vLLM) | Measured tokens/sec | Model tokens/sec | Error |
|---|---|---|---|
| FP16, 2x A100 80GB (TP=2), batch 1 | 25 | 23 | -6.8% |
| FP16, 2x A100 80GB (TP=2), batch 32 | 680 | 627 | -7.8% |
| INT8, 1x H100 80GB, batch 1 | 42 | 45 | +6.3% |
| INT8, 1x H100 80GB, batch 32 | 1,150 | 1,202 | +4.5% |
Method
- Ran the same cost engine the live LLM cost calculator uses across every combination of its inputs: 5 model sizes, 2 weight precisions, 2 deployment types, 3 latency modes, 5 workload shapes, and 3 concurrency levels. That is 900 scenarios and 3,600 comparisons.
- For each scenario the engine picks the cheapest hardware that serves every concurrent user at the speed the latency mode requires. Single-stream decode speed is GPU memory bandwidth divided by weight size. Batching follows the curve we measured. Replicas are set by how many requests fit in KV-cache memory.
- Private cost is a fixed monthly figure: hardware amortized over 36 months, power, and per-GPU maintenance, or GPU rental for cloud. API cost is linear in volume. The break-even volume is therefore exact: private monthly cost divided by the API's blended rate for that workload's input-output mix.
- Self-hosting counts as able to win only when the break-even volume fits inside what the selected hardware can serve at the assumed duty cycle. It says a winning volume exists. It does not say you have that volume.
- API list prices were read from each provider's own pricing page on 2026-09-17.
What this does not show
- Modeled, not measured, outside one benchmark. The throughput model is validated on Llama 3 70B on A100 and H100. It is applied unvalidated to other model sizes and GPUs.
- Decode only. The engine does not price prefill compute, which flatters small GPUs on long-input workloads such as document processing.
- No batching gain is credited beyond batch 32, the largest we measured. Real stacks may do better or worse past that point.
- INT8 cost about 1 point of MMLU on our 70B benchmark. We have not measured the quality cost on other sizes. Validate on your own evals.
- GPU purchase and rental prices are point-in-time and vary widely by provider.
- API tiers are not interchangeable. A cheap tier is not a substitute for a 70B model on every task. Price against the tier your task actually needs.
- The engine prices hardware, power, and per-GPU maintenance. It does not price the engineers who run the stack, which is why results are shown with and without an operations floor.
- Privacy, data residency, latency control, and model control are not priced at all. They are often the real reason to self-host.
Check our work
Download the full dataset (CSV), or re-run the study yourself. These two files are the exact code that produced every number on this page:
- study.mjs — Runs every scenario, checks the model against our benchmark, and writes the dataset.
- engine.mts — The cost engine itself: the same code the live calculator runs, with every assumption in it.
curl -O https://www.navyaai.com/data/private-llm-cost-study/study.mjs
curl -O https://www.navyaai.com/data/private-llm-cost-study/engine.mts
node study.mjsNode.js 23.6 or newer. No install, no dependencies. It writes the same CSV you can download above.
The dataset is licensed CC BY 4.0: reuse it freely, with credit to NavyaAI. The scenarios were run by code, and the script and full dataset are published above. The write-up was drafted with AI assistance and reviewed by the named author.
Comparison table
| Factor | Managed API | Self-hosted |
|---|---|---|
| Upfront cost | Low. Pay as usage arrives. | High. GPU, hosting, networking, and engineering work arrive before savings. |
| Unit economics | Simple token pricing, but agent loops and long context can multiply the invoice. | A fixed monthly cost. Cheap per token when the hardware is busy, expensive when it sits idle. |
| Unit cost, September 2026 | Blended at a chatbot's 512:256 input-output mix: $0.53/M on OpenAI GPT-5.6 Luna, $1.75 on Google Gemini 3.7 Flash, $4.67 on Anthropic Claude Sonnet 5, $5.33 on OpenAI GPT-5.6 Terra. List prices verified 2026-09-17. | Median $0.43/M for a 7B model to $1.61/M for a 70B model, at INT8 with the hardware full, before the operations floor. |
| Operational burden | Provider handles serving, scaling, and reliability. | Your team owns uptime, monitoring, upgrades, capacity, and incident response. |
| Best fit | Experiments, variable demand, low concurrency, quality-sensitive workflows. | Many concurrent users, steady traffic, private or margin-sensitive production workloads. |
Worked example
70B chatbot at 60M tokens/day (September 2026 pricing)
- 1,824M tokens/month, steady
- 32 concurrent users, interactive latency
- 512:256 input-output mix (chatbot)
- INT8 weights
- API list prices verified 2026-09-17
| Option | Monthly cost | Cost per million tokens |
|---|---|---|
| Self-host, on-prem: 1x NVIDIA H200 141GB ($37,000 upfront) | $1,589 hardware, power, maintenance. $3,089-$4,089 with operations. | $0.87 before operations, at 62% of capacity |
| Self-host, rented: 2x NVIDIA A100 80GB | $3,212 rental. $4,712-$5,712 with operations. | $1.76 before operations, at 91% of capacity |
| API: OpenAI GPT-5.6 Terra | $9,728 | $5.33 blended |
| API: Anthropic Claude Sonnet 5 | $8,512 | $4.67 blended |
| API: Google Gemini 3.7 Flash | $3,192 | $1.75 blended |
| API: OpenAI GPT-5.6 Luna | $973 | $0.53 blended |
At this volume, on-prem hosting with operations included comes to $3,089-$4,089 a month, against $8,512-$9,728 on mid-tier APIs, $3,192 on Google Gemini 3.7 Flash, and $973 on OpenAI GPT-5.6 Luna. API tiers are not interchangeable: find the cheapest one that clears your quality bar, then ask whether private hosting beats that number.
Frequently asked questions
How much does it cost to host a private LLM?
Median monthly cost across our modeled INT8 scenarios, for hardware, power, and maintenance: $637 for 7B, $803 for 13B, $1,606 for 34B, $3,178 for 70B, $3,509 for 120B+. Add $1,500-$2,500 per month for the people who run it. The bill tracks how many users you serve at once more than it tracks model size, because a GPU that serves 5 people costs the same as one that serves 50.
Is self-hosting still cheaper than an LLM API in 2026?
It depends on which API tier you compare against, and how busy the hardware is. Across 900 modeled scenarios, self-hosting can beat mid-tier pricing (OpenAI GPT-5.6 Terra, Anthropic Claude Sonnet 5) in 81.8%-84.4% of them, Google Gemini 3.7 Flash in 55.6%, and OpenAI GPT-5.6 Luna in 24.2%. Count a $1,500/month operations floor and those fall to 64.9%-67.3%, 40.8%, and 8.2%. Find the cheapest API tier that clears your quality bar first. Private hosting only has to beat that number.
Is self-hosting always cheaper than an LLM API?
No. Self-hosting can be more expensive when utilization is low, the workload changes often, or the team lacks serving operations experience.
What should be measured before self-hosting?
Measure monthly input and output tokens, concurrency, latency target, retry rate, cache hit rate, RAG overhead, provider mix, and expected growth.
How many tokens per day justify self-hosting?
In our modeled scenarios, the median break-even is 10.5M tokens per day against OpenAI GPT-5.6 Terra, 12.4M against Anthropic Claude Sonnet 5, 33M against Google Gemini 3.7 Flash, and 82.2M against OpenAI GPT-5.6 Luna. Those are for hardware cost alone. An operations floor, volatile traffic, or idle hours all push the threshold higher.
What hidden costs does self-hosting add?
The lines most spreadsheets omit: utilization loss versus theoretical capacity, monitoring and observability, model upgrades and re-quantization, incident response, capacity buffer for peaks, and the engineering time that owns all of it. These form a fixed monthly floor that exists at any volume.
How do I estimate the total cost of ownership for LLM inference?
Sum four buckets. (1) Serving: GPU amortization or rental, power, and hosting, divided by realistic utilization — not theoretical throughput. (2) Operations floor: monitoring, upgrades, on-call, capacity buffer (typically $1,500-$2,500/month for one production stack). (3) Workflow multiplier: retries, RAG overhead, and agent loops that multiply raw token volume 2-50x. (4) For APIs: tokens × price plus the same workflow multiplier. Compare both totals at your real monthly volume — that comparison, not the price sheet, is your TCO.
References & related
Apply this to your stack
Request a free AI inference audit before changing providers or buying GPUs.
Share your monthly spend, token volume, model stack, RAG or agent pattern, and latency target. NavyaAI will identify the first cost levers to inspect.
Request Free Audit