The Compute Line Item: Budgeting AI Inference as Cost of Goods Sold

For most of the last decade, software finance rested on a comfortable assumption: the marginal cost of serving one more customer rounds to zero. That assumption built the SaaS multiple. It is no longer true for the fastest-growing category of software being funded today. When a product calls a frontier model on every user action, each interaction carries a real, metered, non-trivial cost that scales linearly with usage. The gross margin of an AI-native company is therefore a function of engineering decisions made daily by people who have never been asked to think about gross margin.

The finance failure is not that inference is expensive. It is that inference spend is usually invisible in the management accounts. It arrives as a single monthly invoice from a cloud provider or model vendor, is coded to a technology or R&D cost centre, and never touches cost of revenue. The result is a P&L that reports an 80% gross margin while the business is actually operating in the low fifties. Founders make hiring, pricing and fundraising decisions against a number that does not exist, and the correction arrives at the worst possible moment: during a quality of earnings review, three weeks before a term sheet is meant to be signed.

The remedy is unglamorous and entirely mechanical. Every inference call must be attributed to a customer, a feature and a request outcome. The spend must be reclassified below the revenue line. The unit economics must be expressed as cost per successful task rather than cost per token. Once those three things are true, the levers that actually move margin become visible, and most of them are cheap. We have seen companies move from 51% to 74% inference-adjusted gross margin in a single quarter without touching headcount, purely by instrumenting what they were already spending.

Model vendors quote price per million tokens because it is the unit they can meter. It is not the unit that determines your cost of revenue. A single user request in a well-built product rarely maps to a single model call. It maps to a retrieval step, a planning call, two or three tool invocations, a synthesis pass and, frequently, a retry when the first output fails validation. The user perceives one action. The invoice records six. When a finance team divides monthly inference spend by monthly requests, the resulting figure is often three to five times higher than the number engineering quoted from the vendor price list.

Retries are the most under-modelled component. In agentic products, a failure rate of 10 to 15% on the first attempt is normal and often acceptable from a product standpoint, because the retry is invisible to the user. From a cost standpoint it is anything but invisible: a 15% failure rate with a full re-run of the chain adds roughly 15% to variable cost, and if the retry uses a larger model as a fallback, it can add substantially more. The correct denominator is successful tasks completed, because that is the unit the customer pays for. Everything else is waste, and waste that is not measured is waste that compounds.

Context length is the second distortion. Prompt size grows quietly as teams add examples, system instructions, retrieved documents and conversation history. Each addition is individually defensible and collectively expensive, because input tokens are charged on every single call, including every retry. A product that ships a 2,000-token system prompt in January and a 9,000-token system prompt in June has quadrupled the floor cost of every interaction without a single line item appearing in any budget review. We recommend a standing monthly review of average prompt size per feature, treated with the same seriousness as a review of headcount per function.

The accounting question is straightforward once it is asked plainly: is this cost incurred to deliver the service the customer is paying for? Inference consumed when a paying customer uses the product is cost of revenue. Inference consumed to train, evaluate, fine-tune or experiment is research and development. Inference consumed by free-tier users is a sales and marketing cost, and should be reported as such so that the true blended customer acquisition cost is visible. Most companies have all three flowing through one invoice and one cost centre, which makes each of those judgements impossible to audit.

The instrumentation requirement is a tag on every request. At minimum: environment, customer identifier, feature, model, token counts in and out, latency and outcome. That tag set is sufficient to allocate spend into the three buckets automatically, and to produce a gross margin by customer cohort that survives diligence. It is a two-week engineering project that most teams defer for eighteen months, usually because nobody has quantified the cost of not doing it. The cost of not doing it is the difference in valuation between a company that can prove its unit economics and one that asks the buyer to take them on faith.

The reclassification will hurt the reported number, and that is the point. A business reporting 79% gross margin that is actually at 54% is a business with a pricing problem it does not yet know it has. Finding that out internally, twelve months before a raise, leaves room to fix it through routing, caching, packaging or price. Finding it out during confirmatory diligence leaves room only to renegotiate the price of the equity being sold. In every engagement where we have run this reclassification early, the discovered margin gap was material, and in every case the founders said afterwards that the uncomfortable version of the number was the more useful one.

Routing is the highest-return lever and the least used. Most product surfaces do not require a frontier model. Classification, extraction, routing, summarisation of short inputs and formatting can typically be served by a small model at a fraction of the cost, with quality that is indistinguishable to the user on that specific task. A routing layer that sends the easy 70% of traffic to a cheap model and reserves the expensive model for genuinely hard requests routinely cuts blended cost by half. The engineering effort is modest. The reason it is skipped is organisational, not technical: nobody owns the cost line, so nobody is measured on reducing it.

Caching is the second lever, and it is nearly free. Semantic caching of repeated or near-identical requests, prompt caching of the static portion of the context window, and memoisation of deterministic tool outputs together eliminate a surprising share of spend in products with concentrated usage patterns. In document-heavy workflows we have observed cache hit rates above 40%, each hit representing a call that was paid for once and served many times. The prerequisite is again instrumentation: you cannot cache what you have not measured for repetition.

Distillation and fine-tuning are the third lever, and the one with real trade-offs. Training a smaller model on the outputs of a larger one produces a cheap, fast, narrow specialist that outperforms the frontier model on that narrow task. The cost is optionality: the distilled model is now a dependency with its own maintenance burden and its own drift, and swapping the underlying architecture becomes a project rather than a configuration change. We advise distilling only where the task is stable, high-volume and economically material, which in practice means one or two surfaces in a typical product, not the whole system.

Cloud and model vendors offer meaningful discounts for committed spend, typically 20 to 40% against list in exchange for a one to three year floor. The discount is real. The risk is that a variable cost has been converted into a fixed one at precisely the stage where demand is least predictable. A commitment sized to the growth plan is a bet that the growth plan is correct. If usage lands at 60% of forecast, the effective unit cost is higher than list price would have been, and the company is paying for compute it cannot consume while also carrying the reputational discomfort of a shortfall discussion with the vendor.

The disciplined approach is to size the commitment to the trough, not the forecast. Take the lowest plausible monthly consumption over the commitment term, apply the discount to that base, and buy the remainder on demand. The blended rate is worse than the aggressive commitment on paper and better in every scenario where the forecast is wrong, which is most scenarios. Where a vendor offers ramped commitments, the ramp should trail actual consumption rather than lead it, and every step-up should require the same internal approval as a headcount requisition of equivalent annual cost.

There is also a balance sheet dimension that founders consistently overlook. A multi-year compute commitment is a contractual obligation that will be disclosed in diligence, modelled as debt-like by a sophisticated buyer, and deducted from enterprise value in the bridge to equity value. A three-year commitment at $200,000 per month is a $7.2 million obligation. It may still be the right decision, but it should be made with the same rigour as taking on a term loan of that size, because economically that is what it is.

The diligence question set has matured quickly. Two years ago, investors asked what model a company used. Today they ask for gross margin with inference allocated below the line, cost per successful task by feature, the trend in that cost over the last six quarters, the share of traffic served by non-frontier models, and the terms of any committed compute. They ask what happens to margin if model prices fall 40% and what happens if they rise 40%, because both have occurred within recent memory and neither is forecastable.

The companies that answer these questions crisply are rewarded with a materially different conversation. Margin that improves as volume grows signals engineering leverage and defensibility. Margin that is flat signals a business reselling someone else's model with a thin interface on top, which is a position that prices accordingly. The number itself matters less than the demonstrated ability to move it: a company at 58% margin that has improved 14 points in three quarters through routing and caching is a better asset than one sitting statically at 68%.

The practical instruction is to build the inference-adjusted margin view now, review it monthly alongside the management accounts, and give one named person ownership of the cost line. In our experience the ownership question is the one that determines the outcome. Where inference cost belongs to nobody, it grows at the rate of product ambition. Where it belongs to someone with a target, it becomes an engineering problem, and engineering problems of this shape are usually solved within two quarters.