✏️ The Math Behind LLM Pricing 06: The Compute Ledger of Training, RL, and Inference

I. First, Define N: Compute-Active Parameters

One easy source of confusion: what does N mean?

In this post, N means the number of parameters that actually participate in multiply-adds for each token. Think of it as N_compute:

• For a dense model: N_compute ≈ N_total
• For an MoE model: N_compute ≈ N_active, the activated experts plus shared parameters

Why not total parameters? Because this post is about compute, not storage. A MoE model may have 1T total parameters, but each token only visits a subset of them. Its memory footprint cares about N_total; its per-token FLOPs are closer to N_active.

🔑 Short version: in this post, N means “how many parameters this token actually computes through,” not “how many parameters exist in the model file.”

II. The Three Ledgers: Pretraining, RL, Inference

Over a model’s lifecycle, compute is mainly spent in three places:

1. Pretraining: feed massive text corpora so the model learns language and the world
2. RL / post-training: feed high-value tasks so the model learns reasoning, preferences, and alignment
3. Inference: users actually call the model, continuously consuming GPU time

They feel like different activities:

• Pretraining is like building the highway
• RL is like tuning signs, rules, speed limits, and navigation
• Inference is the traffic that runs every day

But all three eventually reduce to the same scarce resource: GPU time.

So the real question is:

If an AI lab has a fixed compute budget, how much should go to pretraining, how much to RL, and how much to inference?

III. 6ND: The First-Order Formula for Training Compute

For pretraining, the standard first-order estimate is:

where:

• N: compute-active parameters per token
• D: training tokens
• 6: the rough FLOPs coefficient for forward + backward

Where Does the 6 Come From?

Split training into two passes.

Forward pass: the model reads the token and predicts.

• Each parameter roughly does one multiply and one add
• So one token’s forward pass is about 2N

Backward pass: the model computes gradients and updates parameters.

• Backward propagation needs gradients, gradient flow, and update accumulation
• Roughly speaking, it costs about 2x the forward pass, or 4N

Together:

Training D tokens gives:

IV. Inference Compute: Forward Only

Inference does not update model weights. It only runs the forward pass.

So each token costs approximately:

Let D_inference be the total number of tokens the model processes over its commercial lifetime: user inputs, system prompts, tool results, model outputs, and every other token that really goes through the model.

Then:

This is the same worldview as the previous posts. Here we are not expanding memory-bound vs KV-bound vs batching mechanics; we are compressing them into a lifecycle compute ledger.

V. RL Is in the Middle: Part Inference, Part Training

RL / post-training is the easiest part to miscount, because it is not simply “train on a bit more data.”

A typical RL loop has at least two kinds of work:

1. Rollout: the model generates answers, reasoning traces, code, or action trajectories. This looks like inference: mostly forward passes.
2. Update: the model updates from rewards, preference signals, verifier outcomes, or policy gradients. This looks like training: it needs backward passes.

RL also tends to run less efficiently than pretraining:

• Autoregressive decode is more serial
• Rollouts may sample many candidates and keep only some
• Reward models, verifiers, and evaluators consume compute too
• Smaller batches, variable-length sequences, and online data streams all reduce MFU

So the cleanest expression is not “RL’s coefficient is exactly 4” or “exactly 6.” Instead:

Here α is the effective compute coefficient per RL token. But be careful: its size depends on what you count as a D_RL token.

How Large Is α? First Define the Token Counter

It depends on what you call D_RL:

• If D_RL means all rollout tokens generated, evaluated, and filtered inside the RL loop, α is smaller. In long-CoT RL, many tokens are just forward-generation tokens, so the coefficient can move toward 2-3.
• If D_RL means only high-quality tokens that actually enter gradient updates, then rollout, verifier, and multi-sample costs are hidden behind each update token; the effective α can be greater than 6.
• If a task uses many samples, self-checks, or long reasoning traces, whether α rises or falls depends on whether those generated tokens are included in the denominator or treated as hidden overhead.

This is why RL is hard to count: it contains both inference-like generation and training-like updates, and modern reasoning-model training is pushing more of the cost into the generation side.

VI. The Key Heuristic: Equal Marginal Returns, Not Literal Equal Costs

Now to the central question: how should pretraining, RL, and inference be allocated?

Strictly speaking, the optimum is not “all three total costs must be exactly equal.” The real condition is:

The last unit of GPU time should produce similar marginal value no matter which ledger receives it.

If pretraining has much higher marginal value than inference, the model is undertrained. If inference returns far more value than further training, the model is ready to be deployed more widely. If RL’s marginal value rises, the bottleneck may have moved from “the model lacks knowledge” to “the model cannot reliably use what it knows.”

But in many power-law systems, “similar marginal returns” often leads to a rough pattern:

Pretraining, RL, and inference compute land in the same order of magnitude, rather than one consuming 99% while the others are rounding errors.

That is the heuristic we will use.

VII. What Happens When the Three Ledgers Balance?

Write the three terms:

If lifecycle compute is roughly balanced:

The beautiful part: N cancels.

The model’s size disappears from the ratio. What remains is a relationship between token counts.

Pretraining vs Inference

So:

🔑 A successful model’s lifetime inference tokens should be in the same order of magnitude as its pretraining tokens, and may be a few times larger.

Not exactly equal. The point is: if pretraining is 100T tokens, lifetime inference should not be 1T. It should be in the hundreds-of-trillions range.

Pretraining vs RL

So:

If α = 6, each RL token is as expensive as a full training token:

If α = 4, some tokens are generated only while others are used for updates:

If long rollouts dominate and α moves toward 2-3:

🔑 Under the rollout-token convention, RL token count can absolutely exceed pretraining token count.

But if you count only “update tokens” and hide rollout / verifier / multi-sample cost behind each update token, the effective α may exceed 6, in which case D_RL becomes smaller than D_pretrain. That is not a contradiction. It is a different denominator.

VIII. This Explains Why Models Are “Overtrained”

Now apply this lens to Chinchilla.

The classic Chinchilla scaling-law takeaway can be remembered roughly as:

That is: if you only care about the best model under a fixed training-compute budget, the compute-optimal token count is about 20 tokens per parameter.

Take a simplified model with 100B compute-active parameters:

But the lifecycle ledger points in a different direction.

If a frontier model’s commercial lifetime inference volume is 100T-600T tokens, then:

That is 15-100× above the 2T Chinchilla-style point.

🔑 “Overtrained relative to Chinchilla” does not mean Chinchilla was wrong. It means Chinchilla optimized a narrower objective: training compute only, not the long inference bill that follows.

The real commercial target is not “best model for a fixed training budget.” It is:

Best lifecycle ROI across training + RL + inference.

Under that objective, somewhat smaller, sparser, and trained for longer can be rational: you spend more upfront training compute to lower active inference compute and improve quality, then earn it back over massive inference volume.

IX. A Fermi Estimate: Infer Training Scale from Inference Volume

We do not need any company’s internal numbers to make an order-of-magnitude estimate.

Suppose a frontier model family processes on average:

10M-100M tokens / second

and its main commercial lifetime is:

1-3 months ≈ 2.6M-7.8M seconds

Then total inference volume is:

or:

roughly 30T-800T tokens

Using D_inference ≈ 3D_pretrain:

D_pretrain ≈ 10T-260T tokens

This range is intentionally wide. Real deployments involve model variants, cache hits, routing policies, free vs paid users, input/output mix, and model retirement schedules. But it is enough to show the point:

🔑 If a model is truly used at massive scale, pretraining on tens to hundreds of trillions of tokens is not surprising.

That is not merely “training harder for benchmarks.” It falls naturally out of lifecycle economics.

X. RL’s Industrial Role: Not a Small Tail, a Second Major Ledger

Many people still think of RL as “a small amount of human preference data at the end.” That intuition is outdated.

For stronger models, RL / post-training may include:

• Math, code, tool-use, and long-task rollouts
• Multiple samples filtered by verifiers
• Reward models, process rewards, and rule-based judges
• Self-play, self-rewriting, and self-critiquing
• Product-specific behavior constraints and safety training

It may not contain as many raw tokens as pretraining, but it can consume a comparable amount of GPU time.

From the formula:

Under the rollout-token convention, if α moves toward 2-3, then for RL compute to catch up with pretraining, D_RL naturally becomes several times D_pretrain:

🔑 That is where “RL tokens can exceed pretraining tokens” comes from: not because each token is more expensive, but because rollout tokens are cheaper, so you do more of them.

So when you see frontier labs emphasize RL, reasoning models, verifiers, and long-horizon tasks, do not read it only as “better data.” A better framing is:

Once pretraining’s marginal returns decline, the next GPU may be more valuable in RL than in more undifferentiated web text.

The bottleneck then shifts: once rollout generation is no longer the only constraint, high-quality verifiable RL environments become critical. Code sandboxes, math datasets, tool tasks, and automatic evaluators are all ways to expand RL capacity.

XI. What This Framework Helps You Judge

1. A Model Company’s Compute Planning

If a company trains aggressively but has no inference distribution, it may have built an asset that is not used enough.

If a company has massive inference traffic but underinvests in training and RL, it may be stretching an old model too far and eventually get squeezed by both quality and cost.

A healthy frontier lab has the three ledgers reinforcing one another: pretraining creates capability, RL turns capability into reliable behavior, and inference turns behavior into revenue and feedback.

2. Architecture Choices

Why does MoE matter? Because it makes N_compute smaller than N_total.

In all three ledgers, N is a multiplier. Lowering active compute per token improves pretraining, RL, and inference at once. MoE, KV compression, speculative decoding, routing, and cache reuse are all trying to shrink that multiplier or amortize it better.

3. Product-Level Routing

If you are routing between models, you should not only look at the API price per token.

At the same apparent capability level, a model with stronger pretraining, better RL, and a healthier inference stack may produce:

• Fewer retries
• Fewer failed tool calls
• Less need for long prompt patches
• Less human fallback

All of that returns to real cost. The cheapest token does not always produce the cheapest completed task.

XII. One-Paragraph Summary

XIII. Two Exercises: Use the Ledger

Exercise 1: A 100B Model Trained on Only 2T Tokens

Question: suppose a startup trains a 100B active-parameter model on only 2T tokens, close to the Chinchilla point. Under the “training + inference ledger” view, what is wrong with its compute allocation? If its goal is to serve many users, how should it adjust?

The intuitive answer: training and inference are badly imbalanced.

Compute it:

If the model is meant to serve a large market, suppose its lifetime inference volume is 200T tokens:

Inference compute is more than 30x pretraining compute. If this model is supposed to support massive inference volume, it was trained too lightly.

But there is one important flip: which direction you adjust depends on what is fixed.

View A: inference demand is fixed

The company must serve that many users, so inference compute is not optional. Then the answer is: increase training investment.

Concretely, do not train the same 100B model for only 2T tokens. Train it on 60T-100T tokens so training compute catches up with expected inference scale.

🔑 This is where overtraining comes from: not from the training team randomly wanting to burn more GPU, but from future inference scale pulling the training target upward.

View B: training budget is fixed

If the company can only afford the 2T-token training run, then the conclusion flips: do not serve that much inference volume. Limit users, serve only high-paying customers, or position the model as a research / lower-traffic model rather than a mass-market product.

In the real world, View A is more common. AI businesses are usually built around scaling usage; inference demand is pulled by the market, while training is a controllable investment. So the common path is: given expected inference scale, infer how far training should go.

The startup has three adjustment paths:

1. Train the same 100B model on 60T-100T tokens: inference cost stays the same, quality rises, and the ledger is healthier.
2. Train a larger model: capability may improve, but per-token inference cost rises, so this may not be lifecycle-optimal.
3. Train a sparser MoE model: larger total capacity, smaller active compute, more room to absorb training data while keeping inference cost low.

🔑 Modern frontier systems lean toward sparse MoE because architecture, training mix, and inference economics are one coupled decision.

📌 Applied to Tbox: when judging whether a model is “trained enough,” do not only compare it to Chinchilla. Compare its training volume with its expected serving scale. A small model trained on 100T tokens may be preparing for mass deployment; one trained on 2T tokens may be closer to a research or low-traffic deployment.

Exercise 2: Can RL Tokens Exceed Pretraining Tokens?

Question: if RL keeps taking a larger share of frontier-model compute, can RL token count exceed pretraining token count? Under what conditions?

Return to the equation:

So:

Answer: yes, if α falls below 6 and RL’s marginal returns have not saturated.

The numbers make it clear:

🔑 Smaller α means more RL tokens. Cheaper tokens should be done more.

This is the key trend in long-CoT RL: one rollout may generate thousands or tens of thousands of tokens, while only part of the trajectory drives gradient updates.

• Rollout generation: mostly forward pass, α ≈ 2
• Gradient update: forward + backward, α ≈ 6
• Weighted average: if generation dominates, α moves toward 2-3

Then:

So RL token count can plausibly reach 2-3x pretraining token count.

For RL to formally exceed pretraining, at least three conditions need to hold:

1. RL is dominated by long rollout inference: lots of generation, relatively fewer updates, so α falls.
2. RL still has strong marginal returns: the model has not saturated on RL.
3. There are enough high-quality, verifiable RL environments: otherwise more generated tokens are just noise.

The third condition may be the real bottleneck. Compute can scale and rollouts can be generated, but verifiable tasks, real environments, automatic evaluators, code sandboxes, and math datasets are harder to scale.

📌 Applied to Tbox: if your product is an agent, code-generation system, math solver, or data-analysis workflow with verifiable outputs, then user interactions plus success / failure signals can themselves become an RL data source. That is an underrated strategic asset for application-layer companies.