Tokens per Watt: The Metric That Prices an AI Factory

Tokens per watt is the number of output tokens an AI system generates per second for each watt of electrical power it consumes, calculated as tokens per second divided by watts. Specifically, its inverse, joules per token (often written as watts per token), is the energy required to produce a single token. The metric thus converts an AI factory’s scarcest input, grid power, into its only saleable output, and it is the denominator behind every tokens-per-dollar number an operator quotes.

The figure, however, is not a constant for a chip. The same accelerator can post a 10x to 100x range depending on whether power is measured at the GPU or the utility meter, and on the model, precision, batch size, context length, and per-user latency target. Accordingly, this guide gives the formula, published benchmark figures by accelerator generation with their exact test context, a 1 MW worked example from the meter to tokens per dollar, and the measurement standards that make the numbers comparable.

What is tokens per watt and how is it calculated?

The base formula is throughput divided by power:

[
\text{tokens per watt} = \frac{\text{tokens per second}}{\text{watts}} = \frac{\text{tokens}}{\text{joule}}
]

Because one watt is one joule per second, tokens per second per watt reduces to tokens per joule. Conversely, the inverse is energy per token:

[
\text{joules per token} = \frac{\text{watts} \times \text{seconds}}{\text{tokens produced}} = \frac{1}{\text{tokens per watt}}
]

The EuroMLSys 2025 paper “Advocating Energy-per-Token” defines the metric exactly this way, as power multiplied by wall-clock time divided by tokens processed, and argues it should sit alongside latency and throughput as a first-class serving metric ([1]). The ML.ENERGY leaderboard likewise uses the same identity, stating that “Throughput per Watt = Work / Energy” ([2]).

Similarly, at data center scale, the industry reports the same quantity in larger units. SemiAnalysis’s InferenceMAX benchmark, for example, reports “tokens/s per all-in provisioned MW of total utility power,” and NVIDIA reports “throughput per megawatt” ([3]; [4]). Divide tokens per second per MW by 1,000,000 to get tokens per second per watt; divide 1,000,000 by tokens per second per MW to get joules per token.

Notably, two conventions matter when reading any published figure. First, “Tokens” almost always means output (decode) tokens, with input tokens folded into a fixed sequence length such as 1K in, 8K out. Second, “Watts” can mean nameplate thermal design power (TDP), measured GPU power, server power at the wall, or utility power for the whole facility. SemiAnalysis notes that “TDP is not the same as expected average power,” which is why its metric uses provisioned utility power rather than chip TDP ([3]).

Chip-level, server-level, rack-level, and facility-level tokens per watt

In fact, every step from the GPU die to the utility meter adds power that produces no tokens. As a result, a tokens-per-watt figure is meaningless without stating which boundary it uses.

First, chip level divides throughput by GPU power only. The Hugging Face AI Energy Score, for instance, measures GPU energy via nvidia-smi and scores “solely on GPU energy,” excluding CPU and RAM ([5]). Chip-level numbers are therefore the highest an operator will ever see.

Second, server level adds host CPUs, memory, NICs, fans, and power supply losses. MLCommons’ MLPerf Power rules, similarly, require AC power to be measured “at the wall” for the system under test, including host CPU, accelerators, memory, and fans, during the same run that produces the performance score ([6]). An NVIDIA DGX H100 illustrates the gap: eight H100 SXM GPUs at up to 700 W each total 5.6 kW, whereas the system’s maximum power is 10.2 kW, so GPUs are about 55 percent of server power at nameplate ([7]; [8]).

Third, rack level adds in-rack networking and, for rack-scale systems, the NVLink switch trays and host CPUs. GB300 NVL72, for example, combines 72 Blackwell Ultra GPUs with 36 Grace CPUs in one liquid-cooled rack, all drawing power against the same token output ([9]).

Finally, facility level divides by utility power, which adds cooling, power distribution losses, and everything else captured by Power Usage Effectiveness (PUE), the ratio of total facility energy to IT energy standardized as ISO/IEC 30134-2:2016 ([10]). The Uptime Institute’s 2025 survey put the industry average PUE at 1.54, flat for roughly six years, while Google reported a 2025 fleet-wide PUE of 1.09 ([11]; [12]). NVIDIA cites roughly 1.25 PUE for Blackwell deployments with about 20 percent air cooling, 1.1 for fully liquid-cooled Vera Rubin at 45°C inlet water, and furthermore warns that at gigawatt scale “up to 40% of the power can be lost before it reaches compute” ([13]).

Hence the practical rule: a chip-level figure of 1 token per joule becomes roughly 0.55 tokens per joule at the server wall in a DGX H100 and roughly 0.36 tokens per joule at the meter of a PUE 1.54 facility. Because only the facility-level figure ties to an electricity bill, only the facility-level figure belongs in a tokens-per-dollar calculation. Why PUE alone understates AI-hall losses is covered in Beyond PUE.

Published tokens per watt and energy per token by accelerator generation

The table lists only figures traceable to a primary or reputable technical source, with the exact test context, and flags vendor figures. Independent figures come from SemiAnalysis InferenceMAX/InferenceX (open-source, run nightly on NVIDIA and AMD hardware), the ML.ENERGY leaderboard, and peer-reviewed work. Where a source reports tokens per second per MW, joules per token is 1,000,000 divided by that figure.

Three cautions, however, apply. First, InferenceX comparison pages are labeled “Interpolated from real benchmark data” and update as software improves; the GB300 NVL72 figure near 95 to 97 tok/s/user differs between the two pages fetched the same day (2.6M vs 5.8M tok/s/MW) because each interpolates from a different configuration set. Second, NVIDIA’s own pages give both “5x improvement in throughput (TPS per MW)” and “50x” for GB300 NVL72 versus Hopper, depending on workload and latency target ([9]; [17]). Third, AMD has stated that the NVIDIA 50x tokens-per-watt claim used FP4 with multi-token prediction on Blackwell against FP8 on Hopper at a 1K/1K sequence, and that “each choice favors NVIDIA’s result” ([25]). The independent numbers in the table, which show GB300 NVL72 at roughly 7.6x H100 at 45 tok/s/user and 23x at 97 tok/s/user on DeepSeek R1, are therefore the defensible range.

Why does tokens per watt vary 10x to 100x for the same hardware?

In fact, tokens per watt is a property of a configuration, not a chip. Specifically, five variables dominate.

Latency target (interactivity). Serving fewer concurrent users at higher tokens per second per user leaves more of the GPU idle per token. In the InferenceX H100 data above, moving from 45 to 97 tok/s/user cut tokens per MW from 539,778 to 112,644, a 4.8x drop, while GB300 NVL72 fell 1.6x over the same range ([16]). Cost likewise follows the same curve: B200 serving DeepSeek R1 in FP4 cost about $0.56 per million output tokens at 50 tok/s/user but about $4 at 125 tok/s/user, “2.5x speed for ~7x price” ([26]).

Batch size. Specifically, larger batches amortize weight reads across more tokens. ML.ENERGY reports that increasing batch size cuts energy per token 3x to 5x, and TokenPowerBench similarly found the steepest drop between batch 32 and 256, with a two-to-three-fold spread across batch sizes up to 1,024 ([2]; [27]). Batch size and latency target are two views of one trade: higher batch, higher tokens per watt, but fewer tokens per second per user.

Model size and architecture. TokenPowerBench measured that going from Llama 3 1B to 70B raised energy per token 7.3x while parameter count grew 70x ([27]). Mixture-of-experts models, however, narrow the gap: ML.ENERGY measured Qwen3 30B-A3B (3B active parameters) at 3.56x lower energy per token than dense Qwen3 32B ([2]). Reasoning, meanwhile, swamps all of it: Hugging Face’s AI Energy Score v2 found reasoning models use about 30x more energy per query on average, and the same weights can span 150x to 700x, with DeepSeek-R1-Distill-Llama-70B at 49.53 Wh per 1,000 queries without reasoning and 7,626.53 Wh with it ([28]).

Quantization. TokenPowerBench likewise found FP8 cut Llama 3 405B energy per token about 30 percent versus FP16 ([27]). The effect is conditional, however: ML.ENERGY measured FP8 using a median 30 percent more energy than BF16 at batch 8 to 16 but 11 percent less at batch 65 to 256, because quantization overhead only pays off when the GPU is compute-bound ([2]). In fact, FP4 on Blackwell versus FP8 on Hopper accounts for a large share of the vendor multiples cited above.

Context length. Attention cost grows with the KV cache. TokenPowerBench measured that raising Llama 3 70B context from 2K to 10K tokens roughly tripled energy per token ([27]). NVIDIA’s headline comparisons use 32K input and 8K output sequences because long-context, agentic workloads are where rack-scale NVLink domains show the largest gains ([17]).

Notably, software moves the number without any hardware change. NVIDIA reported B200 cost on GPT-OSS-120B falling from $0.11 to $0.02 per million tokens in two months of releases, and speculative decoding tripled GPT-OSS throughput from 6,000 to 30,000 tokens per second per GPU at 100 tok/s/user ([17]; [29]). Any tokens-per-watt figure therefore needs a date as well as a context. The Inference Paradox covers why these gains raise total power demand rather than lowering it.

Worked example: 1 MW at the utility meter to tokens per dollar

The chain below uses only published inputs; specifically, it assumes an H100 SXM fleet in DGX H100 servers serving DeepSeek-R1-0528 at 45 tok/s/user, 100 percent busy for a 720-hour month, and shows two facility efficiencies. Real utilization is lower, however, and every output scales linearly with it.

Accordingly, three results follow. First, at 100 percent utilization electricity is about 4 percent of rental-plus-electricity cost, consistent with SemiAnalysis’s finding that “colocation rent and electricity cost typically make up less than 20% of TCO” and that a 20 percent tokens-per-MW deficit moves TCO by under 4 percent ([3]). Second, cutting PUE from 1.54 to 1.09 raises tokens per month 41 percent on the same meter, which is the case for liquid cooling in power-constrained sites. Third, the PUE 1.09 column lands at 526,524 tok/s per MW, within 3 percent of InferenceX’s independent 539,778 tok/s per all-in provisioned MW for the same operating point, which consequently validates the chain.

The example uses nameplate server power. Measured average draw during inference is lower, so a site sized on TDP can usually host more GPUs than step 4 implies; that headroom, however, is not modeled here. Colocation rent in step 13 is a wholesale market rate for 250 to 500 kW blocks and excludes GPU capital, so it is not comparable to step 12 ([32]). The EIA price, finally, is the U.S. average industrial rate; the 2025 annual average was 8.62 ¢/kWh and the trailing 12 months through June 2026 averaged 8.87 ¢/kWh, although a specific site’s tariff can sit well outside that range ([30]).

Tokens per dollar and the “tokens per watt per dollar” composite

Tokens per dollar is tokens per watt multiplied by watts per dollar. The second term is where the capital stack enters: GPU depreciation, rack and network cost, facility construction, and financing all show up as dollars per provisioned watt per month, while electricity shows up as dollars per watt-hour. NVIDIA’s comparison table makes the decomposition explicit by listing $/GPU-hour, tokens per second per GPU, tokens per second per MW, and $/million tokens side by side: GB300 NVL72 at $2.65 per GPU-hour against HGX H200 at $1.41 is still 35x cheaper per token because throughput per GPU is 65x higher ([17]).

Operators, accordingly, use tokens per dollar to price capacity and tokens per watt to size it. When a site is power-limited, tokens per watt sets revenue per megawatt regardless of hardware price; when a site has spare power but limited capital, tokens per dollar of hardware dominates instead. AMD’s MI355X positioning shows the second lens: AMD, for example, claims up to 40 percent more tokens per dollar than B200 on Llama 3.1 405B in FP4 using CoreWeave B200 pricing, and Signal65 measured 1.42x to 2.15x B200 tokens per dollar across GPT-OSS-120B, Kimi-K2.6, and Qwen3-Next-80B at market GPU-hour rates, while the same chip trails GB300 NVL72 on tokens per MW in the InferenceX data above ([33]; [34]; [24]). In other words, a chip can win one metric while losing the other.

“Tokens per watt per dollar” is the composite some operators use to rank hardware on both axes at once, dividing tokens per watt by the all-in cost of the watt. It is useful as a screen, although it has three limits. First, it hides the latency target, so a high score can describe a batch configuration no interactive customer would accept. Second, it depends on a $/GPU-hour or $/kW-month input that varies by contract and market: Lambda lists H100 SXM at $3.99 to $6.16 per GPU-hour depending on on-demand versus cluster terms, while InferenceX’s $0.439 per million tokens for the same H100 operating point implies an all-in cost near $1.17 per GPU-hour ([31]; [16]). Third, it moves with software: the B200 5x cost improvement in two months came with no change in watts or dollars per GPU ([17]). Consequently, the composite should be reported with its model, precision, interactivity, sequence length, price basis, and date.

Watts per token: the inverse conversion

“Watts per token” and “watt per token” mean energy per token; the strictly correct unit is joules per token (watt-seconds per token), because a watt is a rate and a token is a count. The conversion is therefore:

[
\text{joules per token} = \frac{1}{\text{tokens per second per watt}} \qquad
\text{Wh per 1,000 tokens} = \frac{\text{joules per token} \times 1{,}000}{3{,}600}
]

Worked conversions from the figures above: H100 at 539,778 tok/s per MW is 0.54 tokens per joule, 1.85 J per token, or 0.51 Wh per 1,000 tokens; GB300 NVL72 at 4,105,187 tok/s per MW is 0.24 J per token or 0.068 Wh per 1,000 tokens; the H200 vendor baseline of 54,000 tok/s per MW is 18.5 J per token or 5.1 Wh per 1,000 tokens ([16]; [17]). Google’s disclosed median of 0.24 Wh per Gemini Apps text prompt equals 864 joules per prompt, which is the whole-prompt figure rather than per token ([35]).

Uptime Institute frames the metric the same way, as “joules per token (or its derivatives such as tokens per watt)” ([36]). When a vendor quotes “watts per token,” therefore, ask for the sequence length, the interactivity target, and whether the watts are chip TDP or utility power.

How is tokens per watt measured? Standards and disclosures

No single standard governs tokens per watt yet. Instead, seven efforts define how it is measured today, each with a different power boundary.

The boundary problem is quantified in Google’s paper: the same Gemini prompt measures 0.10 Wh under an active-chip-only method and 0.24 Wh under the full-stack method, while published Llama 3.1 70B estimates range from about 580 to 3,600 prompts per kWh, a 6x spread driven by methodology rather than hardware ([40]). Microsoft Research reached a similar conclusion, estimating a median 0.34 Wh per query for models over 200B parameters on an H100 node including utilization and PUE, and finding that non-production benchmark estimates overstate energy by 4x to 20x because they omit batching ([42]).

The stakes are national. Lawrence Berkeley National Laboratory estimated U.S. data centers used 176 TWh in 2023, 4.4 percent of national electricity, and projected 325 to 580 TWh by 2028 ([43]); EPRI, similarly, projects 9 to 17 percent of U.S. electricity by 2030 ([44]).

How SAVRN builds for this

SAVRN designs AI factories against facility-level tokens per watt, not chip TDP, because the utility meter is the only boundary that appears on an invoice. Three design choices therefore follow from the numbers above.

The first is cooling. The worked example shows a 41 percent difference in tokens per month between PUE 1.54 and PUE 1.09 on the same 1 MW meter. SAVRN’s water-free AI data center designs use direct-to-chip liquid cooling to run rack-scale systems such as GB300 NVL72 at the low-PUE end of that range, so the megawatt buys tokens rather than fans and chillers.

The second is power sourcing. Because electricity is a small share of token cost at market GPU rates but the binding constraint on how many GPUs a site can host, SAVRN’s self-powered site model prioritizes firm megawatts and short interconnect timelines over the cheapest tariff.

The third is measurement. SAVRN instruments halls at the meter, the PDU, and the rack so that tokens per watt can be reported at each boundary alongside the model, precision, interactivity, and sequence length that produced it. Capacity contracts are written against a benchmarked throughput envelope rather than a GPU count, which is the only way a buyer can hold a supplier to a tokens-per-dollar figure once software and utilization start moving the number.

Chad Harris, Founder, SAVRN