A drop-in inference runtime, purpose-optimized per model architecture and per accelerator. Same weights, same API, same answers. Measured on your hardware before you commit to anything.
Decode throughput on identical hardware, in tokens per second: Envex Runtime 335. The leading open-source engine scores 207 tuned and 112 at stock defaults, and a second open-source engine scores 162. A major vendor runtime would not load the model. Full table below.
Kimi K2.6, 1T MoE · 2-node GB200 · 10K prompt, 512 out.
Inference has overtaken training as the dominant line in AI compute spend, and it compounds with usage instead of tapering off. Yet nearly every organization runs it on inference engines written to be general: correct on hundreds of models, tuned for none of them.
of a frontier model's achievable throughput is what a stock open-source engine actually delivers on a new architecture. Measured, same hardware, same weights.
annual cost of one 8-GPU H100 node at $2.50 per GPU-hour reserved, below the $3.44 market average. A mid-size fleet is 50 to 200 nodes, and the waste scales linearly.
the GPUs bought to hit a latency target that better software would have hit on the hardware already racked.
When an endpoint is too slow or too expensive, the reflex is to add GPUs, because the software is treated as a fixed input. It is not. It is the single largest untapped variable in the AI cost equation, and the one nobody owns.
Agentic and coding workloads are decode-bound: long contexts, long outputs, thousands of sequential steps. Tokens per second is the wall-clock time of the agent. Slow inference does not just cost more, it ships a worse product.
The leading open-weight models now sit close enough to closed frontier models that self-hosting is a rational decision rather than a compromise. The volume is moving onto customer-owned hardware.
Multi-head latent attention, sparse attention, mixture-of-experts routing with hundreds of experts, hybrid linear-attention state, and new low-bit numeric formats. Each one breaks assumptions baked into general-purpose kernels.
A new architecture lands with a correctness path first. The fast path arrives months later, if it arrives. During that window the model runs at a third to two thirds of what the silicon can do.
GPU supply is tight and boards are asking about utilization. "Serve the same traffic on a third fewer GPUs" is now a finance conversation, not an engineering nicety.
It does not close once and stay closed. It reopens with every frontier release, every new numeric format, and every new accelerator generation. A general-purpose engine cannot prioritize your model over the other four hundred.
Same OpenAI-compatible API, same weights, same outputs under a quality gate we prove to you. What changes is how many tokens each GPU produces.
services: inference:- image: your-current-inference-server:latest+ image: registry.envex.ai/runtime:2026.08 volumes: - /models:/models ports: - "8000:8000"
1.6x decode tokens per second on the fleet already racked. Serve more traffic without a purchase order, or serve today's traffic on a third fewer GPUs.
Models the vendor documentation says need newer, larger GPUs, running at full context on the previous generation. A 120B model at its full 131K window on one GPU instead of eight.
4-bit key-value cache compression lifts the concurrent-session ceiling by a third on identical memory. Density is revenue for anyone reselling tokens.
Not a hosted API. We never touch your weights, your prompts, or your traffic. Not a model compressor that quietly trades accuracy for speed. Not a consulting shop: the deliverable is licensed software with a version number and a rollback path.
Decode throughput, tokens per second, single stream. Higher is better.
| Runtime | Decode tok/s | Relative |
|---|---|---|
|
Envex Runtime
4-bit spec decode plus fabric all-reduce
|
335 | 1.00x |
|
Leading open-source engine
engine A
best native configuration, including its own speculative decoding
|
207 | 0.62x |
|
Second major open-source engine
best native configuration
|
162 | 0.48x |
|
The same leading engine
engine A
stock deployment, no speculative decoding
|
112 | 0.33x |
|
A major vendor's proprietary runtime
would not load this model, architecture not registered
|
n/a | n/a |
engine A Rows two and four are the same engine. One is expertly tuned, the other is at defaults. Tuning alone is worth 1.85x before Envex does anything. Most fleets run the fourth row.
over the best open-source engine at its own best configuration
over the second major open-source engine at its own best configuration
over a stock deployment, which is what most fleets actually run
measurement drift across the window, bracketed by open and close anchors
A repeatable process applied to whatever ships next, every row measured under the same methodology.
| Model | Hardware | Starting point | With Envex | Gain | Note |
|---|---|---|---|---|---|
| Kimi K2.6, 1T MoE | 2-node GB200 | 112 tok/s stock | 335 tok/s | 3.0x | same window |
| Kimi K2.6, 1T MoE | 2 nodes of 8x H100 | 46 tok/s stock | 100 tok/s | 2.2x | full 256K context |
| Kimi K3 | 4 nodes, 16 GB200 | 137 tok/s clean baseline | 160 tok/s | +17% | ahead of the second major open-source engine's best, 156 |
| GLM-5.2 | 2-node GB200 | 130 tok/s tuned stock | 188 tok/s | +44% | quality gate 12/12 |
| GPT-OSS-120B | 4x H100 | 230 tok/s vendor build | 276 tok/s | +20% | |
| GPT-OSS-120B | 1x H100 | will not run at all | 124 tok/s at 131K context | 8 GPUs to 1 | first of its kind |
Each model was brought up, tuned, quality-gated and promoted while the open-source ecosystem was still landing basic correctness patches. Speed of bring-up is itself the product.
Nothing is promoted on speed alone. Coding benchmarks, long-context needle retrieval, semantic-regression harnesses and stopping-behaviour checks all run before any configuration ships. Faster output that is subtly worse output is a regression, and we treat it as one.
The key-value cache, not the weights, is what caps how many concurrent sessions a GPU can hold. Compress it without hurting quality and every GPU serves more users at the same latency.
more context admitted on identical memory. 59K to 80K tokens on the same 8x H100 node, with no measured quality loss.
faster than the naive implementation. A custom fused decode kernel reads 4-bit packed cache directly instead of decompressing the whole pool every step.
coding-benchmark pass rate under compression, inside the model's published uncompressed band.
A 120B mixture-of-experts model at its full 131K context on a single H100. 61 GB of weights plus 2 GB of compressed cache, with 15 GB spare. Five upstream expert-routing backends each failed differently at single-GPU, so we wrote a new MXFP4-native fused kernel that keeps weights packed end to end. The reference deployment for this model is eight GPUs.
No single trick gets you 2x. The gain comes from stacking independent wins that each survive their own A/B, and from knowing, per architecture, which ones stack and which cancel.
MXFP4-native fused expert-routing kernels. Packed sub-byte key-value decode kernels. Low-bit matrix paths native to current-generation silicon. Shape allow-lists that stop the biggest projections from silently falling back to a generic library. This is where the largest single wins live.
Tree-structured drafting on latent attention, not supported upstream, built by us. Draft-model training and evaluation in house. Acceptance-rate tuning per workload shape, because the right draft depth for a chat turn is the wrong one for a coding agent.
NVFP4, MXFP4 and INT4 target paths. Rotated 4-bit key-value cache. Per-layer numeric policy. All of it paired with quality gates strong enough to catch the regressions that throughput benchmarks are blind to.
Half of per-token GPU time at low concurrency can be cross-node all-reduce. We fuse it onto the fastest available interconnect path, move draft-model collectives onto the same fabric, and search the tensor, pipeline and expert parallel layout instead of accepting a default.
The customer receives a pinned container image, a launch manifest with every load-bearing setting recorded, the measured benchmark matrix, the quality-gate result, and a rollback path. Reproducible by digest, not a wiki page of tips.
The kernels can be copied eventually. The discipline that tells you which kernel is worth writing, and stops you shipping a mirage, is what compounds.
Every claim is a pre-registered hypothesis with its falsifier written down before the run. If no result could change our mind, we do not run it.
Minimum detectable effect fixed at 5%. Twelve repeats, warmups discarded, bootstrap confidence intervals. Anything smaller is noise, and gets called noise.
Same-window A/B only. Open and close anchors bracket thermal and clock drift. A window that drifts more than 3% is void, not normalized.
Cross-engine comparisons are gated on byte-identical prompts, greedy decoding, and verified-equal served token counts. Fail a gate and the row is excluded, not caveated.
Nothing promotes on speed alone. Quality harness, long-context retrieval and stopping behaviour run first. We have rejected our own work: a draft model we trained ourselves lost to the public one, so we shipped the public one.
Over 2,400 documented experiment records, dead ends included. Knowing which forty optimizations do not work on a given architecture is worth as much as the six that do, and it is why the next bring-up is faster than the last.
Merged contributions in the open-source serving and speculative-decoding runtimes this industry runs on, including native support for a speculative-decoding path now upstream. We upstream the non-differentiating fixes and keep the differentiated kernels proprietary.
The arithmetic turns on one number: tokens per second per GPU. Raise it by half and a third of the fleet stops being necessary.
GPUs freed per 1,000 running today
GPU cost removed per GPU-hour today
$2.50 blended reserved GPU-hour · 1.5x throughput gain, the low end of what we measure · 24/7 utilization.
Deliberately low: tracked reserved listings for this GPU class average $3.44 per GPU-hour and on-demand averages $3.54, so $2.50 understates the saving rather than inflating it.
This is GPU spend avoided, stated before the cost of the Envex licence. The benchmark runs first, so the gain is measured on your own hardware before there is anything to agree.
One 8-GPU H100 node costs about $175K per year. At 1.5x it delivers the same work for $117K, which is $58K per year of GPU spend avoided on a single node.
They compete on price per token, so throughput per GPU is directly their margin. Largest fleets, shortest technical sales cycle, and they resell the gain to their own customers.
Banks, health systems, defence, national AI programs. They must self-host open weights, because a hosted API is legally unavailable to them. Air-gapped installs supported. No telemetry required, no weights leave the environment, no dependency on an Envex-hosted service at runtime.
Coding agents, document processing, long-context analysis. Decode-bound by nature, so latency is felt directly as product quality.
| Runs on your hardware | Kernel depth on new architectures | Multi-vendor | Whose interest it serves | |
|---|---|---|---|---|
| Open-source inference engines | Yes | General purpose. Correctness first, fast path months later. | Partial | The community. No SLA, no accountability for your model. |
| GPU vendor runtimes | Yes | Deep, but vendor-reference models first. | Single vendor | Selling more of that vendor's GPUs. Efficiency is not the goal. |
| Hosted inference providers | No, theirs | Deep, and entirely internal | No | Themselves. They capture the efficiency gain, you get a bill. |
| Custom-silicon companies | No, their chips | Not applicable, different substrate | No | Selling their hardware. Requires abandoning your fleet. |
| Envex AI | Yes | Purpose-built per architecture | NVIDIA today. AMD by design, bring-up underway. | Yours. You keep every dollar of the saving except our fee. |
The organizations with the most to save own their GPUs and cannot use a hosted API. Their only options today are free-and-generic or hardware-vendor-aligned. There is no independent, multi-vendor, kernel-deep runtime they can license.
Every engagement opens with a measured, same-window A/B on your own hardware, with your model and your workload, before you commit to anything. Two to four weeks. If we do not beat your current stack, we say so and you keep the report.
Or write directly: [email protected]
We have your details and will be in touch to scope the benchmark.