Relion (Intel) and Altus (AMD) run the identical NVIDIA B300/B200 HGX silicon as a DGX — so this isn't a performance trade. The win is density, efficiency, sourcing leverage, and an integrated build/manage wrap that lowers CapEx, OpEx, and time-to-production. Numbers below track your current sizing.
Relion/Altus pack 8×B300 into 4U vs a DGX B300's 10U. Same GPUs, ~40% of the rack space — which cuts racks, PDUs, structured cabling, and data-center floor, the CapEx that never shows on the GPU line item.
DTC liquid cooling runs a ~1.1–1.15 PUE vs ~1.4–1.5 for air. At Blackwell's 1.4 kW/GPU that overhead compounds every hour for the asset's life — power is the dominant OpEx line, and cooling efficiency is where it's won.
Altus (EPYC) and Relion (Xeon 6) let you right-size the CPU to the workload and play vendors against each other on price and lead time — leverage a fixed single-vendor appliance doesn't give you.
Penguin ships the outcome, not a BOM: reference-architecture-validated integration, ClusterWareAI / MemoryAI / ComputeAI, and managed services. Fewer months to first token, lower ops headcount, one throat to choke.
Reasoning & inference are a permanent, high-utilization load — the worst case to rent. At your utilization and cloud rate, here's owned-infra TCO vs paying a hyperscaler per GPU-hour over the horizon.
Three forces are moving frontier compute back inside the walls — and B300 is built for the first of them.
Blackwell Ultra (B300) exists for reasoning. Reasoning models burn compute at generation time — long chains of thought, test-time scaling, agentic loops — moving the cost center from a one-off training run to a permanent inference load. That load is cheapest on iron you own. The 288 GB HBM and FP4 throughput are built to serve it.
Sovereign AI is the fastest-growing driver of on-prem GPU demand: nations and enterprises require models trained and served on infrastructure they own and control, in-country. An on-prem AI factory keeps weights, data, and inference inside your walls — no hyperscaler tenancy, no cross-border data flow, no capacity queue.
Regulated data can't live in shared cloud. Healthcare (HIPAA), finance, and government (FedRAMP High / IL5+, EU AI Act, GDPR residency) need air-gap, residency, and auditability a multi-tenant hyperscaler can't guarantee. On-prem brings the model to the data — compliance as a property of the build.
All three point the same way: sustained, sensitive, owned workloads. That's precisely the profile where buy beats rent and where an integrated partner — hardware + software + managed services — turns a capital project into a running AI factory. See the Buy vs Rent tab for the number.
Build up to 5 racks. Per-phase current, breaker & connector; DTC liquid/air heat split, CFM & loop flow; and a full structured-cabling BOM (DAC/AOC/optic + fiber) from the standard 8-GPU port topology, with slack for slide-out service. Every field recalculates live. Save the BOM to JSON and reload it next session. Typical/reference values — override any field with real numbers.
| Qty | Spares | Order | Speed | Part | Connector | Length | Used by |
|---|
| Qty | Link class | Speed | Media type | Length | Description |
|---|
A B300 costs about the same as a house and can do 15.3 petaFLOPS. When the KV cache outgrows HBM, it does nothing at all — very quickly, while drawing 1.4 kW. This tab answers one question with numbers instead of vibes: are your GPUs actually starving, and if so is CXL memory the cheapest way to feed them? Measure first (DCGM below), then do the arithmetic. If compute is your real constraint, this tab will say so plainly and point you at GPUs instead.
Don't guess whether you're memory-bound. NVIDIA Data Center GPU Manager already knows. Run this against a production serving node under real load — not a benchmark — for a few minutes at 1-second resolution:
Or scrape the same field IDs from dcgm-exporter into Prometheus if you'd rather look at a week than a minute. The field IDs are what matter — the transport doesn't.
| DCGM field | ID | Reading that points here | What it actually means |
|---|---|---|---|
| DCGM_FI_DEV_GPU_UTIL | 203 | > 85% | The one to distrust on its own — it only means a kernel was resident, not that math happened. Read it alongside 1004 rather than instead of it. |
| DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | 1004 | < 35% | The truth. Fraction of cycles the tensor pipes were actually busy. High 203 + low 1004 = the GPU is holding a seat, not working. |
| DCGM_FI_PROF_DRAM_ACTIVE | 1005 | < 30% | Not even HBM bandwidth is saturated. You're not compute-bound or bandwidth-bound — you're waiting on something off-chip. That's the memory wall. |
| DCGM_FI_PROF_DRAM_ACTIVE | 1005 | > 70% w/ low 1004 | A different wall. Bandwidth-bound rather than capacity-bound — added capacity won't move this one. Worth resolving first, then reassessing capacity. |
| DCGM_FI_DEV_FB_USED / FB_TOTAL | 252 / 250 | > 95% | HBM is full. You are evicting and recomputing KV, or rejecting/truncating requests. This is the capacity wall MemoryAI is built for. |
| DCGM_FI_PROF_PCIE_RX_BYTES | 1010 | sustained + bursty | You are already paging KV cache across PCIe. You've built a worse version of CXL by hand. Textbook case. |
| DCGM_FI_DEV_POWER_USAGE | 155 | < 60% of TDP | Under 840 W on a 1,400 W B300 while "busy" suggests the silicon is waiting rather than working — corroborating evidence, not a verdict on its own. |
| DCGM_FI_PROF_SM_OCCUPANCY | 1003 | low + low 1004 | Batches too small to fill the machine — usually because KV headroom caps concurrency. More memory raises the batch ceiling. |
KV cache per token is 2 × layers × KV-heads × head-dim × precision-bytes. Multiply by context and concurrency and compare against the HBM you have left after weights. Everything here is editable — put your real model in.
Most CXL literature is written in the fleet-efficiency register, and for good reason: Microsoft's Pond paper justified pooling with ~25% of Azure DRAM stranded and 50% of VMs never touching 50% of their rented memory. That is a textbook overcommit statistic, and for that use case the analogy is exact.
It transfers poorly to KV cache, and carrying the assumption across tends to under-provision the tier. Overcommit rests on two conditions that do not hold here:
① Demand is anti-correlated. Tenants peak at different times, so peak-of-sum beats sum-of-peaks. But LLM serving follows the same diurnal curve on every replica and a viral hour hits all of them at once — the pool runs dry exactly when you need it.
② There are cold pages to reclaim. Ballooning works because guests hoard idle memory. A KV cache is the hot working set of in-flight requests. Nothing about it is idle. There is nothing to steal.
Size this tier to the working set, not to average utilization — which is what Step 2 does, per replica, deliberately. And note the far tier is a design point, not a failure mode: at 200–500 ns it is NUMA-far, not swap-far. You plan to sit on it permanently.
If two tenants on your iron are commercial enemies, the memory math changes — and the feature that saves you the most memory becomes the one you are obligated to switch off. Set the isolation model in the controls above and Step 2 recomputes.
| Separates… | Mechanism | Enforcement | Reality check |
|---|---|---|---|
| Capacity | CXL MLD / DCD | Hardware — memory controllers, up to 16 logical devices by LD-ID, each to a different host | CXL's uncontested lane. Nothing in NVIDIA's stack pools memory across mutually-untrusting hosts. But it is still mostly pilots at hyperscale as of 2026. |
| Compute | NVIDIA MIG | Hardware — crossbar ports, L2 cache banks, memory controllers and DRAM address buses assigned uniquely per instance | Commonly mislabelled "software/logical" partitioning. It is not. It has shipped in production multi-tenant clouds since A100. |
| Trust | NVIDIA Confidential Computing | Hardware — signing key fused at manufacture, never exposed to software/firmware/host; remote attestation; encrypted VRAM; NVLink encryption across up to 8 Blackwell GPUs | Explicitly a don't-trust-the-operator design. ~2–5% overhead published for H100 inference. |
| Nothing, together | MIG + NVLink | Mutually exclusive | Enabling MIG disables NVLink P2P. You choose hard intra-GPU partitioning or the fabric — never both on the same GPU. Any comparison treating "MIG + NVLink" as one multi-tenant story is comparing something that cannot be built. |
Sharing KV for common prefixes across tenants is the single biggest memory saving in multi-tenant serving — and it is a documented cross-tenant data-leak vector, not an optimisation you get for free. Cache hits return measurably faster, so an attacker probes time-to-first-token, learns which prefixes are resident, and incrementally reconstructs another tenant's prompt — system instructions, PII, medical and financial detail — with no memory access and no special privilege.
This is not theoretical: it carries a vLLM security advisory (GHSA-4qjh-9fv9-r85r) and a research literature (The Early Bird Catches the Leak, InputSnatch), with PrefixWall and SafeKV as proposed mitigations.
Set isolation to hostile in the controls and the model forfeits that saving, because in a real Pepsi-and-Coke deployment you must. The forfeited capacity is shown as its own line — that number is the price of isolation, and it is usually the honest reason to buy more memory rather than a reason not to.
The comparison everyone makes is CXL vs NVMe. That's the wrong axis — NVMe at ~100 µs was never in the inference hot path, it's where cold data goes to retire. The comparison that decides the PO is CXL vs buying more GPUs to reach the same served throughput.
Capacity and idle-GPU capital are the internal arguments. TTFT is the one your customer actually contracts on, and it's the metric the memory tier moves most directly. It's also fully derivable rather than asserted, so here it is derived.
TTFT is set by prefill. On a cache miss you recompute the prompt's KV — roughly 2·P·tokens of FLOPs across the replica. On a cache hit you instead move that KV from wherever it lives, so the tier's bandwidth decides. Which gives the test that actually settles the tier debate: a tier slower than recomputing isn't a cache at all — it's a slower way to arrive at the same answer.
A separate question from whether, and one that lands earlier: specify the memory tier with the cluster, or retrofit it once the workload proves it needs it? The honest answer is that install time is cheaper and later is better-informed, and which matters more depends on how well you already know your workload.
One integration, one validation, one change window. Adding a memory tier to a live cluster means a fresh qualification cycle, a maintenance window against a running SLA, and re-tuning a serving stack that people now depend on. Doing it inside the original bring-up is materially less disruptive.
Supply is the real argument. DDR5 is tight across three suppliers and CXL capacity is allocated. A tier specified in the original order is a tier you have; a tier you decide you want in nine months is a lead-time conversation.
You size power, cooling and rack space once. Retrofitting means finding kW and U in a rack that was planned without them — often the binding constraint, and the one nobody costs in advance.
You are buying against a forecast. Every number on this page is workload-specific, and pre-production traffic estimates are routinely wrong in both directions. Ninety days of real DCGM data is worth more than any sizing exercise, including this one.
Capital sits idle if you are wrong. Unused CXL is expensive shelf-space in a market where the same spend could have been GPUs — and if you turn out to be compute-bound, that is exactly the wrong asset.
The technology is moving. CXL 3.x features and controller generations are still landing. A tier bought a year later is a better tier, at a price that may well have moved.
Buy the headroom, defer the capacity. The costs that are painful to retrofit are structural: PCIe lanes, rack U, power envelope, cooling, and the validation cycle. The cost that is easy to defer is DRAM itself. Specify a platform and a rack design that can accept the memory tier — then populate it when your own telemetry says to.
Concretely: reserve the U and the kW at design time, confirm the platform's CXL AIC slot count with your Penguin rep, and treat the modules as a follow-on order gated on 90 days of production DCGM. That converts an expensive forecasting bet into a cheap option.
Two cases override this. If your workload is already in production elsewhere and you have the telemetry, you are not forecasting — specify it day one and take the integration saving. If you are on a hard SLA from launch with long context or high concurrency, the retrofit window may never be politically available, and day one is the only realistic answer.
Tensor-active (1004) already above ~60% means the silicon is working and there is little idle capacity for memory to unlock. GPUs are the better dollar today. Worth rechecking as you scale concurrency — capacity is commonly the next constraint after compute. This is the most common misdiagnosis, because GPU_UTIL 203 reads ~99% in both cases.
Offline batch inference, classification, embeddings, short prompts, low concurrency — cold data can comfortably afford a 100 µs fetch. NVMe is the right tier here, at roughly 150× less per GB. Take the cheaper win now; the calculus changes if the workload turns interactive or agentic.
CXL rides DDR5, and 2026 DRAM is in a hard shortage — DDR5 roughly tripled-to-quadrupled from mid-2025, with three suppliers and no new entrants. NAND has a broader base and trended down. Dedicated CXL supply contracts carry real allocation risk, so the case is strongest where GPU spend is your dominant cost line. A current quote is worth having before you decide either way.
High concurrency, long context (32K+), agentic multi-step loops, prefix-cache reuse across sessions — anywhere KV working set exceeds HBM and GPUs visibly stall. CXL costs materially less per GB than HBM while sitting ~200–500× closer than SSD. Modest memory spend unlocks GPU capacity that would otherwise cost far more to add in GPUs.
Every figure on the MemoryAI tab is a guess until it's your telemetry. NVIDIA Data Center GPU Manager already collects everything the decision needs. Export it, drop the file here, and the model re-runs on measured data. The file never leaves this browser — parsing is local JavaScript, there is no upload, no server, no telemetry. Close the tab and it's gone.
All four are read-only and safe to run on a production node. Capture under real serving load — a quiet node or a synthetic benchmark will tell you a comfortable lie. Aim for at least 5 minutes; an hour is better; a week through Prometheus is best.
dcgm-exporter easiest · recommendedIf you run the DCGM exporter (it ships with NVIDIA GPU Operator on every Kubernetes GPU cluster), it's already serving exactly what we need on port 9400. One curl, one file:
On Kubernetes, port-forward first: kubectl -n gpu-operator port-forward svc/nvidia-dcgm-exporter 9400:9400
Caveat: this is an instantaneous snapshot — one moment in time. Good for a quick read, weak for a purchase decision. Take several minutes apart and upload the largest, or use method 2. If your exporter doesn't publish the profiling fields, set DCGM_EXPORTER_COLLECTORS to a counters file that includes DCGM_FI_PROF_PIPE_TENSOR_ACTIVE and DCGM_FI_PROF_DRAM_ACTIVE — they are commonly commented out of the default config, which is the single most frequent reason this tab reports "tensor field missing."
If dcgm-exporter feeds Prometheus, ask for a real time window. This gives averages and percentiles over days rather than one instant — which is what actually justifies capital:
On macOS use date -u -v-7d instead of date -u -d '7 days ago'. A week at 5-minute steps is a few MB — well within what this page parses. If the response is truncated, narrow the window or widen the step.
dcgmi dmon — no Prometheus required bare metalStraight from the DCGM CLI on the node. This is the one to use on a standalone box or an air-gapped site. Field IDs are explicit, so nothing depends on exporter config:
-d 1000 = sample every 1000 ms, -c 300 = 300 samples ≈ 5 minutes. Raise -c for a longer window (-c 3600 ≈ 1 hour). Drop -c entirely and it runs until you Ctrl-C.
Requires nv-hostengine running (sudo systemctl start nvidia-dcgm). Profiling fields 1001–1010 need DCGM 2.0+ and are unavailable inside some containers unless the DCGM socket is mounted — if those columns come back N/A, run it on the host rather than in the pod.
Any CSV with a header row works, as long as column names contain either the DCGM field name (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) or the dmon short name (TENSO). Grafana's panel → Inspect → Data → Download CSV produces this directly. Extra columns are ignored.
Minimum useful set: tensor-active and DRAM-active. Everything else sharpens the picture but the verdict can be reached without it.
| DCGM field | ID | dmon | Unit | What this page does with it |
|---|---|---|---|---|
| DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | 1004 | TENSO | ratio 0–1 | Required. Becomes "tensor-active now" — the idle-capital calculation rests entirely on this. |
| DCGM_FI_PROF_DRAM_ACTIVE | 1005 | DRAMA | ratio 0–1 | Required. Separates a capacity wall (CXL helps) from a bandwidth wall (CXL does not). |
| DCGM_FI_DEV_FB_TOTAL | 250 | FBTTL | MiB | Sets HBM per GPU automatically — no need to tell us what you own. |
| DCGM_FI_DEV_FB_USED | 252 | FBUSD | MiB | Peak framebuffer occupancy. >95% is the capacity wall showing itself. |
| DCGM_FI_DEV_GPU_UTIL | 203 | GPUTL | % | Only used to expose the gap against 1004 — the "busy but not working" delta. |
| DCGM_FI_PROF_SM_OCCUPANCY | 1003 | SMOCC | ratio 0–1 | Low occupancy + low tensor = batch ceiling, usually KV-headroom imposed. |
| DCGM_FI_PROF_PCIE_RX_BYTES | 1010 | PCIRX | bytes/s | Sustained inbound = you're already paging KV over PCIe by hand. |
| DCGM_FI_DEV_POWER_USAGE | 155 | POWER | W | Draw far under TDP while "busy" corroborates idle silicon. |
GPU count is inferred from distinct entity/GPU identifiers in the file. Percentiles are computed across all samples and all GPUs; the per-GPU table below the summary is where you'll spot a single starving rank dragging an otherwise healthy fleet.
Six gates, in order. Most of them resolve somewhere other than a memory purchase — which is the point, because the fastest way to trust a recommendation is to see what would have overturned it. Each outcome carries the condition that would bring you back. Your current numbers from the MemoryAI tab light the path you're actually on; change them there or upload real telemetry on the DCGM tab and this redraws.
These are routinely compared as if they compete. They don't — each separates a different thing, and the third one is a trap that gets drawn on architecture diagrams constantly.
| Choose… | When | Because |
|---|---|---|
| GPUs first | tensor-active (1004) > 60% | Silicon is working, so little idle capacity is available for memory to unlock. Recheck as concurrency grows. Most common misdiagnosis — GPU_UTIL reads ~99% either way. |
| Bandwidth first | DRAM-active (1005) > 70% with tensor low | Bandwidth-bound rather than capacity-bound. Resolve that, then reassess — capacity is often the next wall. |
| Headroom today | KV working set fits HBM headroom | Nothing to solve right now. Step 2 gives the concurrency at which you would cross — check against it when traffic changes. |
| NVMe tier | batch, short context, latency-tolerant | Cold data affords 100 µs and NAND is ~150× less per GB. Reopens if the work turns interactive or agentic. |
| MemoryAI (CXL) | KV spills, GPUs stall, work is interactive | ~200–500 ns keeps the GPU fed where NVMe cannot, and costs far less than the GPUs it frees. |
| MemoryAI + isolation | …and tenants are hostile | Prefix dedup must be off — it's a TTFT side channel. Budget the forfeited capacity explicitly. |
| Pilot first | return under ~2× | Thin returns do not yet justify allocation risk in a short DRAM market. A current quote often moves this. |
Every figure this tool leans on, with its source. Vendor numbers are labelled as vendor numbers and separated from what the tool derives itself — so when you quote something in front of a customer, you know exactly which kind of claim you're making and who stands behind it.
The row-by-row breakdown an installer works from, and the asset tagging schema that has to exist before the first rack lands. Derived from whatever the Rack Planner's Advanced mode is currently set to — change the target there and this follows.
| Row | Racks | Identifiers | Contents | GPUs | Load | Notes |
|---|
| Rack | Row | SU | Type | Devices | Ports | Load |
|---|
Microsoft's Cloud Adoption Framework and AWS's tagging guidance both assume a hyperscaler owns the substrate. The provider asserts jurisdiction, vets the staff, controls physical custody and attests to it. A sovereign on-prem AI factory inverts that: you are the substrate, so the guarantees a cloud provider would have made on your behalf have to become properties you carry yourself. That is the whole adaptation — the four commercial categories below are lifted straight from CAF and AWS, and the fifth is the one their frameworks have no reason to include.