I re-ran every benchmark on this blog on bare metal — here's what survived

Every benchmark post on this blog ends with the same honesty clause: this ran on a shared 4-core box — trust the ratios over the absolute milliseconds. I wrote that sentence so many times it became furniture. But an asterisk you never test is just decoration. So when a dedicated machine became available, I did the only honest thing you can do with your own disclaimers: re-ran everything and checked.

This post is the audit. Same harnesses, same MIN-of-trials methodology, full publishable scale — but on a bare-metal host with SQL Server pinned to isolated, clock-clamped cores. Six experiments, roughly 130 configurations, every headline claim re-tested. The score: every ordering survived. One magnitude didn’t. And the two questions the noisy box couldn’t answer now have answers, one of them with an execution plan as the receipt.

The quiet box

The lab is a bare-metal Ryzen 9 5900HX (8 cores / 16 threads) running the SQL Server 2025 container in a carved-out corner of the machine:

CPUs 8–15 are evacuated: host processes, kernel workqueues, and IRQs are confined to CPUs 0–7; the SQL container is the only thing allowed on the isolated cores (cpuset + a dedicated cgroup slice).
Clocks clamped to the 3.3 GHz base frequency on the SQL cores. That sacrifices peak speed for repeatability — no boost-clock lottery deciding which trial wins.
Swap off, memory hard-capped, performance governor.

The receipts, before any benchmark was trusted: a SELECT 1 round-trip floors at 182 µs (median 212) versus ~350 µs on the shared box — 40% quieter with a far tighter tail; two identical load-test runs land within 1.2% throughput with identical p50/p99; and saturating the host cores with stress-ng leaves the SQL cores ≥99.6% idle. Inside the harnesses the same story: the no-op baseline dropped from ~43 to ~30 µs, and trial-to-trial spreads that were ±40–170 µs on the shared box collapsed to ±1–4 µs.

One thing this deliberately is not: a comparison of absolute speed. Clamped clocks mean the lab’s absolutes aren’t comparable to the old box’s — or to yours. What transfers is what always transfers: orderings, ratios, verdicts.

The scoreboard

Post	Published claim	Bare metal	Verdict
IIF vs CASE	it’s a tie, by construction	2-way forms land 0.006 ns/row apart	✅ confirmed
ISNULL vs COALESCE	ISNULL ~2× per evaluation	1.4–2.1× depending on NULL rate; 4-arg chain 1.72×	✅ confirmed, range honest-ified
+ vs CONCAT	`+` costs 1.8× at 8 pieces	1.53×; 2-piece “tie” resolves to a mild CONCAT win	✅ confirmed, magnitude trimmed
SET vs SELECT vs DECLARE	statement count is the lever	identical 5-method ordering; separate ~1.3× combined	✅ confirmed
STRING_AGG vs XML PATH	correct XML swings 3× slower ↔ 2.7× faster	swings 2.8× slower ↔ 3.4× faster; STRING_AGG flat	✅ confirmed, swing wider
Temp table vs table variable	inversion at scale; RECOMPILE worst	populate 5.1×, join 4.7×, RECOMPILE still worst	✅ confirmed, stronger
DROP temp tables	cacheability is a 15× lever vs the DROP debate	14.7×	✅ confirmed, eerily precisely
UTF-8 collations	storage table; seek-vs-scan trap; ASCII inserts ~86% of NVARCHAR	storage within 0.2%; trap ~4,000×; ASCII inserts 183%	⚠️ one claim flipped

Two patterns worth noticing. Ratios within a family of methods reproduced remarkably well — the DROP post’s 15× lever came back at 14.7× on different silicon, which is the kind of agreement you don’t dare hope for. And absolutes did exactly what absolutes do: most configs ran 20–35% faster on the quiet box, while the branch-heavy scalar expressions ran slower — clamped 3.3 GHz beats a noisy shared core on most things and loses on pure straight-line CPU. If you ever needed a demonstration that absolute microseconds don’t travel between hosts, there it is.

The one that flipped

The UTF-8 post reported ASCII inserts into a _UTF8 column at ~86% of NVARCHAR’s time — fewer bytes, less log, faster write — while hedging that precise write numbers needed a quiet host. On the quiet host, the claim inverts: ASCII inserts into the UTF-8 column cost ~1.8× the NVARCHAR insert (2.12 s vs 1.16 s per 200k rows, all six trials tight; legacy VARCHAR flips identically).

The mechanism makes it interesting rather than embarrassing. The benchmark’s source data is NVARCHAR — like most real pipelines, because that’s what drivers send — so inserting into any VARCHAR target pays a per-row UTF-16→UTF-8 conversion, while the NVARCHAR target is a straight copy. On the old box the write path was IO/log-bound: halved byte volume won, conversion hid in the wait time. On bare-metal NVMe the writes are fast enough that the conversion CPU is the bottleneck, and the byte saving can’t pay for it. Neither measurement is wrong; they’re two ends of a spectrum, and “UTF-8 inserts are cheaper” turns out to be an IO-bound-regime fact, not a general one. That’s a sharper and more useful truth than the one it replaces — but it’s a flip, and it’s corrected in the original post.

The two questions the noisy box couldn’t answer

Did the CJK UTF-8 scan really go parallel? The strangest result in the UTF-8 study was the one combination that scanned faster despite being 1.5× bigger: the CJK table under UTF-8. I hypothesized the bigger scan had crossed the optimizer’s parallelism cost threshold, and this time the harness captures the cached plans as receipts. It had: the CJK/UTF-8 retrieve is the only combination of seven that compiles parallel — DOP 8, ~74 ms of worker time compressed into ~12 ms of wall clock, while every sibling runs serial with worker ≈ elapsed. At the smaller smoke scale, it stays serial and the anomaly vanishes, which is exactly what a cost-threshold effect should do. Hypothesis, prediction, plan XML: closed.

Is there a per-length effect in variable assignment? The variable-declaration post had to punt: any per-length effect was smaller than the shared box’s noise floor. With ±1–4 µs spreads it’s answerable. Declaring a string variable is length-independent — flat within ±1 µs from 1 to 4,000 characters; a real null result now, not a shrug. Assigning a literal is not: cost is flat to ~100 characters, then roughly doubles at 1,000. And at 4,000 characters something type-dependent and genuinely weird happens — nvarchar assignment cost collapses below the 1-character case while varchar quintuples, rock-steady across trials, on both hosts. That one is now an open investigation instead of a hand-wave. Quiet hardware doesn’t just confirm answers; it uncovers better questions.

Did benchmarking on a noisy box actually work?

This was the real question of the audit. The whole methodology post is a set of defences against contention — duration-based loops, interleaved shuffled passes, MIN across trials, a no-op baseline — and the bare-metal re-run is the first time I could score them.

The score is good: across ~130 configurations, not a single ordering inverted. Every “A is faster than B” survived. MIN-of-trials on a contended host recovered the true ranking every time, which is what the additive-noise argument promised it would. Magnitudes were softer — ratios drifted by 10–25% (1.8× became 1.53×; 2.5× became 1.9×), and effects near the noise floor were invisible — but the shared box never invented a result. Its published numbers failed in exactly the two ways the posts themselves flagged as risks: effects below the noise floor (declared unmeasurable at the time, correctly), and one claim whose truth depended on the IO regime rather than on noise at all.

So the honest hierarchy of trust for cross-host benchmark claims, now with evidence:

Orderings travel. If it was faster there, it’ll very likely be faster here.
Ratios mostly travel — treat any published ratio as ±25% until re-measured.
Absolutes don’t travel at all. Not between hosts, not between clock policies.
Regime-dependent claims are the dangerous ones — anything whose cause is “IO is slow” or “CPU is contended” can flip outright when the bottleneck moves. Noise defences can’t save you there; only naming the regime can.

What changed on the blog

Every re-verified post now carries a dated “Re-verified on bare metal” section with its bare-metal numbers, the UTF-8 post’s write-path claim is corrected, and both result sets — shared box and lab, plus the captured host environment and execution plans — are in the public repo: github.com/dkultasev/tsql-benchmarks. If you want a claim about your workload measured this carefully, get in touch.