For anyone choosing an AI platform — and anyone using one every day — this report answers one question: how much does enterprise governance cost in latency? We measured every request through Brutor’s full pipeline (auth, RBAC, governance, cost tracking, quota enforcement, guardrails, semantic cache, and routing) against the same models called directly. The answer is none of the speed your users feel — and that means security, control, and observability don’t have to be a trade-off you negotiate with your engineering team every quarter.
We tested the speed of the Brutor AI Gateway and compared it to competitive solutions. Here are the highlights:
Now let’s look at the benchmark results in detail.
The headline numbers, with the path that produced them.
Median (P50) latency at concurrency = 1 — the typical user experience. Each row tests the same upstream model both directly (client → provider) and through Brutor (client → Brutor → provider), so the delta is entirely the proxy’s contribution.
| Path | Direct (ms) | Via Brutor (ms) | Overhead |
|---|---|---|---|
| OpenAI (translation) | 2,437 | 2,372 | −64 ms |
| Anthropic (OpenAI-translated) | 1,438 | 1,438 | +0.5 ms |
| Anthropic (native pipeline) | 1,438 | 1,471 | +33 ms |
| Claude Code (native + beta headers) | 1,438 | 1,491 | +53 ms |
The OpenAI median is negative because Brutor’s pooled HTTP/2 connections are warmer than the benchmark client’s per-call TLS handshake. We don’t lead with that number; the headline read is “the proxy adds no detectable cost at median” — that’s what’s reproducible across runs.
What that latency buys you.
The ~0–53 ms above is the combined cost of everything below. Most competing AI gateways charge more latency for fewer features.
JWT decode & API-key lookup
Bearer JWT or service API key validated, scope + expiry checked, tenant context resolved.
Resource group access
Workspace + ancestor-chain governance loaded; per-group model availability gate enforced.
Budget & rate limits
Per-tenant budget cap, per-model RPM/TPM, daily/monthly token ceilings — all atomic.
Behavioural rules
Temperature ceiling, context-window cap, banned-tool filter, mandatory system-prompt fragments.
PII & pattern scan
Bidirectional content-policy scan on prompt and response. Detect-only or block, per group.
5 strategies, configurable
Weighted, least-busy, latency-based, cost-based, usage-based — picked per workspace.
Wire-format normalization
Anthropic / Bedrock / Groq SSE → OpenAI-compatible chunks, transparently for the client.
Usage tracking
Token counting, cost calculation, audit row insert via background writer — never blocks the request.
Logs + traces + metrics
Structured proxy log, OpenTelemetry span, Prometheus counters. Every request, every time.
Per-mode policy
Audio voice/format whitelist, image size whitelist, video duration cap, embedding dim guard.
Tool + agent gating
Tool whitelist, approval workflow trigger, agent-to-agent delegation chain depth limits.
Vector lookup, infinite hit
Two-layer cache (exact via Redis SHA-256 + semantic via Qdrant similarity). On hit: skip the upstream call entirely. Tenant-scoped, with per-group time-bucketing and exclusion rules. The dashboard tracks Value-per-Token — cost saved per dollar spent on tokens. Real-world tenants typically see 1.3× to 1.8× VPT within weeks.
Why proxy CPU isn’t the bottleneck.
The end-to-end numbers above are dominated by network, TLS, and the provider’s own inference. To isolate Brutor’s own work, we run Criterion microbenchmarks against in-memory fixtures. Total proxy-internal CPU per chat request: under 5 ms.
How does Brutor compare to competition?
We could not do an honest comparison of Brutor against another AI gateway with the same feature set — simply because there isn’t another gateway with this combination of governance depth and Rust-native speed. We compared our gateway to leading competitors on the market who offer very different approaches to governance and have different architectures and feature sets. Here is what we concluded:
- Generic API gateways — built in C or Go. They promise single-digit-millisecond overhead and deliver it, because they don’t do anything AI-specific: no model routing, no guardrails, no semantic cache, no per-token cost tracking, no MCP or A2A awareness. If you adopt one for your AI traffic you still have to build the entire governance layer above it. Brutor charges 0–53 ms more than that minimal proxy floor — and ships every one of those capabilities production-ready.
- Python-based AI gateways — built in Python on FastAPI. They have AI-specific features, but the runtime tax shows: typical end-to-end overhead lands in the 100–500 ms range. That’s the kind of latency users feel; it’s the kind of latency that quietly inflates inference bills on streaming responses; it’s the kind of latency that makes engineering teams quietly route around the gateway for “real-time” use cases — which is exactly when governance matters most.
- Node-based and edge-worker AI gateways — built on Node.js or Cloudflare Workers. They narrow the latency gap, but the feature set narrows with them: logging, simple caching, basic fallbacks. Anything resembling enterprise governance still has to be built above.
Brutor AI Gateway offers a unique advantage: the full enterprise feature set at generic-API-gateway-class latency — a category of its own. The Rust runtime, async I/O, pooled HTTP/2 connections, and DashMap-backed concurrent state are why — and the consequence is that you don’t have to choose. You don’t trade governance for speed, you don’t trade speed for control, and you don’t have to defend either choice to the team that asks why their requests got slower the day audit logging turned on.
Run the benchmark yourself.
No vendor benchmark is worth anything if you can’t reproduce it. The same tool we used to generate these numbers ships with the proxy — reproduce it in under 5 minutes:
export BENCHMARK_OPENAI_KEY=sk-…
export BENCHMARK_ANTHROPIC_KEY=sk-ant-…
# 2. Point at your running proxy
export BENCHMARK_PROXY_URL=http://localhost:8100
export BENCHMARK_PROXY_API_KEY=sk_brutor_…
# 3. Run
cargo run –release –bin benchmark
Raw JSON + summary markdown drop into benchmark-results/ with timestamps. Every run is checked into the repo for trend analysis.
What these numbers mean for you.
Our benchmark testing and the resulting numbers address one of the main concerns IT managers have when looking into adopting an AI governance solution — the worry that putting governance, cost control, and guardrails between users and the model will slow things down. The numbers say it doesn’t, in any way users feel.
That changes the conversation. Governance stops being a tax that engineering teams quietly route around to ship something that feels fast. Security and finance stop having to choose between visibility and developer happiness. And your employees get on with making the most of AI — with the controls, audit trails, and cost guardrails your organization needs already running underneath, every request, every time.
If you’re evaluating an AI platform, this is one of the bars to measure against. Explore Brutor AI Platform and all its features 
* Pass-through reliability: 100.0%. Zero proxy-introduced errors across 480 requests in the May 2026 run. The only failures observed were upstream HTTP 429s from Anthropic at concurrency ≥ 10 — those occur at identical rates on direct calls, and Brutor faithfully forwards them.
