Five gateway integration mistakes we made so you don't have to
2026-11-13
Five gateway integration mistakes we made so you don't have to
A proxy gateway looks like a thin wrapper until you meet your first production
request that is slow, duplicated, or billed wrong. These are the failure modes
that actually bit us, in the order they usually appear.
1. Metering from the response body instead of upstream usage
Counting tokens as len(body)/4 is not metering, it is astrology. It drifts on
multilingual content, it ignores the system prompt your wrapper injects, and it
quietly undercounts on code. The v0.4 metering reads the upstream usage field
(JSON and the final SSE chunk) and flags fallbacks as estimated — the flag is
the point, because an invoice nobody can audit is worse than no invoice. Read
the metering section of the docs before you copy any
token-budget code from the internet.
2. Charging on replay
Network retries are a feature of the internet. If a retry re-runs the upstream
call and re-bills it, a single flaky minute can double a customer's bill.
Idempotency via X-Request-Id is checked before the upstream forward, not
after, so a replayed request reuses the first result and the first ledger entry.
This is also why the video endpoints check replay before forwarding — task
creation costs real money upstream, and the bill must follow the first call, not
the unluckiest one.
3. Betting on streaming you do not have
"Streaming supported" with a full-body buffer is a lie clients will punish in
production: token-by-token UIs stall, and proxies that assume chunk pacing
misjudge timeouts. If you buffer, say so in the docs and let clients opt into
what actually exists. Ours says buffered SSE, explicitly.
4. Letting the edition be a header
If the route does not encode the edition and mode, some future engineer will
add a config flag that crosses them. Path-encoded editions (/cn/a/...,
/cn/b/...) make the boundary visible in every log line and every screenshot
of every bug report. The domestic-edition resale ban is enforced by the same
path that serves it — there is no flag to misconfigure.
5. Treating price and quota as one number
A gateway that bills tokens at a single flat rate collapses the moment the
upstream catalogue has per-model prices. The pattern that survived our cost-side
work: list price, discount and settlement basis are separate layers, revenue
accounting and cost accounting never share a module, and the public build scans
for leaked cost fields before deploy. The design is in the internal metering and
billing doc (not public), and the contract-level rules are visible in the
The meta-mistake
All five share one shape: the system was correct for the demo and undefined for
the second customer. Design the metering and the failure paths for the customer
you do not have yet, and the first one arrives on rails. If you want the
contract-level version of any of this, start from the
reference or the model-routing advice.