tokencoup
Edition: International · 国内版 Language: EN · ZH
Get API keys

Five gateway integration mistakes we made so you don't have to

2026-11-13

Five gateway integration mistakes we made so you don't have to

A proxy gateway looks like a thin wrapper until you meet your first production

request that is slow, duplicated, or billed wrong. These are the failure modes

that actually bit us, in the order they usually appear.

1. Metering from the response body instead of upstream usage

Counting tokens as len(body)/4 is not metering, it is astrology. It drifts on

multilingual content, it ignores the system prompt your wrapper injects, and it

quietly undercounts on code. The v0.4 metering reads the upstream usage field

(JSON and the final SSE chunk) and flags fallbacks as estimated — the flag is

the point, because an invoice nobody can audit is worse than no invoice. Read

the metering section of the docs before you copy any

token-budget code from the internet.

2. Charging on replay

Network retries are a feature of the internet. If a retry re-runs the upstream

call and re-bills it, a single flaky minute can double a customer's bill.

Idempotency via X-Request-Id is checked before the upstream forward, not

after, so a replayed request reuses the first result and the first ledger entry.

This is also why the video endpoints check replay before forwarding — task

creation costs real money upstream, and the bill must follow the first call, not

the unluckiest one.

3. Betting on streaming you do not have

"Streaming supported" with a full-body buffer is a lie clients will punish in

production: token-by-token UIs stall, and proxies that assume chunk pacing

misjudge timeouts. If you buffer, say so in the docs and let clients opt into

what actually exists. Ours says buffered SSE, explicitly.

4. Letting the edition be a header

If the route does not encode the edition and mode, some future engineer will

add a config flag that crosses them. Path-encoded editions (/intl/a/...,

/intl/b/...) make the boundary visible in every log line and every screenshot

of every bug report. The domestic-edition resale ban is enforced by the same

path that serves it — there is no flag to misconfigure.

5. Treating price and quota as one number

A gateway that bills tokens at a single flat rate collapses the moment the

upstream catalogue has per-model prices. The pattern that survived our cost-side

work: list price, discount and settlement basis are separate layers, revenue

accounting and cost accounting never share a module, and the public build scans

for leaked cost fields before deploy. The design is in the internal metering and

billing doc (not public), and the contract-level rules are visible in the

API reference.

The meta-mistake

All five share one shape: the system was correct for the demo and undefined for

the second customer. Design the metering and the failure paths for the customer

you do not have yet, and the first one arrives on rails. If you want the

contract-level version of any of this, start from the

reference or the model-routing advice.

Back to blog