API Rate-Limit Testing for Abuse and Cost Control
Measure work, then enforce a budget
Resource-consumption review begins with cost measurement. Profile report generation, password verification, search fan-out, image conversion, webhook retries, and messages sent to paid providers. Identify which key represents a fair budget: account, tenant, API credential, device, IP, queue worker, or provider account. One global request counter cannot describe all of these costs.
Run controlled tests at baseline, just below threshold, threshold, and concurrent threshold. A CSV export limit is passing only when the next job receives documented queue or 429 behavior and no object-storage object, database snapshot, or provider charge is created. Check alternate routes and parameter encodings; a /reports/export cap that /reports?format=csv bypasses is not protection. For GraphQL, complexity and batch size may be budget dimensions rather than request count.
A failure record includes cost metric, quota key, window, retry output, queue depth, and downstream call count. Repair may add input maxima before parsing, per-tenant concurrency, idempotency keys, or circuit breaking at a paid provider. The closure test proves rejected work has no partial side effect and proves normal traffic recovers after window reset; capacity owner chooses thresholds from observed service limits.
Budget enforcement needs observability without leaking internals to callers. Count accepted and rejected units, queue wait, downstream request count, and retry outcomes by operation. Distinguish a client retry after timeout from a new expensive request with idempotency key. A burst test should stop at approved ceiling and include cleanup verification. If third-party quota is lower than gateway quota, provider boundary wins; configure circuit behavior so failures do not create retry storms.
Choose a quota dimension from cost
A rate limit begins with work accounting. List every expensive action: password checks, report generation, image conversion, search fan-out, file download, webhook retry, and paid SMS or email. For each action, name the unit that should absorb cost. A login attempt may use account plus IP; a report job may use tenant plus concurrent worker count; a public search may use credential plus query complexity. One per-IP counter neither stops distributed traffic nor treats a shared office fairly.
Measure baseline duration, database reads, worker occupancy, response bytes, queue depth, and external requests using a synthetic fixture. Then run four controlled cases: normal traffic, exactly at quota, one request above quota, and many parallel callers. For POST /reports/export, the above-quota result should be a documented 429 or queue refusal. It must create no object-storage file, no retained queue job, and no provider charge. Record retry headers and reset behavior; a client must not be told to retry earlier than server accepts it.
Distributed and identity tests
Use several IPs for one account, several accounts from one IP, several credentials under one tenant, and one credential across several routes that cause same downstream work. These cases show whether policy key matches abuse model. An attacker may distribute source addresses while consuming one tenant’s report budget; a shared NAT may carry many legitimate users. Test alternate endpoint forms such as /exports and /reports?format=csv, otherwise an operation limit can be bypassed by representation.
During a burst, cancel selected clients halfway through. Worker slots, temporary files, and partial uploads should be released. Retry with an idempotency key and verify it does not reserve capacity twice. A queue is not a limit unless backlog is bounded and rejected tasks cannot pile up. If provider quota is lower than application quota, provider result becomes a failure boundary and circuit behavior must prevent retry storms.
Evidence and closure
Keep quota configuration version, traffic script parameters, identity dimension, timestamps, latency samples, queue measurements, provider-call count, and cleanup result. Capacity owner decides threshold from measured headroom and recovery target, then signs the operational rule. Remediate with request-size caps before parsing, pagination caps, per-tenant concurrency, token budgets, idempotency, or provider circuit breaking. Closure reruns the rejected case, proves no partial side effect, and runs an allowed case after window reset. A legacy integration exception names route, quota, consumer, end date, and monitoring signal; it never disables all limits.
Recovery behavior
A limit is also an availability decision. Confirm ordinary requests recover when window resets and that one rejected caller does not lock unrelated tenant work. Examine clock skew between gateway and application counters; inconsistent reset times make clients retry in bursts. When counters are unavailable, define fail-open or fail-closed per operation. Login and paid messaging may require conservative refusal, while a low-cost public read may tolerate degradation. Record that choice with cost evidence.
Run a distributed script slowly enough to observe each layer. Compare gateway counter, application budget, queue admission, and provider response. If layers disagree, decide which one supplies authoritative rejection and make response deterministic. Test tenant deletion or credential rotation: old quota entries must not grant a new principal unexpected capacity. The final artifact includes cleanup confirmation for temporary files, jobs, and provider test messages.
Resource-budget run record
| Operation and quota | Owner | Evidence | Expected result | Observed result / status | Exception | Closure |
|---|---|---|---|---|---|---|
POST /reports/export; 5 concurrent jobs per tenant | Capacity owner | Run RL-206; quota config v12; queue, storage, and provider counters | Sixth request returns 429 with no queued job, file, or provider call | 429; queue delta 0; files 0; provider calls 0 — closed | None | Allowed request passed after reset; temporary files cleaned; run signed 2026-06-12 |