Safe URL Fetching: SSRF-Resistant Design Worksheet
Safe URL fetching needs a policy engine, not a regex. RFC 3986 parsing distinguishes scheme, host, port, userinfo, and path; unsafe normalization creates bypasses.
Design worksheet
| Question | Safe answer | Example |
|---|---|---|
| Who chooses host? | configured allowlist | calendar feed provider |
| Which scheme? | HTTPS only unless required | reject file: |
| Redirects? | revalidate each hop | public URL redirects to private IP |
| DNS? | resolve then check resulting address | rebinding denied |
A URL preview and a document importer need different policies: preview may forbid all private addresses; importer may use a fixed vendor hostname. OWASP’s SSRF prevention guidance supports allowlists and network segmentation.
Decision rule: reject any destination not positively authorized after parsing, resolution, and redirect handling.
Build safely
Use a hardened HTTP client with disabled implicit redirects, bounded response size, egress filtering, and separate network identity. This worksheet does not make arbitrary internet content trustworthy. Source review can inspect fetch helpers.
Make destination policy observable
Use a real URL parser, then evaluate scheme, hostname, port, and all resolved addresses before opening a connection. Disable implicit redirects; when a redirect is allowed, parse and authorize its next target as a new request. Set low connect, read, total-time, and response-byte limits. Separate network identity from main application credentials and block ambient proxy settings where they are not needed.
For a fixed partner, a configured hostname allowlist is stronger than accepting a complete URL. For open previews, public-address checks and egress policy are both needed. Test userinfo ambiguity, unsupported schemes, loopback literals, private IPv6, changed DNS answer, redirect, nonstandard port, and oversized response. Owner: feature service owner. Pass: policy stops unauthorized destinations before useful response processing. Fail: protected address or credentialed service is reached. Exceptions record purpose, target, safeguards, approver, remediation date, and expiry.
Fetch contract example
A documented policy can state HTTPS only, configured partner host, allowed port, redirects disabled, maximum response size, and bounded timeouts. These are examples, not universal settings; owner chooses limits from feature need and resource budget. Log policy decision and resolved addresses, not full response content. Treat user callback URLs as a separate capability.
Fetch decision trace
Expected output: normalized URL, policy version, allowed host rule, resolved addresses, selected port, redirect count, byte limit, timeout result. This trace lets maintainers diagnose destination refusal without retrying blindly.
Worked failure: A preview URL uses userinfo to resemble an allowed host while connecting elsewhere. Parsing must expose the real authority and policy must reject it.
Edge case: A partner can legitimately change addresses. Re-evaluate each connection against public-address rules; do not pin a stale approval to a hostname string.
Closure test: Force a redirect to a private IPv6 target during controlled testing. Closure passes when no socket opens and trace contains the denied address.
Resolver and connector separation
The policy decision must survive the gap between DNS lookup and socket creation. Resolve using controlled resolver behavior, retain returned addresses, and connect only to an address that passed current policy. A later resolver answer must start a new evaluation. If client library pools connections, verify it does not reuse a connection for a differently authorized host. HTTPS certificate validation remains necessary; destination allowlisting does not authenticate a server.
Build test fixtures that return a public address during validation and a prohibited address during connection simulation. Include an HTTP redirect with a relative Location value, an internationalized hostname that normalizes unexpectedly, and an explicit port outside policy. Expected output is a distinct reason code for parse, host, address, redirect, port, limit, or timeout failure. This classification helps owners fix policy safely without logging sensitive remote content.
Connection telemetry should include destination address actually dialed, not only hostname submitted. Test a response that exceeds byte limit after valid headers. Expected result is bounded abort with no cache entry, parser invocation, or retry to another address.
Response handling is part of destination safety. Do not pass fetched bytes directly to image, archive, markup, or document parsers without type and size controls. A safe network destination can still return content that exhausts memory or exploits a downstream parser. Record content type as an observation, not trust signal. Test chunked or unknown-length responses and ensure byte accounting stops reading at policy limit. This separates SSRF prevention from content-processing defense while preserving both controls.
Final fetch test records submitted URL, resolved addresses, redirect decisions, and byte-limit result without retaining sensitive query data. Expected result: only approved destinations and response sizes reach downstream processing. Edge case: DNS answer changes between validation and connection; compare actual dialed address. Close when denied paths create no parser work, cache entry, or alternate retry.