Tuning Caddy timeouts against stalled scrapers
Short header and body deadlines keep reverse-proxy worker pools healthy.
Default reverse-proxy timeouts are generous so slow residential links rarely drop uploads. On a public-facing micro host, however, poorly written scrapers frequently open dozens of connections, send half a header block, and sit idle until the proxy gives up.
Tighten server deadlines at the edge
In Caddy's global servers block, cap max_header_size to 16 KiB and
set explicit read_header (5s), read_body (8s), write
(20s), and idle (30s) timeouts. On the upstream transport, match
max_conns_per_host to the backend's worker limit so excess connections queue or
fail fast at the proxy instead of piling up inside Python.
Avoid logging client bodies or unbounded paths
Edge access logs fill small disks surprisingly fast during vulnerability scans. If the
application already keeps a structured, bounded SQLite summary with automatic pruning, disable
edge access logs and keep error logs scoped so a burst of malformed TLS probes cannot exhaust
/var/log.
Did this work on your setup, or did you spot a mistake? Send the editor a private note or leave a public note for the next reader.