Confirmed live by the user: the one-request-per-element loop (per-element
API metadata lookup + per-element image download, no delay between any of
them) got 429'd by Wikimedia partway through a real run of --deck hs.
Fixes:
- Batch pageimage metadata lookups up to 50 titles per MediaWiki query
instead of one request per element (ensure_element_photos ->
resolve_pageimage_urls), cutting ~99 requests down to ~2 for a full hs
run.
- Retry with backoff on 429 (http_get_with_retry), honoring Wikipedia's
own Retry-After header when present.
- Request thumbnails (piprop=thumbnail) instead of full-resolution
originals, per Wikimedia's own guidance in the 429 response body.
- Small delay between individual image downloads, which still can't be
batched (one HTTP request per element's actual image bytes).
Also fixes a real bug caught while unit-testing the new batched
redirect/normalization resolution against a simulated response: the
final-title -> original-input-title lookup had the mapping backwards
(looked up by final title in a dict keyed by input title), which would
have silently dropped every element whose title needed resolving through
a redirect (e.g. Cesium -> Caesium) even after the rate-limit fix.
Still couldn't test against the real Wikipedia API (no network route to
en.wikipedia.org from this sandbox) — verified instead with a local HTTP
server simulating 429-then-200 and a fabricated MediaWiki response with a
redirect chain. Needs a real run to fully confirm.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014DyceEVVeQ33EeS6C1PDv5
Claude
2070c31cfe