Fetcher: retry with backoff, per-site politeness, and concurrent feed refreshing

agent-pi commented

2026-09-13 01:55:34 +00:00

Collaborator

Problem

The fetcher is single-attempt and single-threaded: each feed is fetched sequentially, every article page fetch runs one after another with no delay between same-site requests, and a failed fetch is never retried. Politeness delays without concurrency would just serialize everything further, so the three concerns need to land together.

The onestepoffthegrid WAF ban (see the issue-33 A/B and fixes) was triggered or worsened by exactly this shape: six same-host article pages hammered back-to-back with one identity. Politeness would likely have avoided tripping it at all; retries recover the rest.

Goal

Feeds and article pages are fetched with retry + exponential backoff on transient failures, a minimum interval between requests to the same host, and feeds refreshed concurrently so one slow or throttled site never blocks updates of the others — all with the SSRF guard intact on every request and hop.

## Problem The fetcher is single-attempt and single-threaded: each feed is fetched sequentially, every article page fetch runs one after another with no delay between same-site requests, and a failed fetch is never retried. Politeness delays without concurrency would just serialize everything further, so the three concerns need to land together. The onestepoffthegrid WAF ban (see the issue-33 A/B and fixes) was triggered or worsened by exactly this shape: six same-host article pages hammered back-to-back with one identity. Politeness would likely have avoided tripping it at all; retries recover the rest. ## Goal Feeds and article pages are fetched with retry + exponential backoff on transient failures, a minimum interval between requests to the same host, and feeds refreshed concurrently so one slow or throttled site never blocks updates of the others — all with the SSRF guard intact on every request and hop.

agent-pi commented

2026-09-13 01:55:57 +00:00

Author

Collaborator

Findings (current state)

fetch_all_feeds iterates feeds sequentially; _fetch_feed_response makes exactly one attempt per URL (no retry, no Retry-After handling); article full-text fetches make two (browser identity, then extractor-identity retry — added in the WAF fix) with no delay between same-host hits.
Every full-text ingest batch hits all of one feed's new article pages back-to-back — the shape that tripped onestepoffthegrid's WAF.
httpx2 is used synchronously today; the project architecture note ("async for most I/O, synchronous SQLite is fine") leaves room for an asyncio client. SQLite writes must stay serialized regardless.
The SSRF guard (_validate_feed_url + per-hop re-validation) and _log_safe must survive the rework untouched.

Plan of attack (design decided at implementation)

Retry with backoff: transient failures (429, 5xx, network errors) retried 2–3 times with exponential backoff + jitter, honouring Retry-After when present. Non-transient 4xx (403/404) stay fail-fast; the article-page extractor-identity retry composes with this.
Per-site politeness: per-host last-request tracking with a configurable minimum interval (a few seconds) shared by feed fetches, article page fetches, and media downloads.
Concurrency: refresh feeds concurrently (asyncio with httpx2.AsyncClient, or a bounded thread pool — decided during implementation), so per-host politeness delays never block unrelated feeds. SQLite writes remain on one side of the boundary.
Tests: backoff schedule, Retry-After honouring, per-host isolation (a slow/blocked host does not delay another), guard preserved on every hop.

Deliberately out of scope: trafilatura's downloader stays unused (see the fetcher discussion — SSRF guard and header control live in our fetcher).

No implementation until this gets a nod.

## Findings (current state) - `fetch_all_feeds` iterates feeds sequentially; `_fetch_feed_response` makes exactly one attempt per URL (no retry, no `Retry-After` handling); article full-text fetches make two (browser identity, then extractor-identity retry — added in the WAF fix) with no delay between same-host hits. - Every full-text ingest batch hits all of one feed's new article pages back-to-back — the shape that tripped onestepoffthegrid's WAF. - httpx2 is used synchronously today; the project architecture note ("async for most I/O, synchronous SQLite is fine") leaves room for an asyncio client. SQLite writes must stay serialized regardless. - The SSRF guard (`_validate_feed_url` + per-hop re-validation) and `_log_safe` must survive the rework untouched. ## Plan of attack (design decided at implementation) 1. **Retry with backoff**: transient failures (429, 5xx, network errors) retried 2–3 times with exponential backoff + jitter, honouring `Retry-After` when present. Non-transient 4xx (403/404) stay fail-fast; the article-page extractor-identity retry composes with this. 2. **Per-site politeness**: per-host last-request tracking with a configurable minimum interval (a few seconds) shared by feed fetches, article page fetches, and media downloads. 3. **Concurrency**: refresh feeds concurrently (asyncio with `httpx2.AsyncClient`, or a bounded thread pool — decided during implementation), so per-host politeness delays never block unrelated feeds. SQLite writes remain on one side of the boundary. 4. Tests: backoff schedule, `Retry-After` honouring, per-host isolation (a slow/blocked host does not delay another), guard preserved on every hop. Deliberately out of scope: trafilatura's downloader stays unused (see the fetcher discussion — SSRF guard and header control live in our fetcher). No implementation until this gets a nod.

marvin8 commented

2026-09-13 02:05:32 +00:00

Owner

More thoughts on this:

Should we use the stamina library for the retries? I have used it with great success in some of my other projects.
instead of asyncio, could we use python 3.14 free threaded and the new tonio library for truly parallel execution?

More thoughts on this: - Should we use the stamina library for the retries? I have used it with great success in some of my other projects. - instead of asyncio, could we use python 3.14 free threaded and the new tonio library for truly parallel execution?

Rows
Columns

Fetcher: retry with backoff, per-site politeness, and concurrent feed refreshing #38

Problem

Goal

Findings (current state)

Plan of attack (design decided at implementation)