marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

The Driven feed fetch fails with 403 (server rejects non-browser User-Agent) #2

New issue
Closed
opened 2026-09-10 03:02:46 +00:00 by agent-pi · 3 comments
agent-pi commented 2026-09-10 03:02:46 +00:00
Collaborator
Copy link

Problem

Fetching The Driven's RSS feed (https://thedriven.io/feed/) fails with 403 Forbidden from yunjin's fetcher, so no articles from that feed are ever imported. The same URL loads fine in a browser, so the feed itself is healthy — the server is rejecting yunjin's requests based on their headers.

## Problem Fetching The Driven's RSS feed (`https://thedriven.io/feed/`) fails with `403 Forbidden` from yunjin's fetcher, so no articles from that feed are ever imported. The same URL loads fine in a browser, so the feed itself is healthy — the server is rejecting yunjin's requests based on their headers.
agent-pi commented 2026-09-10 03:03:37 +00:00
Author
Collaborator
Copy link

Findings

Reproduced during the Sept 2026 probe: httpx.get("https://thedriven.io/feed/") with default headers returns 403 Forbidden; the identical request with a browser User-Agent (Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0) returns 200 and the feed parses normally. All 15 other feeds in the FreshRSS OPML fetch fine without the header.

Plan of attack

  1. Add a browser-like default User-Agent header to the fetch service's httpx client, overridable via config.
  2. Regression test asserting the header is sent on feed requests.
## Findings Reproduced during the Sept 2026 probe: `httpx.get("https://thedriven.io/feed/")` with default headers returns `403 Forbidden`; the identical request with a browser User-Agent (`Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0`) returns 200 and the feed parses normally. All 15 other feeds in the FreshRSS OPML fetch fine without the header. ## Plan of attack 1. Add a browser-like default User-Agent header to the fetch service's httpx client, overridable via config. 2. Regression test asserting the header is sent on feed requests.
agent-pi commented 2026-09-11 20:56:10 +00:00
Author
Collaborator
Copy link

Plan of attack: SSRF guard (WuMing finding, Marvin8 approved option c)

Same branch/PR as the User-Agent fix (#16, still unmerged — the finding is on its diff).

  1. _validate_feed_url() in fetcher.py, called before the HTTP request:
    • scheme allow-list: http/https only, always enforced
    • resolve host via socket.getaddrinfo; block if any resolved address is private, loopback, link-local, reserved, unspecified, or multicast (IPv4-mapped IPv6 unwrapped); fail closed on resolution errors
    • opt-out: YUNJIN_ALLOW_PRIVATE_FEED_HOSTS truthy skips the private-range check for LAN feeds (scheme check still enforced)
    • violations raise FetchError with a specific message
  2. Known limitations documented in the docstring: redirects are followed without re-validation, and validation is TOCTOU vs. the client's own resolution (DNS rebinding) — full pinning needs a custom transport, out of scope here.
  3. TDD: ~10 tests, all getaddrinfo mocked (no live network).
## Plan of attack: SSRF guard (WuMing finding, Marvin8 approved option c) Same branch/PR as the User-Agent fix (#16, still unmerged — the finding is on its diff). 1. `_validate_feed_url()` in `fetcher.py`, called before the HTTP request: - scheme allow-list: `http`/`https` only, always enforced - resolve host via `socket.getaddrinfo`; block if any resolved address is private, loopback, link-local, reserved, unspecified, or multicast (IPv4-mapped IPv6 unwrapped); fail closed on resolution errors - opt-out: `YUNJIN_ALLOW_PRIVATE_FEED_HOSTS` truthy skips the private-range check for LAN feeds (scheme check still enforced) - violations raise `FetchError` with a specific message 2. Known limitations documented in the docstring: redirects are followed without re-validation, and validation is TOCTOU vs. the client's own resolution (DNS rebinding) — full pinning needs a custom transport, out of scope here. 3. TDD: ~10 tests, all `getaddrinfo` mocked (no live network).
agent-pi commented 2026-09-11 21:54:54 +00:00
Author
Collaborator
Copy link

Resolved by PR #16 (merged as cdc5e00): browser-like User-Agent for feed fetches with YUNJIN_HTTP_USER_AGENT override, plus the maintainer-approved SSRF guard (scheme allow-list, private/loopback/link-local blocking with YUNJIN_ALLOW_PRIVATE_FEED_HOSTS opt-out, per-hop redirect re-validation). DNS-rebinding pinning declined as a documented limitation.

Resolved by PR #16 (merged as cdc5e00): browser-like User-Agent for feed fetches with `YUNJIN_HTTP_USER_AGENT` override, plus the maintainer-approved SSRF guard (scheme allow-list, private/loopback/link-local blocking with `YUNJIN_ALLOW_PRIVATE_FEED_HOSTS` opt-out, per-hop redirect re-validation). DNS-rebinding pinning declined as a documented limitation.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#2
No description provided.