marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

Automatic feed discovery when adding feeds #37

New issue
Open
opened 2026-09-13 01:41:02 +00:00 by agent-pi · 1 comment
agent-pi commented 2026-09-13 01:41:02 +00:00
Collaborator
Copy link

Problem

Adding a feed requires the exact RSS/Atom URL. Hunting down that URL on every site is tedious — most sites hide it behind a link in the footer or a standard path like /feed.

Goal

Let the reader discover feed URLs automatically from a site's homepage, so adding a feed can start from the plain site address.

## Problem Adding a feed requires the exact RSS/Atom URL. Hunting down that URL on every site is tedious — most sites hide it behind a link in the footer or a standard path like `/feed`. ## Goal Let the reader discover feed URLs automatically from a site's homepage, so adding a feed can start from the plain site address.
agent-pi commented 2026-09-13 01:41:26 +00:00
Author
Collaborator
Copy link

Findings

  • trafilatura (already a dependency since #23) ships a feeds module for exactly this: determine_feed(htmlstring, params) parses a homepage's HTML and extracts feed URLs from link rel="alternate" tags plus common patterns (verified in the installed package, trafilatura/feeds.py). Its higher-level find_feed_urls(url) also exists but fetches pages itself, which would bypass the SSRF guard every other fetch goes through.
  • SSRF-safe approach: fetch the homepage ourselves through the existing guarded pipeline (_validate_feed_url + _fetch_feed_response, redirects re-validated), then hand the HTML to determine_feed for candidate extraction. Candidate URLs are re-validated through the same guard before being offered/stored.
  • FeedParameters (from courlan, used by determine_feed) carries the base URL for resolving relative feed links.

Plan of attack

  1. Discovery service: services/feed_discovery.py — discover_feeds(conn, site_url) -> list[str]: guarded homepage fetch, determine_feed for candidates, each candidate re-validated (SSRF guard) and de-duplicated. Returns the candidate list, optionally confirming each parses as a feed (feedparser smoke parse).
  2. Web UX: on the feeds page, a "Discover feeds" form (plain POST, no JS): submit a site URL → page lists discovered candidates, each with its own "Add feed" button pre-filling the existing add-feed flow.
  3. CLI: yunjin discover-feeds <site-url> prints candidates (keeps CLI/feature parity).
  4. Tests: recorded/fake HTML fixtures (link-tag feed, path-convention feed, none-found), SSRF rejection of discovered candidates, web flow test.

Branch

feat/issue-37-feed-discovery

Commits

  1. :sparkles: Add feed discovery service on trafilatura's link scanning
  2. :sparkles: Add feed discovery to the feeds page and CLI

No implementation until the approach gets a nod.

## Findings - `trafilatura` (already a dependency since #23) ships a feeds module for exactly this: `determine_feed(htmlstring, params)` parses a homepage's HTML and extracts feed URLs from `link rel="alternate"` tags plus common patterns (verified in the installed package, `trafilatura/feeds.py`). Its higher-level `find_feed_urls(url)` also exists but **fetches pages itself**, which would bypass the SSRF guard every other fetch goes through. - SSRF-safe approach: fetch the homepage ourselves through the existing guarded pipeline (`_validate_feed_url` + `_fetch_feed_response`, redirects re-validated), then hand the HTML to `determine_feed` for candidate extraction. Candidate URLs are re-validated through the same guard before being offered/stored. - `FeedParameters` (from courlan, used by `determine_feed`) carries the base URL for resolving relative feed links. ## Plan of attack 1. **Discovery service**: `services/feed_discovery.py` — `discover_feeds(conn, site_url) -> list[str]`: guarded homepage fetch, `determine_feed` for candidates, each candidate re-validated (SSRF guard) and de-duplicated. Returns the candidate list, optionally confirming each parses as a feed (feedparser smoke parse). 2. **Web UX**: on the feeds page, a "Discover feeds" form (plain POST, no JS): submit a site URL → page lists discovered candidates, each with its own "Add feed" button pre-filling the existing add-feed flow. 3. **CLI**: `yunjin discover-feeds <site-url>` prints candidates (keeps CLI/feature parity). 4. Tests: recorded/fake HTML fixtures (link-tag feed, path-convention feed, none-found), SSRF rejection of discovered candidates, web flow test. ## Branch `feat/issue-37-feed-discovery` ## Commits 1. `:sparkles: Add feed discovery service on trafilatura's link scanning` 2. `:sparkles: Add feed discovery to the feeds page and CLI` No implementation until the approach gets a nod.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#37
No description provided.