marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

Download full text of articles #23

New issue
Closed
opened 2026-09-11 22:28:37 +00:00 by agent-pi · 6 comments
agent-pi commented 2026-09-11 22:28:37 +00:00
Collaborator
Copy link

Problem

Article content comes from whatever the RSS feed includes, which is often only a summary — the reader rarely shows the complete story.

Goal

  • Download the full text of articles so the reader can show the whole article, not just the feed's teaser
## Problem Article content comes from whatever the RSS feed includes, which is often only a summary — the reader rarely shows the complete story. ## Goal - Download the full text of articles so the reader can show the whole article, not just the feed's teaser
agent-pi commented 2026-09-11 22:29:22 +00:00
Author
Collaborator
Copy link

Findings

  • media_extractor._fetch_and_extract_full_article already fetches full article HTML for image extraction — that fetch seam is reusable for text.
  • Turning HTML into readable text is the real design decision: extraction approach/lib needs choosing (with evidence, per house rules) — no dependency decision yet.
  • articles already store a content column; fetch politeness matters (per-origin limits; skip when the feed already delivered full content).

Plan of attack

  1. Design pass: extraction approach + when to fetch (only on short/summary-only content?).
  2. Extend the fetch/extract service; store extracted text.
  3. Reader display of full text; tests with recorded fixtures (no live API calls, per project convention).
## Findings - `media_extractor._fetch_and_extract_full_article` already fetches full article HTML for image extraction — that fetch seam is reusable for text. - Turning HTML into readable text is the real design decision: extraction approach/lib needs choosing (with evidence, per house rules) — no dependency decision yet. - `articles` already store a content column; fetch politeness matters (per-origin limits; skip when the feed already delivered full content). ## Plan of attack 1. Design pass: extraction approach + when to fetch (only on short/summary-only content?). 2. Extend the fetch/extract service; store extracted text. 3. Reader display of full text; tests with recorded fixtures (no live API calls, per project convention).
agent-pi commented 2026-09-12 05:41:53 +00:00
Author
Collaborator
Copy link

Design note (added during planning, from discussion with Marvin8)

There are two ways to source the full text, and the choice should be made — deliberately — when this issue is implemented:

  1. At ingest: fetch and extract full text in the fetch/extraction pipeline, store it, reader just displays it. No UI impact.
  2. Lazily, on demand: the reader shows the feed teaser; a "Load full article" action fetches, extracts, stores, and swaps in the full text for that article. Only articles someone actually opens cost a fetch (politeness + bandwidth), and the slow network operation gets a proper in-place loading state.

htmx consideration: variant 2 is exactly the interaction style htmx is built for (fragment swap, hx-indicator spinner). If we choose lazy loading, implementing this issue is the natural moment to evaluate and decide whether to introduce htmx to yunjin — vendored locally (offline/LAN-safe, like the bundled Atkinson font), real <form> elements kept as the no-JS fallback, and the AGENTS.md no-JavaScript rule amended deliberately to permit it. If we choose ingest-time fetching, htmx stays off the table for now.

Either way the decision and its rationale get recorded in the findings comment when work starts.

## Design note (added during planning, from discussion with Marvin8) There are two ways to source the full text, and the choice should be made — deliberately — when this issue is implemented: 1. **At ingest**: fetch and extract full text in the fetch/extraction pipeline, store it, reader just displays it. No UI impact. 2. **Lazily, on demand**: the reader shows the feed teaser; a "Load full article" action fetches, extracts, stores, and swaps in the full text for that article. Only articles someone actually opens cost a fetch (politeness + bandwidth), and the slow network operation gets a proper in-place loading state. **htmx consideration:** variant 2 is exactly the interaction style htmx is built for (fragment swap, hx-indicator spinner). If we choose lazy loading, implementing this issue is the natural moment to evaluate and decide whether to introduce htmx to yunjin — vendored locally (offline/LAN-safe, like the bundled Atkinson font), real `<form>` elements kept as the no-JS fallback, and the AGENTS.md no-JavaScript rule amended deliberately to permit it. If we choose ingest-time fetching, htmx stays off the table for now. Either way the decision and its rationale get recorded in the findings comment when work starts.
marvin8 commented 2026-09-12 09:13:13 +00:00
Owner
Copy link

Some of the article full texts will probably be in HTML format. This issue should also determine if we can convert the downloaded full text to markdown and display that.

Some of the article full texts will probably be in HTML format. This issue should also determine if we can convert the downloaded full text to markdown and display that.
agent-pi commented 2026-09-12 21:22:57 +00:00
Author
Collaborator
Copy link

Decisions (from the discussion)

  • Fetch timing: at ingest (option A) — full text is stored during feed refresh, reader is instant.
  • Extraction: trafilatura, verified by spike: v2.2.0 fetched a live Guardian article from the local db and produced clean main content via its native output_format="markdown" (~4 KB markdown from 348 KB HTML). One dependency covers extraction and HTML→markdown. Weight measured: 17 packages / 59 MB site-packages (lxml 12 MB is the biggest; no compilers, all wheels).
  • full_text feeds every LLM step: the aggregator prompt (today: title + 300 chars of feed teaser per article) and the summarizer teasers (today: first 150 chars of feed content) both prefer full_text when present.
  • Images kept — hotlinked as they appear in the extracted markdown (local download via the media pipeline would be a follow-up).

Design

  1. Schema: alembic migration adds articles.full_text (TEXT, NULL) + articles.full_text_fetched_at. NULL text = no clean extraction or pre-migration article; teaser behaviour unchanged.
  2. Ingest: in fetch_feed, after _process_feed_entries, for each newly created article: GET the article URL (existing browser-like UA, timeout), skip non-HTML content types, extract markdown with trafilatura, store. full_text_fetched_at records the attempt; failures leave full_text NULL and are not retried.
  3. LLM steps: llm_aggregator._build_article_entry and aggregate_summarizer source from full_text first (aggregator cap raised to 4000 chars/article for prompt economy — a constant, easy to tune), falling back to feed content.
  4. Display: article page renders stored markdown through mistune + nh3 sanitization (external content → never raw HTML); teaser fallback unchanged.

Branch

feat/issue-23-full-text

Commits

  1. :heavy_plus_sign: Add trafilatura, mistune and nh3
  2. :sparkles: Store article full text during feed ingestion
  3. :sparkles: Feed article full text to LLM aggregation and summaries
  4. :sparkles: Render stored full text on the article page

PR (after CI-green locally)

curl -X POST -H "Authorization: token $FORGE_AGENT_PI_TOKEN" -H "Content-Type: application/json" \
  -d "{\"title\":\"Download and store full article text at ingest\",\"body\":\"…Closes #23\",\"head\":\"feat/issue-23-full-text\",\"base\":\"main\"}" \
  https://forge.marvin8.zone/api/v1/repos/marvin8/yunjin/pulls
## Decisions (from the discussion) - **Fetch timing: at ingest** (option A) — full text is stored during feed refresh, reader is instant. - **Extraction: trafilatura**, verified by spike: v2.2.0 fetched a live Guardian article from the local db and produced clean main content via its native `output_format="markdown"` (~4 KB markdown from 348 KB HTML). One dependency covers extraction *and* HTML→markdown. Weight measured: 17 packages / 59 MB site-packages (lxml 12 MB is the biggest; no compilers, all wheels). - **full_text feeds every LLM step**: the aggregator prompt (today: title + 300 chars of feed teaser per article) and the summarizer teasers (today: first 150 chars of feed content) both prefer `full_text` when present. - **Images kept** — hotlinked as they appear in the extracted markdown (local download via the media pipeline would be a follow-up). ## Design 1. **Schema**: alembic migration adds `articles.full_text` (TEXT, NULL) + `articles.full_text_fetched_at`. NULL text = no clean extraction or pre-migration article; teaser behaviour unchanged. 2. **Ingest**: in `fetch_feed`, after `_process_feed_entries`, for each newly created article: GET the article URL (existing browser-like UA, timeout), skip non-HTML content types, extract markdown with trafilatura, store. `full_text_fetched_at` records the attempt; failures leave `full_text` NULL and are not retried. 3. **LLM steps**: `llm_aggregator._build_article_entry` and `aggregate_summarizer` source from `full_text` first (aggregator cap raised to 4000 chars/article for prompt economy — a constant, easy to tune), falling back to feed content. 4. **Display**: article page renders stored markdown through mistune + nh3 sanitization (external content → never raw HTML); teaser fallback unchanged. ## Branch `feat/issue-23-full-text` ## Commits 1. `:heavy_plus_sign: Add trafilatura, mistune and nh3` 2. `:sparkles: Store article full text during feed ingestion` 3. `:sparkles: Feed article full text to LLM aggregation and summaries` 4. `:sparkles: Render stored full text on the article page` ## PR (after CI-green locally) ``` curl -X POST -H "Authorization: token $FORGE_AGENT_PI_TOKEN" -H "Content-Type: application/json" \ -d "{\"title\":\"Download and store full article text at ingest\",\"body\":\"…Closes #23\",\"head\":\"feat/issue-23-full-text\",\"base\":\"main\"}" \ https://forge.marvin8.zone/api/v1/repos/marvin8/yunjin/pulls ```
agent-pi commented 2026-09-12 22:42:17 +00:00
Author
Collaborator
Copy link

Follow-up decision: grouping input reverted to teasers

An A/B comparison (identical 75-article set, fresh refresh() runs differing only in whether full_text feeds the prompt) showed full-text prompts make grouping slightly more fragmented: 4 co-grouped pairs split (Victorian-election pair; a solar/consumer-energy policy trio) vs 1 merge (two distinct energy podcasts), with no visible grouping gain. Decision: the aggregation prompt reverts to feed teasers; full text stays in the summarizer teasers and article display, where it clearly wins. Implemented in the issue-34 branch (stacked on PR #32).

## Follow-up decision: grouping input reverted to teasers An A/B comparison (identical 75-article set, fresh `refresh()` runs differing only in whether `full_text` feeds the prompt) showed full-text prompts make grouping slightly **more** fragmented: 4 co-grouped pairs split (Victorian-election pair; a solar/consumer-energy policy trio) vs 1 merge (two distinct energy podcasts), with no visible grouping gain. Decision: the **aggregation prompt reverts to feed teasers**; full text stays in the summarizer teasers and article display, where it clearly wins. Implemented in the issue-34 branch (stacked on PR #32).
agent-pi commented 2026-09-12 23:29:23 +00:00
Author
Collaborator
Copy link

Implemented in PR #32 (merged as dbf5046): full text fetched and stored as markdown at ingest (trafilatura, SSRF-guarded, images kept); articles.full_text + full_text_fetched_at via alembic; article page renders it (mistune + nh3 sanitized); summarizer teasers sourced from it. Follow-ups: grouping prompt reverted to teasers after the A/B (see comment above), tracked with the refresh-robustness fix in PR #35.

Implemented in PR #32 (merged as dbf5046): full text fetched and stored as markdown at ingest (trafilatura, SSRF-guarded, images kept); `articles.full_text` + `full_text_fetched_at` via alembic; article page renders it (mistune + nh3 sanitized); summarizer teasers sourced from it. Follow-ups: grouping prompt reverted to teasers after the A/B (see comment above), tracked with the refresh-robustness fix in PR #35.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#23
No description provided.