marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

Featured image selection picks site banners and ads instead of the article image #33

New issue
Closed
opened 2026-09-12 22:36:17 +00:00 by agent-pi · 3 comments
agent-pi commented 2026-09-12 22:36:17 +00:00
Collaborator
Copy link
No description provided.
agent-pi commented 2026-09-12 22:36:33 +00:00
Author
Collaborator
Copy link

Findings

  • Reader cards use db.media.get_first_image — the earliest-stored media row for an article. Whichever candidate MediaExtractor stored first wins.
  • MediaExtractor.extract_media_urls unions candidates from media-RSS, <img> tags, og:image, <picture> and bare URLs; nothing ranks them, so an image scraped from page chrome (site banner/logo) or an ad slot frequently lands first.
  • WordPress sites are the worst case reported, but their og:image is usually the correct featured image (Yoast derives it from the post thumbnail).
  • During full-text ingest (issue #23) we already download the article page, so page metadata is available at zero extra cost.

Plan of attack

  1. og:image preference: during full-text ingest, extract og:image (trafilatura extract_metadata) and store it flagged as preferred (new column or a ranked field on media).
  2. Ranking at selection time: get_first_image becomes get_featured_image — preferred/og image first, then media-RSS, then in-content images.
  3. Banner suppression: an image URL appearing across many articles of the same feed is site chrome — exclude it from selection (per-feed URL frequency check).
  4. Tests with fixtures: og:image preferred, banner deduped, fallback to current behaviour.

Branch: feat/issue-33-featured-image. Commits: (1) :sparkles: Record og:image during full-text ingest, (2) :sparkles: Rank featured image selection and suppress site banners.

## Findings - Reader cards use ``db.media.get_first_image`` — the **earliest-stored** media row for an article. Whichever candidate ``MediaExtractor`` stored first wins. - ``MediaExtractor.extract_media_urls`` unions candidates from media-RSS, ``<img>`` tags, og:image, ``<picture>`` and bare URLs; nothing ranks them, so an image scraped from page chrome (site banner/logo) or an ad slot frequently lands first. - WordPress sites are the worst case reported, but their ``og:image`` is usually the *correct* featured image (Yoast derives it from the post thumbnail). - During full-text ingest (issue #23) we already download the article page, so page metadata is available at zero extra cost. ## Plan of attack 1. **og:image preference**: during full-text ingest, extract ``og:image`` (trafilatura ``extract_metadata``) and store it flagged as preferred (new column or a ranked field on media). 2. **Ranking at selection time**: ``get_first_image`` becomes ``get_featured_image`` — preferred/og image first, then media-RSS, then in-content images. 3. **Banner suppression**: an image URL appearing across many articles of the same feed is site chrome — exclude it from selection (per-feed URL frequency check). 4. Tests with fixtures: og:image preferred, banner deduped, fallback to current behaviour. Branch: ``feat/issue-33-featured-image``. Commits: (1) ``:sparkles: Record og:image during full-text ingest``, (2) ``:sparkles: Rank featured image selection and suppress site banners``.
agent-pi commented 2026-09-13 00:10:55 +00:00
Author
Collaborator
Copy link

Agreed design (supersedes the earlier sketch)

Evidence gathered from the live db changed the approach: on onestepoffthegrid the junk (site logo SVG, rss.png, "Supported-by" sponsor banner) recurs across articles, and — surprise — that site's own og:image is also a sponsor graphic. So ranking alone cannot fix it; recurrence-based suppression is the workhorse, og:image is just one ranked candidate. Separately, the ABC "multiple images" turned out to be the same asset URL with different resize query params (40 rows → 7 real images), so dedupe needs URL normalization, not perceptual hashing (dhash measured at ~20 MB for dhash+Pillow and adds false-positive risk; skipped).

Selection happens at ingest; the winner is downloaded once.

  1. Candidates per new article, ranked entry (feed media-RSS/content, stored today) → og (trafilatura page metadata, available during full-text ingest) → content (images from entry content / stored markdown). URLs sanitized (strips the embedded-newline bug), .svg skipped.
  2. Normalize + rank: dedupe on query-stripped base URL (collapses ABC resize variants); same-base variants ranked by parsed dimension params, largest wins.
  3. Suppress site chrome: drop candidates whose base URL appears on ≥ 3 distinct articles of the same feed (kills all the observed junk).
  4. Persist: alembic column articles.featured_media_id; winner gets a media row (created if new) and the article points at it. media_downloader gains a single-image download (it currently has no production caller) so exactly the featured image is fetched locally; reader serves local file with remote fallback.
  5. Reader: cards prefer the featured image; articles without one keep today's behaviour (zero impact on pre-existing rows).

Author avatars: known limitation (below og/entry in rank; caught only if feed-ubiquitous) — review after a week of use.

Branch

feat/issue-33-featured-image

Commits

  1. :sparkles: Add featured image candidate selection service
  2. :sparkles: Choose and download the featured image at ingest
  3. :sparkles: Serve the featured image on reader cards
## Agreed design (supersedes the earlier sketch) Evidence gathered from the live db changed the approach: on onestepoffthegrid the junk (site logo SVG, `rss.png`, "Supported-by" sponsor banner) **recurs across articles**, and — surprise — that site's own `og:image` is *also* a sponsor graphic. So ranking alone cannot fix it; recurrence-based suppression is the workhorse, og:image is just one ranked candidate. Separately, the ABC "multiple images" turned out to be the **same asset URL with different resize query params** (40 rows → 7 real images), so dedupe needs URL normalization, not perceptual hashing (dhash measured at ~20 MB for dhash+Pillow and adds false-positive risk; skipped). **Selection happens at ingest; the winner is downloaded once.** 1. **Candidates** per new article, ranked `entry` (feed media-RSS/content, stored today) → `og` (trafilatura page metadata, available during full-text ingest) → `content` (images from entry content / stored markdown). URLs sanitized (strips the embedded-newline bug), `.svg` skipped. 2. **Normalize + rank**: dedupe on query-stripped base URL (collapses ABC resize variants); same-base variants ranked by parsed dimension params, largest wins. 3. **Suppress site chrome**: drop candidates whose base URL appears on ≥ 3 distinct articles of the same feed (kills all the observed junk). 4. **Persist**: alembic column `articles.featured_media_id`; winner gets a media row (created if new) and the article points at it. `media_downloader` gains a single-image download (it currently has no production caller) so exactly the featured image is fetched locally; reader serves local file with remote fallback. 5. **Reader**: cards prefer the featured image; articles without one keep today's behaviour (zero impact on pre-existing rows). Author avatars: known limitation (below og/entry in rank; caught only if feed-ubiquitous) — review after a week of use. ## Branch `feat/issue-33-featured-image` ## Commits 1. `:sparkles: Add featured image candidate selection service` 2. `:sparkles: Choose and download the featured image at ingest` 3. `:sparkles: Serve the featured image on reader cards`
agent-pi commented 2026-09-13 02:02:37 +00:00
Author
Collaborator
Copy link

Implemented in PR #36 (merged as b5980fd): featured image selection at ingest — ranked candidates (entry → og:image → content), resize-variant collapse via base-URL normalization, site-chrome suppression at ≥ 3 same-feed occurrences, SVG exclusion; articles.featured_media_id + single-image download; reader cards prefer the featured image with legacy fallback. Hardened during review: media assignment validated, ingest log sanitization. Post-merge additions: /media serving resolved absolutely, article fetches send browser headers with extractor-identity retry and reject error pages.

Implemented in PR #36 (merged as b5980fd): featured image selection at ingest — ranked candidates (entry → og:image → content), resize-variant collapse via base-URL normalization, site-chrome suppression at ≥ 3 same-feed occurrences, SVG exclusion; `articles.featured_media_id` + single-image download; reader cards prefer the featured image with legacy fallback. Hardened during review: media assignment validated, ingest log sanitization. Post-merge additions: `/media` serving resolved absolutely, article fetches send browser headers with extractor-identity retry and reject error pages.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#33
No description provided.