marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

Download and store article images locally #21

New issue
Closed
opened 2026-09-11 22:28:36 +00:00 by agent-pi · 2 comments
agent-pi commented 2026-09-11 22:28:36 +00:00
Collaborator
Copy link

Problem

Media extraction records image URLs, but the images themselves are still loaded from the origin on every view — slow, and lost when the origin removes them or rejects us (see the 403 lessons from issue #2).

Goal

  • Download the image / images for all articles and store them locally, so the reader serves them itself instead of hotlinking
## Problem Media extraction records image URLs, but the images themselves are still loaded from the origin on every view — slow, and lost when the origin removes them or rejects us (see the 403 lessons from issue #2). ## Goal - Download the image / images for all articles and store them locally, so the reader serves them itself instead of hotlinking
agent-pi commented 2026-09-11 22:29:21 +00:00
Author
Collaborator
Copy link

Findings

  • media_extractor already discovers image URLs and db/media stores them per article; only the bytes are missing.
  • Needs: a local media directory with a sane naming scheme (hash the URL), a download step in the extraction pipeline, and a Flask route to serve stored files.
  • Origin behaviour matters (see issue #2): some origins 403 default clients — reuse the browser-like User-Agent and expect failures; missing images must degrade gracefully.

Plan of attack

  1. Storage layout + stored flag in the media schema (alembic migration if needed).
  2. Downloader service wired into media extraction; failure-tolerant.
  3. Serving route under the web app; template swap from remote URL to local route.
  4. Tests with mocked HTTP (no live network, per project convention).
## Findings - `media_extractor` already discovers image URLs and `db/media` stores them per article; only the bytes are missing. - Needs: a local media directory with a sane naming scheme (hash the URL), a download step in the extraction pipeline, and a Flask route to serve stored files. - Origin behaviour matters (see issue #2): some origins 403 default clients — reuse the browser-like User-Agent and expect failures; missing images must degrade gracefully. ## Plan of attack 1. Storage layout + `stored` flag in the media schema (alembic migration if needed). 2. Downloader service wired into media extraction; failure-tolerant. 3. Serving route under the web app; template swap from remote URL to local route. 4. Tests with mocked HTTP (no live network, per project convention).
agent-pi commented 2026-09-12 09:10:26 +00:00
Author
Collaborator
Copy link

Resolved by PR #29 (merged): article images are downloaded at fetch time (browser User-Agent, raster-only whitelist excluding SVG, streamed with a hard size cap, SSRF guard with per-hop redirect re-validation) and served locally from /media/<hash> with templates preferring the local copy. Files live in a media/ directory beside the database; the media table gained local_filename/downloaded_at. Also folded in: the RenewEconomy extraction fix (images carried only in the entry summary are now discovered) and the missing get_next_aggregate_with_unread test lost in PR #28.

Resolved by PR #29 (merged): article images are downloaded at fetch time (browser User-Agent, raster-only whitelist excluding SVG, streamed with a hard size cap, SSRF guard with per-hop redirect re-validation) and served locally from `/media/<hash>` with templates preferring the local copy. Files live in a `media/` directory beside the database; the media table gained `local_filename`/`downloaded_at`. Also folded in: the RenewEconomy extraction fix (images carried only in the entry summary are now discovered) and the missing `get_next_aggregate_with_unread` test lost in PR #28.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#21
No description provided.