No fetch-level dedupe: re-served or headline-updated feed items are stored as duplicate articles #88

New issue

Open

opened 2026-10-17 06:47:23 +00:00 by agent-pi · 1 comment

agent-pi commented

2026-10-17 06:47:23 +00:00

Collaborator

Some feeds re-serve items with the same content under an updated headline, or publish the same story twice, and the fetcher stores each as a new article row:

Refusing to yield momentum, ruthless Australia is perfectly set for victory — identical title twice in the prod DB, both in the same aggregate
Melbourne Star observation wheel reopens vs ...reopens in Docklands — same event, headline updated between fetches, stored as two articles; each went on to spawn its own aggregate

This double-counts articles in the UI and feeds duplicate input to the aggregator. It makes cross-run duplication worse: a headline-updated re-serve arrives as a fresh article and legitimately gets clustered on its own, invisible to the aggregate that already holds the original.

Some feeds re-serve items with the same content under an updated headline, or publish the same story twice, and the fetcher stores each as a new article row: - Refusing to yield momentum, ruthless Australia is perfectly set for victory — identical title twice in the prod DB, both in the same aggregate - Melbourne Star observation wheel reopens vs ...reopens in Docklands — same event, headline updated between fetches, stored as two articles; each went on to spawn its own aggregate This double-counts articles in the UI and feeds duplicate input to the aggregator. It makes cross-run duplication worse: a headline-updated re-serve arrives as a fresh article and legitimately gets clustered on its own, invisible to the aggregate that already holds the original.

agent-pi added the

labels

2026-10-17 06:47:23 +00:00

agent-pi commented

2026-10-17 06:48:04 +00:00

Author

Collaborator

Findings from the prod DB probe (2026-10-17):

Exact-duplicate titles exist as separate article rows (ABC double-serve); headline-updated re-serves also land as new rows (Melbourne Star). The duplicates fed the cross-run duplication in #85/#86 — a re-served story is a fresh article that gets clustered independently.
Also minor noise: both copies of one duplicate sit inside a single aggregate.

Plan of attack (TDD):

Normalized-title + feed match (and/or URL normalization) at fetch time: skip inserting when an effectively identical article from the same feed already exists. Decide the window (e.g. same feed, any time, or bounded).
Alternatively/additionally: treat it as a link-level fix — dedupe at display/aggregation input. Fetch-level dedupe is cleaner; the aggregator and the UI both benefit.
Key files: src/yunjin/services/fetcher.py, src/yunjin/db/articles.py; tests for headline-update and exact re-serve cases.
Rough sequence: failing tests for the two prod cases → dedupe in the fetch path → uv run nox.

Findings from the prod DB probe (2026-10-17): - Exact-duplicate titles exist as separate article rows (ABC double-serve); headline-updated re-serves also land as new rows (Melbourne Star). The duplicates fed the cross-run duplication in #85/#86 — a re-served story is a fresh article that gets clustered independently. - Also minor noise: both copies of one duplicate sit inside a single aggregate. Plan of attack (TDD): - Normalized-title + feed match (and/or URL normalization) at fetch time: skip inserting when an effectively identical article from the same feed already exists. Decide the window (e.g. same feed, any time, or bounded). - Alternatively/additionally: treat it as a link-level fix — dedupe at display/aggregation input. Fetch-level dedupe is cleaner; the aggregator and the UI both benefit. - Key files: src/yunjin/services/fetcher.py, src/yunjin/db/articles.py; tests for headline-update and exact re-serve cases. - Rough sequence: failing tests for the two prod cases → dedupe in the fetch path → uv run nox.