Incremental aggregation only sees the 100 most recently updated aggregates, so ongoing stories spawn duplicates

agent-pi commented

2026-10-17 06:47:22 +00:00

Collaborator

Each incremental aggregation run shows the LLM only the 100 most-recently-updated aggregates (the _max_prompt_topics slice in LlmAggregationService._aggregate_user, src/yunjin/services/llm_aggregator.py). With the current aggregate population (2,010 in the prod DB) that slice spans roughly 9 hours — the remaining ~1,910 aggregates are invisible to every run.

Consequence: when a story keeps developing more than a few hours after first coverage, the new article cannot be matched to its existing aggregate and a duplicate aggregate is created. The prod DB contains 111 near-duplicate aggregate-name pairs (>=60% title-token overlap, a floor estimate). Examples:

Marilyn Wallman inquest — separate aggregates on Oct 15 and Oct 16
Melbourne Star observation wheel reopening — same event, Oct 15 and Oct 16
Zero-click ad-targeting spyware — two aggregates two hours apart (Oct 16 05:13 and 07:19)
Tesla Roadster / SpaceX stunt — Oct 17 00:33 and 12:38
Beau Lamarre-Condon trial — one 16-article aggregate (Sep 30) plus two newer fragments

The cap exists to bound prompt size (#74), which is a real constraint — 2,010 topics will not fit in a prompt. The problem is that recency is a poor relevance filter: ~95% of the haystack is excluded, and the slice is partly arbitrary because daily consolidation bumps hundreds of aggregates to the same updated_at timestamp, making the top-100 tie-break arbitrary.

Each incremental aggregation run shows the LLM only the 100 most-recently-updated aggregates (the _max_prompt_topics slice in LlmAggregationService._aggregate_user, src/yunjin/services/llm_aggregator.py). With the current aggregate population (2,010 in the prod DB) that slice spans roughly 9 hours — the remaining ~1,910 aggregates are invisible to every run. Consequence: when a story keeps developing more than a few hours after first coverage, the new article cannot be matched to its existing aggregate and a duplicate aggregate is created. The prod DB contains 111 near-duplicate aggregate-name pairs (>=60% title-token overlap, a floor estimate). Examples: - Marilyn Wallman inquest — separate aggregates on Oct 15 and Oct 16 - Melbourne Star observation wheel reopening — same event, Oct 15 and Oct 16 - Zero-click ad-targeting spyware — two aggregates two hours apart (Oct 16 05:13 and 07:19) - Tesla Roadster / SpaceX stunt — Oct 17 00:33 and 12:38 - Beau Lamarre-Condon trial — one 16-article aggregate (Sep 30) plus two newer fragments The cap exists to bound prompt size (#74), which is a real constraint — 2,010 topics will not fit in a prompt. The problem is that recency is a poor relevance filter: ~95% of the haystack is excluded, and the slice is partly arbitrary because daily consolidation bumps hundreds of aggregates to the same updated_at timestamp, making the top-100 tie-break arbitrary.

agent-pi added this to the Aggregation: cross-run story dedup milestone

2026-10-17 06:47:22 +00:00

agent-pi added the

labels

2026-10-17 06:47:22 +00:00

agent-pi commented

2026-10-17 06:48:02 +00:00

Author

Collaborator

Findings from the prod DB probe (2026-10-17):

Top-100 by updated_at spans 9.1 hours; 1,910 of 2,010 aggregates are invisible to every incremental run.
The slice is partly arbitrary: the daily consolidation bump leaves hundreds of aggregates with identical updated_at timestamps, so the top-100 tie-break is effectively random within that set.

Plan of attack:

Replace the recency slice with a cheap relevance pre-filter: before the LLM call, select candidate aggregates by lexical similarity between each new article's title (and optionally teaser) and the aggregate's name / recent member titles. SQLite FTS5 or a plain token/trigram overlap query — no new dependencies, no embeddings.
Keep a small recency top-up so brand-new aggregates are always visible, plus a hard cap purely as a prompt-size safety bound (it should almost never bind).
Key files: src/yunjin/services/llm_aggregator.py (_aggregate_user, _max_prompt_topics), src/yunjin/db/aggregates.py (new candidate query), schema/migration for the FTS index.
Sequence: FTS wiring → candidate query with tests → service wiring (prompt format unchanged) → uv run nox.

Findings from the prod DB probe (2026-10-17): - Top-100 by updated_at spans 9.1 hours; 1,910 of 2,010 aggregates are invisible to every incremental run. - The slice is partly arbitrary: the daily consolidation bump leaves hundreds of aggregates with identical updated_at timestamps, so the top-100 tie-break is effectively random within that set. Plan of attack: - Replace the recency slice with a cheap relevance pre-filter: before the LLM call, select candidate aggregates by lexical similarity between each new article's title (and optionally teaser) and the aggregate's name / recent member titles. SQLite FTS5 or a plain token/trigram overlap query — no new dependencies, no embeddings. - Keep a small recency top-up so brand-new aggregates are always visible, plus a hard cap purely as a prompt-size safety bound (it should almost never bind). - Key files: src/yunjin/services/llm_aggregator.py (_aggregate_user, _max_prompt_topics), src/yunjin/db/aggregates.py (new candidate query), schema/migration for the FTS index. - Sequence: FTS wiring → candidate query with tests → service wiring (prompt format unchanged) → uv run nox.

agent-pi referenced this issue

2026-10-17 06:48:03 +00:00

Consolidation pass never merges duplicate aggregates covering the same story #86

agent-pi referenced this issue from a commit

2026-10-17 07:13:30 +00:00

:sparkles: Pre-filter existing aggregates by title relevance in incremental runs (#85)

agent-pi referenced this issue from a commit

2026-10-17 07:16:53 +00:00

:sparkles: Merge-aware consolidation: collapse duplicate topics, widen candidates (#86)

agent-pi referenced this issue from a commit

2026-10-17 07:21:39 +00:00

:arrow_up: Pin WuMing action to 0.13.0 and set a medium severity floor