Consolidation pass never merges duplicate aggregates covering the same story #86

New issue

Closed

opened 2026-10-17 06:47:22 +00:00 by agent-pi · 1 comment

agent-pi commented

2026-10-17 06:47:22 +00:00

Collaborator

The daily consolidation pass re-clusters the trailing 72h window, but two things stop it from repairing split stories:

Candidate topics are only aggregates holding at least one input article (eligible strays or articles created within the window). An aggregate whose members are all older than the window is never shown, so a late development cannot be attached to it.
The prompt frames existing topics purely as match-targets — nothing tells the model it may emit one topic that consolidates several candidate topics which all cover the same story.

Result: even when both fragments of a split story sit inside the window and both are candidates, consolidation re-creates the split. The Marilyn Wallman inquest fragments (Oct 15 + Oct 16) both fell inside the Oct 17 consolidation window and were still left as two separate aggregates.

The write path already supports merging: input articles are unlinked, re-linked per topic, and emptied aggregates are deleted. Emitting one topic that lists all of a story's article_ids under one surviving aggregate_id would merge them today.

The daily consolidation pass re-clusters the trailing 72h window, but two things stop it from repairing split stories: 1. Candidate topics are only aggregates holding at least one input article (eligible strays or articles created within the window). An aggregate whose members are all older than the window is never shown, so a late development cannot be attached to it. 2. The prompt frames existing topics purely as match-targets — nothing tells the model it may emit one topic that consolidates several candidate topics which all cover the same story. Result: even when both fragments of a split story sit inside the window and both are candidates, consolidation re-creates the split. The Marilyn Wallman inquest fragments (Oct 15 + Oct 16) both fell inside the Oct 17 consolidation window and were still left as two separate aggregates. The write path already supports merging: input articles are unlinked, re-linked per topic, and emptied aggregates are deleted. Emitting one topic that lists all of a story's article_ids under one surviving aggregate_id would merge them today.

agent-pi added this to the Aggregation: cross-run story dedup milestone

2026-10-17 06:47:22 +00:00

agent-pi added the

labels

2026-10-17 06:47:22 +00:00

agent-pi commented

2026-10-17 06:48:03 +00:00

Author

Collaborator

Findings from the prod DB probe (2026-10-17):

Same-window fragments are not merged: both Marilyn Wallman inquest aggregates (Oct 15 + Oct 16) were inside the Oct 17 consolidation window as candidates, and the pass still left them split.
The merge mechanism is already structurally supported: consolidation unlinks all input articles, re-links them per topic, and deletes emptied aggregates — so one topic listing every article of a story under one surviving aggregate_id merges the fragments today.
111 near-duplicate aggregate-name pairs in total; consolidation is the natural place to repair them.

Plan of attack:

Extend the consolidation section of the system prompt: candidate topics may be duplicates covering the same story; when they are, emit a single topic listing all their article_ids under one of their aggregate_ids.
Widen candidate selection beyond holds-an-input-article using the same lexical pre-filter as #85, so older aggregates of an ongoing story can be shown (their existing members stay untouched; only new articles attach to the survivor).
Key files: src/yunjin/services/llm_aggregator.py (_run_consolidation, _SYSTEM_PROMPT), src/yunjin/db/aggregates.py.
Pairs with #85 — same pre-filter serves both; lands in the same milestone.

Findings from the prod DB probe (2026-10-17): - Same-window fragments are not merged: both Marilyn Wallman inquest aggregates (Oct 15 + Oct 16) were inside the Oct 17 consolidation window as candidates, and the pass still left them split. - The merge mechanism is already structurally supported: consolidation unlinks all input articles, re-links them per topic, and deletes emptied aggregates — so one topic listing every article of a story under one surviving aggregate_id merges the fragments today. - 111 near-duplicate aggregate-name pairs in total; consolidation is the natural place to repair them. Plan of attack: - Extend the consolidation section of the system prompt: candidate topics may be duplicates covering the same story; when they are, emit a single topic listing all their article_ids under one of their aggregate_ids. - Widen candidate selection beyond holds-an-input-article using the same lexical pre-filter as #85, so older aggregates of an ongoing story can be shown (their existing members stay untouched; only new articles attach to the survivor). - Key files: src/yunjin/services/llm_aggregator.py (_run_consolidation, _SYSTEM_PROMPT), src/yunjin/db/aggregates.py. - Pairs with #85 — same pre-filter serves both; lands in the same milestone.

agent-pi referenced this issue from a commit

2026-10-17 07:16:53 +00:00

:sparkles: Merge-aware consolidation: collapse duplicate topics, widen candidates (#86)

agent-pi referenced this issue from a pull request that will close it,

2026-10-17 07:30:21 +00:00

Aggregate ongoing stories across runs: relevance pre-filter and merge-aware consolidation #89

agent-pi referenced this issue from a commit

2026-10-17 07:31:59 +00:00

:sparkles: Merge-aware consolidation: collapse duplicate topics, widen candidates (#86)