Aggregate ongoing stories across runs: relevance pre-filter and merge-aware consolidation #89

Manually merged

marvin8 merged 4 commits from feat/issue-85-86-cross-run-story-dedup into main 2026-10-17 08:02:23 +00:00

Conversation 1 Commits 4 Files changed 6 +616 -15

agent-pi commented 2026-10-17 07:30:21 +00:00

Collaborator

Copy link

Fixes the dominant aggregation defect measured in the prod DB (2026-10-17 probe): the same ongoing news story accumulating multiple separate aggregates — 111 near-duplicate aggregate-name pairs, with concrete splits like the Marilyn Wallman inquest, Melbourne Star reopening, zero-click spyware (two hours apart), Tesla Roadster stunt, and Beau Lamarre-Condon trial.

Two combined changes (milestone: Aggregation: cross-run story dedup):

Relevance pre-filter for incremental matching (#85)

• New aggregate_search FTS5 index (aggregate name + latest member titles), kept in sync by triggers and backfilled on upgrade — verified against the prod DB copy: 2,010/2,010 rows, and every known split-story case now surfaces its twins as top candidates.
• Incremental runs select EXISTING TOPICS by per-article bm25 relevance first; the recency-ordered remainder only fills leftover slots. The YUNJIN_MAX_PROMPT_TOPICS cap stays purely as a prompt-size bound. Previously the cap sliced the 100 most-recently-updated aggregates ≈ a 9-hour window, leaving 95% of the haystack invisible.

Merge-aware consolidation (#86)

• Consolidation candidates widen beyond holds-an-input-article via the same pre-filter, so earlier coverage of a story (members outside the 72h window) becomes attachable.
• The consolidation prompt gains merge rules — duplicate candidate topics collapse into one surviving aggregate_id; the write path already supported merges (unlink → relink → delete empties). Incremental runs keep the base system prompt byte-identical.

Also rides along: the WuMing action is pinned to 0.13.0 (latest release) with a medium severity floor.

Test-first throughout; full nox green (ty, ruff, complexipy, pysentry, pytest 3.12–3.14, deptry, zaojun).

Closes #85
Closes #86

Fixes the dominant aggregation defect measured in the prod DB (2026-10-17 probe): the same ongoing news story accumulating multiple separate aggregates — 111 near-duplicate aggregate-name pairs, with concrete splits like the Marilyn Wallman inquest, Melbourne Star reopening, zero-click spyware (two hours apart), Tesla Roadster stunt, and Beau Lamarre-Condon trial. Two combined changes (milestone: Aggregation: cross-run story dedup): **Relevance pre-filter for incremental matching (#85)** - New aggregate_search FTS5 index (aggregate name + latest member titles), kept in sync by triggers and backfilled on upgrade — verified against the prod DB copy: 2,010/2,010 rows, and every known split-story case now surfaces its twins as top candidates. - Incremental runs select EXISTING TOPICS by per-article bm25 relevance first; the recency-ordered remainder only fills leftover slots. The YUNJIN_MAX_PROMPT_TOPICS cap stays purely as a prompt-size bound. Previously the cap sliced the 100 most-recently-updated aggregates ≈ a 9-hour window, leaving 95% of the haystack invisible. **Merge-aware consolidation (#86)** - Consolidation candidates widen beyond holds-an-input-article via the same pre-filter, so earlier coverage of a story (members outside the 72h window) becomes attachable. - The consolidation prompt gains merge rules — duplicate candidate topics collapse into one surviving aggregate_id; the write path already supported merges (unlink → relink → delete empties). Incremental runs keep the base system prompt byte-identical. Also rides along: the WuMing action is pinned to 0.13.0 (latest release) with a medium severity floor. Test-first throughout; full nox green (ty, ruff, complexipy, pysentry, pytest 3.12–3.14, deptry, zaojun). Closes #85 Closes #86

agent-pi added 4 commits 2026-10-17 07:30:21 +00:00

✨ Pre-filter existing aggregates by title relevance in incremental runs (#85) ... 1a2685f16f

The incremental prompt showed only the 100 most-recently-updated
aggregates: with the prod population that is a ~9-hour window, so
95% of the haystack was invisible and any story that developed
later than that spawned a duplicate aggregate (111 near-duplicate
name pairs in the prod DB).

An aggregate_search FTS5 index (aggregate name + latest member
titles, kept in sync by triggers, backfilled on upgrade) now lets
each new article vote for its lexically relevant aggregates; the
recency-ordered remainder only fills leftover slots and the
YUNJIN_MAX_PROMPT_TOPICS cap stays as the prompt-size bound.

Closes #85

✨ Merge-aware consolidation: collapse duplicate topics, widen candidates (#86) ... bd34598edc

Consolidation could never repair split stories: the prompt framed
existing topics purely as match-targets and candidates were limited
to aggregates holding an input article, so an ongoing story whose
earlier coverage fell outside the 72h window was invisible even to
the nightly pass (both Marilyn Wallman inquest fragments sat inside
the Oct 17 window and were still left split).

The consolidation prompt now carries merge rules — duplicate topics
collapse into one surviving aggregate_id, and topics describing
earlier coverage accept new articles — while candidates widen via
the aggregate_search index from #85. The write path already
supported merges: unlink, relink, delete emptied aggregates.

Closes #86

⬆️ Pin WuMing action to 0.13.0 and set a medium severity floor ... 6947d16e33

Rides along the cross-run story dedup PR; the review floor keeps
low-severity findings out of WuMing's inline comments.

Refs #85

🐛 Strip aggregate_search FTS objects in the legacy-database migration test ... 680596b89a

The pre-settings simulation rebuilds the schema at head and rewinds
alembic_version, but a database genuinely created at 20260911 never
had the FTS triggers — leaving them in place fired
article_aggregates_fts_ai against a mid-rebuild aggregates table
when the data-ownership migration re-ran its drop-and-rename.

agent-pi force-pushed feat/issue-85-86-cross-run-story-dedup from 680596b89a to 2b981a882c 2026-10-17 07:31:59 +00:00 Compare

marvin8 approved these changes 2026-10-17 08:01:11 +00:00

marvin8 manually merged commit a947fc83c5 into main 2026-10-17 08:02:23 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

marvin8

Labels

Clear labels   

Compat/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Compat/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

marvin8/yunjin!89

No description provided.