Ongoing stories produce multiple aggregates instead of one that updates over time

agent-pi commented

2026-10-06 10:07:57 +00:00

Collaborator

Some ongoing news stories end up spread across multiple aggregates instead of forming one aggregate that keeps updating. Three recent examples:

A number of articles about Beau Lamar did not all get grouped together.
The airline pilot stabbed mid-air, apparently by his co-pilot, produced multiple separate aggregates over the last few days.
Alan Jones is currently in court on sex offence charges: many articles, many aggregates — effectively one per day (day 1, day 2, day 3 of coverage). That feels like it should be a single "Alan Jones court case" aggregate that gets updated as new articles arrive.

The common thread: a story that develops over days — court cases, ongoing incidents — should end up as one aggregate that accumulates later articles, not a fresh aggregate per batch of articles or per day of coverage.

Some ongoing news stories end up spread across multiple aggregates instead of forming one aggregate that keeps updating. Three recent examples: - A number of articles about Beau Lamar did not all get grouped together. - The airline pilot stabbed mid-air, apparently by his co-pilot, produced multiple separate aggregates over the last few days. - Alan Jones is currently in court on sex offence charges: many articles, many aggregates — effectively one per day (day 1, day 2, day 3 of coverage). That feels like it should be a single "Alan Jones court case" aggregate that gets updated as new articles arrive. The common thread: a story that develops over days — court cases, ongoing incidents — should end up as one aggregate that accumulates later articles, not a fresh aggregate per batch of articles or per day of coverage.

agent-pi commented

2026-10-06 20:56:26 +00:00

Author

Collaborator

Investigation findings (prod DB copy + `llm_aggregator.py`)

Investigated against a fresh prod DB copy (taken 2026-10-07) and the incremental pipeline in src/yunjin/services/llm_aggregator.py.

Data first — corrections to two examples in the issue body

Against prod data, two of the three examples do not show fragmentation:

Alan Jones: exactly one aggregate (#6, "Alan Jones criminal trial: final complainant evidence") holds all 5 trial articles, Sep 24 → Oct 6. Incremental matching worked here — nearly every teaser names the story, so matching is trivial.
Beau Lamarre-Condon: all 11 matching articles sit in one aggregate (#5), Sep 28 → Oct 6. Also fine on prod.

If those read as "a new aggregate per day" in the UI, two other defects plausibly create that impression (both real, below): the aggregate's name is rewritten on every update, so the same group reappears daily under a new title ("…athlete takes witness box" → "…final complainant evidence"), indistinguishable from a fresh aggregate; and the instance on this NUC (port 8080) has had a dead LLM key since Oct 3 (401 on every hourly run), so everything since then is unaggregated there. If any reading was done on that instance, it would show exactly this kind of breakage. (Key renewal is ops, separate from this issue.)

The real fragmentation signal

The pilot story is split on prod: aggregate #124 "Flydubai FZ1073 emergency landing heroism awards" (Sep 30) vs #256 "Stabbed pilot Captain Smit Machchhar attack" (Oct 1) — same incident, two groups created a day apart. #124's summary had drifted to the awards/misinformation sub-aspect; the Oct 1 article about the pilot's background didn't "clearly belong", so a new group formed. Once #256 existed with a name anchored on the attack, later articles matched into it fine (Oct 2–4 all landed there).
Singleton explosion: 719 of 898 aggregates (80%) hold exactly one article; 2 hold zero. Contrast: the Sep 30 from-scratch run clustered 282 articles into 28 aggregates with zero singletons. Every incremental run since Oct 1 creates ~80–85% singletons (Oct 6: 221 created, 178 singletons).
Mechanism: articles arrive in small hourly batches (~6–17) and are matched against an ever-growing EXISTING TOPICS list — now 898 name+summary entries in every prompt. With thin summaries and the system prompt's conservative rule ("create a new group if an article does not fit any existing group well"), the model creates a new group for almost everything. A developing story with 2–3 articles a day spread across hourly runs then produces one thin singleton per batch — the "effectively one per day" pattern — and nothing ever consolidates them afterwards.
Stranded articles: 169 of 1,678 articles inserted before the latest watermark have no aggregate at all. The prompt does not require the LLM to assign every article, and the watermark (max(updated_at) over the user's aggregates) means an unassigned article is never reconsidered — stranding is permanent.
TTL expiry (7 days) is a continuity risk for slow stories: _expire_stale deletes aggregates not updated for 7 days, and their article links go with them. Those articles are then permanently unaggregated (the watermark has moved past them). Too early to observe in prod (DB starts Sep 30), but structural in code.

Root cause

The incremental match step is the weak link, not the design concept: matching tiny batches against a huge, summary-drifting topic list fails silently, and no pass ever consolidates the singletons it leaves behind. From-scratch clustering (same LLM, same prompt shape) demonstrably groups well.

Plan of attack (for discussion — nothing implemented)

Consolidation window — aggregate over a rolling window (e.g. last 48–72 h, already-aggregated articles included) so strays from earlier hourly batches meet each other; or a nightly bounded re-cluster of the trailing 7 days to merge singletons.
Require total assignment — prompt change: every new article must appear in exactly one topic (existing or new); no silent stranding.
Stable aggregate identity — stop rewriting the name on update (keep the creation-time name; let summary text evolve), and include each existing topic's key article titles in EXISTING TOPICS so matching has stable anchors instead of drifted summaries.
Cap the prompt — 898 topics in every hourly prompt is a cost and quality problem; e.g. only recently-updated topics plus a bounded recency slice.
TTL must not strand — when an aggregate expires, release its articles back into the eligible pool instead of deleting their links, so a story that resumes after a gap can re-form.

## Investigation findings (prod DB copy + `llm_aggregator.py`) Investigated against a fresh prod DB copy (taken 2026-10-07) and the incremental pipeline in `src/yunjin/services/llm_aggregator.py`. ### Data first — corrections to two examples in the issue body Against prod data, two of the three examples do **not** show fragmentation: - **Alan Jones**: exactly one aggregate (#6, "Alan Jones criminal trial: final complainant evidence") holds all 5 trial articles, Sep 24 → Oct 6. Incremental matching worked here — nearly every teaser names the story, so matching is trivial. - **Beau Lamarre-Condon**: all 11 matching articles sit in one aggregate (#5), Sep 28 → Oct 6. Also fine on prod. If those read as "a new aggregate per day" in the UI, two other defects plausibly create that impression (both real, below): the aggregate's **name is rewritten on every update**, so the same group reappears daily under a new title ("…athlete takes witness box" → "…final complainant evidence"), indistinguishable from a fresh aggregate; and the instance on this NUC (port 8080) has had a dead LLM key since Oct 3 (401 on every hourly run), so everything since then is unaggregated there. If any reading was done on that instance, it would show exactly this kind of breakage. (Key renewal is ops, separate from this issue.) ### The real fragmentation signal - **The pilot story is split on prod**: aggregate #124 "Flydubai FZ1073 emergency landing heroism awards" (Sep 30) vs #256 "Stabbed pilot Captain Smit Machchhar attack" (Oct 1) — same incident, two groups created a day apart. #124's summary had drifted to the awards/misinformation *sub-aspect*; the Oct 1 article about the pilot's background didn't "clearly belong", so a new group formed. Once #256 existed with a name anchored on the attack, later articles matched into it fine (Oct 2–4 all landed there). - **Singleton explosion**: 719 of 898 aggregates (80%) hold exactly one article; 2 hold zero. Contrast: the Sep 30 from-scratch run clustered 282 articles into 28 aggregates with **zero** singletons. Every incremental run since Oct 1 creates ~80–85% singletons (Oct 6: 221 created, 178 singletons). - **Mechanism**: articles arrive in small hourly batches (~6–17) and are matched against an ever-growing EXISTING TOPICS list — now 898 name+summary entries in every prompt. With thin summaries and the system prompt's conservative rule ("create a new group if an article does not fit any existing group well"), the model creates a new group for almost everything. A developing story with 2–3 articles a day spread across hourly runs then produces one thin singleton per batch — the "effectively one per day" pattern — and nothing ever consolidates them afterwards. - **Stranded articles**: 169 of 1,678 articles inserted before the latest watermark have no aggregate at all. The prompt does not require the LLM to assign every article, and the watermark (`max(updated_at)` over the user's aggregates) means an unassigned article is never reconsidered — stranding is permanent. - **TTL expiry (7 days) is a continuity risk for slow stories**: `_expire_stale` deletes aggregates not updated for 7 days, and their article links go with them. Those articles are then permanently unaggregated (the watermark has moved past them). Too early to observe in prod (DB starts Sep 30), but structural in code. ### Root cause The incremental match step is the weak link, not the design concept: matching tiny batches against a huge, summary-drifting topic list fails silently, and no pass ever consolidates the singletons it leaves behind. From-scratch clustering (same LLM, same prompt shape) demonstrably groups well. ### Plan of attack (for discussion — nothing implemented) 1. **Consolidation window** — aggregate over a rolling window (e.g. last 48–72 h, already-aggregated articles included) so strays from earlier hourly batches meet each other; or a nightly bounded re-cluster of the trailing 7 days to merge singletons. 2. **Require total assignment** — prompt change: every new article must appear in exactly one topic (existing or new); no silent stranding. 3. **Stable aggregate identity** — stop rewriting the name on update (keep the creation-time name; let summary text evolve), and include each existing topic's key article titles in EXISTING TOPICS so matching has stable anchors instead of drifted summaries. 4. **Cap the prompt** — 898 topics in every hourly prompt is a cost *and* quality problem; e.g. only recently-updated topics plus a bounded recency slice. 5. **TTL must not strand** — when an aggregate expires, release its articles back into the eligible pool instead of deleting their links, so a story that resumes after a gap can re-form.

agent-pi referenced this issue

2026-10-06 21:26:28 +00:00

Every aggregation run sends all existing aggregates to the LLM — prompt grows without bound #74

agent-pi referenced this issue

2026-10-06 21:26:49 +00:00

Every aggregation run sends all existing aggregates to the LLM — prompt grows without bound #74

agent-pi commented

2026-10-06 21:26:49 +00:00

Author

Collaborator

Decision recorded (2026-10-07)

Items 1, 2, 3 and 5 from the plan of attack will be implemented as one piece of work on this issue:

Consolidation window — rolling re-aggregation so strays from earlier batches meet each other
Total assignment — every article lands in exactly one topic; no permanent stranding
Stable aggregate identity — creation-time name kept on update; member article titles shown to the matcher
TTL must not strand — expired aggregates release their articles back into the eligible pool

Item 4 (cap the prompt) is split out to #74 and deferred until after this work, since the cap design depends on the end-state topic universe and has an old-story duplicate risk (see #74 for sequencing).

Implementation has not started; this comment records scope only.

## Decision recorded (2026-10-07) Items **1, 2, 3 and 5** from the plan of attack will be implemented as **one piece of work on this issue**: 1. Consolidation window — rolling re-aggregation so strays from earlier batches meet each other 2. Total assignment — every article lands in exactly one topic; no permanent stranding 3. Stable aggregate identity — creation-time name kept on update; member article titles shown to the matcher 5. TTL must not strand — expired aggregates release their articles back into the eligible pool Item **4** (cap the prompt) is split out to **#74** and deferred until after this work, since the cap design depends on the end-state topic universe and has an old-story duplicate risk (see #74 for sequencing). Implementation has **not** started; this comment records scope only.

agent-pi referenced this issue

2026-10-06 23:16:41 +00:00

Aggregate summaries sometimes describe the article instead of summarising it #70

agent-pi referenced this issue

2026-10-09 20:57:15 +00:00

Aggregate summaries sometimes describe the article instead of summarising it #70