LLM refresh() silently produces an empty grouping when the model hallucinates aggregate ids #34

New issue

Closed

opened 2026-09-12 22:41:13 +00:00 by agent-pi · 2 comments

agent-pi commented 2026-09-12 22:41:13 +00:00

Collaborator

Copy link

No description provided.

agent-pi commented 2026-09-12 22:42:01 +00:00

Author

Collaborator

Copy link

Findings

Reproduced during the grouping A/B experiment (2026-09-13): refresh() clears all aggregates, but the model's response can still carry hallucinated aggregate ids in its topics. _apply_topics then:

1. "updates" aggregates that no longer exist (a no-op), and
2. links articles via add_article_to_aggregate, whose foreign-key violations are swallowed by except sqlite3.IntegrityError: pass.

Net effect: an entirely empty grouping with a success log — observed live (0 aggregates, 0 mappings after "Aggregation complete: 66 topic(s)").

Plan of attack

1. In _apply_topics, when a topic references an aggregate id, verify it exists; if it does not, log a warning and create a new aggregate instead (degrade gracefully, never drop the articles).
2. Log swallowed IntegrityError link failures instead of passing silently.
3. Tests: hallucinated id degrades to create; existing id still updates.

Stacked on PR #32 (touches llm_aggregator code introduced there). Branch: fix/issue-34-refresh-robustness. Commit: :bug: Degrade gracefully when the LLM invents aggregate ids.

## Findings Reproduced during the grouping A/B experiment (2026-09-13): `refresh()` clears all aggregates, but the model's response can still carry **hallucinated aggregate ids** in its topics. `_apply_topics` then: 1. "updates" aggregates that no longer exist (a no-op), and 2. links articles via `add_article_to_aggregate`, whose foreign-key violations are swallowed by `except sqlite3.IntegrityError: pass`. Net effect: an entirely empty grouping with a success log — observed live (0 aggregates, 0 mappings after "Aggregation complete: 66 topic(s)"). ## Plan of attack 1. In `_apply_topics`, when a topic references an aggregate id, verify it exists; if it does not, log a warning and **create a new aggregate** instead (degrade gracefully, never drop the articles). 2. Log swallowed `IntegrityError` link failures instead of passing silently. 3. Tests: hallucinated id degrades to create; existing id still updates. Stacked on PR #32 (touches `llm_aggregator` code introduced there). Branch: `fix/issue-34-refresh-robustness`. Commit: `:bug: Degrade gracefully when the LLM invents aggregate ids`.

agent-pi referenced this issue from a pull request that will close it, 2026-09-12 22:50:56 +00:00

Harden LLM refresh against invented ids; group on teasers #35

agent-pi commented 2026-09-12 23:43:30 +00:00

Author

Collaborator

Copy link

Implemented in PR #35 (merged as e471b34): _apply_topics now verifies referenced aggregate ids exist — invented ids degrade to creating a new aggregate (articles never dropped), and skipped article links are logged instead of silently passing. Grouping input reverted to feed teasers per the A/B on issue #23 (full text stays in the summarizer and article display).

Implemented in PR #35 (merged as e471b34): `_apply_topics` now verifies referenced aggregate ids exist — invented ids degrade to creating a new aggregate (articles never dropped), and skipped article links are logged instead of silently passing. Grouping input reverted to feed teasers per the A/B on issue #23 (full text stays in the summarizer and article display).

agent-pi closed this issue 2026-09-12 23:43:30 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

0.4.0

0.3.0

0.2.0

0.1.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

marvin8/yunjin#34

No description provided.