Aggregate summaries sometimes describe the article instead of summarising it

agent-pi commented

2026-10-06 09:58:17 +00:00

Collaborator

Some aggregate summaries contain lines that describe what kind of article it is rather than what it actually says, e.g.:

"The piece is a standalone music and culture feature rather than coverage of a news event."

Statements like this are unhelpful. The aggregate title and summary are where I go to find out what the article is about — a line that classifies or characterises the article instead of summarising its content tells me nothing I would not learn within seconds of reading it anyway.

Summaries should stick to what the article(s) actually say: no commentary on the genre, type, or newsworthiness of the piece, and no statements about what it is or is not coverage of. Just the actual summary.

Some aggregate summaries contain lines that describe what *kind* of article it is rather than what it actually says, e.g.: > "The piece is a standalone music and culture feature rather than coverage of a news event." Statements like this are unhelpful. The aggregate title and summary are where I go to find out what the article is about — a line that classifies or characterises the article instead of summarising its content tells me nothing I would not learn within seconds of reading it anyway. Summaries should stick to what the article(s) actually say: no commentary on the genre, type, or newsworthiness of the piece, and no statements about what it is or is not coverage of. Just the actual summary.

agent-pi commented

2026-10-06 23:16:41 +00:00

Author

Collaborator

Findings (prod DB copy + source, 2026-10-07)

Where the text comes from

aggregates.summary is written by the grouping LLM call in services/llm_aggregator.py (name\n\n{topic.summary} per topic). services/aggregate_summarizer.py is not involved — it's a word-frequency/snippet extractor with no production callers (tests only — also a dead-code candidate). So this is a prompt problem, not a post-processing one.

How prevalent

Across 898 prod aggregates:

187 (~21%) carry meta-subject phrasing — "coverage focuses/describes…", "reports cover…"
129 (~14%) genre-contrast — "…rather than news coverage", "rather than a single news event"
112 (~12%) genre terms — "live blog", "roundup", "opinion piece", "news event"
309 (34%) contain "standalone" — the single biggest offender

(Patterns overlap; roughly a third of all aggregates carry some classification or commentary.)

Root cause

The system prompt asks each group for "an overview of 2-3 paragraphs that condenses and paraphrases the group's articles together: what happened, the key facts, and any context or differing angles across sources."

But ~80% of aggregates hold exactly one article. With nothing to condense "together", the model fills its paragraph quota with classification and grouping rationale — the two failure modes in the issue:

Genre classification: "The piece is a standalone music and culture feature rather than coverage of a news event."
Grouping rationale leaking into prose: "The win is a standalone racing milestone, separate from other group 1 results on the same day" — that clause is the model explaining its clustering decision, mirroring the prompt's own grouping vocabulary ("same specific news story or event" / "different stories … must be in SEPARATE groups").

The "2–3 paragraphs" instruction plus the singleton-dominated group population is a padding machine.

Plan of attack (for discussion — nothing implemented)

Rewrite the overview instruction (primary fix, prompt-only):
- Keep: what happened, key facts, context.
- Forbid: describing the type/genre/format/newsworthiness of the article(s) — no "opinion piece", "roundup", "live blog", "standalone story", "news event", "advertising feature" classifications.
- Forbid: explaining or justifying the grouping (no "separate from…", "rather than coverage of…").
- Make length adaptive: one paragraph for a single article; two or three only when genuinely condensing multiple sources. This removes the padding incentive at its root.
Optional monitoring: a post-parse regex sanity check (same patterns as the prevalence count above) that logs occurrences, so we can see whether the prompt fix holds over time. Not a hard failure — a bad style check must never lose a good grouping.
Existing text: summaries regenerate on every aggregate update, so old offenders linger until their group next updates; a one-off refresh would clear them all immediately (at a full re-cluster cost — interacts with the #71 consolidation work, so probably don't).

Prompt-construction tests will need updating to assert the new instruction lines (recorded fixtures, no live calls).

## Findings (prod DB copy + source, 2026-10-07) ### Where the text comes from `aggregates.summary` is written by the **grouping LLM call** in `services/llm_aggregator.py` (`name\n\n{topic.summary}` per topic). `services/aggregate_summarizer.py` is *not* involved — it's a word-frequency/snippet extractor with no production callers (tests only — also a dead-code candidate). So this is a prompt problem, not a post-processing one. ### How prevalent Across 898 prod aggregates: - 187 (~21%) carry meta-subject phrasing — "coverage focuses/describes…", "reports cover…" - 129 (~14%) genre-contrast — "…rather than news coverage", "rather than a single news event" - 112 (~12%) genre terms — "live blog", "roundup", "opinion piece", "news event" - **309 (34%) contain "standalone"** — the single biggest offender (Patterns overlap; roughly a third of all aggregates carry some classification or commentary.) ### Root cause The system prompt asks each group for *"an overview of 2-3 paragraphs that condenses and paraphrases the group's articles together: what happened, the key facts, and any context or differing angles across sources."* But ~80% of aggregates hold exactly **one article**. With nothing to condense "together", the model fills its paragraph quota with classification and grouping rationale — the two failure modes in the issue: 1. **Genre classification**: "The piece is a standalone music and culture feature rather than coverage of a news event." 2. **Grouping rationale leaking into prose**: "The win is a standalone racing milestone, *separate from other group 1 results on the same day*" — that clause is the model explaining its clustering decision, mirroring the prompt's own grouping vocabulary ("same specific news story or event" / "different stories … must be in SEPARATE groups"). The "2–3 paragraphs" instruction plus the singleton-dominated group population is a padding machine. ### Plan of attack (for discussion — nothing implemented) 1. **Rewrite the overview instruction** (primary fix, prompt-only): - Keep: what happened, key facts, context. - Forbid: describing the type/genre/format/newsworthiness of the article(s) — no "opinion piece", "roundup", "live blog", "standalone story", "news event", "advertising feature" classifications. - Forbid: explaining or justifying the grouping (no "separate from…", "rather than coverage of…"). - Make length adaptive: one paragraph for a single article; two or three only when genuinely condensing multiple sources. This removes the padding incentive at its root. 2. **Optional monitoring**: a post-parse regex sanity check (same patterns as the prevalence count above) that logs occurrences, so we can see whether the prompt fix holds over time. Not a hard failure — a bad style check must never lose a good grouping. 3. **Existing text**: summaries regenerate on every aggregate update, so old offenders linger until their group next updates; a one-off `refresh` would clear them all immediately (at a full re-cluster cost — interacts with the #71 consolidation work, so probably don't). Prompt-construction tests will need updating to assert the new instruction lines (recorded fixtures, no live calls).

agent-pi commented

2026-10-09 20:57:15 +00:00

Author

Collaborator

Implementation plan (approved 2026-10-08)

Branch: fix/issue-70-summary-prompt

Commits:

:white_check_mark: failing prompt-construction test asserting the summary instruction bans classification/grouping-rationale phrasing and mandates adaptive length
:bug: rewrite the overview rule in the aggregation system prompt

Prompt changes (all in _SYSTEM_PROMPT, services/llm_aggregator.py):

Summaries state what happened, key facts, context/differing angles — nothing else.
Forbidden: describing the type/genre/format/newsworthiness of the article(s) — no "standalone", "live blog", "roundup", "opinion piece", "news event", "advertising feature" style classifications.
Forbidden: explaining or justifying the grouping — no "separate from…", "rather than coverage of…".
Adaptive length: one paragraph for a single-article group; two or three only when genuinely condensing multiple sources.

Sequencing: lands before the #71 rework so that branch builds on the final prompt wording. Tests use recorded fixtures only; no live LLM calls.

## Implementation plan (approved 2026-10-08) **Branch**: `fix/issue-70-summary-prompt` **Commits**: - `:white_check_mark:` failing prompt-construction test asserting the summary instruction bans classification/grouping-rationale phrasing and mandates adaptive length - `:bug:` rewrite the overview rule in the aggregation system prompt **Prompt changes** (all in `_SYSTEM_PROMPT`, `services/llm_aggregator.py`): - Summaries state what happened, key facts, context/differing angles — nothing else. - Forbidden: describing the type/genre/format/newsworthiness of the article(s) — no "standalone", "live blog", "roundup", "opinion piece", "news event", "advertising feature" style classifications. - Forbidden: explaining or justifying the grouping — no "separate from…", "rather than coverage of…". - Adaptive length: one paragraph for a single-article group; two or three only when genuinely condensing multiple sources. Sequencing: lands before the #71 rework so that branch builds on the final prompt wording. Tests use recorded fixtures only; no live LLM calls.

agent-pi referenced this issue from a commit

2026-10-09 23:38:25 +00:00

:white_check_mark: Test summary prompt bans classification and grouping rationale

agent-pi referenced this issue from a commit

2026-10-09 23:38:25 +00:00

:bug: Summaries describe content only, never classify the article

agent-pi referenced this issue from a pull request that will close it,

2026-10-09 23:38:41 +00:00

Summaries describe content only, never classify the article #77