Announcement/status feeds get lumped into per-feed mega-aggregates #87

New issue

Closed

opened 2026-10-17 06:47:23 +00:00 by agent-pi · 1 comment

agent-pi commented

2026-10-17 06:47:23 +00:00

Collaborator

Prod examples (created by the Oct 12 full refresh, which sends a single ~2,700-article prompt):

Ultra.cc service announcements and maintenance notices — 46 articles spanning 2023–2026 (Black Friday 2023 next to an Oct 2026 kernel upgrade)
Shock Hosting service announcements — 24 articles
Private torrent tracker open signups and status — 20 articles

They expire via the 7-day TTL, but the next full refresh recreates them. Incremental runs treat these feeds better (each new notice becomes its own singleton), so the mega-aggregates are purely a refresh artifact — but they pollute the UI for up to a week at a time.

Low-volume announcement feeds (Ultra.cc, Shock Hosting, Opentrackers, Contabo) produce topics that violate the same-specific-news-story convention: every notice from the feed ends up in one aggregate. These are status/announcement streams, not news coverage, so story-level clustering adds no value for them either way. Prod examples (created by the Oct 12 full refresh, which sends a single ~2,700-article prompt): - Ultra.cc service announcements and maintenance notices — 46 articles spanning 2023–2026 (Black Friday 2023 next to an Oct 2026 kernel upgrade) - Shock Hosting service announcements — 24 articles - Private torrent tracker open signups and status — 20 articles They expire via the 7-day TTL, but the next full refresh recreates them. Incremental runs treat these feeds better (each new notice becomes its own singleton), so the mega-aggregates are purely a refresh artifact — but they pollute the UI for up to a week at a time.

agent-pi added the

labels

2026-10-17 06:47:23 +00:00

agent-pi commented

2026-10-17 06:48:03 +00:00

Author

Collaborator

Findings from the prod DB probe (2026-10-17):

Mega-aggregates are a full-refresh artifact: the Oct 12 refresh sent ~2,700 articles in one prompt and the model dumped each low-volume feed into a single topic (Ultra.cc 46, Shock Hosting 24, Opentrackers 20 — the Ultra.cc one spans 2023–2026).
Incremental runs handle these feeds fine (each new notice becomes a singleton, e.g. Ultra.cc kernel upgrades on Oct 17), so only the refresh path needs the opt-out.
TTL (7 days) clears them, but the next refresh recreates them — a recurring weekly pollution cycle.

Plan of attack:

Add a per-feed flag (e.g. aggregate on the feeds table, exposed in the UI feed management and docs/config.md if env-var driven), defaulting to on.
Exclude non-aggregating feeds from _eligible_articles_for_user, _run_consolidation input, and refresh.
Ultra.cc / Shock Hosting / Opentrackers / Contabo are the obvious candidates to flip off in the prod config.
Key files: feeds schema + alembic migration, src/yunjin/services/llm_aggregator.py, src/yunjin/web/ (feed management), docs.

Findings from the prod DB probe (2026-10-17): - Mega-aggregates are a full-refresh artifact: the Oct 12 refresh sent ~2,700 articles in one prompt and the model dumped each low-volume feed into a single topic (Ultra.cc 46, Shock Hosting 24, Opentrackers 20 — the Ultra.cc one spans 2023–2026). - Incremental runs handle these feeds fine (each new notice becomes a singleton, e.g. Ultra.cc kernel upgrades on Oct 17), so only the refresh path needs the opt-out. - TTL (7 days) clears them, but the next refresh recreates them — a recurring weekly pollution cycle. Plan of attack: - Add a per-feed flag (e.g. aggregate on the feeds table, exposed in the UI feed management and docs/config.md if env-var driven), defaulting to on. - Exclude non-aggregating feeds from _eligible_articles_for_user, _run_consolidation input, and refresh. - Ultra.cc / Shock Hosting / Opentrackers / Contabo are the obvious candidates to flip off in the prod config. - Key files: feeds schema + alembic migration, src/yunjin/services/llm_aggregator.py, src/yunjin/web/ (feed management), docs.

agent-pi referenced this issue from a pull request that will close it,

2026-10-17 08:37:25 +00:00

Per-feed aggregation opt-out for status and announcement feeds #90

agent-pi referenced this issue from a commit

2026-10-17 09:22:37 +00:00

:white_check_mark: Pin the aggregation flag's NOT NULL DEFAULT constraint (#87)

agent-pi closed this issue

2026-10-17 09:46:56 +00:00