#74 - Every aggregation run sends all existing aggregates to the LLM — prompt grows without bound - marvin8/yunjin

agent-pi commented

2026-10-06 21:26:28 +00:00

Collaborator

Each hourly aggregation run sends the user's entire set of aggregate summaries into the LLM prompt as EXISTING TOPICS. As of 7 Oct 2026 that is ~900 entries, and it grows with every aggregate created.

Consequences:

It is the dominant token cost of aggregation, paid 24 times a day and growing forever.
Matching new articles against such a large haystack is part of why match rates are poor (see the #71 investigation).

This was item 4 of the #71 plan of attack, deliberately split out: the cap design depends on what shape the candidate-topic universe has after #71's main work (consolidation window, total assignment, stable identity, TTL release) lands, and an early cap risks duplicate aggregates for older stories.

Each hourly aggregation run sends the user's entire set of aggregate summaries into the LLM prompt as EXISTING TOPICS. As of 7 Oct 2026 that is ~900 entries, and it grows with every aggregate created. Consequences: - It is the dominant token cost of aggregation, paid 24 times a day and growing forever. - Matching new articles against such a large haystack is part of why match rates are poor (see the #71 investigation). This was item 4 of the #71 plan of attack, deliberately split out: the cap design depends on what shape the candidate-topic universe has after #71's main work (consolidation window, total assignment, stable identity, TTL release) lands, and an early cap risks duplicate aggregates for older stories.

agent-pi commented

2026-10-06 21:26:49 +00:00

Author

Collaborator

Findings (from the #71 investigation, 2026-10-07 prod DB)

Every hourly run builds EXISTING TOPICS from all of the user's aggregates: 898 entries as of 7 Oct, each with name + full summary, paid 24×/day.
Aggregate count only grows in practice: ~80% of aggregates are single-article and nothing consolidates them (until the #71 work lands), so the prompt grows roughly linearly with time.
The Sep 30 from-scratch run (no existing topics, big batch) produced 0 singletons from 282 articles; incremental runs against the large topic list produce ~85% singletons — prompt size is implicated in match quality, not just cost.

Approach sketch (deferred until after #71's items 1, 2, 3, 5)

Cap EXISTING TOPICS to a candidate set — e.g. aggregates updated within the last N days plus the most recently created slice — instead of everything. Intended to land after #71's main work, because:

Sequencing: the consolidation window and TTL-release change which aggregates exist and how fresh they stay; the cap should be designed against that end state.
Regression risk: anything outside the cap can never be matched to, so an article about an older-but-live story creates a duplicate aggregate. The 7-day TTL currently bounds how old live aggregates get, which is what makes a recency-based cap viable at all — but TTL-release (item 5) weakens that bound, so the interplay needs deciding first.

## Findings (from the #71 investigation, 2026-10-07 prod DB) - Every hourly run builds EXISTING TOPICS from **all** of the user's aggregates: 898 entries as of 7 Oct, each with name + full summary, paid 24×/day. - Aggregate count only grows in practice: ~80% of aggregates are single-article and nothing consolidates them (until the #71 work lands), so the prompt grows roughly linearly with time. - The Sep 30 from-scratch run (no existing topics, big batch) produced 0 singletons from 282 articles; incremental runs against the large topic list produce ~85% singletons — prompt size is implicated in match quality, not just cost. ## Approach sketch (deferred until after #71's items 1, 2, 3, 5) Cap EXISTING TOPICS to a candidate set — e.g. aggregates updated within the last N days plus the most recently created slice — instead of everything. Intended to land **after** #71's main work, because: 1. **Sequencing**: the consolidation window and TTL-release change which aggregates exist and how fresh they stay; the cap should be designed against that end state. 2. **Regression risk**: anything outside the cap can never be matched to, so an article about an older-but-live story creates a duplicate aggregate. The 7-day TTL currently bounds how old live aggregates get, which is what makes a recency-based cap viable at all — but TTL-release (item 5) weakens that bound, so the interplay needs deciding first.

agent-pi referenced this issue

2026-10-06 21:26:49 +00:00

Ongoing stories produce multiple aggregates instead of one that updates over time #71

agent-pi commented

2026-10-11 00:35:11 +00:00

Author

Collaborator

Plan of attack (design approved 2026-10-11, cap = 100)

Findings from exploration (post-#71)

The unbounded prompt is the incremental path only. The consolidation pass already builds EXISTING TOPICS from candidates (aggregates holding at least one input article) — bounded by construction.
TTL expiry (7-day, #71) now bounds the aggregate universe at roughly “touched this week”: 590 of 898 in the 7 Oct prod snapshot. So post-TTL steady state is ~590 entries, not 898 — still far too many for match quality, and it regrows with bursty activity.
get_all_aggregates already returns most-recently-updated first, so the cap is a slice in the service layer — no db-layer change (cap selection is business logic).
Prompt entry = id + name + up to 5 recent member titles ≈ 50–80 tokens → ~30–47k tokens per hourly run at steady state.

Approach

_max_prompt_topics() helper reading YUNJIN_MAX_PROMPT_TOPICS via the existing _env_int convention (unset/empty/non-numeric/below-1 → default 100).
In aggregate()'s incremental path: slice user_aggregatesto the cap before_aggregate_prompt_payloads` — the most recently updated aggregates are kept, which is exactly the ordering the DB query already provides.
Consolidation path untouched.
Duplicate risk for cap-excluded stories is bounded: nightly consolidation re-clusters every aggregate holding an eligible (≤30d) article, so it heals duplicates unless the story is fully past the eligibility horizon — and those age out via TTL within 7 days.
TDD: monkeypatched _llm_request captures the prompt; assert over-cap users get only the K most recently updated topics; helper default/validation cases.
Docs: YUNJIN_MAX_PROMPT_TOPICS row in the LLM aggregation table of docs/config.md.

Sequence

feat/issue-74-cap-existing-topics from main
:sparkles: Cap incremental EXISTING TOPICS to the most recently updated (test + implementation)
:memo: Document YUNJIN_MAX_PROMPT_TOPICS (docs/config.md)
Full nox, push, PR into main, ci-watch

## Plan of attack (design approved 2026-10-11, cap = 100) ### Findings from exploration (post-#71) - **The unbounded prompt is the incremental path only.** The consolidation pass already builds EXISTING TOPICS from `candidates` (aggregates holding at least one input article) — bounded by construction. - **TTL expiry (7-day, #71) now bounds the aggregate universe** at roughly “touched this week”: 590 of 898 in the 7 Oct prod snapshot. So post-TTL steady state is ~590 entries, not 898 — still far too many for match quality, and it regrows with bursty activity. - **`get_all_aggregates` already returns most-recently-updated first**, so the cap is a slice in the service layer — no db-layer change (cap selection is business logic). - Prompt entry = id + name + up to 5 recent member titles ≈ 50–80 tokens → ~30–47k tokens per hourly run at steady state. ### Approach - `_max_prompt_topics()` helper reading `YUNJIN_MAX_PROMPT_TOPICS` via the existing `_env_int` convention (unset/empty/non-numeric/below-1 → default **100**). - In `aggregate()'s incremental path: slice `user_aggregates` to the cap before `_aggregate_prompt_payloads` — the most recently updated aggregates are kept, which is exactly the ordering the DB query already provides. - Consolidation path untouched. - Duplicate risk for cap-excluded stories is bounded: nightly consolidation re-clusters every aggregate holding an eligible (≤30d) article, so it heals duplicates unless the story is fully past the eligibility horizon — and those age out via TTL within 7 days. - TDD: monkeypatched `_llm_request` captures the prompt; assert over-cap users get only the K most recently updated topics; helper default/validation cases. - Docs: `YUNJIN_MAX_PROMPT_TOPICS` row in the LLM aggregation table of `docs/config.md`. ### Sequence 1. `feat/issue-74-cap-existing-topics` from `main` 2. `:sparkles: Cap incremental EXISTING TOPICS to the most recently updated` (test + implementation) 3. `:memo: Document YUNJIN_MAX_PROMPT_TOPICS` (`docs/config.md`) 4. Full `nox`, push, PR into `main`, ci-watch

agent-pi referenced this issue from a pull request that will close it,

2026-10-11 01:18:25 +00:00

Cap incremental EXISTING TOPICS to the most recently updated aggregates #80

agent-pi commented

2026-10-11 03:04:18 +00:00

Author

Collaborator

Closed by PR #80 (merged as 6b5d1cc): incremental EXISTING TOPICS now capped to the 100 most recently updated aggregates (YUNJIN_MAX_PROMPT_TOPICS); consolidation passes unaffected. Prompt cost drops ~5-8x vs post-TTL steady state.

Closed by PR #80 (merged as 6b5d1cc): incremental EXISTING TOPICS now capped to the 100 most recently updated aggregates (YUNJIN_MAX_PROMPT_TOPICS); consolidation passes unaffected. Prompt cost drops ~5-8x vs post-TTL steady state.

agent-pi closed this issue

2026-10-11 03:04:19 +00:00

agent-pi referenced this issue

2026-10-17 06:47:22 +00:00

Incremental aggregation only sees the 100 most recently updated aggregates, so ongoing stories spawn duplicates #85