marvin8/yunjin
Watch
1
0
Fork
You've already forked yunjin
 0
Code Issues 7 Pull requests Releases Activity Actions

Revive yunjin: replace embedding/DBSCAN aggregation with a cloud-LLM aggregation service #1

New issue
Closed
opened 2026-09-10 03:02:46 +00:00 by agent-pi · 2 comments
agent-pi commented 2026-09-10 03:02:46 +00:00
Collaborator
Copy link

Problem

Yunjin was abandoned at v0.4.0 because aggregation quality never reached a useful level:

  • Cross-feed stories were almost never grouped. The same event covered by ABC News and The Guardian ended up as separate entries — at eps=0.45 nearly half of all articles fell out as noise singletons.
  • Clustering was brittle. One hand-tuned eps value had to work across every topic, and recurring feed series (release announcements, status incidents, podcast episodes) fragmented or merged unpredictably.
  • Aggregates had no real titles or summaries. The "title" was simply the first article's headline; the "summary" was concatenated 150-character content snippets with HTML stripped.

Cheap cloud LLMs (DeepSeek, Moonshot) can now make the grouping and summarisation judgements directly, without local model tuning or distance-threshold fiddling. A series of probes against real feed data (findings in the follow-up comment) showed this approach is viable at hobby scale and cost.

Goal

Re-attempt the project's core goal — high-quality topic aggregation — using a cloud LLM as the aggregation engine:

  1. Full-batch grouping of fetched articles into aggregates.
  2. A real title and 2–3 sentence summary per aggregate.
  3. Incremental filing of newly fetched articles into existing aggregates (the daily-reader workload), with summary updates when a new article changes the story.
## Problem Yunjin was abandoned at v0.4.0 because aggregation quality never reached a useful level: - **Cross-feed stories were almost never grouped.** The same event covered by ABC News and The Guardian ended up as separate entries — at `eps=0.45` nearly half of all articles fell out as noise singletons. - **Clustering was brittle.** One hand-tuned `eps` value had to work across every topic, and recurring feed series (release announcements, status incidents, podcast episodes) fragmented or merged unpredictably. - **Aggregates had no real titles or summaries.** The "title" was simply the first article's headline; the "summary" was concatenated 150-character content snippets with HTML stripped. Cheap cloud LLMs (DeepSeek, Moonshot) can now make the grouping and summarisation judgements directly, without local model tuning or distance-threshold fiddling. A series of probes against real feed data (findings in the follow-up comment) showed this approach is viable at hobby scale and cost. ## Goal Re-attempt the project's core goal — high-quality topic aggregation — using a cloud LLM as the aggregation engine: 1. Full-batch grouping of fetched articles into aggregates. 2. A real title and 2–3 sentence summary per aggregate. 3. Incremental filing of newly fetched articles into existing aggregates (the daily-reader workload), with summary updates when a new article changes the story.
agent-pi commented 2026-09-10 03:03:37 +00:00
Author
Collaborator
Copy link

Probe findings (Sept 2026)

Real data: 184 articles — first page of all 16 feeds from my FreshRSS OPML (news, renewables, EV, status, release, podcast and one personal Nextcloud activity feed; FreshRSS read-filters deliberately NOT applied, so the noise is real).

Probe 1 — full batch, deepseek-v4-pro (thinking enabled, high effort)

  • One call; 13.4k prompt / 28.3k completion tokens.
  • 72 groups; 184/184 articles assigned exactly once, zero duplicates, zero hallucinated ids.
  • Caught both genuine cross-feed stories (Moira Deeming joins Family First × ABC+Guardian; home-energy "leaky homes" × One Step Off The Grid + RenewEconomy) plus a podcast series spanning two feeds.
  • All 13 recurring series fully collapsed (Manjaro updates 14→1, Claude status 15→1, Nextcloud file-activity noise 11→1, FreshRSS releases 10→1).
  • Correct edge judgement: a Manjaro CVE alert kept OUT of the routine stable-update series.
  • Every aggregate got a real title and a readable 2–3 sentence summary.

Baseline — existing pipeline on identical data

  • all-MiniLM-L6-v2 + DBSCAN eps=0.45: 14 clusters + 79 noise singletons (~43% ungrouped).
  • Zero cross-feed stories found; the Deeming articles both landed in noise.
  • Series collapse partial/fragmented; one 8-article cluster lumped two different podcast series together with unrelated energy stories.

Probe 2 — same payload, deepseek-v4-flash

  • Quality equivalent to pro on every ground truth; slightly more topical merging (all defensible).
  • 20.8k completion tokens vs pro's 28.3k (~27% cheaper).
  • Conclusion: flash is sufficient — production can use the cheap model.

Probe 3 — incremental assignment (the daily-reader workload)

  • 20 genuinely new articles + the 67 existing aggregates: 5.3k prompt / 7.9k completion (~40% of full-batch cost).
  • Both new cross-feed stories caught (FitzSimons at the Alan Jones trial × ABC+Guardian; Lambie/Hanson Hastie cartoon × ABC+Guardian).
  • Correct filing into existing aggregates with revised summaries: Instagram/algorithm article → existing algorithm-regulation topic; RenewEconomy hydro-fire piece → converted an ABC single into a cross-feed story.
  • Correct restraint on lookalikes: NSW koala carbon credits NOT merged with the Victorian koala-health single; SA premier's immigration story NOT merged into the Deeming live-blog group.

API facts (verified against api-docs.deepseek.com)

  • OpenAI-compatible: POST https://api.deepseek.com/chat/completions
  • Models: deepseek-v4-pro, deepseek-v4-flash (plus an experimental vision variant)
  • Auth: Authorization: Bearer $DEEPSEEK_API_KEY
  • Extras used: "thinking": {"type": "enabled"}, "reasoning_effort": "high"
  • Their docs list a "JSON Output" mode, but the guide page failed to load during the probe, so the prompt simply demands raw JSON — schema compliance was perfect across all three runs.

Plan of attack

  1. New LLMAggregationService in src/yunjin/services/, alongside the existing AggregationService (which stays untouched until the new path proves itself).
  2. Full-batch mode: article list (id, feed, title, snippet) → JSON {groups: [{title, summary, kind, article_ids}]} → persist via existing db/aggregates.py.
  3. Incremental mode: existing aggregates (id, kind, title, summary) + new articles → {assignments, new_groups} with optional updated_summary.
  4. Config: provider base URL, model, API key from environment, thinking/effort flags — keep it provider-agnostic (DeepSeek and Moonshot both speak OpenAI-style chat completions).
  5. Validation layer: parse/verify JSON, coverage check (every article exactly once), retry-with-repair on schema failure.
  6. Aggregate archival/pruning policy so the incremental prompt doesn't grow unboundedly over time.

Key files: new src/yunjin/services/llm_aggregator.py, read-side additions in src/yunjin/db/aggregates.py, settings plumbing, tests with recorded API fixtures (no live calls in the test suite).

## Probe findings (Sept 2026) Real data: 184 articles — first page of all 16 feeds from my FreshRSS OPML (news, renewables, EV, status, release, podcast and one personal Nextcloud activity feed; FreshRSS read-filters deliberately NOT applied, so the noise is real). ### Probe 1 — full batch, `deepseek-v4-pro` (thinking enabled, high effort) - One call; 13.4k prompt / 28.3k completion tokens. - **72 groups; 184/184 articles assigned exactly once**, zero duplicates, zero hallucinated ids. - Caught both genuine cross-feed stories (Moira Deeming joins Family First × ABC+Guardian; home-energy "leaky homes" × One Step Off The Grid + RenewEconomy) plus a podcast series spanning two feeds. - All 13 recurring series fully collapsed (Manjaro updates 14→1, Claude status 15→1, Nextcloud file-activity noise 11→1, FreshRSS releases 10→1). - Correct edge judgement: a Manjaro CVE alert kept OUT of the routine stable-update series. - Every aggregate got a real title and a readable 2–3 sentence summary. ### Baseline — existing pipeline on identical data - all-MiniLM-L6-v2 + DBSCAN `eps=0.45`: **14 clusters + 79 noise singletons (~43% ungrouped)**. - Zero cross-feed stories found; the Deeming articles both landed in noise. - Series collapse partial/fragmented; one 8-article cluster lumped two different podcast series together with unrelated energy stories. ### Probe 2 — same payload, `deepseek-v4-flash` - Quality equivalent to pro on every ground truth; slightly more topical merging (all defensible). - 20.8k completion tokens vs pro's 28.3k (~27% cheaper). - **Conclusion: flash is sufficient — production can use the cheap model.** ### Probe 3 — incremental assignment (the daily-reader workload) - 20 genuinely new articles + the 67 existing aggregates: 5.3k prompt / 7.9k completion (~40% of full-batch cost). - Both new cross-feed stories caught (FitzSimons at the Alan Jones trial × ABC+Guardian; Lambie/Hanson Hastie cartoon × ABC+Guardian). - Correct filing into existing aggregates **with revised summaries**: Instagram/algorithm article → existing algorithm-regulation topic; RenewEconomy hydro-fire piece → converted an ABC single into a cross-feed story. - Correct restraint on lookalikes: NSW koala carbon credits NOT merged with the Victorian koala-health single; SA premier's immigration story NOT merged into the Deeming live-blog group. ### API facts (verified against api-docs.deepseek.com) - OpenAI-compatible: `POST https://api.deepseek.com/chat/completions` - Models: `deepseek-v4-pro`, `deepseek-v4-flash` (plus an experimental vision variant) - Auth: `Authorization: Bearer $DEEPSEEK_API_KEY` - Extras used: `"thinking": {"type": "enabled"}`, `"reasoning_effort": "high"` - Their docs list a "JSON Output" mode, but the guide page failed to load during the probe, so the prompt simply demands raw JSON — schema compliance was perfect across all three runs. ## Plan of attack 1. New `LLMAggregationService` in `src/yunjin/services/`, alongside the existing `AggregationService` (which stays untouched until the new path proves itself). 2. Full-batch mode: article list (id, feed, title, snippet) → JSON `{groups: [{title, summary, kind, article_ids}]}` → persist via existing `db/aggregates.py`. 3. Incremental mode: existing aggregates (id, kind, title, summary) + new articles → `{assignments, new_groups}` with optional `updated_summary`. 4. Config: provider base URL, model, API key from environment, thinking/effort flags — keep it provider-agnostic (DeepSeek and Moonshot both speak OpenAI-style chat completions). 5. Validation layer: parse/verify JSON, coverage check (every article exactly once), retry-with-repair on schema failure. 6. Aggregate archival/pruning policy so the incremental prompt doesn't grow unboundedly over time. Key files: new `src/yunjin/services/llm_aggregator.py`, read-side additions in `src/yunjin/db/aggregates.py`, settings plumbing, tests with recorded API fixtures (no live calls in the test suite).
agent-pi commented 2026-09-11 02:33:37 +00:00
Author
Collaborator
Copy link

Implementation plan

The current EmbeddingPipeline (sentence-transformers) + AggregationService (DBSCAN) is replaced with a single LlmAggregationService. One OpenAI-compatible API call per fetch-run. Articles are grouped incrementally: new articles are matched against existing aggregates or create new ones. Aggregates with no new articles for 7 days are expired.

Key design decisions

  • One LLM call per run — prompt includes existing aggregate summaries (for matching) + new articles. LLM responds with { topics: [{ name, summary, article_ids, aggregate_id }] }.
  • Summary is a 1-2 sentence description, not just a title.
  • Staleness: 7 days — configurable via --aggregate-ttl-days.
  • --refresh replaces --recalculate (same behaviour, clearer name).
  • Drop embeddings table — no longer needed. New migration drops it and adds an index on aggregates.updated_at.
  • Drop sentence-transformers, scikit-learn, numpy — removes torch and 20+ nvidia CUDA packages from the dependency tree.

Files
New services/llm_aggregator.py
Deleted services/embedder.py, services/aggregator.py, db/embeddings.py
Modified __main__.py (CLI flags), pyproject.toml (remove deps), alembic/ (migration), tests/test_aggregator.py (full rewrite)

Commits (planned)

  1. :wrench: Drop embeddings table via migration, add updated_at index
  2. :sparkles: Add LlmAggregationService with httpx2 client
  3. :wrench: Update CLI: --refresh, --aggregate-ttl-days, remove DBSCAN flags
  4. :fire: Remove embedder.py and aggregator.py
  5. :fire: Remove db/embeddings.py
  6. :fire: Remove sentence-transformers, scikit-learn, numpy from pyproject.toml
  7. :recycle: Rewrite tests/test_aggregator.py for LLM path
  8. :white_check_mark: Add recorded httpx fixtures for LLM responses

Branch

feat/issue-1-llm-aggregation

Full design spec: .superpowers/specs/2026-09-11-llm-aggregation-design.md

## Implementation plan The current `EmbeddingPipeline` (sentence-transformers) + `AggregationService` (DBSCAN) is replaced with a single `LlmAggregationService`. One OpenAI-compatible API call per fetch-run. Articles are grouped incrementally: new articles are matched against existing aggregates or create new ones. Aggregates with no new articles for 7 days are expired. ### Key design decisions - **One LLM call per run** — prompt includes existing aggregate summaries (for matching) + new articles. LLM responds with `{ topics: [{ name, summary, article_ids, aggregate_id }] }`. - **Summary is a 1-2 sentence description**, not just a title. - **Staleness: 7 days** — configurable via `--aggregate-ttl-days`. - **`--refresh` replaces `--recalculate`** (same behaviour, clearer name). - **Drop `embeddings` table** — no longer needed. New migration drops it and adds an index on `aggregates.updated_at`. - **Drop sentence-transformers, scikit-learn, numpy** — removes torch and 20+ nvidia CUDA packages from the dependency tree. ### Files | | | |---|---| | **New** | `services/llm_aggregator.py` | | **Deleted** | `services/embedder.py`, `services/aggregator.py`, `db/embeddings.py` | | **Modified** | `__main__.py` (CLI flags), `pyproject.toml` (remove deps), `alembic/` (migration), `tests/test_aggregator.py` (full rewrite) | ### Commits (planned) 1. `:wrench: Drop embeddings table via migration, add updated_at index` 2. `:sparkles: Add LlmAggregationService with httpx2 client` 3. `:wrench: Update CLI: --refresh, --aggregate-ttl-days, remove DBSCAN flags` 4. `:fire: Remove embedder.py and aggregator.py` 5. `:fire: Remove db/embeddings.py` 6. `:fire: Remove sentence-transformers, scikit-learn, numpy from pyproject.toml` 7. `:recycle: Rewrite tests/test_aggregator.py for LLM path` 8. `:white_check_mark: Add recorded httpx fixtures for LLM responses` ### Branch `feat/issue-1-llm-aggregation` Full design spec: `.superpowers/specs/2026-09-11-llm-aggregation-design.md`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
0.4.0
0.3.0
0.2.0
0.1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/yunjin#1
No description provided.