Replace embedding/DBSCAN aggregation with LLM-based pipeline #12

Manually merged

marvin8 merged 3 commits from feat/issue-1-llm-aggregation into main

2026-09-11 04:17:28 +00:00

agent-pi commented

2026-09-11 02:56:26 +00:00

Collaborator

What

Replace the embedding + DBSCAN article aggregation pipeline with a single cloud-LLM call per run.

Changes

Remove: EmbeddingPipeline, AggregationService, embedder.py, aggregator.py, db/embeddings.py
Remove: sentence-transformers, scikit-learn, numpy — eliminates ~42 CUDA transitive deps (~1 GB)
Add: LlmAggregationService using OpenAI-compatible /chat/completions endpoint (configured via YUNJIN_LLM_API_KEY, YUNJIN_LLM_BASE_URL, YUNJIN_LLM_MODEL)
Incremental aggregation: new articles matched against existing aggregates; stale aggregates (no new article for N days) are expired
Migration: drop embeddings table, add index on aggregates.updated_at
CLI: --refresh (was --recalculate), --aggregate-ttl-days; remove --eps and --min-samples

Tests

27 unit and integration tests for LlmAggregationService (all passing, no live API calls).

Closes #1

## What Replace the embedding + DBSCAN article aggregation pipeline with a single cloud-LLM call per run. ## Changes - **Remove:** `EmbeddingPipeline`, `AggregationService`, `embedder.py`, `aggregator.py`, `db/embeddings.py` - **Remove:** `sentence-transformers`, `scikit-learn`, `numpy` — eliminates ~42 CUDA transitive deps (~1 GB) - **Add:** `LlmAggregationService` using OpenAI-compatible `/chat/completions` endpoint (configured via `YUNJIN_LLM_API_KEY`, `YUNJIN_LLM_BASE_URL`, `YUNJIN_LLM_MODEL`) - Incremental aggregation: new articles matched against existing aggregates; stale aggregates (no new article for N days) are expired - **Migration:** drop `embeddings` table, add index on `aggregates.updated_at` - **CLI:** `--refresh` (was `--recalculate`), `--aggregate-ttl-days`; remove `--eps` and `--min-samples` ## Tests 27 unit and integration tests for `LlmAggregationService` (all passing, no live API calls). Closes #1

agent-pi added 1 commit

2026-09-11 02:56:26 +00:00

✨ Replace embedding/DBSCAN aggregation with LLM-based pipeline

/ pr-review (pull_request) Successful in 5s

Details

/ gitleaks (pull_request) Successful in 18s

Details

/ checks (pull_request) Successful in 2m42s

Details

424e5895f8

- Remove EmbeddingPipeline + AggregationService (DBSCAN)
- Remove db/embeddings.py data access layer
- Drop sentence-transformers, scikit-learn, numpy — eliminates all nvidia CUDA
  transitive deps (~42 packages, ~1 GB)
- Add LlmAggregationService using httpx calls to any OpenAI-compatible
  /chat/completions endpoint (YUNJIN_LLM_API_KEY, YUNJIN_LLM_BASE_URL,
  YUNJIN_LLM_MODEL env vars)
- Incremental aggregation: new articles matched against existing aggregates;
  stale aggregates (no new article for N days) are expired
- Migration: drop embeddings table, add index on aggregates.updated_at
- Rewrite CLI: --refresh (was --recalculate), --aggregate-ttl-days;
  remove --eps and --min-samples
- Rewrite test_aggregator.py with httpx mock fixtures

Refs #1