Phase C: Wire ML confidence scores into the FenLiu review UI #64

New issue

Closed

opened 2026-07-03 07:29:02 +00:00 by coding-agent-marvin8 · 1 comment

coding-agent-marvin8 commented

2026-07-03 07:29:02 +00:00

(Migrated from codeberg.org)

The 110-day checkpoint analysis (2026-07-03) confirmed the model is ready: 86.1% CV accuracy, +24.9 pp margin over the spam_score heuristic (which has degraded to 61.2% due to Bluesky bridge traffic). The 90-day data minimum is met.

Scope:

Alembic migration: add ml_confidence_score REAL column to posts
New MLInferenceService singleton — loads model at startup, scores posts at fetch time
Review UI: ML confidence badge per post, uncertain-zone indicator, missing-model banner
Queue: uncertainty-first sort (closest to 0.5 first), ML confidence range filter
Nightly retraining job in APScheduler (2am); artifacts picked up on daily container restart

Out of scope: per-stream model thresholds, hot-swap reload, auto-filtering.

FenLiu has a trained XGBoost classifier (Phase B complete) that predicts approval probability for Fediverse posts. Phase C wires that model into the running web app so the reviewer can see ML confidence scores alongside the existing spam score, and so the review queue can be sorted by model uncertainty. The 110-day checkpoint analysis (2026-07-03) confirmed the model is ready: 86.1% CV accuracy, +24.9 pp margin over the spam_score heuristic (which has degraded to 61.2% due to Bluesky bridge traffic). The 90-day data minimum is met. **Scope:** - Alembic migration: add `ml_confidence_score REAL` column to `posts` - New `MLInferenceService` singleton — loads model at startup, scores posts at fetch time - Review UI: ML confidence badge per post, uncertain-zone indicator, missing-model banner - Queue: uncertainty-first sort (closest to 0.5 first), ML confidence range filter - Nightly retraining job in APScheduler (2am); artifacts picked up on daily container restart **Out of scope:** per-stream model thresholds, hot-swap reload, auto-filtering.

coding-agent-marvin8 commented

2026-07-03 07:29:20 +00:00

(Migrated from codeberg.org)

Exploration findings (2026-07-03):

save_fetched_posts in services/post_storage.py is where spam scoring runs — ML scoring slots in right after calculate_spam_score on the same post_data dict.
StreamScheduler in services/scheduler.py uses APScheduler's AsyncIOScheduler — a cron job at 2am fits the existing pattern cleanly.
load_artifacts() / transform() in training/model.py and training/features.py are the inference entry points.
_render_post_row in liveviews.py:1388 renders the spam score badge — ML badge goes alongside it.
Post model has no ml_confidence_score column yet; Alembic migration needed.
Existing sort options live in liveviews.py with a guard at line 494: {"created_at", "spam_score"}.

Plan of attack:

Alembic migration — add ml_confidence_score REAL nullable column to posts, add to Post model.
MLInferenceService — new services/ml_inference.py singleton; load() at startup in main.py lifespan; graceful degradation when artifacts absent.
Wire scoring — save_fetched_posts calls ml_inference_service.score_post(post_data) and writes result to db_post.ml_confidence_score.
Review UI — missing-model banner; ML confidence badge with agree/disagree/uncertain indicators; new ml_confidence_min/ml_confidence_max filter inputs.
Queue ordering — ml_uncertainty sort option (abs(ml_confidence_score - 0.5) ASC, NULLs last); becomes default when model loaded.
Nightly retraining — _retraining_job cron at 2am in StreamScheduler; saves .prev rollback artifacts; skips if < 500 rows.

Branch: feat/phase-c-ml-integration

**Exploration findings (2026-07-03):** - `save_fetched_posts` in `services/post_storage.py` is where spam scoring runs — ML scoring slots in right after `calculate_spam_score` on the same `post_data` dict. - `StreamScheduler` in `services/scheduler.py` uses APScheduler's `AsyncIOScheduler` — a cron job at 2am fits the existing pattern cleanly. - `load_artifacts()` / `transform()` in `training/model.py` and `training/features.py` are the inference entry points. - `_render_post_row` in `liveviews.py:1388` renders the spam score badge — ML badge goes alongside it. - `Post` model has no `ml_confidence_score` column yet; Alembic migration needed. - Existing sort options live in `liveviews.py` with a guard at line 494: `{"created_at", "spam_score"}`. **Plan of attack:** 1. Alembic migration — add `ml_confidence_score REAL` nullable column to `posts`, add to `Post` model. 2. `MLInferenceService` — new `services/ml_inference.py` singleton; `load()` at startup in `main.py` lifespan; graceful degradation when artifacts absent. 3. Wire scoring — `save_fetched_posts` calls `ml_inference_service.score_post(post_data)` and writes result to `db_post.ml_confidence_score`. 4. Review UI — missing-model banner; ML confidence badge with agree/disagree/uncertain indicators; new `ml_confidence_min`/`ml_confidence_max` filter inputs. 5. Queue ordering — `ml_uncertainty` sort option (`abs(ml_confidence_score - 0.5) ASC`, NULLs last); becomes default when model loaded. 6. Nightly retraining — `_retraining_job` cron at 2am in `StreamScheduler`; saves `.prev` rollback artifacts; skips if < 500 rows. Branch: `feat/phase-c-ml-integration`

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

marvin8/dujiangyan#64

No description provided.

Rows
Columns