Remove XGBoost ML + spam scorer; add image-aware AI curation #93

Manually merged

marvin8 merged 33 commits from feat/issue-80-collect into main 2026-08-15 00:20:19 +00:00

Conversation 9 Commits 33 Files changed 109 +2025 -9420

agent-pi commented 2026-08-14 04:35:14 +00:00

Collaborator

Copy link

Collecting branch for the AI-curation rework: replaces Fenliu's XGBoost ML model and rule-based spam scorer with a two-stage AI classifier that checks post text for promotion and the image for a real cat.

Sub-PRs merged into this branch:

• PR #81 — remove XGBoost ML + spam scorer; add AI text/vision classifiers
• PR #86 — inline vision classification instead of the Batch API
• PR #88 — review filter fixes
• PR #89 — reconcile models with DB + fix test-isolation flakiness
• PR #90 — rename reviewer_notes to auto_reject_reason
• PR #92 — remove stale review.html template
• PR #95 — wire AI auto-reject into the fetch flow
• PR #98 — classify posts on manual fetch paths

Closes #80
Closes #82
Closes #83
Closes #84
Closes #85
Closes #87
Closes #91
Closes #94
Closes #96

Collecting branch for the AI-curation rework: replaces Fenliu's XGBoost ML model and rule-based spam scorer with a two-stage AI classifier that checks post text for promotion and the image for a real cat. Sub-PRs merged into this branch: - PR #81 — remove XGBoost ML + spam scorer; add AI text/vision classifiers - PR #86 — inline vision classification instead of the Batch API - PR #88 — review filter fixes - PR #89 — reconcile models with DB + fix test-isolation flakiness - PR #90 — rename `reviewer_notes` to `auto_reject_reason` - PR #92 — remove stale `review.html` template - PR #95 — wire AI auto-reject into the fetch flow - PR #98 — classify posts on manual fetch paths Closes #80 Closes #82 Closes #83 Closes #84 Closes #85 Closes #87 Closes #91 Closes #94 Closes #96

agent-pi added 25 commits 2026-08-14 04:35:14 +00:00

🔧 point source references at forge.marvin8.zone 2d481d3343

🔥 remove XGBoost ML and spam scorer 31c716b3ea

✨ add AI classification settings fdcec0f173

✨ add AI text and vision classifiers c65e1a63be

🗃️ add AI classification columns 02a2897cc0

✨ add overnight batch vision processor 2b3b50473b

✨ wire AI classification into scheduler 80d22ca33c

✨ add AI badge and filters to review UI 5246c44317

📝 document AI classification and update release notes c7cb138ef4

📝 track stale review.html cleanup in TODO fd7669a0eb

🔀 Merge branch 'feat/issue-80-remove-ml-add-vision-ai' into feat/issue-80-collect 17bc44c5a7

♻️ classify images inline instead of via batch API 46ae38a906

🐛 set SECRET_KEY in AI tests to fix flaky app startup 0bf8f26c19

🔀 Merge branch 'feat/issue-84-inline-vision' into feat/issue-80-collect fe019862da

🐛 fix AI review filters applying and tidy filter bar f06605d2cd

🐛 fix checkbox filter value parsing and dropdown width a56bd6799e

🔀 Merge branch 'feat/issue-87-review-filter-fixes' into feat/issue-80-collect f2ef56af0e

🗃️ reconcile models with DB and drop legacy api_key d87c9406f3

✅ add schema-reconciliation metadata test 801124a4f5

🐛 fix flaky tests: per-request ContextVar overrides replace patch() isolation ... b22895bfa9

App-level tests patched get_db across a module list with process-global
patch(), which tryke's single-process junit reporter exposed as
order-dependent flakiness against the shared module-level in-memory
engine. Replace it with a TestAppWrapper that injects _db_session_override
and _auth_override per request, which get_db() and the UI auth middleware
already consult. Also widen isolated in-memory DB name entropy and drop
test_ui_auth's module-level create_all on the shared engine.

🔀 Merge branch 'feat/issue-83-alembic-drift' into feat/issue-80-collect 08ede21838

♻️ rename reviewer_notes to auto_reject_reason and drop duplicate ... 8ab5cc82ef

posts.reviewer_notes was written only by reject_blocked_posts, and
review_feedback.reviewer_notes was a never-read duplicate. Rename the
post field to auto_reject_reason, drop the feedback column, remove the
vestigial PATCH input field, and update the migration, tests, and docs.

🔀 Merge branch 'feat/issue-85-auto-reject-reason' into feat/issue-80-collect ac71032a9b

🔥 remove stale review.html template ... 545f0e5e30

The review page renders from liveviews.py (Python-built HTML), so the
804-line templates/review.html was unused. Tick the TODO item.

🔀 Merge branch 'feat/issue-91-remove-stale-review-template' into feat/issue-80-collect c0b1eafd1c

forgejo-actions commented 2026-08-14 04:35:20 +00:00

Copy link

This PR is too large to review automatically (14451 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14451 lines; limit is 3000). Please consider splitting it into smaller PRs.

forgejo-actions commented 2026-08-14 04:35:21 +00:00

Copy link

This PR is too large to review automatically (14451 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14451 lines; limit is 3000). Please consider splitting it into smaller PRs.

marvin8 added 3 commits 2026-08-14 07:19:00 +00:00

📝 add prose follow-up posts to Release-Notes eae71f69ee

✨ wire AI auto-reject into the fetch flow ... 723a3e0e7e

The ai_auto_reject setting was stored but never consumed. Add
reject_ai_flagged_posts() to reject unreviewed posts the AI would reject
(ai_would_reject=True) when the setting is enabled, and call it in the
scheduler after inline classification.

🔀 Merge branch 'feat/issue-94-wire-ai-auto-reject' into feat/issue-80-collect dca551f552

forgejo-actions commented 2026-08-14 07:19:09 +00:00

Copy link

This PR is too large to review automatically (14660 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14660 lines; limit is 3000). Please consider splitting it into smaller PRs.

forgejo-actions commented 2026-08-14 07:19:09 +00:00

Copy link

This PR is too large to review automatically (14660 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14660 lines; limit is 3000). Please consider splitting it into smaller PRs.

marvin8 added 3 commits 2026-08-14 21:09:29 +00:00

✨ classify posts on manual fetch paths too ... f2a8e141fe

Extract the scheduler's inline classification into a shared
classify_posts() in services/ai_pipeline.py and call it (plus
reject_ai_flagged_posts) from the manual POST /hashtags/{id}/fetch
endpoint and the review page's _fetch_all_active_streams, so every path
that saves new posts also classifies them.

⬆️ bump deps ... 10485efb47

Signed-off-by: Marvin8 <marvin8@tuta.io>

🔀 Merge branch 'feat/issue-96-classify-manual-fetch' into feat/issue-80-collect c592953b0e

forgejo-actions commented 2026-08-14 21:09:39 +00:00

Copy link

This PR is too large to review automatically (14965 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14965 lines; limit is 3000). Please consider splitting it into smaller PRs.

forgejo-actions commented 2026-08-14 21:09:40 +00:00

Copy link

This PR is too large to review automatically (14965 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (14965 lines; limit is 3000). Please consider splitting it into smaller PRs.

marvin8 added 2 commits 2026-08-14 22:38:50 +00:00

🐛 fix SQLite 'database is locked' under concurrent fetches ... 5e125affe0

Add a 30s busy_timeout (PRAGMA busy_timeout) on each SQLite connection so
writers wait instead of erroring, and commit the saved posts + deterministic
auto-reject before the slow AI classification in all three fetch paths so
the write lock is released during image download + vision.

🔀 Merge branch 'feat/issue-97-sqlite-locking' into feat/issue-80-collect 8128841cc4

forgejo-actions commented 2026-08-14 22:38:57 +00:00

Copy link

This PR is too large to review automatically (15058 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (15058 lines; limit is 3000). Please consider splitting it into smaller PRs.

forgejo-actions commented 2026-08-14 22:38:58 +00:00

Copy link

This PR is too large to review automatically (15058 lines; limit is 3000). Please consider splitting it into smaller PRs.

This PR is too large to review automatically (15058 lines; limit is 3000). Please consider splitting it into smaller PRs.

marvin8 approved these changes 2026-08-15 00:19:29 +00:00

marvin8 manually merged commit 645b52db05 into main 2026-08-15 00:20:19 +00:00

marvin8 deleted branch feat/issue-80-collect 2026-08-15 00:21:03 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

marvin8

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

marvin8/dujiangyan!93

No description provided.