marvin8/dujiangyan
Watch
1
0
Fork
You've already forked dujiangyan
 0
Code Issues 4 Pull requests Projects Releases 48 Wiki Activity Actions

Fix SQLite 'database is locked' under concurrent fetches #99

Manually merged
marvin8 merged 1 commit from feat/issue-97-sqlite-locking into feat/issue-80-collect 2026-08-14 22:38:53 +00:00
Conversation 1 Commits 1 Files changed 5 +52
agent-pi commented 2026-08-14 21:20:33 +00:00
Collaborator
Copy link

Fixes the sqlite3.OperationalError: database is locked seen when a manual "Fetch All" overlaps the scheduler's fetch job.

Two changes:

  1. database.py: a SQLite connect hook runs PRAGMA busy_timeout = 30000 on every connection, so a writer waits up to 30s instead of erroring immediately.
  2. All three fetch paths (scheduler._fetch_stream_job, fetch_hashtag_stream, liveview _fetch_all_active_streams) now db.commit() the saved posts + deterministic auto-reject before the slow AI classification, releasing the write lock during the network-bound image download + vision calls, then a second commit for the classification + AI auto-reject.

Test: test_database.py asserts the busy-timeout hook runs the PRAGMA.

Refs #97

Fixes the `sqlite3.OperationalError: database is locked` seen when a manual "Fetch All" overlaps the scheduler's fetch job. Two changes: 1. `database.py`: a SQLite `connect` hook runs `PRAGMA busy_timeout = 30000` on every connection, so a writer waits up to 30s instead of erroring immediately. 2. All three fetch paths (`scheduler._fetch_stream_job`, `fetch_hashtag_stream`, liveview `_fetch_all_active_streams`) now `db.commit()` the saved posts + deterministic auto-reject *before* the slow AI classification, releasing the write lock during the network-bound image download + vision calls, then a second commit for the classification + AI auto-reject. Test: `test_database.py` asserts the busy-timeout hook runs the PRAGMA. Refs #97
🐛 fix SQLite 'database is locked' under concurrent fetches
All checks were successful
/ gitleaks (pull_request) Successful in 15s
Details
/ pr-review (pull_request) Successful in 34s
Details
/ checks (pull_request) Successful in 2m45s
Details
/ publish (pull_request) Has been skipped
Details
/ publish-container (pull_request) Has been skipped
Details
/ deploy-docs (pull_request) Has been skipped
Details
5e125affe0
 
Add a 30s busy_timeout (PRAGMA busy_timeout) on each SQLite connection so
writers wait instead of erroring, and commit the saved posts + deterministic
auto-reject before the slow AI classification in all three fetch paths so
the write lock is released during image download + vision.
marvin8 approved these changes 2026-08-14 21:59:29 +00:00
marvin8 manually merged commit 8128841cc4 into feat/issue-80-collect 2026-08-14 22:38:53 +00:00
marvin8 deleted branch feat/issue-97-sqlite-locking 2026-08-14 22:39:08 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
marvin8
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
marvin8/dujiangyan!99
No description provided.