marvin8/cang
Watch
1
0
Fork
You've already forked cang
 0
Code Issues 1 Pull requests Releases 28 Activity Actions

Move processed videos and snapshots out of incoming folders into output_dir #112

New issue
Closed
opened 2026-07-14 19:50:01 +00:00 by coding-agent-marvin8 · 1 comment
coding-agent-marvin8 commented 2026-07-14 19:50:01 +00:00 (Migrated from codeberg.org)
Copy link

Today the incoming camera folders are a mix of processed and unprocessed media: transcoded .dav sources stay behind unless delete_after_transcode is set, and snapshots are served from the incoming tree forever (the snapshots table points there). There is no way to look at an incoming folder and know what has been processed.

Anything still sitting in an incoming folder should mean "not processed yet". Processed snapshots and videos should live in cang's own storage (output_dir) and be removed from incoming, including .idx sidecars and emptied date/hour directories. Existing deployments (DB rows pointing into incoming, already-transcoded sources still in incoming, per-clip snapshot copy dirs) need to migrate to the new scheme automatically.

Today the incoming camera folders are a mix of processed and unprocessed media: transcoded .dav sources stay behind unless delete_after_transcode is set, and snapshots are served from the incoming tree forever (the snapshots table points there). There is no way to look at an incoming folder and know what has been processed. Anything still sitting in an incoming folder should mean "not processed yet". Processed snapshots and videos should live in cang's own storage (output_dir) and be removed from incoming, including .idx sidecars and emptied date/hour directories. Existing deployments (DB rows pointing into incoming, already-transcoded sources still in incoming, per-clip snapshot copy dirs) need to migrate to the new scheme automatically.
coding-agent-marvin8 commented 2026-07-14 19:50:43 +00:00 (Migrated from codeberg.org)
Copy link

Findings from exploration:

  • The clip snapshot album routes already query the snapshots table by camera + time range; the per-clip copy dirs written by _persist_snapshots are referenced by nothing except the thumbnail jpg. They can go.
  • _run_cleanup already deletes snapshot files at their DB paths (inside incoming), so retention-driven deletion from incoming is existing behaviour; this change just moves the files earlier, at ingest.

Plan of attack:

  • New cang/storage.py: canonical paths (<camera>/clips/<date>/…, <camera>/snapshots/<date>/…), copy_into_store, prune_empty_dirs.
  • Snapshot ingest: copy → insert row with canonical path → commit → unlink incoming original; labels derived from DB columns.
  • Videos: unconditional source deletion after successful transcode (+ .idx); delete_after_transcode removed (deprecation warning); done-row-with-lingering-source deleted on scan.
  • Thumbnails resolved from the snapshots table at clip-done time; adapter loses filesystem snapshot↔clip matching; scan order flips to snapshots first.
  • New cang/migrate.py: idempotent startup migration (DB-row-driven; never deletes anything from incoming that has no verified copy in output_dir).
  • Sequence: storage helpers → db helpers → snapshot ingest → unconditional deletion → clips/ layout → adapter cleanup → authed video route (#113) → migration → docs.
Findings from exploration: - The clip snapshot album routes already query the snapshots table by camera + time range; the per-clip copy dirs written by _persist_snapshots are referenced by nothing except the thumbnail jpg. They can go. - _run_cleanup already deletes snapshot files at their DB paths (inside incoming), so retention-driven deletion from incoming is existing behaviour; this change just moves the files earlier, at ingest. Plan of attack: - New cang/storage.py: canonical paths (`<camera>/clips/<date>/…`, `<camera>/snapshots/<date>/…`), copy_into_store, prune_empty_dirs. - Snapshot ingest: copy → insert row with canonical path → commit → unlink incoming original; labels derived from DB columns. - Videos: unconditional source deletion after successful transcode (+ .idx); delete_after_transcode removed (deprecation warning); done-row-with-lingering-source deleted on scan. - Thumbnails resolved from the snapshots table at clip-done time; adapter loses filesystem snapshot↔clip matching; scan order flips to snapshots first. - New cang/migrate.py: idempotent startup migration (DB-row-driven; never deletes anything from incoming that has no verified copy in output_dir). - Sequence: storage helpers → db helpers → snapshot ingest → unconditional deletion → clips/<date> layout → adapter cleanup → authed video route (#113) → migration → docs.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
pages
2026.8.4
2026.7.25
2026.7.24
2026.7.23
2026.7.22
0.10.1
0.10.0
0.9.0
0.8.0
0.7.0
0.6.0
0.5.0
0.4.2
0.4.1
0.4.0
0.3.12
0.3.11
0.3.10
0.3.9
0.3.8
0.3.7
0.3.6
0.3.5
0.3.4
0.3.3
0.3.2
0.3.1
0.3.0
0.2.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
marvin8/cang#112
No description provided.