Redesign cang as snapshot-first NVR: events UI, snapshot DB, tiered retention #102

New issue

Closed

opened 2026-07-08 23:24:32 +00:00 by coding-agent-marvin8 · 1 comment

coding-agent-marvin8 commented

2026-07-08 23:24:32 +00:00

(Migrated from codeberg.org)

Cang currently treats video clips as the primary artefact, with snapshots as secondary metadata attached to clips. This design is unsatisfying because:

The Dahua cameras produce glitchy .dav files (firmware issue, not fixable on our side), making the video clips unreliable as the primary view
The snapshot ingestion is incomplete: only JPGs that match a clip's time window are tracked; background [F] snapshots between clips are never recorded
There is a long-standing bug where JPG files are never deleted from the incoming directory — only .dav files get cleaned up
The daily review workflow is better served by a quick visual scan of motion events (thumbnail grid) than by scrubbing through video

The camera already does reliable motion detection and marks snapshot files with [M] (motion) vs [F] (first-frame/background). This gives us a clean event signal without any video analysis.

Proposed changes:

Ingest all camera JPGs into a new snapshots DB table, not just those matching clips
Group consecutive [M] snapshots (gap < configurable threshold) into events
Make events the primary view on the day page (thumbnail grid linking to slideshows)
Demote video clips to a secondary collapsible section
Add tiered retention: motion snapshots 30 days (configurable), background snapshots 14 days, video clips 7 days
Fix the JPG cleanup bug — snapshot file deletion now owned by the retention job
Remove the 'development paused' note from README

Cang currently treats video clips as the primary artefact, with snapshots as secondary metadata attached to clips. This design is unsatisfying because: - The Dahua cameras produce glitchy .dav files (firmware issue, not fixable on our side), making the video clips unreliable as the primary view - The snapshot ingestion is incomplete: only JPGs that match a clip's time window are tracked; background [F] snapshots between clips are never recorded - There is a long-standing bug where JPG files are never deleted from the incoming directory — only .dav files get cleaned up - The daily review workflow is better served by a quick visual scan of motion events (thumbnail grid) than by scrubbing through video The camera already does reliable motion detection and marks snapshot files with [M] (motion) vs [F] (first-frame/background). This gives us a clean event signal without any video analysis. **Proposed changes:** 1. Ingest *all* camera JPGs into a new `snapshots` DB table, not just those matching clips 2. Group consecutive [M] snapshots (gap < configurable threshold) into events 3. Make events the primary view on the day page (thumbnail grid linking to slideshows) 4. Demote video clips to a secondary collapsible section 5. Add tiered retention: motion snapshots 30 days (configurable), background snapshots 14 days, video clips 7 days 6. Fix the JPG cleanup bug — snapshot file deletion now owned by the retention job 7. Remove the 'development paused' note from README

coding-agent-marvin8 commented

2026-07-08 23:24:51 +00:00

(Migrated from codeberg.org)

Findings from test data

The incoming directory for one camera was examined. Key observations:

Snapshot structure: <serial>/<YYYY-MM-DD>/<channel>/jpg/<HH>/<MM>/<SS>[FLAG][0@0][0].jpg
[M] snapshots fire at ~1fps during motion events, and notably appear before the corresponding .dav clip starts recording
[F] snapshots fire at ~30–60s intervals as background coverage
The [M] flag in the filename stem is the event signal — no video analysis needed
At 4–5 cameras, ~620MB/day/camera in snapshots; with 4TB available, 30-day retention is very comfortable

Plan of attack

New module: src/cang/events.py — pure function group_into_events(snapshots, gap_seconds=60) that clusters consecutive [M] rows into events when the gap between them is < threshold.

Key file changes (TDD, each with a commit):

adapters/base.py + adapters/dahua.py — add IncomingSnapshot dataclass and scan_snapshots() that walks the full jpg/ tree
db.py — migration v3: snapshots table with unique index on path; new queries: insert_snapshot, list_snapshots_for_day, list_days_with_snapshot_stats, delete_expired_snapshots, rename delete_expired → delete_expired_clips
config.py + pipeline.py — add retention_days / background_retention_days / video_retention_days to ServerConfig; simplify _delete_source to only delete .dav
web/lifespan.py — add _scan_snapshots_for_camera and _run_cleanup wired into startup and periodic scans
events.py — gap-based event grouper
Web layer — routes/clips.py (events grid on day view, clips in <details>), routes/snapshots.py (DB-backed, time-window filtering), updated templates
README — remove 'development paused' note

Full implementation plan: docs/superpowers/plans/2026-07-09-snapshot-first-nvr.md

## Findings from test data The incoming directory for one camera was examined. Key observations: - Snapshot structure: `<serial>/<YYYY-MM-DD>/<channel>/jpg/<HH>/<MM>/<SS>[FLAG][0@0][0].jpg` - [M] snapshots fire at ~1fps during motion events, and notably appear *before* the corresponding .dav clip starts recording - [F] snapshots fire at ~30–60s intervals as background coverage - The [M] flag in the filename stem is the event signal — no video analysis needed - At 4–5 cameras, ~620MB/day/camera in snapshots; with 4TB available, 30-day retention is very comfortable ## Plan of attack **New module:** `src/cang/events.py` — pure function `group_into_events(snapshots, gap_seconds=60)` that clusters consecutive [M] rows into events when the gap between them is < threshold. **Key file changes (TDD, each with a commit):** 1. `adapters/base.py` + `adapters/dahua.py` — add `IncomingSnapshot` dataclass and `scan_snapshots()` that walks the full jpg/ tree 2. `db.py` — migration v3: `snapshots` table with unique index on path; new queries: `insert_snapshot`, `list_snapshots_for_day`, `list_days_with_snapshot_stats`, `delete_expired_snapshots`, rename `delete_expired` → `delete_expired_clips` 3. `config.py` + `pipeline.py` — add `retention_days` / `background_retention_days` / `video_retention_days` to `ServerConfig`; simplify `_delete_source` to only delete .dav 4. `web/lifespan.py` — add `_scan_snapshots_for_camera` and `_run_cleanup` wired into startup and periodic scans 5. `events.py` — gap-based event grouper 6. Web layer — `routes/clips.py` (events grid on day view, clips in `<details>`), `routes/snapshots.py` (DB-backed, time-window filtering), updated templates 7. README — remove 'development paused' note Full implementation plan: `docs/superpowers/plans/2026-07-09-snapshot-first-nvr.md`

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

marvin8/cang#102

No description provided.

Rows
Columns