mqtt_logger: incomplete MQTT payloads partially overwrite the cached field set

agent-pi commented

2026-09-09 01:33:04 +00:00

Collaborator

Records are emitted from a cached field set that each topic refreshes when it publishes. When a JSON topic delivers a payload that is missing one of its expected fields, the fields processed before the missing one have already been written into the cache. The next emitted record can therefore mix values from a half-delivered payload with values cached from earlier publishes, producing internally inconsistent records with no outward sign. Analysis code has no way to detect that this happened.

Observed in the field: schema-boundary records (e.g. sun elevation present but gate states absent for days after a cutover) show the cache refresh process is not atomic in practice.

Records are emitted from a cached field set that each topic refreshes when it publishes. When a JSON topic delivers a payload that is missing one of its expected fields, the fields processed before the missing one have already been written into the cache. The next emitted record can therefore mix values from a half-delivered payload with values cached from earlier publishes, producing internally inconsistent records with no outward sign. Analysis code has no way to detect that this happened. Observed in the field: schema-boundary records (e.g. sun elevation present but gate states absent for days after a cutover) show the cache refresh process is not atomic in practice.

agent-pi commented

2026-09-09 01:33:37 +00:00

Author

Collaborator

Findings

The merge loop in on_message writes each field into the latest cache as it iterates; a missing field raises KeyError partway through, leaving the fields before it already overwritten. fresh is not marked, so no record is emitted from the bad payload — but the poisoned cache leaks into the next good emit.
Reproduced with a unit test: publishing a complete zappi/readings payload then an incomplete one (first fields present with different values, later fields absent) leaves zappi_charge_mode mutated (Fast instead of Eco). Test initially passed with same-value sentinels — partial merge was invisible — so the test uses differing values.

Plan of attack

Extract all fields for a topic into a temporary dict first; merge into latest only if every field is present (single atomic update).
TDD per repo conventions: test_mqtt_logger.py as a standalone PEP 723 unittest script (style of test_analyze.py); cases: complete payload updates all fields, incomplete payload mutates nothing, cache recovers on next complete payload.
Commit: :bug: reject incomplete MQTT payloads without touching the field cache on branch fix/issue-2-mqtt-logger-quality.

**Findings** - The merge loop in `on_message` writes each field into the `latest` cache as it iterates; a missing field raises KeyError partway through, leaving the fields before it already overwritten. `fresh` is not marked, so no record is emitted from the bad payload — but the poisoned cache leaks into the next good emit. - Reproduced with a unit test: publishing a complete `zappi/readings` payload then an incomplete one (first fields present with different values, later fields absent) leaves `zappi_charge_mode` mutated (`Fast` instead of `Eco`). Test initially passed with same-value sentinels — partial merge was invisible — so the test uses differing values. **Plan of attack** - Extract all fields for a topic into a temporary dict first; merge into `latest` only if every field is present (single atomic update). - TDD per repo conventions: `test_mqtt_logger.py` as a standalone PEP 723 unittest script (style of `test_analyze.py`); cases: complete payload updates all fields, incomplete payload mutates nothing, cache recovers on next complete payload. - Commit: `:bug: reject incomplete MQTT payloads without touching the field cache` on branch `fix/issue-2-mqtt-logger-quality`.

agent-pi referenced this issue from a commit

2026-09-09 01:35:22 +00:00

:bug: reject incomplete MQTT payloads without touching the field cache

agent-pi referenced this issue from a pull request that will close it,

2026-09-09 01:35:40 +00:00

Harden mqtt_logger against partial payloads, stale cache values, and silent reconnects #4

agent-pi referenced this issue from a commit

2026-09-09 03:37:55 +00:00

:bug: reject incomplete MQTT payloads without touching the field cache