Verify findings against full file content before posting #102

Manually merged
marvin8 merged 19 commits from feat/issue-82-verify-findings into main 2026-10-16 20:56:14 +00:00
Collaborator

Adds a verification pass that fact-checks each surviving finding against the changed file's full content before it is posted.

  • One LLM call per finding, bounded to 4 concurrent calls; only an affirmative rejected verdict drops a finding — every failure mode fails open.
  • File contents are fetched lazily per file with findings (seeded from already-attached full contents) and cached; unfetchable, failed, or oversized fetches skip verification with findings kept.
  • WUMING_VERIFY=false disables the pass; enabled by default (fail-closed parsing — a typo raises ValueError).

Reviewed against the wave-1 spec (Review quality & convergence milestone). Whole-branch fresh-context review passed: one critical (fetch-failure crash, fixed with RED→GREEN test), one documentation gap (CLAUDE.md symlink target uncommitted, fixed), release-notes entry added.

Closes #82

Adds a verification pass that fact-checks each surviving finding against the changed file's full content before it is posted. - One LLM call per finding, bounded to 4 concurrent calls; only an affirmative `rejected` verdict drops a finding — every failure mode fails open. - File contents are fetched lazily per file with findings (seeded from already-attached full contents) and cached; unfetchable, failed, or oversized fetches skip verification with findings kept. - `WUMING_VERIFY=false` disables the pass; enabled by default (fail-closed parsing — a typo raises `ValueError`). Reviewed against the wave-1 spec (`Review quality & convergence` milestone). Whole-branch fresh-context review passed: one critical (fetch-failure crash, fixed with RED→GREEN test), one documentation gap (CLAUDE.md symlink target uncommitted, fixed), release-notes entry added. Closes #82
forgejo-actions left a comment

WuMing

Found 7 issue(s). See inline comments below.

## WuMing Found **7** issue(s). See inline comments below.
README.md Outdated
@ -165,0 +167,4 @@
Before any finding is posted, WuMing fact-checks it: a dedicated LLM call receives the finding plus the full content of the changed file and decides whether the file actually contradicts the finding's factual claims. Only findings the file demonstrably contradicts are dropped — the verifier never judges style, importance, or severity, and it only sees the changed file itself, so claims about other files ("no test exists") are not verified.
Verification is fail-open: a provider error or unparseable verdict keeps the finding. Files whose content cannot be fetched, or that exceed 2000 lines, skip verification entirely. The pass adds one LLM call per finding (at most 4 run concurrently), after the cheap filters — severity floor, line-in-diff, deduplication — so LLM spend goes only to findings that would otherwise be posted. Set `WUMING_VERIFY=false` to disable.

docs [LOW]

This states a fixed 2000-line cutoff for skipping verification, but no such limit appears anywhere else in the documentation: AGENTS.md line 104 only says verification is "skipped for unfetchable/oversized files", and the adjacent WUMING_MAX_FILE_LINES setting (line 160) is user-configurable and defaults to 0 (disabled). As written, users cannot tell whether the verification cutoff is the hard-coded 2000 or is derived from WUMING_MAX_FILE_LINES. Confirm the value against the implementation and either document the constant consistently in AGENTS.md too, or reference the configurable setting. If the number is not actually fixed, remove it and describe the behaviour instead.

**docs** [LOW] This states a fixed 2000-line cutoff for skipping verification, but no such limit appears anywhere else in the documentation: AGENTS.md line 104 only says verification is "skipped for unfetchable/oversized files", and the adjacent `WUMING_MAX_FILE_LINES` setting (line 160) is user-configurable and defaults to 0 (disabled). As written, users cannot tell whether the verification cutoff is the hard-coded 2000 or is derived from `WUMING_MAX_FILE_LINES`. Confirm the value against the implementation and either document the constant consistently in AGENTS.md too, or reference the configurable setting. If the number is not actually fixed, remove it and describe the behaviour instead. <!-- wuming:sha256:90696f0fa49ef4955f510cf80e651565e978d293b46a106a30b7fa16dfd4d586 -->
@ -3,3 +3,3 @@
---
wuming 0.10.0
wuming 0.11.0

docs [MEDIUM]

The top-of-file release heading was renamed from wuming 0.10.0 to wuming 0.11.0 instead of adding a new section, so the pre-existing entries that follow it — the SBOM attestation (line 8), cosign signing (line 9) and the openrouter backend (line 10) — are now attributed to 0.11.0 even though they shipped in 0.10.0 (the diff shows no second heading was introduced). If 0.10.0 was already released, this silently deletes that release's notes and mislabels its features. Fix by restoring a wuming 0.10.0 heading above the existing bullets and putting the new verification bullet under a new wuming 0.11.0 heading placed above it. If 0.10.0 was never released and this is a deliberate rename of the unreleased draft, say so in the PR body to avoid the ambiguity.

**docs** [MEDIUM] The top-of-file release heading was renamed from `wuming 0.10.0` to `wuming 0.11.0` instead of adding a new section, so the pre-existing entries that follow it — the SBOM attestation (line 8), cosign signing (line 9) and the `openrouter` backend (line 10) — are now attributed to 0.11.0 even though they shipped in 0.10.0 (the diff shows no second heading was introduced). If 0.10.0 was already released, this silently deletes that release's notes and mislabels its features. Fix by restoring a `wuming 0.10.0` heading above the existing bullets and putting the new verification bullet under a new `wuming 0.11.0` heading placed above it. If 0.10.0 was never released and this is a deliberate rename of the unreleased draft, say so in the PR body to avoid the ambiguity. <!-- wuming:sha256:ea0a23b9ef2f430ef18a371b78d6dfb1bab5a6ed031ed0d33bf0381a0bb2cd5b -->
marvin8 marked this conversation as resolved
@ -0,0 +78,4 @@
if verdict not in _VALID_VERDICTS:
return None
reason = item.get("reason")
return (verdict, str(reason) if reason else "")

code [LOW]

parse_verdict's docstring says the reason defaults to an empty string when it is 'not a string', but the implementation converts truthy non-string values to strings (e.g. integer 123 -> '123', list [1] -> '[1]'). Make the type check explicit with isinstance(reason, str) or update the docstring to match the actual behavior.

**code** [LOW] parse_verdict's docstring says the reason defaults to an empty string when it is 'not a string', but the implementation converts truthy non-string values to strings (e.g. integer 123 -> '123', list [1] -> '[1]'). Make the type check explicit with isinstance(reason, str) or update the docstring to match the actual behavior. <!-- wuming:sha256:99ed87f801dbe489a545015c8020ee740c29b6dba98a751a78f48d02625e127e -->
marvin8 marked this conversation as resolved
@ -0,0 +127,4 @@
except Exception as exc:
logger.warning(
"verifier call failed — keeping finding",
extra={"path": comment.path, "line": comment.line, "error": repr(exc)},

security [MEDIUM]

A09: Logging repr(exc) from backend failures can leak sensitive data (API keys, tokens, URLs with credentials, or response bodies) into application logs. Exception messages from the LLM backend or HTTP client are untrusted and may embed secrets. Remediation: log only a sanitized error message or exception type, and avoid repr(exc) for external/HTTP errors.

**security** [MEDIUM] A09: Logging `repr(exc)` from backend failures can leak sensitive data (API keys, tokens, URLs with credentials, or response bodies) into application logs. Exception messages from the LLM backend or HTTP client are untrusted and may embed secrets. Remediation: log only a sanitized error message or exception type, and avoid `repr(exc)` for external/HTTP errors. <!-- wuming:sha256:ccf7d2f2f737ae21756f76937eb42dceb6c6305a51915e8317e1d4f64959a5b6 -->
marvin8 marked this conversation as resolved
@ -0,0 +176,4 @@
return_exceptions=True,
)
for path, content in zip(to_fetch, fetched, strict=True):
if isinstance(content, BaseException):

code [MEDIUM]

fetch_file_contents uses asyncio.gather(return_exceptions=True) and then catches BaseException, which swallows asyncio.CancelledError. If the surrounding task is cancelled while fetches are in flight, the cancellation signal is returned as an exception in the results list and silently treated as a fetch failure, so the function returns normally instead of propagating cancellation. Re-raise asyncio.CancelledError before treating it as a fetch failure, and add a regression test.

**code** [MEDIUM] fetch_file_contents uses asyncio.gather(return_exceptions=True) and then catches BaseException, which swallows asyncio.CancelledError. If the surrounding task is cancelled while fetches are in flight, the cancellation signal is returned as an exception in the results list and silently treated as a fetch failure, so the function returns normally instead of propagating cancellation. Re-raise asyncio.CancelledError before treating it as a fetch failure, and add a regression test. <!-- wuming:sha256:6bddfb9c148b7f53426a90118475e72c8a7fd169e718e80f7d542a7ff61860a4 -->
@ -0,0 +179,4 @@
if isinstance(content, BaseException):
logger.warning(
"file content fetch failed — skipping verification for file",
extra={"path": path, "error": repr(content)},

security [MEDIUM]

A09: Logging repr(content) for a BaseException returned by asyncio.gather(return_exceptions=True) can expose sensitive forgejo/HTTP error details (e.g., API tokens in URLs, response payloads) in logs. Remediation: log a generic fetch-failure message and record the exception type/name without repr(); route full exception details to debug-level or scrubbed telemetry only.

**security** [MEDIUM] A09: Logging `repr(content)` for a `BaseException` returned by `asyncio.gather(return_exceptions=True)` can expose sensitive forgejo/HTTP error details (e.g., API tokens in URLs, response payloads) in logs. Remediation: log a generic fetch-failure message and record the exception type/name without `repr()`; route full exception details to debug-level or scrubbed telemetry only. <!-- wuming:sha256:29b4da2478ddabf14d30eb0a971d475ef53a980e0c17f48d2ca0ce2c018b49e0 -->
@ -0,0 +46,4 @@
expect(parse_verdict(raw)).to_equal(expected)
@test("parse_verdict returns None for anything that is not a clean verdict").cases(

code [LOW]

No test exercises parse_verdict's markdown-fence extraction (verify.py lines 67-70). Add cases for fenced JSON, fences with a language tag (e.g. ```json), and responses that mix explanatory text with a fenced verdict block to prevent regressions in fence stripping.

**code** [LOW] No test exercises parse_verdict's markdown-fence extraction (verify.py lines 67-70). Add cases for fenced JSON, fences with a language tag (e.g. ```json), and responses that mix explanatory text with a fenced verdict block to prevent regressions in fence stripping. <!-- wuming:sha256:522d7faa212077395726da6857cda28e53afd9022c522d6577350a8371d78818 -->
Author
Collaborator

Release-Notes.md

docs [MEDIUM]

👀 Valid catch, but this needs Marvin8's call rather than a fix from me: the git history shows the 0.11.0 release commit (d173561, changelog update) shipped while this thread still read wuming 0.10.0 — the release cycle itself left the header stale. My line-bump followed the AGENTS.md rule ("update the version line from pyproject.toml"), which produced exactly the misattribution you describe. The clean fix would be moving the three 0.10.0 bullets out of the intro post, but how the intro post interacts with just release bullet extraction is a release-workflow decision. Escalated to Marvin8.

[`Release-Notes.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5536) > **docs** [MEDIUM] 👀 Valid catch, but this needs Marvin8's call rather than a fix from me: the git history shows the 0.11.0 release commit (`d173561`, changelog update) shipped while this thread still read `wuming 0.10.0` — the release cycle itself left the header stale. My line-bump followed the AGENTS.md rule ("update the version line from pyproject.toml"), which produced exactly the misattribution you describe. The clean fix would be moving the three 0.10.0 bullets out of the intro post, but how the intro post interacts with `just release` bullet extraction is a release-workflow decision. Escalated to Marvin8.
Author
Collaborator

README.md

docs [LOW]

Fixed in commit 6749594 — the value is confirmed: VERIFICATION_MAX_FILE_LINES = 2000, a module constant in src/wuming/verify.py (not derived from WUMING_MAX_FILE_LINES). The AGENTS.md env-var entry now states the hard-coded 2000-line cap explicitly.

[`README.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5537) > **docs** [LOW] Fixed in commit `6749594` — the value is confirmed: `VERIFICATION_MAX_FILE_LINES = 2000`, a module constant in `src/wuming/verify.py` (not derived from `WUMING_MAX_FILE_LINES`). The AGENTS.md env-var entry now states the hard-coded 2000-line cap explicitly.
Author
Collaborator

src/wuming/verify.py

security [MEDIUM]

Fixed in commit 634ee13 — the WARNING log now carries only the exception type name (error_type); the full repr() moved to a DEBUG line, visible under WUMING_VERBOSE. Regression test test_fetch_failure_log_sanitized asserts the message text never reaches WARNING-level records.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5535) > **security** [MEDIUM] Fixed in commit `634ee13` — the WARNING log now carries only the exception type name (`error_type`); the full `repr()` moved to a DEBUG line, visible under `WUMING_VERBOSE`. Regression test `test_fetch_failure_log_sanitized` asserts the message text never reaches WARNING-level records.
Author
Collaborator

src/wuming/verify.py

code [MEDIUM]

Fixed in commit 634ee13 — asyncio.CancelledError is re-raised before the fetch-failure mapping. Regression test test_fetch_cancellation_propagates pins it.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5531) > **code** [MEDIUM] Fixed in commit `634ee13` — `asyncio.CancelledError` is re-raised before the fetch-failure mapping. Regression test `test_fetch_cancellation_propagates` pins it.
Author
Collaborator

src/wuming/verify.py

code [LOW]

Fixed in commit 634ee13 — the docstring now matches the implementation: absent/null reason defaults to empty string, truthy non-string values are coerced with str().

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5532) > **code** [LOW] Fixed in commit `634ee13` — the docstring now matches the implementation: absent/null reason defaults to empty string, truthy non-string values are coerced with `str()`.
Author
Collaborator

src/wuming/verify.py

security [MEDIUM]

Fixed in commit 634ee13 — same treatment as the fetch-failure site: WARNING carries only error_type, full repr() moved to DEBUG. Regression test test_verify_failure_log_sanitized asserts backend exception message text never appears in WARNING-level records.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5534) > **security** [MEDIUM] Fixed in commit `634ee13` — same treatment as the fetch-failure site: WARNING carries only `error_type`, full `repr()` moved to DEBUG. Regression test `test_verify_failure_log_sanitized` asserts backend exception message text never appears in WARNING-level records.
Author
Collaborator

tests/test_verify.py

code [LOW]

Fixed in commit 634ee13 — test_parse_verdict_fences now covers a plain fence, a ```json-tagged fence, and prose-then-fence responses.

[`tests/test_verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5533) > **code** [LOW] Fixed in commit `634ee13` — `test_parse_verdict_fences` now covers a plain fence, a ```json-tagged fence, and prose-then-fence responses.
📝 state the 2000-line verification cap in the env var list
All checks were successful
/ gitleaks (pull_request) Successful in 40s
/ checks (pull_request) Successful in 1m58s
/ deploy-docs (pull_request) Has been skipped
/ publish-container (pull_request) Has been skipped
/ pr-review (pull_request) Successful in 7m22s
674959447a
forgejo-actions left a comment

WuMing

Found 7 issue(s). See inline comments below.

## WuMing Found **7** issue(s). See inline comments below.
README.md Outdated
@ -158,10 +158,17 @@ See the [Forgejo/GitHub Actions guide](https://wuming.marvin8.zone/getting-start
| `WUMING_PROGRESS` | — | `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent |
| `WUMING_MAX_DIFF_LINES` | `0` | Maximum raw diff size in lines. When the diff exceeds this value and it is greater than 0, WuMing posts a single plain PR comment explaining the PR is too large to review automatically, then exits without posting inline comments. Set to `0` to disable. |
| `WUMING_MAX_FILE_LINES` | `0` | Maximum file size (in lines) for full-file context fetching. When greater than 0, WuMing fetches the complete content of each changed file from Forgejo and includes it in the agent's context, giving the model visibility beyond the diff hunks. Files exceeding this line count are excluded. Set to `0` (the default) to disable. |
| `WUMING_VERIFY` | `true` | When `true`, each candidate finding is fact-checked against the changed file's full content before posting; findings whose factual claims the file contradicts are dropped. Verification is fail-open: provider errors keep findings. Set to `false` to disable. |

docs [LOW]

The table row documents true/false but does not mention that parsing is strict (per the PR, an invalid value such as WUMING_VERIFY=yes raises a ValueError rather than falling back to the default). Since the option is enabled by default and fail-closed, users can unintentionally abort a run. State explicitly that only true/false are accepted and any other value is an error.

**docs** [LOW] The table row documents `true`/`false` but does not mention that parsing is strict (per the PR, an invalid value such as `WUMING_VERIFY=yes` raises a `ValueError` rather than falling back to the default). Since the option is enabled by default and fail-closed, users can unintentionally abort a run. State explicitly that only `true`/`false` are accepted and any other value is an error. <!-- wuming:sha256:b8542971fc40c138b60f5139cee0b793126b6d33cd393b4c8cf6178b6162fe9a -->
@ -0,0 +120,4 @@
user_message = (
f"## Finding under review\n\n{_format_finding(comment)}\n\n"
f"## Full file content: {comment.path}\n\n"
f"```\n{file_content}\n```\n\n"

code [MEDIUM]

The file content is wrapped in a plain triple-backtick fence. If file_content itself contains triple backticks, the fence closes early and the rest of the prompt is misinterpreted, which can lead to an incorrect 'rejected' verdict and a valid finding being dropped despite the fail-open design. Escape embedded fences or dynamically choose a fence longer than the longest run of backticks in the content.

**code** [MEDIUM] The file content is wrapped in a plain triple-backtick fence. If file_content itself contains triple backticks, the fence closes early and the rest of the prompt is misinterpreted, which can lead to an incorrect 'rejected' verdict and a valid finding being dropped despite the fail-open design. Escape embedded fences or dynamically choose a fence longer than the longest run of backticks in the content. <!-- wuming:sha256:5814a4e85efb3085dd1a7b33fe67e91f2d6602fd8a29626c2c8d47e438db9443 -->
@ -0,0 +136,4 @@
)
logger.debug(
"verifier call failure detail",
extra={"path": comment.path, "line": comment.line, "error": repr(exc)},

security [MEDIUM]

A02: Logging repr(exc) from backend.complete may leak API keys or bearer tokens if the HTTP exception includes request/response headers. Replace the full exception repr with a sanitized error message or explicitly strip authorization headers before logging.

**security** [MEDIUM] A02: Logging repr(exc) from backend.complete may leak API keys or bearer tokens if the HTTP exception includes request/response headers. Replace the full exception repr with a sanitized error message or explicitly strip authorization headers before logging. <!-- wuming:sha256:1924ab16a870911f43e617196125550d376d2d043cd57c1fcb834b2d7730c8d6 -->
marvin8 marked this conversation as resolved
@ -0,0 +187,4 @@
for path, content in zip(to_fetch, fetched, strict=True):
if isinstance(content, asyncio.CancelledError):
raise content
if isinstance(content, BaseException):

code [LOW]

After re-raising CancelledError, all other BaseException subclasses (including KeyboardInterrupt and SystemExit) are caught and treated as fetch failures. User-interruption signals raised inside get_file_content should propagate instead of being logged and swallowed. Only Exception subclasses should be treated as fetch failures after the cancellation check.

**code** [LOW] After re-raising CancelledError, all other BaseException subclasses (including KeyboardInterrupt and SystemExit) are caught and treated as fetch failures. User-interruption signals raised inside get_file_content should propagate instead of being logged and swallowed. Only Exception subclasses should be treated as fetch failures after the cancellation check. <!-- wuming:sha256:a23a1bbc0fb3eb3231494a334d1249f06aacd8ee28eefe9ce6750f7d9755f3a0 -->
@ -0,0 +194,4 @@
)
logger.debug(
"file content fetch failure detail",
extra={"path": path, "error": repr(content)},

security [MEDIUM]

A02: Logging repr(content) of a Forgejo fetch exception may leak API tokens if the HTTP error includes request headers. Summarize or sanitize the exception instead of logging its full repr.

**security** [MEDIUM] A02: Logging repr(content) of a Forgejo fetch exception may leak API tokens if the HTTP error includes request headers. Summarize or sanitize the exception instead of logging its full repr. <!-- wuming:sha256:2d6a66ed127028e912ca9f1bf6d60cd44ac73131430b9d353737cf148c88392b -->
@ -0,0 +253,4 @@
A ``(kept, dropped)`` pair, both preserving input order.
"""
if progress is not None:
progress.step_start("verify_findings", "Fact-checking findings…")

code [MEDIUM]

If any awaited call inside verify_comments raises (e.g. asyncio.CancelledError from fetch_file_contents or asyncio.gather), progress.step_done is never reached, leaving the progress reporter in a started-but-not-completed state. Wrap the body between step_start and step_done in a try/finally and call step_done in the finally block.

**code** [MEDIUM] If any awaited call inside verify_comments raises (e.g. asyncio.CancelledError from fetch_file_contents or asyncio.gather), progress.step_done is never reached, leaving the progress reporter in a started-but-not-completed state. Wrap the body between step_start and step_done in a try/finally and call step_done in the finally block. <!-- wuming:sha256:f44b5153828c63bb9f559a743868802f0a8ecd74e2202199d6699f0dbe91589d -->
@ -437,3 +438,28 @@ def test_reads_summary_comment(env: Annotated[None, Depends(clean_env)]) -> None
with patch.dict("os.environ", {"WUMING_SUMMARY_COMMENT": "true"}, clear=False):
config = load_config()
expect(config.summary_comment).to_be_truthy()

code [LOW]

There is no test that load_config reads WUMING_VERIFY and wires it into Config.verify_findings. The parser function is unit-tested, but the load_config integration path (default-true behaviour and the typo-raises-ValueError path) is not covered.

**code** [LOW] There is no test that load_config reads WUMING_VERIFY and wires it into Config.verify_findings. The parser function is unit-tested, but the load_config integration path (default-true behaviour and the typo-raises-ValueError path) is not covered. <!-- wuming:sha256:206baa94770e2c7e70b0c6bdbd55e30037fb4c8a479d14e48d5d05b164b8cd43 -->
Author
Collaborator

src/wuming/verify.py

code [MEDIUM]

Fixed in commit 9c7e76e — the body between step_start and step_done now runs in try/finally with step_done in the finally block. Regression test test_verify_comments_progress_step_done_on_exception drives a cancelled fetch through a spy progress reporter and asserts the step closes.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5547) > **code** [MEDIUM] Fixed in commit `9c7e76e` — the body between `step_start` and `step_done` now runs in try/finally with `step_done` in the `finally` block. Regression test `test_verify_comments_progress_step_done_on_exception` drives a cancelled fetch through a spy progress reporter and asserts the step closes.
Author
Collaborator

src/wuming/verify.py

code [MEDIUM]

Fixed in commit 9c7e76e — the fence is now chosen dynamically: longer than the longest backtick run in the file content (minimum 3). Regression test test_verify_finding_fence_escapes_content verifies content containing ``` fences survives verbatim inside a ```` fence.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5548) > **code** [MEDIUM] Fixed in commit `9c7e76e` — the fence is now chosen dynamically: longer than the longest backtick run in the file content (minimum 3). Regression test `test_verify_finding_fence_escapes_content` verifies content containing ``` fences survives verbatim inside a ```` fence.
Author
Collaborator

src/wuming/verify.py

code [LOW]

Fixed in commit 9c7e76e — only Exception subclasses are treated as fetch failures; any other BaseException (CancelledError, KeyboardInterrupt, SystemExit) is re-raised. Regression test uses a KeyboardInterrupt-class stand-in (test_fetch_keyboard_interrupt_propagates) because a real KeyboardInterrupt raised in a task kills the test runner's worker.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5549) > **code** [LOW] Fixed in commit `9c7e76e` — only `Exception` subclasses are treated as fetch failures; any other `BaseException` (CancelledError, KeyboardInterrupt, SystemExit) is re-raised. Regression test uses a KeyboardInterrupt-class stand-in (`test_fetch_keyboard_interrupt_propagates`) because a real KeyboardInterrupt raised in a task kills the test runner's worker.
Author
Collaborator

tests/test_config.py

code [LOW]

Fixed in commit 9c7e76e — three load_config integration tests added: default-true, explicit false wiring, and the typo-raises-ValueError path.

[`tests/test_config.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5550) > **code** [LOW] Fixed in commit `9c7e76e` — three load_config integration tests added: default-true, explicit `false` wiring, and the typo-raises-`ValueError` path.
Author
Collaborator

README.md

docs [LOW]

Fixed in commit 29bd0f0 — the row now states that only true/false (case-insensitive) are accepted and any other value is an error, not a fallback.

[`README.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5553) > **docs** [LOW] Fixed in commit `29bd0f0` — the row now states that only `true`/`false` (case-insensitive) are accepted and any other value is an error, not a fallback.
Author
Collaborator

src/wuming/verify.py

security [MEDIUM]

📌Holding for Marvin8's review — this re-raises the substance of wave-1 finding 5534, where the agreed remedy moved full exception repr() to DEBUG. Two mitigating facts: in this codebase tokens travel in HTTP headers (never URLs or query strings), and httpx2 exception messages carry status lines and URLs but not headers; and DEBUG logs are gated behind WUMING_VERBOSE, which AGENTS.md already documents as sensitive. Stripping the repr from DEBUG entirely would leave only the exception class name for debugging provider failures.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5551) > **security** [MEDIUM] 📌Holding for Marvin8's review — this re-raises the substance of wave-1 finding 5534, where the agreed remedy moved full exception `repr()` to DEBUG. Two mitigating facts: in this codebase tokens travel in HTTP headers (never URLs or query strings), and httpx2 exception messages carry status lines and URLs but not headers; and DEBUG logs are gated behind `WUMING_VERBOSE`, which AGENTS.md already documents as sensitive. Stripping the repr from DEBUG entirely would leave only the exception class name for debugging provider failures.
Author
Collaborator

src/wuming/verify.py

security [MEDIUM]

📌 Same answer as 5551 above — consolidated: the WARNING level is already sanitized (type name only, per wave-1 fix); the question is whether the DEBUG-level repr should exist at all. Holding for Marvin8's decision.

[`src/wuming/verify.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5552) > **security** [MEDIUM] 📌 Same answer as 5551 above — consolidated: the WARNING level is already sanitized (type name only, per wave-1 fix); the question is whether the DEBUG-level repr should exist at all. Holding for Marvin8's decision.
📝 document strict WUMING_VERIFY parsing in the config table
All checks were successful
/ gitleaks (pull_request) Successful in 20s
/ checks (pull_request) Successful in 1m56s
/ deploy-docs (pull_request) Has been skipped
/ publish-container (pull_request) Has been skipped
/ pr-review (pull_request) Successful in 8m39s
29bd0f022c
forgejo-actions left a comment

WuMing

Found 5 issue(s). See inline comments below.

## WuMing Found **5** issue(s). See inline comments below.
@ -101,6 +101,7 @@ tools/
- `WUMING_PROGRESS` — `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent
- `WUMING_MAX_DIFF_LINES` — integer; if the raw diff line count exceeds this value and it is > 0, posts a plain PR comment and exits without inline review (default: `0` = disabled)
- `WUMING_MAX_FILE_LINES` — integer; when > 0, fetches the full content of each changed file from Forgejo and includes it in the agent's context (files exceeding this line count are excluded); (default: `0` = disabled)
- `WUMING_VERIFY` — `true`/`false`; fact-check findings against full file content before posting (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap); (default: `true`)

docs [MEDIUM]

This new entry documents only true/false and the fail-open behaviour, but omits two things the README documents for the same variable: (1) parsing is fail-closed — any value other than true/false (case-insensitive) raises an error rather than silently falling back to the default, and (2) the values are case-insensitive. A reader relying on AGENTS.md could set WUMING_VERIFY=1 or WUMING_VERIFY=yes and get a hard startup failure with no warning from the docs. Add the fail-closed/hard-error note (and the case-insensitivity) so both reference files agree.

**docs** [MEDIUM] This new entry documents only `true`/`false` and the fail-open behaviour, but omits two things the README documents for the same variable: (1) parsing is fail-closed — any value other than `true`/`false` (case-insensitive) raises an error rather than silently falling back to the default, and (2) the values are case-insensitive. A reader relying on AGENTS.md could set `WUMING_VERIFY=1` or `WUMING_VERIFY=yes` and get a hard startup failure with no warning from the docs. Add the fail-closed/hard-error note (and the case-insensitivity) so both reference files agree. <!-- wuming:sha256:dd3062aea932e52d1590f91271abea5c6b272572c2498980b85ef5694ba672c3 -->
@ -165,0 +167,4 @@
Before any finding is posted, WuMing fact-checks it: a dedicated LLM call receives the finding plus the full content of the changed file and decides whether the file actually contradicts the finding's factual claims. Only findings the file demonstrably contradicts are dropped — the verifier never judges style, importance, or severity, and it only sees the changed file itself, so claims about other files ("no test exists") are not verified.
Verification is fail-open: a provider error or unparseable verdict keeps the finding. Files whose content cannot be fetched, or that exceed 2000 lines, skip verification entirely. The pass adds one LLM call per finding (at most 4 run concurrently), after the cheap filters — severity floor, line-in-diff, deduplication — so LLM spend goes only to findings that would otherwise be posted. Set `WUMING_VERIFY=false` to disable.

docs [LOW]

The 2000-line limit is described only as a bare number here ("exceed 2000 lines"), while AGENTS.md calls it a "hard-coded" cap. Because the preceding table row introduces WUMING_MAX_FILE_LINES (default 0) for full-file context fetching, readers may reasonably assume raising WUMING_MAX_FILE_LINES also extends or disables the verification cap. State explicitly that the verification limit is fixed at 2000 lines and is independent of, and not configurable by, WUMING_MAX_FILE_LINES.

**docs** [LOW] The 2000-line limit is described only as a bare number here ("exceed 2000 lines"), while AGENTS.md calls it a "hard-coded" cap. Because the preceding table row introduces `WUMING_MAX_FILE_LINES` (default `0`) for full-file context fetching, readers may reasonably assume raising `WUMING_MAX_FILE_LINES` also extends or disables the verification cap. State explicitly that the verification limit is fixed at 2000 lines and is independent of, and not configurable by, `WUMING_MAX_FILE_LINES`. <!-- wuming:sha256:b065bb750a59dfb1102f27ef583351b90342166d54c2ca80bbafc6be3e97dceb -->
@ -90,6 +91,32 @@ def load_config() -> Config:
dry_run=_parse_bool_flag(os.environ.get("WUMING_DRY_RUN")),
verbose=_parse_bool_flag(os.environ.get("WUMING_VERBOSE")),
summary_comment=_parse_bool_flag(os.environ.get("WUMING_SUMMARY_COMMENT")),
verify_findings=_parse_verify_enabled(os.environ.get("WUMING_VERIFY", "")),

code [LOW]

There is no integration test for load_config mapping WUMING_VERIFY into Config.verify_findings. The unit tests cover _parse_verify_enabled in isolation, but a regression in load_config (e.g., forgetting to pass the env value or using the wrong variable name) would not be caught. Add load_config tests for default-on, explicit false, and typo behavior.

**code** [LOW] There is no integration test for load_config mapping WUMING_VERIFY into Config.verify_findings. The unit tests cover _parse_verify_enabled in isolation, but a regression in load_config (e.g., forgetting to pass the env value or using the wrong variable name) would not be caught. Add load_config tests for default-on, explicit false, and typo behavior. <!-- wuming:sha256:49af13ae5ed502896d10f5e2347ff2c5f2537c712033c3415fd6bdd260a550ec -->
marvin8 marked this conversation as resolved
@ -99,6 +100,26 @@ async def _main() -> None:
raw_comments, position_maps, existing_markers, file_diffs
)
if config.verify_findings and new_comments:

code [MEDIUM]

The verification pass is invoked without any exception handling. Per the PR description, verification must fail open for every failure mode; an unexpected exception from verify_comments (e.g., a bug, malformed LLM response, or unhandled network error) would bubble up and crash the pipeline before comments are published, causing a hard failure instead of preserving findings. Wrap the verify_comments call in a try/except that logs the failure and keeps the original new_comments list.

**code** [MEDIUM] The verification pass is invoked without any exception handling. Per the PR description, verification must fail open for every failure mode; an unexpected exception from verify_comments (e.g., a bug, malformed LLM response, or unhandled network error) would bubble up and crash the pipeline before comments are published, causing a hard failure instead of preserving findings. Wrap the verify_comments call in a try/except that logs the failure and keeps the original new_comments list. <!-- wuming:sha256:6060dc02ad3805541f5b9dbcf64a22a22a7aa269ae7e57f08d70e5bc7a2bcbb5 -->
marvin8 marked this conversation as resolved
@ -911,0 +988,4 @@
_verify_pipeline_env(),
)
expect(verify_mock.await_count).to_equal(1)
survivors = verify_mock.call_args.args[0]

code [LOW]

The verification pipeline tests only inspect verify_mock.call_args.args[0] (the survivors list). They do not assert that verify_comments receives the correct keyword arguments (seeded_contents, progress, backend, forgejo) or that seeded_contents is populated from file_diffs. Add assertions on call_args.kwargs to cover the integration contract.

**code** [LOW] The verification pipeline tests only inspect verify_mock.call_args.args[0] (the survivors list). They do not assert that verify_comments receives the correct keyword arguments (seeded_contents, progress, backend, forgejo) or that seeded_contents is populated from file_diffs. Add assertions on call_args.kwargs to cover the integration contract. <!-- wuming:sha256:56d6ee7a0d98fe761b13ecb73f52deb5dc2d9756696fd5ae0a57cee3648f107e -->
🐛 fail open when the verification pass itself crashes
All checks were successful
/ gitleaks (pull_request) Successful in 21s
/ checks (pull_request) Successful in 1m54s
/ deploy-docs (pull_request) Has been skipped
/ publish-container (pull_request) Has been skipped
/ pr-review (pull_request) Successful in 2m30s
2740a0de32
Author
Collaborator

src/wuming/main.py

code [MEDIUM]

Fixed in commit 2740a0d — the verification call in _main is now wrapped fail-open: any unexpected exception logs a warning (type name; repr at DEBUG) and posts the findings unverified. Regression test test_pipeline_verification_crash_fails_open drives a crashing verifier and asserts the review still posts. Deliberately NOT caught: CancelledError, KeyboardInterrupt, SystemExit — cancellation and interruption must still propagate.

[`src/wuming/main.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5563) > **code** [MEDIUM] Fixed in commit `2740a0d` — the verification call in `_main` is now wrapped fail-open: any unexpected exception logs a warning (type name; repr at DEBUG) and posts the findings unverified. Regression test `test_pipeline_verification_crash_fails_open` drives a crashing verifier and asserts the review still posts. Deliberately NOT caught: `CancelledError`, `KeyboardInterrupt`, `SystemExit` — cancellation and interruption must still propagate.
Author
Collaborator

tests/test_main.py

code [LOW]

Fixed in commit 2740a0d — test_pipeline_verifies_survivors now also asserts seeded_contents == {} (the default-env wiring: no full contents attached), a non-None progress, and the four positional arguments (survivors, forgejo, backend, agent configs).

[`tests/test_main.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5564) > **code** [LOW] Fixed in commit `2740a0d` — `test_pipeline_verifies_survivors` now also asserts `seeded_contents == {}` (the default-env wiring: no full contents attached), a non-None `progress`, and the four positional arguments (survivors, forgejo, backend, agent configs).
Author
Collaborator

src/wuming/config.py

code [LOW]

🔴 Already covered — commit 9c7e76e (in this PR, before your wave) added exactly these load_config integration tests: test_default_verify_findings (default-true), test_reads_verify_findings (explicit false wired through), and test_reads_verify_findings_typo (typo raises ValueError naming the variable). No change needed.

[`src/wuming/config.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5565) > **code** [LOW] 🔴 Already covered — commit `9c7e76e` (in this PR, before your wave) added exactly these load_config integration tests: `test_default_verify_findings` (default-true), `test_reads_verify_findings` (explicit false wired through), and `test_reads_verify_findings_typo` (typo raises `ValueError` naming the variable). No change needed.
Author
Collaborator

AGENTS.md

docs [MEDIUM]

Fixed in commit 2740a0d — the AGENTS.md entry now documents both: values are case-insensitive, and parsing is fail-closed (any other value raises ValueError rather than falling back to the default).

[`AGENTS.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5566) > **docs** [MEDIUM] Fixed in commit `2740a0d` — the AGENTS.md entry now documents both: values are case-insensitive, and parsing is fail-closed (any other value raises `ValueError` rather than falling back to the default).
Author
Collaborator

README.md

docs [LOW]

Fixed in commit 2740a0d — the verification section now states the 2000-line cap is a fixed internal limit, independent of WUMING_MAX_FILE_LINES (which only controls how much context the review agents see).

[`README.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5567) > **docs** [LOW] Fixed in commit `2740a0d` — the verification section now states the 2000-line cap is a fixed internal limit, independent of `WUMING_MAX_FILE_LINES` (which only controls how much context the review agents see).
📝 clear pre-0.11.0 bullets from the release-notes intro post
All checks were successful
/ gitleaks (pull_request) Successful in 16s
/ checks (pull_request) Successful in 2m4s
/ deploy-docs (pull_request) Has been skipped
/ publish-container (pull_request) Has been skipped
/ pr-review (pull_request) Successful in 3m39s
7945d8aa70
Author
Collaborator

Marvin8's rulings on the two held findings

src/wuming/verify.py DEBUG repr · src/wuming/verify.py DEBUG repr (fetch side) · Release-Notes.md heading

  • 5551/5552 (DEBUG-level repr()): keeping as is — in this codebase tokens travel in HTTP headers only, httpx2 exception messages carry status lines/URLs but not headers, and DEBUG is gated behind the documented-sensitive WUMING_VERBOSE. No change.
  • 5536 (release-notes heading): resolved in commit 7945d8a — all pre-0.11.0 bullets cleared from the intro post; the version line stays at 0.11.0 and the historical 0.10.0 content remains in the thread's follow-on posts below.
### Marvin8's rulings on the two held findings [`src/wuming/verify.py` DEBUG repr](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5560) · [`src/wuming/verify.py` DEBUG repr (fetch side)](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5561) · [`Release-Notes.md` heading](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5539) - **5551/5552 (DEBUG-level `repr()`):** keeping as is — in this codebase tokens travel in HTTP headers only, httpx2 exception messages carry status lines/URLs but not headers, and DEBUG is gated behind the documented-sensitive `WUMING_VERBOSE`. No change. - **5536 (release-notes heading):** resolved in commit `7945d8a` — all pre-0.11.0 bullets cleared from the intro post; the version line stays at 0.11.0 and the historical 0.10.0 content remains in the thread's follow-on posts below.
forgejo-actions left a comment

WuMing

Found 3 issue(s). See inline comments below.

## WuMing Found **3** issue(s). See inline comments below.
@ -101,6 +101,7 @@ tools/
- `WUMING_PROGRESS` — `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent
- `WUMING_MAX_DIFF_LINES` — integer; if the raw diff line count exceeds this value and it is > 0, posts a plain PR comment and exits without inline review (default: `0` = disabled)
- `WUMING_MAX_FILE_LINES` — integer; when > 0, fetches the full content of each changed file from Forgejo and includes it in the agent's context (files exceeding this line count are excluded); (default: `0` = disabled)
- `WUMING_VERIFY` — `true`/`false` (case-insensitive); fact-check findings against full file content before posting (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap); parsing is fail-closed — any other value raises `ValueError` rather than falling back to the default; (default: `true`)

docs [LOW]

This entry crams three unrelated facts into one very dense run-on line with two levels of nested parentheses and semicolons: the behavior (fact-check ... before posting), the failure modes (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap), and the parsing rule (parsing is fail-closed ...). Aguably the fail-open and fail-closed sub-clauses are adjacent and easy to misread as a contradiction. Consider splitting into shorter sentences or separate list items (e.g. 'Fail-open on provider errors and crashes; skipped for unfetchable files and files over the internal 2000-line cap. Parsing is fail-closed: any value other than true/false raises ValueError (default: true).') to improve scannability and reduce misreading.

**docs** [LOW] This entry crams three unrelated facts into one very dense run-on line with two levels of nested parentheses and semicolons: the behavior (`fact-check ... before posting`), the failure modes (`fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap`), and the parsing rule (`parsing is fail-closed ...`). Aguably the `fail-open` and `fail-closed` sub-clauses are adjacent and easy to misread as a contradiction. Consider splitting into shorter sentences or separate list items (e.g. 'Fail-open on provider errors and crashes; skipped for unfetchable files and files over the internal 2000-line cap. Parsing is fail-closed: any value other than true/false raises ValueError (default: true).') to improve scannability and reduce misreading. <!-- wuming:sha256:6d249d0ffca5f870ea1b520de623aaf9b7b2781c6433858121a527ae391d753c -->
@ -108,0 +147,4 @@
seeded_contents=seeded_contents,
progress=progress,
)
except Exception as exc:

code [MEDIUM]

When the verification pass crashes, the exception is caught and logged but no stack trace is captured. The warning log only records the exception type and the debug log records repr(exc), but debug logging is often disabled in production, making it impossible to diagnose where/why verification failed. Add exc_info=True to the warning log (or use logger.exception) so the traceback is preserved.

**code** [MEDIUM] When the verification pass crashes, the exception is caught and logged but no stack trace is captured. The warning log only records the exception type and the debug log records repr(exc), but debug logging is often disabled in production, making it impossible to diagnose where/why verification failed. Add exc_info=True to the warning log (or use logger.exception) so the traceback is preserved. <!-- wuming:sha256:9622422b92e7476bb81d9d2c3eb917c55d1287ff8f966adb547d0885cfa7f4fe -->
marvin8 marked this conversation as resolved
@ -911,0 +1000,4 @@
expect(verify_mock.call_args.kwargs["seeded_contents"]).to_equal({})
expect(verify_mock.call_args.kwargs["progress"]).not_.to_be_none()
expect(len(verify_mock.call_args.args)).to_equal(4)
post_body = str(review_post_calls[0].content)

code [LOW]

The posted request body is inspected by calling str() on bytes, which yields the Python repr (e.g., "b'{...}'") rather than the decoded body. While the substring check happens to work for ASCII content, it is fragile and non-idiomatic. Use review_post_calls[0].content.decode("utf-8") or review_post_calls[0].text instead.

**code** [LOW] The posted request body is inspected by calling str() on bytes, which yields the Python repr (e.g., "b'{...}'") rather than the decoded body. While the substring check happens to work for ASCII content, it is fragile and non-idiomatic. Use review_post_calls[0].content.decode("utf-8") or review_post_calls[0].text instead. <!-- wuming:sha256:d705170809376f97c2cce7e2550e3eef70babdd3a1e1792b140154de1049add6 -->
🐛 log traceback on verification crash and tidy review fixes
All checks were successful
/ gitleaks (pull_request) Successful in 45s
/ pr-review (pull_request) Successful in 4m25s
/ checks (pull_request) Successful in 4m57s
/ deploy-docs (pull_request) Has been skipped
/ publish-container (pull_request) Has been skipped
90d2177ea4
Author
Collaborator

src/wuming/main.py

code [MEDIUM]

✅ Fixed in commit pending-push — the crash warning now carries exc_info=True, so the full traceback reaches WARNING-level logs regardless of debug settings. Regression test test_verification_crash_logs_traceback asserts the warning record carries exc_info.

[`src/wuming/main.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5577) > **code** [MEDIUM] ✅ Fixed in commit pending-push — the crash warning now carries `exc_info=True`, so the full traceback reaches WARNING-level logs regardless of debug settings. Regression test `test_verification_crash_logs_traceback` asserts the warning record carries exc_info.
Author
Collaborator

tests/test_main.py

code [LOW]

✅ Fixed in the same commit — all four body assertions now use .content.decode("utf-8") instead of str() on bytes.

[`tests/test_main.py`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5578) > **code** [LOW] ✅ Fixed in the same commit — all four body assertions now use `.content.decode("utf-8")` instead of `str()` on bytes.
Author
Collaborator

AGENTS.md

docs [LOW]

✅ Fixed in the same commit — the entry is rewritten as three short sentences (behaviour, fail-open semantics, fail-closed parsing) instead of nested parentheses.

[`AGENTS.md`](https://forge.marvin8.zone/marvin8/wuming/pulls/102#issuecomment-5579) > **docs** [LOW] ✅ Fixed in the same commit — the entry is rewritten as three short sentences (behaviour, fail-open semantics, fail-closed parsing) instead of nested parentheses.
marvin8 approved these changes 2026-10-16 20:55:16 +00:00
marvin8 manually merged commit 1bb440886c into main 2026-10-16 20:56:14 +00:00
Author
Collaborator

Correction to my reply on wave-3 comment 5565: I answered 🔴 "already covered — commit 9c7e76e" for the load_config WUMING_VERIFY integration tests, but that was wrong: the three tests (test_default_verify_findings, test_reads_verify_findings, test_reads_verify_findings_typo) were appended locally and never committed — 9c7e76e contains only the parser unit tests. They are now properly committed (5c89362) and will land via PR #103 alongside the two deferred minors. Apologies for the false ✅.

**Correction to my reply on wave-3 comment 5565:** I answered 🔴 "already covered — commit `9c7e76e`" for the `load_config` `WUMING_VERIFY` integration tests, but that was wrong: the three tests (`test_default_verify_findings`, `test_reads_verify_findings`, `test_reads_verify_findings_typo`) were appended locally and never committed — `9c7e76e` contains only the parser unit tests. They are now properly committed (`5c89362`) and will land via PR #103 alongside the two deferred minors. Apologies for the false ✅.
Sign in to join this conversation.
No reviewers
No milestone
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
marvin8/wuming!102
No description provided.