Verify findings against full file content before posting #102
No reviewers
Labels
No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No milestone
No assignees
3 participants
Notifications
Due date
No due date set.
Reference
marvin8/wuming!102
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/issue-82-verify-findings"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds a verification pass that fact-checks each surviving finding against the changed file's full content before it is posted.
rejectedverdict drops a finding — every failure mode fails open.WUMING_VERIFY=falsedisables the pass; enabled by default (fail-closed parsing — a typo raisesValueError).Reviewed against the wave-1 spec (
Review quality & convergencemilestone). Whole-branch fresh-context review passed: one critical (fetch-failure crash, fixed with RED→GREEN test), one documentation gap (CLAUDE.md symlink target uncommitted, fixed), release-notes entry added.Closes #82
WuMing
Found 7 issue(s). See inline comments below.
@ -165,0 +167,4 @@Before any finding is posted, WuMing fact-checks it: a dedicated LLM call receives the finding plus the full content of the changed file and decides whether the file actually contradicts the finding's factual claims. Only findings the file demonstrably contradicts are dropped — the verifier never judges style, importance, or severity, and it only sees the changed file itself, so claims about other files ("no test exists") are not verified.Verification is fail-open: a provider error or unparseable verdict keeps the finding. Files whose content cannot be fetched, or that exceed 2000 lines, skip verification entirely. The pass adds one LLM call per finding (at most 4 run concurrently), after the cheap filters — severity floor, line-in-diff, deduplication — so LLM spend goes only to findings that would otherwise be posted. Set `WUMING_VERIFY=false` to disable.docs [LOW]
This states a fixed 2000-line cutoff for skipping verification, but no such limit appears anywhere else in the documentation: AGENTS.md line 104 only says verification is "skipped for unfetchable/oversized files", and the adjacent
WUMING_MAX_FILE_LINESsetting (line 160) is user-configurable and defaults to 0 (disabled). As written, users cannot tell whether the verification cutoff is the hard-coded 2000 or is derived fromWUMING_MAX_FILE_LINES. Confirm the value against the implementation and either document the constant consistently in AGENTS.md too, or reference the configurable setting. If the number is not actually fixed, remove it and describe the behaviour instead.@ -3,3 +3,3 @@---wuming 0.10.0wuming 0.11.0docs [MEDIUM]
The top-of-file release heading was renamed from
wuming 0.10.0towuming 0.11.0instead of adding a new section, so the pre-existing entries that follow it — the SBOM attestation (line 8), cosign signing (line 9) and theopenrouterbackend (line 10) — are now attributed to 0.11.0 even though they shipped in 0.10.0 (the diff shows no second heading was introduced). If 0.10.0 was already released, this silently deletes that release's notes and mislabels its features. Fix by restoring awuming 0.10.0heading above the existing bullets and putting the new verification bullet under a newwuming 0.11.0heading placed above it. If 0.10.0 was never released and this is a deliberate rename of the unreleased draft, say so in the PR body to avoid the ambiguity.@ -0,0 +78,4 @@if verdict not in _VALID_VERDICTS:return Nonereason = item.get("reason")return (verdict, str(reason) if reason else "")code [LOW]
parse_verdict's docstring says the reason defaults to an empty string when it is 'not a string', but the implementation converts truthy non-string values to strings (e.g. integer 123 -> '123', list [1] -> '[1]'). Make the type check explicit with isinstance(reason, str) or update the docstring to match the actual behavior.
@ -0,0 +127,4 @@except Exception as exc:logger.warning("verifier call failed — keeping finding",extra={"path": comment.path, "line": comment.line, "error": repr(exc)},security [MEDIUM]
A09: Logging
repr(exc)from backend failures can leak sensitive data (API keys, tokens, URLs with credentials, or response bodies) into application logs. Exception messages from the LLM backend or HTTP client are untrusted and may embed secrets. Remediation: log only a sanitized error message or exception type, and avoidrepr(exc)for external/HTTP errors.@ -0,0 +176,4 @@return_exceptions=True,)for path, content in zip(to_fetch, fetched, strict=True):if isinstance(content, BaseException):code [MEDIUM]
fetch_file_contents uses asyncio.gather(return_exceptions=True) and then catches BaseException, which swallows asyncio.CancelledError. If the surrounding task is cancelled while fetches are in flight, the cancellation signal is returned as an exception in the results list and silently treated as a fetch failure, so the function returns normally instead of propagating cancellation. Re-raise asyncio.CancelledError before treating it as a fetch failure, and add a regression test.
@ -0,0 +179,4 @@if isinstance(content, BaseException):logger.warning("file content fetch failed — skipping verification for file",extra={"path": path, "error": repr(content)},security [MEDIUM]
A09: Logging
repr(content)for aBaseExceptionreturned byasyncio.gather(return_exceptions=True)can expose sensitive forgejo/HTTP error details (e.g., API tokens in URLs, response payloads) in logs. Remediation: log a generic fetch-failure message and record the exception type/name withoutrepr(); route full exception details to debug-level or scrubbed telemetry only.@ -0,0 +46,4 @@expect(parse_verdict(raw)).to_equal(expected)@test("parse_verdict returns None for anything that is not a clean verdict").cases(code [LOW]
No test exercises parse_verdict's markdown-fence extraction (verify.py lines 67-70). Add cases for fenced JSON, fences with a language tag (e.g. ```json), and responses that mix explanatory text with a fenced verdict block to prevent regressions in fence stripping.
Release-Notes.md👀 Valid catch, but this needs Marvin8's call rather than a fix from me: the git history shows the 0.11.0 release commit (
d173561, changelog update) shipped while this thread still readwuming 0.10.0— the release cycle itself left the header stale. My line-bump followed the AGENTS.md rule ("update the version line from pyproject.toml"), which produced exactly the misattribution you describe. The clean fix would be moving the three 0.10.0 bullets out of the intro post, but how the intro post interacts withjust releasebullet extraction is a release-workflow decision. Escalated to Marvin8.README.mdFixed in commit
6749594— the value is confirmed:VERIFICATION_MAX_FILE_LINES = 2000, a module constant insrc/wuming/verify.py(not derived fromWUMING_MAX_FILE_LINES). The AGENTS.md env-var entry now states the hard-coded 2000-line cap explicitly.src/wuming/verify.pyFixed in commit
634ee13— the WARNING log now carries only the exception type name (error_type); the fullrepr()moved to a DEBUG line, visible underWUMING_VERBOSE. Regression testtest_fetch_failure_log_sanitizedasserts the message text never reaches WARNING-level records.src/wuming/verify.pyFixed in commit
634ee13—asyncio.CancelledErroris re-raised before the fetch-failure mapping. Regression testtest_fetch_cancellation_propagatespins it.src/wuming/verify.pyFixed in commit
634ee13— the docstring now matches the implementation: absent/null reason defaults to empty string, truthy non-string values are coerced withstr().src/wuming/verify.pyFixed in commit
634ee13— same treatment as the fetch-failure site: WARNING carries onlyerror_type, fullrepr()moved to DEBUG. Regression testtest_verify_failure_log_sanitizedasserts backend exception message text never appears in WARNING-level records.tests/test_verify.pyFixed in commit
634ee13—test_parse_verdict_fencesnow covers a plain fence, a ```json-tagged fence, and prose-then-fence responses.WuMing
Found 7 issue(s). See inline comments below.
@ -158,10 +158,17 @@ See the [Forgejo/GitHub Actions guide](https://wuming.marvin8.zone/getting-start| `WUMING_PROGRESS` | — | `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent || `WUMING_MAX_DIFF_LINES` | `0` | Maximum raw diff size in lines. When the diff exceeds this value and it is greater than 0, WuMing posts a single plain PR comment explaining the PR is too large to review automatically, then exits without posting inline comments. Set to `0` to disable. || `WUMING_MAX_FILE_LINES` | `0` | Maximum file size (in lines) for full-file context fetching. When greater than 0, WuMing fetches the complete content of each changed file from Forgejo and includes it in the agent's context, giving the model visibility beyond the diff hunks. Files exceeding this line count are excluded. Set to `0` (the default) to disable. || `WUMING_VERIFY` | `true` | When `true`, each candidate finding is fact-checked against the changed file's full content before posting; findings whose factual claims the file contradicts are dropped. Verification is fail-open: provider errors keep findings. Set to `false` to disable. |docs [LOW]
The table row documents
true/falsebut does not mention that parsing is strict (per the PR, an invalid value such asWUMING_VERIFY=yesraises aValueErrorrather than falling back to the default). Since the option is enabled by default and fail-closed, users can unintentionally abort a run. State explicitly that onlytrue/falseare accepted and any other value is an error.@ -0,0 +120,4 @@user_message = (f"## Finding under review\n\n{_format_finding(comment)}\n\n"f"## Full file content: {comment.path}\n\n"f"```\n{file_content}\n```\n\n"code [MEDIUM]
The file content is wrapped in a plain triple-backtick fence. If file_content itself contains triple backticks, the fence closes early and the rest of the prompt is misinterpreted, which can lead to an incorrect 'rejected' verdict and a valid finding being dropped despite the fail-open design. Escape embedded fences or dynamically choose a fence longer than the longest run of backticks in the content.
@ -0,0 +136,4 @@)logger.debug("verifier call failure detail",extra={"path": comment.path, "line": comment.line, "error": repr(exc)},security [MEDIUM]
A02: Logging repr(exc) from backend.complete may leak API keys or bearer tokens if the HTTP exception includes request/response headers. Replace the full exception repr with a sanitized error message or explicitly strip authorization headers before logging.
@ -0,0 +187,4 @@for path, content in zip(to_fetch, fetched, strict=True):if isinstance(content, asyncio.CancelledError):raise contentif isinstance(content, BaseException):code [LOW]
After re-raising CancelledError, all other BaseException subclasses (including KeyboardInterrupt and SystemExit) are caught and treated as fetch failures. User-interruption signals raised inside get_file_content should propagate instead of being logged and swallowed. Only Exception subclasses should be treated as fetch failures after the cancellation check.
@ -0,0 +194,4 @@)logger.debug("file content fetch failure detail",extra={"path": path, "error": repr(content)},security [MEDIUM]
A02: Logging repr(content) of a Forgejo fetch exception may leak API tokens if the HTTP error includes request headers. Summarize or sanitize the exception instead of logging its full repr.
@ -0,0 +253,4 @@A ``(kept, dropped)`` pair, both preserving input order."""if progress is not None:progress.step_start("verify_findings", "Fact-checking findings…")code [MEDIUM]
If any awaited call inside verify_comments raises (e.g. asyncio.CancelledError from fetch_file_contents or asyncio.gather), progress.step_done is never reached, leaving the progress reporter in a started-but-not-completed state. Wrap the body between step_start and step_done in a try/finally and call step_done in the finally block.
@ -437,3 +438,28 @@ def test_reads_summary_comment(env: Annotated[None, Depends(clean_env)]) -> Nonewith patch.dict("os.environ", {"WUMING_SUMMARY_COMMENT": "true"}, clear=False):config = load_config()expect(config.summary_comment).to_be_truthy()code [LOW]
There is no test that load_config reads WUMING_VERIFY and wires it into Config.verify_findings. The parser function is unit-tested, but the load_config integration path (default-true behaviour and the typo-raises-ValueError path) is not covered.
src/wuming/verify.pyFixed in commit
9c7e76e— the body betweenstep_startandstep_donenow runs in try/finally withstep_donein thefinallyblock. Regression testtest_verify_comments_progress_step_done_on_exceptiondrives a cancelled fetch through a spy progress reporter and asserts the step closes.src/wuming/verify.pyFixed in commit
9c7e76e— the fence is now chosen dynamically: longer than the longest backtick run in the file content (minimum 3). Regression testtest_verify_finding_fence_escapes_contentverifies content containing ``` fences survives verbatim inside a ```` fence.src/wuming/verify.pyFixed in commit
9c7e76e— onlyExceptionsubclasses are treated as fetch failures; any otherBaseException(CancelledError, KeyboardInterrupt, SystemExit) is re-raised. Regression test uses a KeyboardInterrupt-class stand-in (test_fetch_keyboard_interrupt_propagates) because a real KeyboardInterrupt raised in a task kills the test runner's worker.tests/test_config.pyFixed in commit
9c7e76e— three load_config integration tests added: default-true, explicitfalsewiring, and the typo-raises-ValueErrorpath.README.mdFixed in commit
29bd0f0— the row now states that onlytrue/false(case-insensitive) are accepted and any other value is an error, not a fallback.src/wuming/verify.py📌Holding for Marvin8's review — this re-raises the substance of wave-1 finding 5534, where the agreed remedy moved full exception
repr()to DEBUG. Two mitigating facts: in this codebase tokens travel in HTTP headers (never URLs or query strings), and httpx2 exception messages carry status lines and URLs but not headers; and DEBUG logs are gated behindWUMING_VERBOSE, which AGENTS.md already documents as sensitive. Stripping the repr from DEBUG entirely would leave only the exception class name for debugging provider failures.src/wuming/verify.py📌 Same answer as 5551 above — consolidated: the WARNING level is already sanitized (type name only, per wave-1 fix); the question is whether the DEBUG-level repr should exist at all. Holding for Marvin8's decision.
WuMing
Found 5 issue(s). See inline comments below.
@ -101,6 +101,7 @@ tools/- `WUMING_PROGRESS` — `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent- `WUMING_MAX_DIFF_LINES` — integer; if the raw diff line count exceeds this value and it is > 0, posts a plain PR comment and exits without inline review (default: `0` = disabled)- `WUMING_MAX_FILE_LINES` — integer; when > 0, fetches the full content of each changed file from Forgejo and includes it in the agent's context (files exceeding this line count are excluded); (default: `0` = disabled)- `WUMING_VERIFY` — `true`/`false`; fact-check findings against full file content before posting (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap); (default: `true`)docs [MEDIUM]
This new entry documents only
true/falseand the fail-open behaviour, but omits two things the README documents for the same variable: (1) parsing is fail-closed — any value other thantrue/false(case-insensitive) raises an error rather than silently falling back to the default, and (2) the values are case-insensitive. A reader relying on AGENTS.md could setWUMING_VERIFY=1orWUMING_VERIFY=yesand get a hard startup failure with no warning from the docs. Add the fail-closed/hard-error note (and the case-insensitivity) so both reference files agree.@ -165,0 +167,4 @@Before any finding is posted, WuMing fact-checks it: a dedicated LLM call receives the finding plus the full content of the changed file and decides whether the file actually contradicts the finding's factual claims. Only findings the file demonstrably contradicts are dropped — the verifier never judges style, importance, or severity, and it only sees the changed file itself, so claims about other files ("no test exists") are not verified.Verification is fail-open: a provider error or unparseable verdict keeps the finding. Files whose content cannot be fetched, or that exceed 2000 lines, skip verification entirely. The pass adds one LLM call per finding (at most 4 run concurrently), after the cheap filters — severity floor, line-in-diff, deduplication — so LLM spend goes only to findings that would otherwise be posted. Set `WUMING_VERIFY=false` to disable.docs [LOW]
The 2000-line limit is described only as a bare number here ("exceed 2000 lines"), while AGENTS.md calls it a "hard-coded" cap. Because the preceding table row introduces
WUMING_MAX_FILE_LINES(default0) for full-file context fetching, readers may reasonably assume raisingWUMING_MAX_FILE_LINESalso extends or disables the verification cap. State explicitly that the verification limit is fixed at 2000 lines and is independent of, and not configurable by,WUMING_MAX_FILE_LINES.@ -90,6 +91,32 @@ def load_config() -> Config:dry_run=_parse_bool_flag(os.environ.get("WUMING_DRY_RUN")),verbose=_parse_bool_flag(os.environ.get("WUMING_VERBOSE")),summary_comment=_parse_bool_flag(os.environ.get("WUMING_SUMMARY_COMMENT")),verify_findings=_parse_verify_enabled(os.environ.get("WUMING_VERIFY", "")),code [LOW]
There is no integration test for load_config mapping WUMING_VERIFY into Config.verify_findings. The unit tests cover _parse_verify_enabled in isolation, but a regression in load_config (e.g., forgetting to pass the env value or using the wrong variable name) would not be caught. Add load_config tests for default-on, explicit false, and typo behavior.
@ -99,6 +100,26 @@ async def _main() -> None:raw_comments, position_maps, existing_markers, file_diffs)if config.verify_findings and new_comments:code [MEDIUM]
The verification pass is invoked without any exception handling. Per the PR description, verification must fail open for every failure mode; an unexpected exception from verify_comments (e.g., a bug, malformed LLM response, or unhandled network error) would bubble up and crash the pipeline before comments are published, causing a hard failure instead of preserving findings. Wrap the verify_comments call in a try/except that logs the failure and keeps the original new_comments list.
@ -911,0 +988,4 @@_verify_pipeline_env(),)expect(verify_mock.await_count).to_equal(1)survivors = verify_mock.call_args.args[0]code [LOW]
The verification pipeline tests only inspect verify_mock.call_args.args[0] (the survivors list). They do not assert that verify_comments receives the correct keyword arguments (seeded_contents, progress, backend, forgejo) or that seeded_contents is populated from file_diffs. Add assertions on call_args.kwargs to cover the integration contract.
src/wuming/main.pyFixed in commit
2740a0d— the verification call in_mainis now wrapped fail-open: any unexpected exception logs a warning (type name; repr at DEBUG) and posts the findings unverified. Regression testtest_pipeline_verification_crash_fails_opendrives a crashing verifier and asserts the review still posts. Deliberately NOT caught:CancelledError,KeyboardInterrupt,SystemExit— cancellation and interruption must still propagate.tests/test_main.pyFixed in commit
2740a0d—test_pipeline_verifies_survivorsnow also assertsseeded_contents == {}(the default-env wiring: no full contents attached), a non-Noneprogress, and the four positional arguments (survivors, forgejo, backend, agent configs).src/wuming/config.py🔴 Already covered — commit
9c7e76e(in this PR, before your wave) added exactly these load_config integration tests:test_default_verify_findings(default-true),test_reads_verify_findings(explicit false wired through), andtest_reads_verify_findings_typo(typo raisesValueErrornaming the variable). No change needed.AGENTS.mdFixed in commit
2740a0d— the AGENTS.md entry now documents both: values are case-insensitive, and parsing is fail-closed (any other value raisesValueErrorrather than falling back to the default).README.mdFixed in commit
2740a0d— the verification section now states the 2000-line cap is a fixed internal limit, independent ofWUMING_MAX_FILE_LINES(which only controls how much context the review agents see).Marvin8's rulings on the two held findings
src/wuming/verify.pyDEBUG repr ·src/wuming/verify.pyDEBUG repr (fetch side) ·Release-Notes.mdheadingrepr()): keeping as is — in this codebase tokens travel in HTTP headers only, httpx2 exception messages carry status lines/URLs but not headers, and DEBUG is gated behind the documented-sensitiveWUMING_VERBOSE. No change.7945d8a— all pre-0.11.0 bullets cleared from the intro post; the version line stays at 0.11.0 and the historical 0.10.0 content remains in the thread's follow-on posts below.WuMing
Found 3 issue(s). See inline comments below.
@ -101,6 +101,7 @@ tools/- `WUMING_PROGRESS` — `true`/`1` force on; `false`/`0` force off; when unset or empty, auto-detects: enabled when stdout is a TTY and `CI` env var is absent- `WUMING_MAX_DIFF_LINES` — integer; if the raw diff line count exceeds this value and it is > 0, posts a plain PR comment and exits without inline review (default: `0` = disabled)- `WUMING_MAX_FILE_LINES` — integer; when > 0, fetches the full content of each changed file from Forgejo and includes it in the agent's context (files exceeding this line count are excluded); (default: `0` = disabled)- `WUMING_VERIFY` — `true`/`false` (case-insensitive); fact-check findings against full file content before posting (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap); parsing is fail-closed — any other value raises `ValueError` rather than falling back to the default; (default: `true`)docs [LOW]
This entry crams three unrelated facts into one very dense run-on line with two levels of nested parentheses and semicolons: the behavior (
fact-check ... before posting), the failure modes (fail-open; skipped for unfetchable files and files over the hard-coded 2000-line cap), and the parsing rule (parsing is fail-closed ...). Aguably thefail-openandfail-closedsub-clauses are adjacent and easy to misread as a contradiction. Consider splitting into shorter sentences or separate list items (e.g. 'Fail-open on provider errors and crashes; skipped for unfetchable files and files over the internal 2000-line cap. Parsing is fail-closed: any value other than true/false raises ValueError (default: true).') to improve scannability and reduce misreading.@ -108,0 +147,4 @@seeded_contents=seeded_contents,progress=progress,)except Exception as exc:code [MEDIUM]
When the verification pass crashes, the exception is caught and logged but no stack trace is captured. The warning log only records the exception type and the debug log records repr(exc), but debug logging is often disabled in production, making it impossible to diagnose where/why verification failed. Add exc_info=True to the warning log (or use logger.exception) so the traceback is preserved.
@ -911,0 +1000,4 @@expect(verify_mock.call_args.kwargs["seeded_contents"]).to_equal({})expect(verify_mock.call_args.kwargs["progress"]).not_.to_be_none()expect(len(verify_mock.call_args.args)).to_equal(4)post_body = str(review_post_calls[0].content)code [LOW]
The posted request body is inspected by calling str() on bytes, which yields the Python repr (e.g., "b'{...}'") rather than the decoded body. While the substring check happens to work for ASCII content, it is fragile and non-idiomatic. Use review_post_calls[0].content.decode("utf-8") or review_post_calls[0].text instead.
src/wuming/main.py✅ Fixed in commit pending-push — the crash warning now carries
exc_info=True, so the full traceback reaches WARNING-level logs regardless of debug settings. Regression testtest_verification_crash_logs_tracebackasserts the warning record carries exc_info.tests/test_main.py✅ Fixed in the same commit — all four body assertions now use
.content.decode("utf-8")instead ofstr()on bytes.AGENTS.md✅ Fixed in the same commit — the entry is rewritten as three short sentences (behaviour, fail-open semantics, fail-closed parsing) instead of nested parentheses.
Correction to my reply on wave-3 comment 5565: I answered 🔴 "already covered — commit
9c7e76e" for theload_configWUMING_VERIFYintegration tests, but that was wrong: the three tests (test_default_verify_findings,test_reads_verify_findings,test_reads_verify_findings_typo) were appended locally and never committed —9c7e76econtains only the parser unit tests. They are now properly committed (5c89362) and will land via PR #103 alongside the two deferred minors. Apologies for the false ✅.