Repository navigation
Give the legacy audit caller evidence and keep the default audit to verdicts - #26
Open
ccarpene-blk wants to merge 1 commit into
Open
ccarpene-blk wants to merge 1 commit into
ccarpene-blk wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The legacy audit used by the
defaultprompt set saw only the files its findings cited. A route guard or middleware registered beside a handler was invisible to it, so a claim such as a missing authorization check could be confirmed even though the guard exists in a file the auditor never received. The same audit could also return new findings of its own, which bypassed discovery and arrived without the batch context needed to avoid repeating other batches' claims.Caller evidence for every audit batch
internal/cli/scan_audit.go:171). Each batch receives up to about 24K tokens of excerpts with real line numbers, taken round-robin across the batch's claims and never repeating a file the batch already includes.internal/auditctx/auditctx.go:102,:498).internal/cli/scan_audit.go:238). If the rendered context would still exceed the limit, the batch is sent with an omission note in place of the excerpts (:265). The claims' own files are always sent.Audit context omitted: ambiguous caller reference <name> in <file>orAudit context omitted: unresolved or external caller reference <name> in <file>, so a guard that was not retrieved is not mistaken for a guard that does not exist.@/and~/aliases and directoryindexfiles resolve to scanned files; other import prefixes are treated as external. The notes are listed inREADME.md:864.Audit additions in the default set
allow_new_findingssetting (internal/llm/prompt.go:87,:797-798). When it isfalse, the audit request uses a schema whosenew_findingsarray is described as required to be empty (internal/llm/schema.go:175-178); the array is kept for provider compatibility. Any additions a model returns anyway are discarded before verdicts are applied, and their count is recorded asexecution.auditAdditionsDroppedin the run metadata (internal/cli/scan_audit.go:375,internal/sarif/metadata.go:102), omitted when zero.prompts/default/audit.yaml:1setsallow_new_findings: false. Sets that omit the key keep the previous behavior, and a v2 audit has no additions, so the setting has no effect there.README.md:322and theexecutionrow of the SARIF property table document the setting and the counter.Release note
defaultprompt set no longer reports audit-originated findings (auditStatus: "new"). Its audit only issues verdicts on analysis findings. Other v1 prompt sets are unchanged unless they setallow_new_findings: false.Validation
internal/auditctx/caller_regression_test.gocovers mounted and bare middleware references, Go wrappers, neighbouring registrations that must not be pulled in, external imports that must not resolve to a local namesake, import-path boundaries, omission notes and their deduplication.TestLegacyAuditRetrievesMountedGuardWithinBudget(internal/cli/audit_caller_context_test.go:17) runs a legacy audit end to end and checks that the guard is retrieved within the context budget.TestAuditDisablesUnsolicitedAdditions(internal/cli/audit_recovery_test.go) uses a model endpoint that ignores the empty-array instruction and checks the schema sent, the results kept and the dropped count with the setting both on and off; schema tests cover the existing-claims schema.make build,go test -race -count=1 ./...,go vet ./...andgit diff --checkpass; lint is unchanged frommain.Merge order with the #18–#20 stack
This PR targets
mainand conflicts with the open audit-integrity stack (#18 onward) in the legacy audit context code. #19 moves source extraction intointernal/sourceindex, so if this lands first, the stack needs the caller lookup ported intointernal/sourceindexwith the claim-citation step kept ahead of retrieval. That port has been prepared and validated locally (full race suite, vet and lint pass, with a regression test for cited ranges reaching the legacy audit prompt) and can be applied when the stack is restacked.