test(lh-114518): add SCCFM agent harness - #37
Open
huides00 wants to merge 3 commits into
Open
Conversation
Add a local and CI-compatible harness that exercises the SCCFM Codex plugin and canonical skills through real agent sessions backed by deterministic SCCFM and Ansible command doubles. Report safety, functional, quality, and reliability evidence in JSON, Markdown, and an interactive dashboard. Detect stale installed plugins, preserve caches used by open tasks during refresh, and cover routing, secret handling, setup, uninstall, and guarded mutations. Harden command approval for Ansible check mode and document the confirmation and sampling workflows. Bump both plugin manifests to 0.1.2. The full test suite, static checks, skill synchronization, plugin validation, and three installed-agent samples pass.
huides00
requested review from
Scoombe,
afercal and
siddhuwarrier
as code owners
September 9, 2026 11:50
Resolve the system sh executable instead of assuming macOS zsh so the cleanup stub test runs on GitHub's Ubuntu runner.
Record the agent harness and guarded check-mode improvements in the Ansible release history so the 0.41.1 release rehearsal can complete.
Consistency CheckNo consistency issues found. Checker output |
Scoombe
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://jira-eng-rtp3.cisco.com/jira/browse/LH-114518
Description
SCCFM plugin maintainers need repeatable evidence that real agents select the right skill, use safe commands, protect credentials, and stop before unapproved mutations without relying on customer tenants or credentials.
The new local and CI-compatible harness runs real Codex sessions against deterministic SCCFM and Ansible command doubles, records functional, safety, quality, and reliability evidence, and renders JSON, Markdown, and interactive dashboard reports. It also detects stale local plugin installations, safely refreshes them while preserving caches used by open tasks, and strengthens check-mode and exact-command confirmation behavior.
Both Codex and Claude plugin manifests advance to 0.1.2. The full test suite, static checks, skill synchronization, plugin validation, and three installed-plugin samples pass.