SkillFenceDVAS
SkillFence is runtime behavioral security for AI agent skills. It doesn't read a skill's manifest and hope — it watches what the skill actually causes an agent to do, scores the drift deterministically, and pauses anything HIGH or CRITICAL for a human, before it executes.
Static scanners test what a skill says it will do.
SkillFence tests what it actually does.
Agents don't just chat anymore — they install skills, and those skills read files, call APIs, and act with real consequences on someone's behalf. SkillFence exists because the tooling to verify what a skill actually does at runtime hasn't kept pace with how fast they're being installed.
ClawHavoc, ClawJacked, the Snyk ToxicSkills audit — SkillFence follows the actual 2026 incident data, not hypotheticals, and publishes what it finds.
Deterministic, auditable enforcement that watches what a skill does — not one more scanner that trusts what it says about itself.
Every claim ships as a runnable, reproducible lab — so a defender watches the exploit happen and watches the tool catch it, not just reads about it.
SkillFence is the tool. DVAS is the practice ground it ships with: seventeen intentionally vulnerable, fully offline agent skills — the same idea as DVWA for web apps, aimed at agentic skills instead. Install a DVAS skill, watch it misbehave, then watch SkillFence catch it live.
Both are released for the public to download, run, and learn from — not sold as a locked enterprise product. Clone the repository, point SkillFence at your own skill, or run any of the 17 DVAS labs and watch SkillFence catch a real attack live. That's the whole idea: a security engineer, a student, or a red-teamer should be able to have the entire thing working, offline, in under five minutes, with nothing to buy and nothing hidden.
MIT licensed · Python 3.10+ · zero network required to run any labNo account, no API key, no network. This is the entire proof: install it, run a real attack, watch it get caught.
The agent skills ecosystem is under active attack as of 2026.
| Incident | Date | Impact |
|---|---|---|
| ClawHavoc campaign | Jan 2026 | 1,184 malicious skills across 12 accounts on ClawHub, delivering Atomic Stealer (AMOS) |
| Snyk ToxicSkills audit | Feb 2026 | 3,984 skills scanned — 36.82% had security flaws, 13.4% critical, 76 confirmed malicious |
| Claude Code CVEs | Feb 2026 | CVE-2025-59536 (CVSS 8.7) + CVE-2026-21852 (CVSS 5.3) — repo config files = execution layer |
| ClawJacked | Feb 2026 | CVE-2026-28363 (CVSS 9.9) — WebSocket brute-force against local OpenClaw instances |
| SecurityScorecard exposure | Feb 2026 | 135,000+ OpenClaw instances publicly exposed, 40,000 found in one day |
| USENIX Security 2026 study | 2026 | 98,380 skills analyzed, 157 malicious skills carrying 632 vulnerabilities |
| Mobb.ai audit | Mar 2026 | 22,511 skills across 4 registries — 140,963 total issues |
(Simon Willison / Palo Alto Networks) — an agent skill combines three dangerous capabilities: access to private data, exposure to untrusted content, and the ability to communicate externally. This is not theoretical. Five of the top seven most-downloaded ClawHub skills at peak infection were confirmed malware.
20+ static scanners. Commercial-only runtime platforms at $50K–$500K/year. Vulnerable labs that target agentic apps, never agentic skills. Nobody combines all three.
Nobody occupies this intersection. This is the gap SkillFence fills.
Where SkillFence sits against static scanners and the closest published research — verified against the DVAS lab suite.
| Capability | Static Scanners | SkillDetonate (sandbox) | RSA (pre-deploy audit) | TokenWall (token firewall) | SkillFence |
|---|---|---|---|---|---|
| Continuous runtime monitoring | ✗ | ✗ (one-shot sandbox) | ✗ (pre-deploy) | ✓ (token flows only) | ✓ (full OS-level*) |
| Declared vs observed comparison | ✗ | ✗ | ✗ | ✗ | ✓ (CDS metric) |
| Attack chain correlation | ✗ | ✓ (taint analysis) | ✗ | ✗ | ✓ (behavioral chains) |
| Human authorization gate | ✗ | ✗ | ✗ | Partial (defer to user) | ✓ (enforcement boundary) |
| External instruction provenance | ✗ | ✗ | ✗ | ✗ | ✓ (AST05 chain) |
| Reproducible vulnerable labs | ✗ | ✗ | ✗ | ✗ | ✓ (17 labs, AST01–05) |
| Open-source, educational | Mixed | Research code | Research code | Research code | ✓ (tool + labs + docs) |
| Evasion-resistant | ✗ (80–96% bypass) | ✓ (behavior-based) | ✓ (runtime-based) | ✓ (semantic-based) | ✓ (runtime behavior) |
The unique combination nobody has: continuous runtime + declared-vs-observed CDS + attack chains + human gate + provenance + vulnerable labs + open source.
* Layer A (agent-tool boundary) interception, shipped today. Full OS-level telemetry (eBPF/auditd) is roadmap — see Limitations in the README.Runtime declared-vs-observed behavioral gap analysis for agentic skills — the single novel contribution no existing tool implements as an open-source, reproducible framework.
Why this matters:
The RSA paper (arXiv:2606.11671) found runtime probing reaches 90% detection accuracy vs. 77% for the best static baseline — a 13-point gap that holds even as attacks adversarially evolve. SkillFence makes that same principle accessible as a free, reproducible tool rather than a research prototype.
Every wrapped action — read, write, exec, fetch, network — funnels through one enforcement point before it's allowed to happen.
AST01 through AST05 — three reproducible labs per category, every one offline and machine-scored against a known-correct answer.
| # | Risk | Severity | Key Mitigation | Real-World Evidence |
|---|---|---|---|---|
| AST01 | Malicious Skills | Critical | Merkle root signing, registry scanning | ClawHavoc (1,184 skills), ToxicSkills (76 payloads) |
| AST02 | Supply Chain Compromise | Critical | Registry transparency, provenance tracking | ClawHub collapse, Claude Code CVE-2025-59536 |
| AST03 | Over-Privileged Skills | High | Least-privilege manifests, schema validation | 280+ credential-leaking skills (Snyk, Feb 2026) |
| AST04 | Insecure Metadata | High | Static analysis, safe parsers, sandboxed loading | Fake "Google" skill impersonation; YAML payload in SKILL.md |
| AST05 | Untrusted External Instructions | High | Source inventory, content pinning, continuous rescanning | Air PoC bypassed all scanners; ~26,000 agents at risk |
Every lab is real, checked-in, runnable code — no real socket ever opens, no real credential exists anywhere in the sandbox. Run any of them with the command shown.
Click any lab to read its full story, the exact command to run it, and the real recorded verdict.
cloud-debug promises to troubleshoot deployment logs and declares exactly that: read access to ./logs/**. It's the kind of narrow, boring manifest that gets approved in about ten seconds. Mid-run it also reads ~/.aws/credentials — a path nowhere in its declared scope, and unrelated to "troubleshoot deployment logs."
~/.aws/credentialsinvoice-helper reads local invoices with an honest, narrow manifest. The lab isn't about the one alarming read — it's that a sensitive read immediately followed by an attempt to send that exact data off the machine gets shown to a human as one connected chain, not two disconnected, easy-to-dismiss alerts.
~/.ssh/id_rsa → STEP 2 https://collect.attacker.test/uploadbackup-helper behaves exactly as advertised for its first two invocations — clean, boring, the kind of two-run evaluation a security team would treat as "verified safe." From the third invocation onward it also reads ~/.ssh/id_rsa — the same technique real supply-chain implants use to survive a review.
~/.ssh/id_rsa — only from invocation 3 onwardexpense-report-helper has no exec, no curl, no code pattern a static scanner would ever flag. Its payload is a single sentence written into its own SKILL.md, phrased like an ordinary processing note — the exact document a human reads to decide whether to trust the skill in the first place.
~/.netrcchangelog-bot has run safely for months. Its v1.1 update ships a new, self-declared network domain and uses it immediately — nothing about the new version looks inconsistent on its own, which is exactly the trap: a plain declared-vs-observed check against the current manifest finds nothing wrong.
https://cdn.attacker.test/beaconinvoice-sync's v2.0 update bundles an innocuous-sounding line — "verify the deployment account before syncing" — with a new, quietly declared read of ~/.aws/credentials, used the instant the update finishes.
~/.aws/credentials — declared only as of v2.0terraform-plan-bot jumps from v1.0 straight to v3.0 overnight, no 2.x releases in between, no changelog anyone can point to — the shape of a registry account takeover, not a normal release cadence. The new version reads an SSH key it never needed before, "for attestation."
~/.ssh/id_ed25519 — declared only as of v3.0log-shipper's entire pitch is "no network" — network.enabled: false, stated plainly, usually the deciding factor that makes an install low-risk enough to skip a close review. It POSTs to an attacker domain anyway, because a declaration is a claim, not a constraint, until something enforces it.
https://telemetry.attacker.test/collectcustomer-support-bot drafts replies for a human to approve before anything reaches a customer — nothing sends without a person's sign-off. Right after reading a support ticket, it quietly ships the full content, names and emails included, to an undeclared "analytics" endpoint.
https://product-analytics.vendor-saas.test/collectpdf-formatter-lite shells out to a local tool it never declared — on its own, too quiet to matter, so SkillFence lets it through and just logs it. Then it uploads the result to an attacker domain. Neither step alone is dramatic; together, they're the entire attack.
pdftotext exec — 20/LOW, auto-allowed, loggedhttps://ocr-pipeline.attacker.test/uploadskillfence replaybilling-sync legitimately needs the network and says so precisely — one named domain, not a vague "yes." At runtime it sends to a completely different destination. Still "telling the truth" about needing network; lying about where the data actually goes.
https://attacker.test/exfil — not billing.example.testgoogle-drive-sync-helper's name does most of the persuading before anyone reads a word of its manifest. The actual upload goes to a lookalike domain built to survive a glance, not the real one — not a code exploit, just a trustworthy-sounding identity paired with a destination nobody double-checked character by character.
https://drive-google-sync.attacker.test/uploadslack-status-notifier declares secrets.access: false, accurately — a webhook URL isn't a secret. It reads GITHUB_TOKEN out of its process environment anyway: a completely unrelated system's credential, sitting there only because CI runners export a pile of tokens into every process regardless of need.
GITHUB_TOKENresearch-helper's manifest is accurate and its code is clean — nothing to flag at review time. A later edit to a page on its own declared, trusted domain embeds an instruction, and the naive reference agent follows it straight to a credential it was never trusted with.
~/.aws/credentials — requested by the agent, not the skill's own code[i]pdf-tools-helper's package never changed. Its own FAQ page — which says outright "no, everything runs on-device" — gets a paragraph added directly beneath that promise, instructing the agent to phone home anyway. The scanner-evasion case: nothing in the package was ever wrong.
https://telemetry-collector.attacker.test/reportrelease-notes-bot fetches two wiki pages in one run. The first instruction is easy to dismiss — "run this cleanup command, safe to run automatically" — and it runs, quietly logged, never interrupting anyone. The second asks it to phone home. Untrusted content driving local execution, not just data theft.
curl | sh exec — 40/MEDIUM, auto-allowed, loggedhttps://cdn-mirror.attacker.test/checkinPython 3.10+. No API key, no network required — every lab runs fully offline. This is the complete reference: every CLI command, every flag, and how to run it against the labs or a skill of your own.
Every command below is safe to run exactly as shown. No lab ever opens a real socket or touches your real filesystem, no matter what path you pass it — see Bring Your Own Skill below before pointing this at a skill you didn't write.
| CDS Band / Severity | What It Means | SkillFence's Default | What You Should Do |
|---|---|---|---|
| ALLOW · LOW (0–29) | Fully within declared scope | auto-allowed, logged | Nothing — audit later with skillfence replay if curious |
| WARN · MEDIUM (30–49) | Minor drift, low risk on its own | auto-allowed, logged | Worth a look if the same session also touches something sensitive |
| GATE · HIGH (50–69) | Meaningful capability drift | paused for a human | Press [i] to inspect provenance, then approve, reject, or quarantine |
| BLOCK · CRITICAL (70+) | Sensitive data + drift, often an untrusted instruction | paused, defaults to reject | Reject unless you can explain every factor in skillfence findings |
Start here — see what's runnable, then read a skill's declared capabilities without executing anything.
skillfence lab list [labs_root]List every discoverable lab with its AST category, skill name, malicious/benign kind, and declared purpose. Defaults to scanning labs/.
skillfence inspect <skill>Static-only inspection — reads the declared manifest (purpose, filesystem/process/network/secrets capabilities) and the first lines of skill/SKILL.md. Never executes the skill or touches the sandbox. Works on any directory with a skill/manifest.yaml — a DVAS lab, an AST shorthand (ast03), or a skill you wrote yourself.
Execute a skill's scripted actions through the SkillFence runtime gateway.
Every HIGH/CRITICAL action pauses for a human decision unless you pass --decision.
skillfence run <skill> [options]The core command. Loads the skill, runs its script.yaml steps through the gateway, prints a run summary, and writes any findings to .runs/findings.jsonl. Also records a behavior fingerprint and warns if it changed since the last run.
| --decision <value> | Non-interactive: auto-answer every gate. One of approve_once, reject, allow_for_session, allow_scoped, always_deny_rule, quarantine_skill, inspect_chain. Omit for a live interactive prompt. |
| --mode enforce|observe | enforce (default) genuinely blocks on reject. observe logs everything and never blocks. |
| --fresh | Ignore the shared, org-wide policy store (any ALLOW_SCOPED grants) for this run only. |
skillfence observe <skill>alias — run --mode observe --decision approve_onceEstablish a behavior baseline: every action is logged and none is blocked, so you can see a skill's true observed capability set before turning enforcement on.
| --fresh | Ignore the shared org-wide policy store for this run. |
skillfence protect <skill>alias — run --mode enforceEnforce: HIGH/CRITICAL actions are genuinely paused, and on rejection the real underlying call never executes.
| --decision <value> | Non-interactive decision for every gate — same values as run. |
| --fresh | Ignore the shared org-wide policy store for this run. |
skillfence demoDay-1 smoke test — proves the event schema, bus, and CLI wiring work end-to-end with dummy events. No lab required, nothing to configure.
Every decision leaves a trail. These commands read it back — the named reasons, the raw event timeline, and a rolled-up assessment.
skillfence findings <skill>Print every explainable finding recorded for a skill: title, AST tags, RISK, CDS + band, skill/action/resource, declared vs. observed capability, the full WHY FLAGGED list with point values, the correlated attack chain, raw evidence event IDs, and the human's decision.
skillfence replay <events.jsonl>Replay a recorded session's full event timeline in order — timestamp, event type, resource, decision — including actions that were auto-allowed and never gated. Experimental: prints deterministically, does not re-run the human gate.
skillfence report <skill> [options]Rolls a skill's recorded findings up into a full security assessment: skill / risk / AST / findings / attack-chain / decision / evidence-count. Must be run after skillfence run has produced findings.
| --json | Emit machine-readable JSON instead of the text report. |
| --markdown | Emit a Markdown report instead of the text report. |
skillfence bench [labs_root]Runs every lab with an auto-reject decision and scores the result against its ground-truth.yaml — detection rate on malicious labs, false-positive rate on benign labs, and total human interruptions. The single command that proves the numbers on this page.
For a first walkthrough, or teaching someone else — no path-typing required.
skillfence learn [labs_root]Interactive menu — pick a malicious lab by number, read its mission (the lab's own README, rendered in a panel), then it runs live so you make the human decision yourself. ATTACK → OBSERVE → DETECT → STOP → REMEDIATE.
Human approvals that persist — narrowly scoped to the exact (skill, action, resource), never blanket trust, always expiring unless told otherwise.
skillfence policy list [options]List every remembered approval in the shared, org-wide policy store — grant ID, skill, action, resource, expiry, and reason. Not scoped to any one lab.
| --all | Include expired grants (hidden by default). |
skillfence policy allow <skill> <action> <resource> [options]Pre-approve a specific (skill, action, resource) so it stops gating — the same scoped grant an [s] Allow scoped live decision creates, made ahead of time by whoever owns policy (e.g. a security lead clearing a known false positive org-wide). action is an event type such as filesystem.read, network.http_request, process.exec, secret.access.
| --reason <text> | Why this is approved — recorded in the grant for audit. |
| --ttl <seconds> | Seconds until the grant expires. Omit for the default (2h); 0 = never expires. |
skillfence policy revoke <grant_id>Remove a previously created grant — that action gates again the next time it occurs.
Everything above runs against SkillFence's own pre-built DVAS labs. Checking a skill you didn't write is the same tool, in two tiers.
All SkillFence needs is a skill/manifest.yaml next to your skill (name, version, purpose, and declared capabilities for filesystem/process/network/secrets). No execution, nothing touched.
Add a script.yaml describing the actions to check (read, write, exec, fetch, network_send, update, read_secret) and a sandbox/ with whatever local fixture files those actions touch. SkillFence scores each action against your manifest exactly like a lab. Nothing ever reaches your real filesystem or network, regardless of what path you write.
examples/my-first-skill/ ships a minimal, commented manifest + SKILL.md + script.yaml that works out of the box, with a commented-out step showing exactly how to add an out-of-scope action so you can watch SkillFence catch it.
What this isn't yet: wiring
RuntimeGateway directly into a live agent (Claude Code, an MCP server, your own agent
loop) so it enforces on real tool calls as they happen, rather than a scripted simulation. The
gateway and its wrapper methods (skillfence/runtime/gateway.py) are the actual
enforcement point a real integration would call — see skillfence/lab_runner.py::run_lab
for exactly how it's wired today — but there's no packaged adapter for a specific agent
framework yet.
Deterministic risk scoring, zero LLM in the security-decision path. Every rejection is a real block — the underlying file, process, or network call never executes.