Research prototype · not investment advice

Same judge. Second domain.

Name: CACE-Bench DeFi track
Creator: Digital Economy Lab
License: https://opensource.org/licenses/MIT

CACE-Bench measures whether an agent reaches the right verdict, refuses to decide when the facts were unobtainable, and cites only sources that answered. The DeFi track applies the same unchanged judge to GO / NO-GO verdicts on lending markets, vaults and RWA tokens.

—

historical cases reconstructed on chain

—

drafts corrected by on-chain reconstruction

—

false negative of the rules found

Run it in your browser Methodology Issue #5

Research prototype by Digital Economy Lab. Not investment advice. Not a rating of any protocol. Historical cases are public incidents.

The judge does not change

Case→your agent→Narrative→judge (unchanged)→7 metrics with Wilson 95% CI

Credit track	DeFi track
FLAG / CLEAR / ESCALATE	NO-GO / GO / INSUFFICIENT_DATA
sanctions hit	hard gate fails (exit depth, reflexive backing, manipulable oracle)
PEP match	single-borrower concentration
KYC verified	oracle & admin controls verified
country	chain

The judge, the metrics and the confidence intervals are imported from the credit track without a single change. Only the case generator, the source registry and the ground-truth rules are new.

Where the chain corrected the draft

H10 · wrong borrower

Draft: one borrower holds ~100% of debt — true today.
Chain at T0: that address held ~3%; a different address held 89% and exited before the collapse.
What changed: the label survived, the reason did not.

Rule adopted: API data may supply candidate addresses, never amounts.

H09 · not a constant

Draft: xUSD priced at a hardcoded $1.
Chain at T0: an “xUSD/USD” feed reporting issuer-side value (1.248 → 1.262), still 1.266 when xUSD traded near $0.26.
What changed: the case carries a manual label; open question 1a.

H05 · not on the public address still a draft

Draft: single-borrower CRV concentration.
Chain at T0: the founder’s public address held 0.67% of LlamaLend CRV debt.
What changed: kept as a draft; open question 1b.

This case is not in the table above — it is listed under drafts.

The first silent decision the track caught was its own. An early reconstruction of H10 targeted a public “Elixir USDC” vault and returned OK: the reconstructor had counted the vault’s idle balance as collateral and reported 100% concentration. The vault was 100% idle at T0 — not the lending channel at all. The reconstructor now refuses idle-only vaults, the artefact was removed, and H10 was re-scoped. A confident verdict on the wrong object, with every number internally consistent, is exactly the failure this benchmark scores.

Where the rules were wrong

At the block before the attack, Term Finance’s delay module reported txCooldown = 608,400 s — seven days. The rules read that as an adequate timelock and return GO. A public proposal queued for six days removed it, and $8.5M was drained.

Proposed rule change, open in issue #5: a queued change that lowers a control below policy sets controls_safe = False. At least two more such cases are needed before it is adopted.

The point of this block: the benchmark can show where its own rules fail. A split on which the rules separated perfectly would prove nothing.

Run it in your browser

Nothing is sent anywhere. Python runs in your tab via Pyodide; the files below are fetched from this site.

What it loads

/demo/cace_bench.py      judge, metrics, confidence intervals (v0.3.0)
/defi/defi_track.py      DeFi case generator and ground-truth rules
/defi/defi_sources.json  source registry

Run it against your own agent

python examples/defi_adapter.py --cmd "your-agent"
python examples/defi_adapter.py --cmd "python examples/claude_code_agent.py"

What the agent gets: the position and the state of each source — never the labels.
What it returns: verdict, controls_verified, anomaly_basis, citations.
What is forbidden: web search — it leaks the outcome of a historical case.

Co-define the case set — comment on issue #5.

What this does not show

Synthetic split: injected error rates and an unverified registry.
Historical split: a handful of cases on chain and a few drafts; perfect separation would prove nothing.
Model knowledge leakage: cases before the agent model’s training cutoff are reported only as a contamination measurement.
Code-level exploits (Euler v1, Balancer v2) are outside the observable risk surface — listed as unscored controls.
Not investment advice; not a rating of any protocol.