Evaluate scanner behavior on a labeled corpus

evaluate_security_cases() runs llmshieldr scanners over a small labeled corpus and returns action-level metrics. It is designed for repeatable local evaluation, release notes, and adoption reviews; it is not a substitute for a full red-team benchmark.

Usage

evaluate_security_cases(
  cases = NULL,
  policy = "comprehensive",
  reviewer = NULL,
  checks = "rules",
  redaction = NULL,
  scanners = scanner_options()
)

Arguments

cases

Optional data frame. If NULL, the packaged inst/extdata/security_eval_cases.csv corpus is loaded.

policy

A shieldr_policy or built-in policy name.

reviewer

Optional reviewer function or object with $chat().

checks

One of "rules", "nlp", "llm", or "both".

redaction

Optional redaction strategy from redaction_strategy().

scanners

Optional scanner configuration from scanner_options().

Value

A data frame with case metadata, expected and actual actions, matched, latency_ms, and n_findings.

Details

The input corpus should contain at least stage, text, and expected_action columns. If stage is "output", rows are scanned with scan_output(). If stage is "context", each row is scanned as a one-row context data frame with scan_context(). All other stages are scanned with scan_prompt().

The returned data frame includes per-case latency in milliseconds and a Boolean matched column. Use the summary columns to calculate detection rate, false-positive rate, action accuracy, and latency percentiles in vignettes or release notes.

Examples

if (FALSE) { # \dontrun{
results <- evaluate_security_cases(policy = "comprehensive")
mean(results$matched)
} # }