agenteval

agenteval

Your CLAUDE.md is untested. So is your AGENTS.md, your copilot-instructions.md, and your .cursorrules.

agenteval is a linter, benchmarker, and CI gate for AI coding instructions. It finds dead references, token bloat, contradictions, and stale instructions before your agent does. Then it scores agent performance so you can measure whether your instruction changes actually help.

Install

npm install -g agenteval-cli

Or pick your preferred method:

brew tap lukasmetzler/agenteval && brew install agenteval   # Homebrew
curl -fsSL https://raw.githubusercontent.com/lukasmetzler/agenteval/main/install.sh | bash  # Shell

No Bun, no Node at runtime. The binary is self-contained.

Quick Start

agenteval lint                    # Find problems in your instruction files
agenteval lint --explain          # Same, with explanations for each rule
agenteval harvest --dry-run       # Preview what AI commits are in your history
agenteval ci                      # Run all tasks, fail on regressions

What It Catches

• Dead references to files, paths, and headings that don't exist
• Filler phrases that waste context tokens ("make sure to", "it is important that")
• Contradictions between instruction files ("always use X" and "never use X")
• Content overlap and duplication across files
• Token budget overruns that crowd out code context
• Vague instructions without actionable specifics
• Stale instructions referencing code that was refactored weeks ago
• Invalid skill metadata (per Anthropic spec)
• Broken markdown links and heading anchors

Supported Formats

Commands

The Pipeline

flowchart LR
    A["Your CLAUDE.md"] --> B["agenteval lint"]
    B --> C["Fix quality issues"]
    D["Git history"] --> E["agenteval harvest"]
    E --> F["Task YAML files"]
    F --> G["agenteval run"]
    G --> H["Scored results"]
    H --> I["agenteval compare"]
    I --> J{{"Did my instructions improve?"}}

    style A fill:#2d333b,stroke:#444,color:#e6edf3
    style D fill:#2d333b,stroke:#444,color:#e6edf3
    style J fill:#1a7f37,stroke:#2ea043,color:#fff

Lint catches problems statically. Harvest builds benchmarks from your git history. Run scores agent performance. Compare tells you what changed. CI gates regressions before they merge.

CI Integration

Add agenteval to your GitHub Actions workflow with one line:

- uses: lukasmetzler/agenteval@v0
  with:
    command: ci                   # or: lint, harvest --dry-run

Or use the CLI directly in any CI system:

agenteval ci --min-score 0.7 --max-regression 0.05

See the CI Guide for thresholds, configuration, and examples.

Installation Options

Documentation

Contributing

See CONTRIBUTING.md.

License

MIT