Skip to main content

​
Overview

Exercism is a comprehensive coding benchmark that evaluates AI code agents’ ability to solve real-world programming exercises across multiple programming languages. Based on the popular Exercism platform, this evaluation tests not just code generation capabilities, but the full coding workflow including reading specifications, navigating file systems, implementing solutions, and passing test suites. The benchmark places agents in realistic development environments where they must read instructions, understand existing code structures, and modify multiple files to create working solutions—mirroring actual software development workflows.

​
Key Features

  • Multi-Language Support: Tests coding capabilities across 5 programming languages:
    • Python
    • JavaScript
    • Go
    • Java
    • Rust
  • Multiple Code Agent Harnesses: Evaluate models across different code agent frameworks on identical tasks for fair comparison:
    • Aider - AI-powered pair programming tool with git integration
    • OpenCode - OpenAI-compatible code generation tool
    • Claude Code - Claude-based code editor with file system access
    • Roo - VS Code extension with interactive development
    • Codex - OpenAI Codex cli coding agent
  • Realistic Development Environment: Agents work in full file system workspaces with multiple files, build configurations, and test suites
  • Language-Specific Testing: Each language uses its native testing framework and build tools
  • Comprehensive Evaluation: Tests the complete coding workflow from understanding specifications to passing automated tests

​
Usage

Run Exercism evaluation across all languages with the default code agent (codex):
Specify a different code agent:
Evaluate with a specific model:
Hide the Exercism test suites from the agent (tests still run after the edit loop):
With --hidden-tests, the agent works inside a sanitized copy of the repo with the language-specific tests excluded, while the final verification still runs against the full workspace to ensure correctness. Run evaluation for a single language:

​
Dataset Information

The Exercism dataset provides a realistic coding environment for each task, structured to mirror real software development projects.

​
Workspace Structure

Each exercise places the agent in a complete project workspace:

​
Task Format

Agents receive:
  1. Instructions: A prompt describing the coding task and requirements
  2. File System Context: Complete access to all project files
  3. Development Environment: A sandboxed Docker environment with all necessary language tools and dependencies installed

​
Workflow

For each task, the agent must:
  1. Read and understand the INSTRUCTIONS.md file
  2. Navigate the project structure to find relevant files
  3. Analyze existing code and tests to understand requirements
  4. Implement the solution by modifying source files
  5. Ensure the implementation passes all automated tests

​
Scoring

Exercism uses test-based scoring where success is determined by passing the automated test suite for each exercise.

​
Success Criteria

A task is considered successful if and only if:
  1. The test command executes without errors (exit code 0)
  2. All tests in the test suite pass
  3. No compilation or syntax errors occur
  4. Execution completes within the time limit (6 minutes per task)

​
Metrics

  • Overall Accuracy: Success rate across all languages and tasks
  • Per-Language Accuracy: Breakdown of performance by programming language

​
Performance Insights

The multi-agent design allows for unique comparative analysis:
  • Compare how different models perform on identical tasks
  • Evaluate which code agent harness works best for different languages and models
  • Identify model strengths and weaknesses across programming paradigms