> ## Documentation Index
> Fetch the complete documentation index at: https://openbench.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration

> Set openbench model and evaluation behavior

## Configuration Methods

openbench can be configured through multiple methods, in order of precedence:

1. **Command-Line Arguments**

```bash theme={null}
# Run eval with --config CLI flags
bench mmlu --max_tokens=10000 --temperature=0.5
```

See what else you can do with the [openbench CLI.](/cli/overview)

2. **Environment Variables**

```bash theme={null}
# Set configuration defaults
export BENCH_MAX_TOKENS=10000
export BENCH_TEMPERATURE=0.5
```

3. **Configuration Files** (`.env` files)

```bash .env theme={null}
# Write default values to config file
BENCH_MAX_TOKENS=10000
BENCH_TEMPERATURE=0.5
```

```bash theme={null}
# Automatically loaded if in current directory
bench eval mmlu

# Or source manually
source .env
bench eval mmlu
```

If not defined through any of the above methods, parameters resort to **default values**.

## Configuration Parameters

### Model Configuration

| CLI Command | Environment Variable | Description |
| - | - | - |
| `--model` | `BENCH_MODEL` | Model(s) to evaluate. |
| `--model-base-url` | `BENCH_MODEL_BASE_URL` | Base URL for model(s). |
| `--model-role` | `BENCH_MODEL_ROLE` | Map role(s) to specific models. |

<Note>Some providers may support additional parameters. See [provider-specific model configuration](/providers#provider-specific-model-configuration) for extended features.</Note>

### Task Configuration

| CLI Command | Environment Variable | Description |
| - | - | - |
| `--epochs` | `BENCH_EPOCHS` | Number of epochs to run each evaluation. |
| `--epochs-reducer` | `BENCH_EPOCHS_REDUCER` | Reducer(s) applied when aggregating epoch scores. |
| `--limit` | `BENCH_LIMIT` | Limit evaluated samples (single number or range). |
| `--message-limit` | `BENCH_MESSAGE_LIMIT` | Maximum number of messages per sample. |
| `--score / --no-score` | `BENCH_SCORE` | Grade the benchmark, or skip scoring (can score later). |

Use `-T` Flag for Other Task-Specific Arguments\
(e.g. `bench eval graphwalks --model ollama/llama3.1:70b -T task=parents`)

### Generation Settings

| CLI Command | Environment Variable | Description |
| - | - | - |
| `--temperature` | `BENCH_TEMPERATURE` | Model sampling temperature. |
| `--top-p` | `BENCH_TOP_P` | Nucleus sampling. |
| `--max-tokens` | `BENCH_MAX_TOKENS` | Maximum tokens for model response. |
| `--seed` | `BENCH_SEED` | Random seed for deterministic generation. |

### Performance & Concurrency

| CLI Command | Environment Variable | Description |
| - | - | - |
| `--max-connections` | `BENCH_MAX_CONNECTIONS` | Maximum number of parallel requests. |
| `--max-subprocesses` | `BENCH_MAX_SUBPROCESSES` | Maximum number of parallel subprocesses. |
| `--max-tasks` | `BENCH_MAX_TASKS` | Maximum number of tasks to run concurrently. |
| `--timeout` | `BENCH_TIMEOUT` | Timeout per model API request (seconds). |
| `--max-retries` | `BENCH_MAX_RETRIES` | Maximum number of retries for API requests. |
| `--retry-on-error` | `BENCH_RETRY_ON_ERROR` | Retry samples on errors (set number of retries). |
| `--fail-on-error` | `BENCH_FAIL_ON_ERROR` | Failure threshold for sample errors (percentage or count). |
| `--no-fail-on-error` | `BENCH_NO_FAIL_ON_ERROR` | Do not fail evaluation if errors occur. |

### Logging & Output

| CLI Flag | Environment Variable | Description |
| - | - | - |
| `--logfile` | `BENCH_OUTPUT` | Output file for results. |
| `--log-format` | `BENCH_LOG_FORMAT` | Output logging format (`eval` / `json`). |
| `--display` | `BENCH_DISPLAY` | Display type for evaluation progress. |
| `--log-samples / --no-log-samples` | `BENCH_LOG_SAMPLES` | Include or exclude detailed samples in logs. |
| `--log-images / --no-log-images` | `BENCH_LOG_IMAGES` | Include or exclude base64 encoded images in logs. |
| `--log-buffer` | `BENCH_LOG_BUFFER` | Number of samples to buffer before writing logs. |
| `--log-dir` | `BENCH_LOG_DIR` | Directory for log files. |

### Sandbox & Execution

| CLI Flag | Environment Variable | Description |
| - | - | - |
| `--sandbox` | `BENCH_SANDBOX` | Environment to run evaluation (`local` or `docker`). |
| `--sandbox-cleanup / --no-sandbox-cleanup` | `BENCH_SANDBOX_CLEANUP` | Cleanup sandbox environments after tasks (or skip cleanup). |

### Hub Integration

openbench can export logs to a Hugging Face Hub dataset. This is useful if you want to share your results with the community or use them for further analysis.

| CLI Flag | Environment Variable | Description |
| - | - | - |
| `--hub-repo` | `BENCH_HUB_REPO` | Target Hub dataset repo for logs. |
| `--hub-private` | `BENCH_HUB_PRIVATE` | Push Hub dataset as private. |

### Debugging & Inspection

| CLI Flag | Environment Variable | Description |
| - | - | - |
| `--debug` | `BENCH_DEBUG` | Enable debug mode with full stack traces. |
| `--debug-errors` | `BENCH_DEBUG_ERRORS` | Enable debug mode for errors only. |
| `--trace` | `BENCH_TRACE` | Trace message interactions with model. |
| `--alpha` | `BENCH_ALPHA` | Allow running experimental/alpha benchmarks. |

## Configuration Examples

### Simple Custom Configuration

```bash theme={null}
bench eval humaneval \
  --model openai/gpt-4o \
  --temperature 0.5 \
  --max-tokens 512 \
  --epochs 5 \
  --sandbox docker
```

### Model Comparison

```bash theme={null}
bench eval mmlu --limit 500 \
    --model groq/llama-3.3-70b \
    --model openai/gpt-4o \
    --model anthropic/claude-3-5-sonnet
```

### Multi-Model Evaluation

```bash theme={null}
bench eval simpleqa \
  --model-role candidate=groq/llama-3.3-70b \
  --model-role grader=openai/gpt-4o
```

### Batch Configuration

```bash theme={null}
bench eval --model openai/gpt-4.1 \
  math --limit 20 \
  mmlu --limit 20
```

### Interruption Retry Configuration

```bash theme={null}
bench eval-retry logs/incomplete.json \
  --max-retries 3 \
  --retry-on-error
```

### Display Configuration

```bash theme={null}
# Display modes
export BENCH_DISPLAY="rich"           # Rich terminal output (default)
export BENCH_DISPLAY="plain"          # Simple text output
export BENCH_DISPLAY="none"           # No output (logs only)
export BENCH_DISPLAY="conversation"   # Show full conversations

bench eval mmlu --display conversation
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.