OpenClaw model benchmark methodology

Environment record

Every result must include the exact model tag and quantization, hardware, RAM or VRAM, runtime version, OpenClaw version, context setting, task class, and a URL for the raw run records.

Task classes

1. Read one known file and report a verifiable fact.
2. Write a file only at the permitted path.
3. Run a harmless command with one required parameter.
4. Search a supplied local document set.
5. Complete read → transform → write.
6. Recover from one invalid tool argument without looping.

Minimum sample

Each model and task class requires at least 20 attempts before a result enters the public dataset. More runs are required before making a strong comparative claim. Prompts, starting state, timeouts, and tool policy must remain constant across compared candidates.

Metrics

tool_success_rate = successful_tool_tasks / attempted_tool_tasks
parameter_error_rate = invalid_parameter_calls / total_tool_calls
loop_rate = runs_exceeding_call_limit / total_runs

Median latency is reported separately. A fast model with a high wrong-tool or loop rate is not ranked above a slower reliable model solely on speed.

Failure labels

Runs distinguish wrong tool, invalid parameters, refusal, unsupported claim, timeout, context failure, tool loop, and infrastructure failure. Infrastructure failures do not become model failures without evidence, but they remain visible in the raw records.

Publication rule

No score is published without raw-run access and every required environment field. Corrections update the dataset date and are recorded under the site's corrections policy.

Expected result

Another tester can reconstruct the environment, run the same task classes, understand every failure label, and compare the raw records with the published aggregate.