# How to compare agent task success rates across model providers > Run one tool-using suite against several providers with only the model varied, and report success with an interval by case, tokens, and wall time per model. Source: https://voxgig.com/howto/measure-task-success-across-model-providers - Audience: agent-builder - Level: intermediate - Languages: python, typescript - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Fix the tools, the system prompt, the cases, and every request parameter, and vary only the model, through one OpenAI-compatible gateway such as LiteLLM with drop_params off. Grade each run into a failure class, so a protocol failure caused by the route is not counted as the model's mistake. Report each model's rate with a bootstrap interval drawn over cases, and compare two models paired by case before you rank them. --- ## You will need Node 22 or later, Python 3.12 or later, and a tool-using suite whose cases name the tool calls a correct run makes. Verified 2026-10-15 against Node 22.22.0 and Python 3.13.16, with no packages beyond the standard libraries. The three models on this page are scripted stubs behind a local gateway, so the numbers show what the method finds rather than how any provider performs. Choosing an evaluation framework is a separate task, and so is asserting a single tool call. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Braintrust experiments](https://www.braintrust.dev/docs/evaluate/compare-experiments) | A team that wants every run kept and compared case by case against a baseline | Runs go to Braintrust's service or to a self-hosted data plane you operate | You want the comparison in a CI log rather than a dashboard | | [LiteLLM as an OpenAI-compatible gateway](https://docs.litellm.ai/docs/simple_proxy) | One runner and one request shape for every provider | It decides per provider which parameters map, and with `drop_params` it drops the rest unannounced | The parameter it would drop is the behavior you are measuring | | [promptfoo provider matrix](https://www.promptfoo.dev/docs/configuration/guide/) | A config file of providers and tests, run from the CLI or in CI | Each provider is configured in its own dialect, so equal settings across providers are your job | Your team already runs its evaluations in another framework | | [Provider-native SDK runs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/parallel-tool-use) | You are measuring a provider-specific behavior, such as parallel calls or strict tools | One adapter per provider, and each adapter is a place for the comparison to stop being fair | You want several providers compared by the end of the week | A gateway buys one code path, and the code path is where fairness lives. LiteLLM raises an error by default when a provider cannot take a parameter, and silently drops it with [`drop_params`](https://docs.litellm.ai/docs/completion/drop_params) on. Braintrust aligns cases across experiments against a baseline, which is the pairing, and promptfoo reruns each test with [`--repeat`](https://www.promptfoo.dev/docs/usage/command-line/). Check whether a report gives an interval before you rank on it. Native SDKs are the only route that sends each provider exactly what it supports, at the cost of writing the same runner three times. ## Hold everything fixed except the model The runner reads the tools, the system prompt, and 200 cases from `suite/`, and sends the same request body for every model. Only `model` changes: ```ts title="run.ts" for (let turn = 0; turn < Number(opt['max-turns']); turn++) { const res = await fetch(`${base}/v1/chat/completions`, { method: 'POST', headers, // seed only goes to the stub, which needs it to vary between trials. A gateway that // refuses parameters a provider lacks, as LiteLLM does by default, would reject it. body: JSON.stringify({ model, messages, tools, tool_choice: 'auto', parallel_tool_calls: PARALLEL_TOOL_CALLS, ...(stub ? { seed: trial } : {}) }), }) ``` The tools declare `strict: true`, and the request sets `parallel_tool_calls: false`. Both are [OpenAI function-calling](https://developers.openai.com/api/docs/guides/function-calling) parameters, so a gateway has to translate them for every other provider. Each case carries the calls a correct run makes, and `grade.ts` checks a run in a fixed order: the protocol, then the schema, the tool, the arguments, and last the answer. A write tool called on a read task fails as the wrong tool, whatever the answer says. For real models, run the LiteLLM proxy with one `model_name` per provider and point the runner at it: ```yaml title="litellm.yaml" model_list: - model_name: claude litellm_params: model: anthropic/CLAUDE_MODEL_ID api_key: os.environ/ANTHROPIC_API_KEY - model_name: gpt litellm_params: model: openai/OPENAI_MODEL_ID api_key: os.environ/OPENAI_API_KEY - model_name: gemini litellm_params: model: gemini/GEMINI_MODEL_ID api_key: os.environ/GEMINI_API_KEY litellm_settings: # The default. A parameter a provider does not support is an error, not a silent drop. drop_params: false ``` ```bash litellm --config litellm.yaml node --experimental-strip-types run.ts --gateway http://localhost:4000 --models claude,gpt,gemini ``` Those two commands call real models and were not run here. Without `--gateway`, the runner starts `stub-gateway.ts` in a process of its own and uses three scripted models. ## Read the failure classes before the rates Three trials of 200 cases against each stub: ```bash node --experimental-strip-types run.ts --trials 3 --out results/latest.jsonl ``` ```text output 200 cases x 3 trials x 3 models = 1800 runs, parallel_tool_calls: false stub-a 515/600 succeeded 33 wrong tool 32 no tool call 20 wrong arguments stub-b 494/600 succeeded 42 wrong arguments 33 no tool call 31 wrong tool stub-c 388/600 succeeded 141 parallel calls despite parallel_tool_calls: false 30 no tool call 25 wrong arguments 16 wrong tool ``` `stub-c` sits behind a route that drops `parallel_tool_calls`. On every comparison task it asks for both meters' readings in one turn, which the request forbade. Allow parallel calls for everyone and run the two again: ```bash node --experimental-strip-types run.ts --models stub-a,stub-c --parallel --out results/parallel.jsonl ``` ```text output 200 cases x 3 trials x 2 models = 1200 runs, parallel_tool_calls: true stub-a 515/600 succeeded 33 wrong tool 32 no tool call 20 wrong arguments stub-c 514/600 succeeded 32 wrong arguments 30 no tool call 24 wrong tool ``` The 21-point gap was the route. The providers place the switch differently, which is why a gateway has to translate it: OpenAI takes `parallel_tool_calls` at the top of the request, and Anthropic takes [`disable_parallel_tool_use` inside `tool_choice`](https://platform.claude.com/docs/en/agents-and-tools/tool-use/parallel-tool-use).[^1] Decide which behavior you are measuring, then send it to every provider or none. ## Report an interval drawn over cases `stats.py` reads the recorded results, the file one run of `run.ts` wrote, and reports each model's rate with a 95% interval: ```bash python3 stats.py results/recorded.jsonl ``` ```text output model cases runs success 95% CI, by case tokens/run wall ms p50 stub-a 200 600 85.8% [ 83.2%, 88.5%] 984 34 stub-b 200 600 82.3% [ 79.3%, 85.5%] 1174 29 stub-c 200 600 64.7% [ 59.2%, 70.2%] 831 20 ``` The interval is a bootstrap over cases, not runs.[^2] Three trials of a case a model cannot do are three failures of one case, and drawing runs instead would count them as three independent pieces of evidence. One test in `test_stats.py` builds that data and shows the run-level interval coming out narrower than the case-level one. ```python title="stats.py" def bootstrap(values, rounds, seed): """95% percentile interval of the mean, resampling the given per-case values.""" # random() is the one method whose sequence Python promises to keep across versions for a # given seed; choice() is not, so the index is drawn from random() directly. rng = random.Random(seed) n = len(values) means = sorted(sum(values[int(rng.random() * n)] for _ in range(n)) / n for _ in range(rounds)) return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1] ``` Read the other two columns with care. Each provider counts tokens its own way, so `tokens/run` compares what you are billed for, not how much text moved. The wall times here are a stub sleeping a set time per model, measured at the runner; they show the column works, not how fast anything is. ## Do not rank on twenty cases The first twenty cases rank the first two models the other way round: ```bash python3 stats.py results/recorded.jsonl --cases 20 --compare stub-a stub-b python3 stats.py results/recorded.jsonl --compare stub-a stub-b --detect 0.05 ``` ```text output stub-a minus stub-b over 20 cases: -5.0%, 95% CI [-16.7%, +5.0%], inside the noise stub-a minus stub-b over 200 cases: +3.5%, 95% CI [-0.5%, +7.5%], inside the noise cases needed to detect a 5% difference at this spread: 286 ``` The stubs were scripted six points apart. Twenty cases put the wrong one ahead, and two hundred still could not separate them at 95%. The comparison is paired by case, which removes the case-to-case spread both models share, and `--detect` turns the remaining spread into the number of cases a real difference would need. ## Check it worked The grader tests cover each failure class, a number found inside a longer number, and malformed arguments. `run.test.ts` reruns the suite and checks `results/recorded.jsonl` is what it writes, apart from wall time: ```bash node --experimental-strip-types --test grade.test.ts run.test.ts ``` ```text output # tests 11 # suites 0 # pass 11 # fail 0 ``` The statistics tests run under Python's own test runner: ```bash python3 test_stats.py 2>&1 ``` ```text output test_a_failed_request_is_not_a_failed_task (__main__.Stats.test_a_failed_request_is_not_a_failed_task) ... ok test_a_model_missing_a_case_is_refused (__main__.Stats.test_a_model_missing_a_case_is_refused) ... ok test_halving_the_difference_needs_four_times_the_cases (__main__.Stats.test_halving_the_difference_needs_four_times_the_cases) ... ok test_resampling_runs_reports_a_narrower_interval_than_the_data_supports (__main__.Stats.test_resampling_runs_reports_a_narrower_interval_than_the_data_supports) ... ok test_the_interval_is_the_same_for_the_same_seed (__main__.Stats.test_the_interval_is_the_same_for_the_same_seed) ... ok test_the_reported_rate_is_the_one_the_interval_is_drawn_around (__main__.Stats.test_the_reported_rate_is_the_one_the_interval_is_drawn_around) ... ok ``` ## When it goes wrong One provider's rate drops sharply in the middle of a run. The provider is rate limiting you, and a runner that counts a 429 as a failed task scores the provider's quota rather than its model. `run.ts` records a failed request with `success: null`, and `stats.py` leaves it out of the rate. The rate is the mean of the case rates, so a case that lost two of its three requests still counts once. `stats.py` refuses a comparison when a model has no graded run for a case, because the pairing is then broken. The gateway rejects the whole run with an unsupported-parameter error. That is LiteLLM doing its job. Remove the parameter for every provider, or drop that provider from the comparison. Turning on `drop_params` would silence the error by sending each provider a different request. A model's argument JSON does not parse. The runner keeps the call with `null` arguments and the grader fails it as `arguments are not JSON`, rather than crashing the run. ## When not to do this Do not pick a model on a rate difference whose interval includes zero. Run more cases, or treat the models as tied and choose on price or latency. Do not route the comparison through a gateway when the behavior you care about is specific to one provider, for example parallel calls. Use each provider's own SDK for that one measurement. Do not grade on the final answer alone. A model that reads the right number from the wrong call passes an answer check, and so does one that registers a meter while answering a question. The grader checks the calls before the answer to catch both. ## Related how-tos - [Validate JSON model output before you use it](/howto/validate-json-model-output-in-typescript) - [Name tools so an agent picks the right one from fifty](/howto/name-tools-so-agents-pick-the-right-one) - [Cut agent cost with provider prompt caching](/howto/cut-agent-cost-with-prompt-caching) - [Measure the token cost of an agent task against your API](/howto/measure-token-cost-per-agent-task) - [Map one tool definition to MCP, OpenAI, Claude, and Gemini](/howto/convert-tool-definitions-between-mcp-openai-claude-gemini) ## Last verified Verified 2026-10-15 against Node 22.22.0 and Python 3.13.16. Every output block is what the command preceding it printed. The models are scripted stubs behind a local gateway, so no command on this page with an output block called a model. The behavior of LiteLLM, promptfoo, and Braintrust described here is from their documentation, not from a run. [^1]: The two vendors phrase one switch in opposite senses. OpenAI's is a [permission](https://developers.openai.com/api/docs/guides/function-calling), `parallel_tool_calls: false`. Anthropic's is a prohibition, `disable_parallel_tool_use: true`, and [its documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/parallel-tool-use) adds that it is not a top-level request parameter. A gateway translating one into the other moves the setting into another object and turns a false into a true. [^2]: The bootstrap is Bradley Efron's, published in [The Annals of Statistics in 1979](https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full). The paper's title, *Bootstrap Methods: Another Look at the Jackknife*, introduces the method as a second opinion on an older one. `stats.py` draws 2,000 samples with a fixed seed, which makes the interval repeatable on one machine. Across machines it rests on the [Python documentation](https://docs.python.org/3/library/random.html), which guarantees that `random()` keeps its sequence for a seed and declines to say the same of most of the module.