Feature · Comparison Mode
One prompt. Every server. Every model.
Run the same task across any combination of MCP servers and language models. See outputs, tool-call traces, latency, and failures side by side. The eval workflow you currently fake with five terminal tabs.
Side-by-side, every dimension
Three axes: server, model, prompt.
Pick any combination. Hold the prompt fixed; vary servers and models. Hold the model fixed; vary servers. Hold the server fixed; vary models. MCPOrbit runs every cell of the grid concurrently and shows you the outputs, tool-call traces, latency, error class and token cost in one view.
- Run claude-opus-4.7 vs gpt-5 on the same MCP server
- Run two builds of the same server through the same model
- Mix and match — N×M without writing a runner script
linear
claude-opus-4.7287ms> tools/call list_issues
{ "team": "growth", "status": "open" }
< response
{ "issues": [4 items], "cursor": "..." }
postgres
gpt-5-mini412ms> tools/call list_issues
{ "team": "growth", "status": "open" }
< response
{ "issues": [4 items], "cursor": "..." }
Use cases
Three jobs comparison mode does well.
Vetting a new server before integration
Run your real prompts against it. See if the tool selection matches what the README claims.
Regression testing during development
Snapshot a working build. Re-run after a change. Diff the structured outputs and the tool-call traces.
Benchmarking model providers
Same MCP server, three model vendors. Decide based on cost, latency, and behaviour, not on vibes.
The MCP log
Every tool call, with its name, a summary of its arguments, and its duration.
The log records a call once it settles: the tool, the arguments you sent, the full response JSON, and how long it took. Select a row to read the request and response side by side, or copy the whole entry as JSON. When something diverges, the log is the first place you look.
- Full request and response JSON, untruncated
- Duration in milliseconds on every call
- Filter to failures only, with a running error count
- Copy any entry as JSON
MCP log
Honest caveats
What comparison mode is not.
Not an automated eval framework.
MCPOrbit doesn't grade outputs against a rubric. It surfaces them so you can decide.
Not a load-test runner.
Concurrency is one prompt at a time across configurations. If you need 10k requests, you want a load tester.
Not a CI plugin (yet).
Comparison runs are driver-driven from the app. Headless CI integration is on the roadmap, not shipped.
Comparison Mode FAQ
Can I save a comparison configuration?
Yes. Each comparison is a saved configuration with a name, the participating server-model pairs, and the prompt. Reopen any time.
Are tool calls executed against real servers?
Yes. There's no simulation. If your MCP server has side effects, comparison mode will trigger them. Use a staging server when comparing destructive tools.
Do model API calls cost real money?
Yes. MCPOrbit uses your own API keys. Cost is on you. We surface token counts in the log panel so you can budget.
Can I export comparison results?
JSON export ships today. Markdown export is on the roadmap.
What's the practical limit?
No hard cap. Practical limit is whatever your model providers and your patience can handle. We've seen teams run 6×3 grids comfortably.
