Text models
Fable 5.2 vs. GPT-6 Astra for coding
Compare reproducible fixes, investigation quality, and tool behavior.
Score fixes and investigation separately
Measure regression tests for code edits and reproducibility for findings. Record false positives as well as confirmed issues.
Compare cost per completed task
Count retries, review time, and actual charges. Separate the tasks where both pass from those where only one does.
Verify transport and tools independently
Test streaming and function round trips before attributing an integration failure to model quality.