AI developer tools · What shipped

Claude Haiku 5.5 was evaluated on Artificial Analysis Intelligence Index v4.3.2, which incorporates 10 benchmarks including AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity's Last Exam, GDP.pdf, CritPt, AA-Omniscience, and AA-LCR v1.1. The benchmark measures model performance across capability indices for Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, and Economics, alongside specific evaluations for agentic work, coding, document reasoning, physics, medical long-context reasoning, and software engineering tasks. Cost-per-task and output speed metrics are tracked across token types.

AI startups Elorian and Scale AI released Humanity's Sixth Sense (HSS), a new benchmark measuring how models interpret visual scenes through intuitive reasoning. The benchmark contains 522 questions across 288 images and 234 video clips, testing temporal dynamics, physical logic, social understanding, and abstract inference. GPT-6 Astra achieved a pass@1 score of 53.6 compared to the human baseline of 93.1, while Chinese open-weight models performed significantly worse (GLM-5.3-flash scored 26.8, Kimi K3 scored 25.5). P-Zero Research released TasteVal, a benchmark evaluating AI systems' research taste across eight tasks including data curation, model training, and preference modeling. Frontier models' experimental research taste has doubled every three months since December 2025, with Opus 5.5 exceeding human baselines at 2.30x compute efficiency and roughly 1/30 of expert cost per attempt. Reflection announced Beam, a 501 billion parameter open-weight MoE model with 23 billion active parameters, optimized for token efficiency in reasoning and coding. Beam achieves comparable scores to GLM-5.2 while using 3-4x less inference compute, though benchmarks show it trails Chinese frontier models. The company plans to release model weights and technical artifacts later in October. Mistral announced Mistral Large 4, an open-weight model optimized for cybersecurity tasks, planned for release by month-end. Trained on 3,800 NVIDIA Grace Blackwell GPUs, ML4 will be the strongest Western open-weight model upon release, though Chinese models like GLM-5.3 and Kimi K3 maintain higher benchmark scores across most tasks.

Harvey released the Legal Agent Benchmark, a test for evaluating how AI models perform legal work in agentic settings using tools like file operations and bash commands across document types. Muse Spark 1.2 leads overall with a 25.42% task resolution score, while Muse Spark 1.3 Max achieves the highest criteria pass rate at 94.74%, revealing a significant gap between criterion-level performance and full task resolution. The benchmark includes two datasets (public and held-out test set) with task-specific evaluation across legal domains including intellectual property, M&A, data privacy, and banking.