AI developer tools · What shipped

Splunk published a comprehensive guide on multi-agent AI benchmarking that covers evaluation frameworks for measuring collaboration, coordination, and reliability in autonomous agent systems. Key benchmarks discussed include MultiAgentBench for testing coordination protocols across different topologies, BattleAgentBench for competitive scenarios, SOTOPIA-π for social intelligence, MARL-EVAL for statistical rigor in reinforcement learning, ST-WebAgentBench for safety compliance in web workflows, and SupChain-Bench for domain-specific supply chain tasks. The article introduces the CLEAR framework (Cost, Latency, Efficacy, Assurance, Reliability) which showed 0.83 correlation with production success versus 0.41 for accuracy-only metrics. It also highlights emerging trends including cost-normalized scoring, reliability as a first-class metric through pass^k testing, and failure taxonomies over aggregate scores, noting that even top models achieve only 86.33% clean tool-use rates per ToolFailBench evaluation across 1,000 tasks.

Benchmark Radar is a living database and search engine that catalogs AI benchmarks covering LLM evaluation, coding, reasoning, safety, and domain-specific assessments. The v0.11.0 release contains 1,283 source benchmark records with 12,916 numeric score observations across 790 records, drawing from 37 discovery sources including arXiv, Hugging Face Hub, and GitHub. The system combines daily benchmark discovery with searchable catalog retrieval, model-card mentions, score histories, and reproducible analysis. It provides web dashboard access with leaderboard and Pareto frontier views, downloadable evidence, and a command-line interface for offline queries, helping developers locate benchmark datasets, code, and evaluation settings when designing new AI evaluations.

OpenAI released GPT-Live-1 in the API on September 10, 2026, a full-duplex voice model priced at $0.05 per minute billed per second, with backend model charges separate. The model separates voice conversation from backend work, allowing continuous conversation while delegating reasoning and tool use to a paired backend model. GPT-Live-1 showed significant performance gains in benchmarks: 87.0% pass rate on Full Duplex Bench v3 tool-calling (vs 60.0% for GPT-Realtime-2.1), 90.0% on response quality, and notably 0.798s turn-taking latency compared to 1.41s for Realtime-2.1, demonstrating the full-duplex improvement. However, all published benchmarks are system-level measurements dependent on backend pairing and reasoning effort, so developers must test with stated configurations rather than relying on single scores. The API supports audio and text I/O, streaming, function calling, and telephony integration through WebRTC and WebSockets, with 12 voices at launch and no support for images, video, structured outputs, or fine-tuning.

MemPalace, a local-first AI memory system, achieved 96.6% R@5 retrieval recall on the LongMemEval benchmark using semantic search with zero API calls required. The hybrid v4 pipeline reached 98.4% R@5 on held-out test data, and with LLM reranking achieved ≥99% recall. Additional benchmarks show 88.9% R@10 on LoCoMo (hybrid v5), 92.9% average recall on ConvoMem, and 80.3% R@5 on MemBench (ACL 2025 with 8,500 items). The project ships pluggable storage backends including ChromaDB (default), SQLite, Rust-native vectors, Milvus, Qdrant, and pgvector, with published performance comparisons showing the rust_exact backend reduced memory usage by 77% to 557 MB and achieved 7.2–11.8 ms query latency on 334k-row workloads.