AI developer tools · What shipped

MÖVE, a holistic evaluation framework for the German public sector, assessed 39 large language models across three governance dimensions relevant to AI deployment. The benchmark found estimated energy consumption varies 63-fold across models (0.647 to 40.6 Wh per query), with energy efficiency not determined by model size alone. Provider transparency scores reveal significant gaps, particularly in computational resources and energy consumption disclosure, with 84.6% of models providing no training-energy data and 79.5% offering no measurement methodology. On political knowledge evaluation using 4,788 official German party positions, no model achieved consistently accurate outputs, with the highest-performing models reaching only 67.1% accuracy; notably, geographical origin did not predict contextual knowledge, with competitive results from European, US, and Chinese providers. The framework emphasizes that model selection for public institutions cannot rely on performance rankings alone and should incorporate governance requirements including energy efficiency, transparency, and domain-specific accuracy.

This benchmark framework measures Speech-to-Text (STT) service performance across latency and accuracy metrics. Key results show AssemblyAI's universal-3-5-pro and Soniox's stt-rt-v4 leading the Pareto frontier with 1.25-1.44% word error rates and 249-282ms median latency, while NVIDIA's Nemotron 3.0 ASR achieves fastest latency at 221ms median with 1.90% WER. The framework introduces Semantic WER, which uses Claude to evaluate only transcription errors that impact LLM understanding (ignoring punctuation, contractions, filler words), and measures TTFS (Time To Final Segment) latency including P95/P99 percentiles critical for real-time voice agents. Benchmark tested 20 vendors across 1000 samples from the pipecat-ai/smart-turn-data-v3.1-train dataset with results stored in SQLite and publicly available on Hugging Face.