AI developer tools · What shipped

MetaInfer, a knowledge-only LLM inference engine generator, achieved substantial performance gains over state-of-the-art solutions through AI-assisted optimization. In a benchmark on Qwen-3.6-35B-A3B with NVFP4 precision on a single NVIDIA B200, an AI-generated engine called VibeQwen outperformed vLLM 0.25.1 by up to 90% on single-stream decode speed (1,792 vs 943 tokens per second) and delivered 2.3x faster time-to-first-token (12ms vs 28ms). At concurrency level 32, VibeQwen achieved 71% higher throughput (10,307 vs 6,030 output tokens per second). A second experiment applied the same approach to SAM 3.1 image segmentation, achieving 50% throughput improvement (91 images/second on H100 vs Meta's baseline), demonstrating the reusability of the optimization knowledge base across different model architectures and modalities.

Prime released Prime Inference, a production model serving platform supporting both serverless endpoints and reserved capacity for frontier open-source models. The system separates prefill and decode operations on NVIDIA Blackwell GPUs, implements NVFP4 KV compression to increase cache capacity by 50%, and uses block-major (BLHNC) KV layouts to reduce NVLink transfer overhead by 47%. GLM-5.3 deployed on OpenRouter achieves 101 tokens per second per user at 66 concurrent sessions with 100% uptime. The stack combines vLLM, NVIDIA Dynamo, Mooncake, and FlashInfer, with contributions including a native sparse-MLA attention kernel for FlashInfer and structural-tag support in Dynamo for reliable tool-call generation with near-zero error rates in production agent serving.

NVIDIA released the Nemotron family of foundation models with open weights and reproducible training recipes optimized for various deployment scenarios. Key releases include Nemotron 3.5 Lightning (30B total / 3B active parameters with Mamba-2 + Transformer MoE architecture and 1M-token context), Nemotron 3 Super (120B total / 12B active with up to 5x higher throughput than previous versions), and Nemotron 3 Ultra (550B total / 55B active for demanding agentic workloads). These models feature Multi-Token Prediction layers and speculative-decoding drafters (DSpark, DFlash, MTP) for faster inference, with support across vLLM, SGLang, TRT-LLM, Ollama, and llama.cpp frameworks. NVIDIA also released Cosmos 3, an omni-model for physical AI with variants: Cosmos 3 Super (32B reasoner + 32B generator), Cosmos 3 Nano (8B + 8B optimized for RTX PRO workstations), and Cosmos 3 Edge (4B reasoner + 2B generator for real-time robotic deployment). The platform includes specialized models for speech recognition (Parakeet ASR, Canary multilingual), document parsing (Nemotron Parse), multimodal embeddings, and reranking for RAG pipelines, alongside 200+ open datasets covering pretraining, alignment, math reasoning, code generation, and evaluation benchmarks.