Senior AI Engineering

Google Cloud's Gemini Enterprise Agent Platform unterstützt supervised fine-tuning für mehrere Gemini-Modelle, darunter Gemini 3.5 Flash, 2.5 Pro und 2.5 Flash. Bei der Erstellung eines Tuning-Jobs empfiehlt Google, zunächst ohne Änderungen an den Hyperparametern zu starten, da die Standardwerte auf Basis von Benchmarking-Ergebnissen optimiert sind. Die wichtigsten konfigurierbaren Parameter sind: Epochs (automatisch basierend auf Trainings-Datensatzgröße angepasst), Adapter Size (beeinflusst die Anzahl trainierbarer Parameter) und Learning Rate Multiplier (für schnellere oder stabilere Konvergenz). Das System sammelt automatisch Tuning-Metriken wie Training Loss, Token Accuracy und Anzahl Predictions, wobei optionale Validierungs-Metriken für separate Evaluierung verfügbar sind. Für die Produktionsbereitstellung können tuned models über dedizierte Endpoints deployed und mit verschiedenen Ressourcen-Optionen skaliert werden. Google dokumentiert auch Best Practices für supervised fine-tuning in einem separaten Blog-Post.

LLM Model Selection: Matching Models to Tasks by Minimum Required Capability The article addresses a critical best practice for AI engineering production deployment: deliberate LLM model selection based on task requirements rather than defaulting to the most expensive flagship model. Teams often accidentally default to premium models for simple tasks like ticket classification or file reading that cheaper models could handle equally well, inflating costs without quality gains. The core approach involves defining performance thresholds through systematic evaluation—running the same prompt and context across multiple models from different capability tiers, scoring outputs consistently using metrics or assertions, and selecting the cheapest model that meets the quality bar. For agent-based systems, model selection becomes multi-step: different pipeline stages require different capability levels, such as lightweight models for routing decisions versus more capable models for complex reasoning. The article includes a practical case study where Pattern reduced annual costs by $60,000 using benchmarking to confirm a smaller model matched their baseline quality across diverse production data. For teams managing coding agents like Claude Code or Codex, the challenge is identifying when expensive model defaults are unnecessary—Cost Intelligence tools can surface task-level spend patterns showing which sessions triggered premium models for routine work, enabling better workspace-level defaults.

Llama 3 8B fine-tuning erreichte in einem NVIDIA-Test eine 18% Verbesserung der Aufgabengenauigkeit gegenüber dem ungestimmten Basismodell. Der Leitfaden behandelt umfassend LLM-Fine-Tuning-Methoden einschließlich Full Fine-Tuning, Parameter-Efficient Fine-Tuning (PEFT), LoRA/QLoRA und Supervised Fine-Tuning. Wichtige Best Practices sind: hochwertige Datensätze nutzen, das richtige Basismodell wählen, konservative Trainingseinstellungen verwenden, parametereffiziente Methoden bevorzugen, das Modell während des Trainings evaluieren, Trainingsdaten gründlich überprüfen und reale Szenarien vor dem Deployment testen. Ein kritischer Punkt ist die Entscheidung, wann Fine-Tuning sinnvoll ist: Es funktioniert besser als Prompt Engineering und RAG für konsistentes Verhalten über tausende Anfragen, domain-spezifische Fachbegriffe und konsistente Tonalität. Häufige Herausforderungen sind Overfitting bei kleinen Datensätzen, Catastrophic Forgetting, Bias in Trainingsdaten, hohe Rechenkosten bei Full Fine-Tuning und Modelldrift über Zeit. Die praktische Implementierung umfasst sieben Schritte: Datensatzvorbereitung, Wahl des Basismodells, Trainingsparameter-Setting, Modelltraining, Validierung der Performance, Deployment und kontinuierliches Monitoring.

The podcast discusses AI benchmarking and evaluation as critical infrastructure for both industry and government. Vals, an independent evaluation company, has developed methodologies to create realistic, challenging benchmarks that go beyond traditional multiple-choice tests to assess whether models can perform actual professional tasks like coding, legal work, or cybersecurity. Key developments include CyberBench for evaluating offensive and defensive AI capabilities in security contexts, and plans for benchmarks measuring recursive self-improvement, mental health impacts on minors, and military applications. The conversation highlights how static benchmarks quickly become obsolete and emphasizes the importance of third-party independent evaluation rather than companies self-reporting results. Vals has demonstrated the ability to evaluate models within hours of release by scaling infrastructure efficiently. The marketplace for AI evaluation is rapidly professionalized, with government agencies increasingly demanding rigorous, independently-verified benchmarks to inform policy decisions rather than relying on vendor self-attestation. This shift from "evaluations by vibes" to data-driven evidence represents a maturing of AI evaluation practices critical for enterprise ROI decisions and government procurement.