AI World 5 · Serve and Secure · Stage 6

Watch the Drift

Six tickets from a month in production: drift nobody saw, accuracy that fell, a fairness rule, a classifier with no monitoring, a doubled token bill and the auditors' prompts.

Model Monitor and drift
Token metrics and invocation logging
Application inference profiles

Domain: AI and machine learning
World: AI World 5: Serve and Secure (Associate)
Game type: Evidence desk: Each ticket is the reporter's words: find its cause in the evidence.
Time: About 30 minutes on a first attempt (estimated)
Exam guides: MLA-C02Skill 2.3.2: Create model performance baselines and implement drift detection; Skill 4.1.2: Monitor workflows to detect anomalies or errors in data processing or model inference; Skill 4.1.3: Detect changes in data distribution that can affect model performance; Skill 4.2.3: Set up dashboards to monitor performance metrics; Skill 4.2.5: Optimize costs and set cost quotas by using appropriate cost management tools; Skill 4.2.10: Monitor AI-specific cost patterns (for example, token usage optimization, embedding computation costs, vector database storage optimization)
AIP-C01Skill 3.3.4: Implement continuous monitoring and advanced governance controls to support safety audits and regulatory readiness (for example, by using automated detection for misuse, drift, and policy violations, bias drift monitoring, automated alerting and remediation workflows, token-level redaction, response logging, AI output policy filters); Skill 4.3.1: Create holistic observability systems to provide complete visibility into FM application performance (for example, by using operational metrics, performance tracing, FM interaction tracing, business impact metrics with custom dashboards); Skill 4.3.2: Implement comprehensive GenAI monitoring systems to proactively identify issues and evaluate key performance indicators specific to FM implementations (for example, by using CloudWatch to track token usage; prompt effectiveness; hallucination rates; and response quality, anomaly detection for token burst patterns and response drift, Amazon Bedrock Model Invocation Logs to perform detailed request and response analysis, performance benchmarks, cost anomaly detection); Skill 4.3.3: Develop integrated observability solutions to provide actionable insights for FM applications (for example, by using operational metric dashboards, business impact visualizations, compliance monitoring, forensic traceability and audit logging, user interaction tracking, model behavior pattern tracking)