AI World 1 · GenAI Foundations · Stage 4

Teach and Measure

Fine-tune, pre-train, distil or retrieve: match the method to the data, then pick the metric that proves it.

Fine-tuning
Continued pre-training
Distillation
Knowledge Bases
Model evaluation
ROUGE, BLEU, BERTScore

Domain: AI and machine learning
World: AI World 1: GenAI Foundations (Associate)
Game type: Solution board: Drop services into each row's slots, then run records through them.
Time: About 25 minutes on a first attempt (estimated)
Exam guides: AIF-C01Task 2.1: Explain the basic concepts of generative AI (GenAI); Task 3.1: Describe design considerations for applications that use foundation models (FMs); Task 3.3: Describe the training and fine-tuning process for FMs; Task 3.4: Describe methods to evaluate FM performance
MLA-C02Skill 1.2.9: Prepare data for FM fine-tuning, continuous pre-training, and model distillation; Skill 2.1.2: Identify fine-tuning strategies for pre-trained FMs to meet business needs; Skill 2.1.5: Select Retrieval Augmented Generation (RAG) architecture patterns based on use case requirements; Skill 2.3.1: Perform reproducible experiments (for example, by using MLflow on SageMaker AI, Amazon Bedrock evaluations, Amazon Bedrock Prompt Management); Skill 2.3.6: Apply comprehensive model evaluation techniques for traditional ML and GenAI models; Skill 2.3.7: Implement integrated human evaluation frameworks (for example, human-in-the-loop workflows, text generation quality assessment); Skill 2.3.8: Apply natural language processing (NLP) evaluation metrics (for example, bilingual evaluation understudy [BLEU], Recall-Oriented Understudy for Gisting Evaluation [ROUGE], BERTScore, semantic similarity); Skill 2.3.9: Perform AI evaluation (for example, model output assessment, content quality validation, bias detection, LLM-as-a-judge frameworks); Skill 2.3.10: Configure RAG system monitoring, including retrieval accuracy assessment
AIP-C01Skill 1.2.4: Implement FM customization deployment and lifecycle management (for example, by using Amazon SageMaker AI to deploy domain-specific fine-tuned models, parameter-efficient adaptation techniques); Skill 5.1.1: Develop comprehensive assessment frameworks to evaluate the quality and effectiveness of FM outputs beyond traditional ML evaluation approaches (for example, by using metrics for relevance, factual accuracy, consistency, and fluency); Skill 5.1.2: Create systematic model evaluation systems to identify optimal configurations (for example, by using Amazon Bedrock Model Evaluations, A/B testing and canary testing of FMs, multi-model evaluation, cost-performance analysis); Skill 5.1.3: Develop user-centered evaluation mechanisms to continuously improve FM performance based on user experience (for example, by using feedback interfaces, rating systems for model outputs, annotation workflows to assess response quality); Skill 5.1.5: Develop comprehensive assessment systems to ensure thorough evaluation from multiple perspectives for FM outputs (for example, by using RAG evaluation, automated quality assessment with LLM-as-a-Judge techniques, human feedback collection interfaces)