AIF-C01 Domain 3: Applications of Foundation Models

Part 4 of 6 in the AIF-C01 exam prep series for developers. Previous: Domain 2: Fundamentals of GenAI.

Domain 3 is the biggest domain and the most practical one. It covers how you actually build with foundation models: choosing one and tuning its inference parameters, grounding it in your data with RAG, deciding whether and how to customize it, writing and managing prompts, and measuring whether any of it worked.

If you've built an LLM feature, much of this will feel familiar. The exam's twist is cost: for almost every scenario, it wants the cheapest approach that meets the requirement.

Domain at a glance#

Domain 3 is 28% of scored content, about 14 of the 50 scored questions: the largest single domain on the exam.

Task	What it covers
3.1	Describe design considerations for applications that use foundation models
3.2	Choose effective prompt engineering techniques
3.3	Describe the training and fine-tuning process for foundation models
3.4	Describe methods to evaluate foundation model performance

Objective checklist#

#	Objective	Section
3.1.1	Selection criteria for foundation models, including prompt caching	Choosing a foundation model
3.1.2	Effect of inference parameters on responses	Inference parameters
3.1.3	RAG and its business applications (Bedrock Knowledge Bases)	Retrieval Augmented Generation
3.1.4	AWS services that store embeddings in vector databases	Vector stores on AWS
3.1.5	Cost trade-offs of customization approaches, including distillation	The customization ladder
3.1.6	The role of AI agents and their business applications	One-liner below; full coverage in Post 5
3.2.1	Prompt engineering constructs: context, instruction, negative prompts	Anatomy of a prompt
3.2.2	Techniques: chain-of-thought, zero-shot, single-shot, few-shot, templates	Prompting techniques
3.2.3	Benefits and best practices	Best practices
3.2.4	Risks: exposure, poisoning, hijacking, jailbreaking	Prompt risks
3.2.5	Prompt versioning with Bedrock Prompt Management	Bedrock Prompt Management
3.3.1	Key elements of training: pre-training, fine-tuning, continued pre-training, distillation	How foundation models are trained
3.3.2	Fine-tuning methods	Fine-tuning methods
3.3.3	Preparing data for fine-tuning, including RLHF	Preparing data
3.4.1	Evaluation approaches: human-in-the-loop, benchmarks, Bedrock Model Evaluation	Evaluation approaches
3.4.2	Metrics: ROUGE, BLEU, BERTScore, LLM-as-a-judge	Evaluation metrics
3.4.3	Whether a model meets business objectives	Does it meet the business objective?
3.4.4	Evaluating applications built with foundation models: RAG, agents, workflows	Evaluating whole applications
3.4.5	Business alignment metrics	Business alignment metrics

Design considerations#

Choosing a foundation model#

Criterion	What to consider
Cost	Price per input and output token at your volume
Modality	Text, image, audio, video; input only, or generation too?
Latency	Is a user waiting? Smaller models respond faster
Multilingual support	Which languages must it handle well?
Model size and complexity	Larger models reason better but cost more and run slower
Customization	Does it support fine-tuning or distillation if you need it later?
Input/output length	Is the context window big enough for your documents? What's the maximum output?
Prompt caching	Does it support caching for repeated prompt prefixes? That can change the cost math entirely

Beyond the guide's list, check licensing, compliance requirements, and availability in your Region.

Inference parameters#

Parameter	What it controls	Low value	High value
Temperature	Randomness in token selection	Focused, consistent, repeatable	Varied, creative, less predictable
Top-p	Sample only from tokens whose combined probability reaches p	Conservative word choice	Wider vocabulary
Top-k	Sample only from the k most likely tokens	Conservative	More diverse
Maximum tokens	Upper limit on output length	Short answers; lower output cost	Longer answers allowed
Stop sequences	Strings that end generation when produced

Here's where these parameters live in a real request, using the Bedrock Converse API:

import boto3

bedrock = boto3.client("bedrock-runtime")

response = bedrock.converse(
    modelId="<model-id>",
    messages=[{"role": "user", "content": [{"text": "Summarize our refund policy in 3 bullets."}]}],
    inferenceConfig={
        "temperature": 0.2,      # low: focused and consistent
        "topP": 0.9,             # sample from the top 90% of probability mass
        "maxTokens": 300,        # caps output length, and output cost
        "stopSequences": ["###"],
    },
)
print(response["output"]["message"]["content"][0]["text"])

Illustrative only; the exam never asks you to read code. Top-k is model-specific, so Converse passes it separately rather than in inferenceConfig.

Retrieval Augmented Generation#

Retrieval Augmented Generation (RAG) retrieves relevant content from your own data at query time and adds it to the prompt, so the model answers from facts it was never trained on. It's the default answer whenever a scenario needs current, proprietary, or citable information. Its advantages:

No retraining is needed.
Updating the answer is as simple as updating the documents.
Responses can cite their sources.

Rendering diagram...

Figure: The RAG flow at query time. Retrieval happens before generation, which makes it a likely ordering question.

Amazon Bedrock Knowledge Bases is AWS's managed RAG capability, and it now comes in two types:

Managed Knowledge Base (generally available since June 17, 2026). AWS runs storage, indexing, and retrieval. Native connectors cover Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, and a web crawler. It adds hybrid search, reranking, and agentic multi-hop retrieval.
Custom knowledge base. You bring and manage your own vector store (see below) and control the pipeline.

Applications call it in one of two ways:

Retrieve: returns the relevant chunks, and you build the prompt yourself.
RetrieveAndGenerate: retrieves, builds the prompt, calls the model, and returns an answer with citations.

Typical business applications:

Customer support assistants that answer from product documentation.
Internal knowledge assistants over HR, IT, and policy documents.
Legal and compliance Q&A with citations.
Sales enablement over product and pricing material.

Vector stores on AWS#

Service	Why you'd choose it
Amazon OpenSearch Service (including Serverless)	Purpose-built search engine with vector (k-NN) search and hybrid keyword-plus-semantic search. The default answer for vector search at scale
Amazon Aurora PostgreSQL-Compatible	Relational database with the `pgvector` extension; keeps vectors next to your relational data
Amazon RDS for PostgreSQL	Same `pgvector` approach on standard RDS
Amazon Neptune	Graph database with vector search (Neptune Analytics); useful when relationships between entities matter (GraphRAG)
Amazon DocumentDB	Vector search for MongoDB-compatible JSON document workloads

Knowledge Bases also supports Amazon S3 Vectors for low-cost vector storage, and third-party stores such as Pinecone, Redis, and MongoDB Atlas.

The customization ladder#

Every step up this ladder costs more, needs more data, and takes longer. The exam almost always wants the lowest rung that solves the problem.

Rendering diagram...

Figure: Cost, effort, and data requirements rise from left to right. Distillation is a specialized form of fine-tuning whose goal is lower running cost.

Approach	Cost	Data needed	Changes model weights?	Use when
Prompt engineering / in-context learning	Lowest	A few examples in the prompt	No	Always try first
RAG	Low to medium	A document collection	No	Needs current, proprietary, or citable facts
Fine-tuning	High	Labeled prompt-response examples	Yes	Needs a specific style, format, or task behavior
Model distillation	Medium to high up front; lowers ongoing cost	Prompts, plus outputs from a large "teacher" model	Yes (a small "student" model)	One narrow, high-volume task where inference cost dominates
Continued pre-training	Higher	Large unlabeled domain corpus	Yes	The model doesn't understand the domain's vocabulary at all
Pre-training from scratch	Highest	Massive corpus and compute	Builds them	Almost never, for a practitioner

In-context learning means giving the model examples or context in the prompt itself, so it "learns" the task without any training. Few-shot prompting is the common case.

Distillation is the only approach on the ladder that reduces ongoing inference cost. A large teacher model generates high-quality outputs, a small student model is trained on them, and you deploy the student for near-teacher quality at a fraction of the price per call.

The role of AI agents#

AI agents (3.1.6) use a foundation model to plan and execute multi-step tasks, calling tools and taking actions rather than only generating text. Business applications include:

resolving customer support cases end to end
processing claims
running IT operations
modernizing code

Prompt engineering#

Anatomy of a prompt#

A well-structured prompt has up to four parts, plus optional negative instructions:

Part	Purpose
Instruction	The task: what the model should do
Context	Background, role, or reference material the model needs
Input data	The specific content to work on
Output indicator	The format, length, or structure you want back
Negative prompt	What the model should not do or include

You are a support specialist for an online bookstore.        <- context (role)
Answer the customer's question using only the policy below.  <- instruction
Do not promise refunds outside the policy.                   <- negative prompt
Do not mention competitors.

Policy: {{refund_policy}}                                    <- context (reference)
Question: {{customer_question}}                              <- input data

Reply in at most 3 sentences, in a friendly tone.            <- output indicator

The {{placeholders}} make this a prompt template: a reusable structure with variables filled in at runtime.

Prompting techniques#

Technique	How it works	Best for
Zero-shot	The instruction alone, with no examples	Simple, common tasks the model already handles well
Single-shot (one-shot)	One example of input and desired output	Showing a format
Few-shot	Several examples	Teaching a pattern, labeling scheme, or style
Chain-of-thought	Ask the model to reason step by step before answering	Math, logic, multi-step reasoning
Prompt templates	Reusable prompt structures with variables	Consistency across many requests and teams

Best practices#

The guide lists these benefits and best practices:

Response quality improvement. Better prompts measurably improve output.
Experimentation. Iterate and compare variants rather than guessing.
Guardrails. Constrain what the model may discuss or reveal.
Discovery. Explore what the model can do before committing to an approach.
Specificity and concision. Be exact about the task, format, and audience, and cut filler.
Using multiple comments. Break complex instructions into several clear statements rather than one long paragraph.

Prompt risks#

Risk	What happens	Mitigations
Exposure (prompt leaking)	The model reveals its system prompt or sensitive data from its context	Keep secrets out of prompts; Guardrails sensitive-information filters; output filtering
Poisoning	Malicious or biased content is planted in training data, RAG sources, or templates	Curate and validate sources; restrict who can change them; version prompts
Hijacking (prompt injection)	Input text overrides the original instructions ("ignore previous instructions…")	Separate instructions from data; Guardrails prompt-attack filter; least-privilege tools
Jailbreaking	Role-play or clever framing tricks the model past its safety training	Guardrails content and prompt-attack filters; monitoring

Bedrock Prompt Management#

Amazon Bedrock Prompt Management, new to the guide as objective 3.2.5, turns prompts from strings buried in code into managed, versioned resources. With it you can:

Create prompts with variables, and attach a model and inference configuration.
Save immutable versions, and have applications and Bedrock Flows reference a specific version.
Compare variants side by side, and test them against different models without redeploying the application.
Optimize prompts automatically for a target model.

Training and fine-tuning#

How foundation models are trained#

Element	Data	Purpose
Pre-training	Massive unlabeled data, self-supervised	Learn language and general knowledge; the most expensive step
Fine-tuning	Smaller, labeled, task-specific data	Adapt behavior to a task, format, or style
Continued pre-training	Large unlabeled domain data	Teach a domain's vocabulary and knowledge (legal, medical, financial). The exam guide calls this continuous pre-training; same thing
Distillation	Outputs from a teacher model	Transfer a large model's skill on a task into a smaller, cheaper model

Fine-tuning methods#

Method	What it is
Instruction tuning	Train on instruction-and-response pairs so the model follows instructions better
Domain adaptation	Specialize a model for a particular field
Transfer learning	Reuse a pre-trained model's knowledge as the starting point for a new task. Fine-tuning is a form of transfer learning
Continued pre-training	Keep pre-training on unlabeled domain text

On Bedrock, fine-tuning uses labeled prompt-and-completion examples stored in S3, and continued pre-training uses unlabeled text. The resulting custom model is private to your account. Serving it can require dedicated capacity, which adds ongoing cost to the one-time training cost.

Preparing data for fine-tuning#

Consideration	Why it matters
Data curation	Remove errors, duplicates, and low-quality examples. Quality beats quantity
Governance	Track where data came from, who can access it, and whether you're allowed to use it
Size	Enough examples to learn the pattern; more isn't always better
Labeling	Accurate, consistent labels; bad labels teach bad behavior
Representativeness	Data must reflect the real inputs and users the model will see, or it will be biased
RLHF	Humans rank model outputs, a reward model learns their preferences, and the model is tuned toward preferred responses

Evaluating foundation model performance#

Evaluation approaches#

Amazon Bedrock Model Evaluation supports three modes:

Approach	How it works	Cost	Best for
Automatic (programmatic)	Algorithmic metrics against reference answers, on built-in or custom datasets	Lowest	Tasks with clear reference answers
LLM-as-a-judge	A judge model scores outputs against criteria such as correctness, completeness, and harmfulness, with explanations	Medium	Open-ended quality at scale
Human-in-the-loop	Your team or an AWS-managed workforce rates outputs	Highest	Nuance, subjective quality, high-stakes sign-off

Benchmark datasets are standardized test sets for comparing models on the same tasks. Use public benchmarks for general capability and your own curated datasets for your use case.

Evaluation metrics#

Metric	Measures	Typical task
ROUGE	Overlap of words and phrases with a reference, recall-oriented	Summarization
BLEU	Overlap with reference translations, precision-oriented, with a penalty for short outputs	Machine translation
BERTScore	Semantic similarity using embeddings, so paraphrases still score well	Any generation where meaning matters more than exact wording
Perplexity	How "surprised" a model is by text; lower is better	Language model quality
LLM-as-a-judge	A model grades outputs against a rubric	Open-ended generation at scale

The recall-versus-precision distinction between ROUGE and BLEU is easiest to see in their core ratios:

\text{ROUGE-N} = \frac{\text{n-grams shared with the reference}}{\text{n-grams in the reference}} \qquad \text{BLEU precision} = \frac{\text{n-grams shared with the reference}}{\text{n-grams in the generated output}}

ROUGE asks how much of the reference you covered, which suits summaries. BLEU asks how much of your output was correct, which suits translations.

Does it meet the business objective?#

A model can score well and still fail the business. Objective 3.4.3 names productivity (time saved per task), user engagement (do people use it and come back?), and task engineering (does it reliably complete the defined task?).

Evaluating whole applications#

A good model inside a bad pipeline still gives bad answers. So objective 3.4.4 asks you to evaluate the application, not just the model.

RAG applications fail in one of two places:

Stage	Metrics	If it's bad
Retrieval	Context relevance, context coverage	Fix chunking, embeddings, or reranking
Generation	Correctness, completeness, faithfulness (is the answer supported by the retrieved context?), citation precision and coverage	Fix the prompt or the model; add grounding checks

Bedrock Evaluations can evaluate Knowledge Bases directly, either retrieval only or retrieve-and-generate.

Workflows are evaluated step by step and end to end: did each step produce valid output, and did the whole flow reach the right result?

Agents are evaluated on their trajectory: did they pick the right tools, in the right order, and finish the task? Post 5 covers this.

To evaluate…	Use
A model's responses	Bedrock Model Evaluation
A RAG application	Bedrock Knowledge Base (RAG) evaluation
An agent's tool use and task completion	AgentCore Evaluations (Post 5)

Business alignment metrics#

Objective 3.4.5 is new, and it names three metrics:

Metric	What it tells you
Task completion rate	Share of interactions where the user's goal was achieved
User satisfaction	How users rate the experience (surveys, thumbs up/down, CSAT)
Cost per interaction	Total cost (tokens, infrastructure, human escalation) divided by interactions

The exam distinguishes three tiers of metrics, and the right tier depends on who's asking:

Tier	Examples	Answers the question
Model	Accuracy, F1, ROUGE, BLEU, BERTScore	Is the model good?
Application	Task completion rate, latency, cost per interaction, deflection rate	Does the system work?
Business	ROI, conversion rate, ARPU, customer lifetime value	Is it worth it?

Service cheat sheet#

Service or feature	One line
Bedrock Knowledge Bases	Managed RAG: ingest, embed, store, retrieve, and generate with citations
Bedrock Managed Knowledge Base	Knowledge Base type where AWS runs storage and retrieval, with native connectors
Bedrock Prompt Management	Versioned, reusable prompts with variables and model configuration
Bedrock Flows	Deterministic workflows linking prompts, knowledge bases, and functions
Bedrock Model Evaluation	Automatic, LLM-as-a-judge, and human evaluation of models and RAG
Bedrock model customization	Fine-tuning, continued pre-training, and distillation
Bedrock Guardrails	Filters for harmful content, PII, denied topics, prompt attacks, and ungrounded answers
OpenSearch Service	Default vector and hybrid search store
Aurora / RDS for PostgreSQL	Relational databases with pgvector
Neptune	Graph database with vector search

Commonly confused#

If the scenario says…	Answer	Not…	Because
Answers must reflect documents updated daily	RAG	Fine-tuning	Fine-tuning freezes knowledge at training time
Responses must always follow our brand voice and format	Fine-tuning	RAG	RAG adds facts, not behavior
Model doesn't understand clinical terminology; we have lots of unlabeled notes	Continued pre-training	Fine-tuning	Unlabeled domain text, vocabulary gap
One high-volume task, inference bill too high, quality must hold	Distillation	Smaller base model	The student learns the task from the teacher
Model gives inconsistent answers	Lower temperature	Higher top-k	Less randomness means more consistency
Track and roll back prompt changes	Prompt Management	Prompt templates alone	Templates are structure; Prompt Management adds versioning
Fixed, predictable sequence of prompt steps	Bedrock Flows	Agents	Deterministic, not model-directed
Evaluate summaries against references	ROUGE	BLEU	ROUGE is recall-oriented, for summarization
Evaluate translations against references	BLEU	ROUGE	BLEU is precision-oriented, for translation
Wording differs but meaning matches	BERTScore	BLEU	Embedding-based similarity handles paraphrases
RAG answers include facts that aren't in the retrieved context	Faithfulness is low (generation problem)	Context relevance	Retrieval was fine; the model added things

Practice questions#

Q1. A support team wants an assistant that answers questions from 5,000 internal PDF documents, which change weekly. Answers must cite the source document, and cost should be kept low. What should they use?

A. Fine-tune a foundation model on the PDFs every week
B. Amazon Bedrock Knowledge Bases
C. Continued pre-training on the PDFs
D. Amazon Bedrock Guardrails

Show answer

Answer: B. RAG grounds responses in the documents, returns citations, and updating the documents updates the answers, with no training.

A and C bake knowledge into model weights. That's expensive, stale within a week, and gives no citations.
D filters inputs and outputs; it doesn't retrieve anything.

Q2 (ordering). Order these customization approaches from lowest to highest cost and effort.

A. Fine-tuning
B. Pre-training from scratch
C. Prompt engineering
D. Continued pre-training
E. Retrieval Augmented Generation

Show answer

C → E → A → D → B.

Prompting needs no infrastructure. RAG adds retrieval but no training. Fine-tuning trains on labeled data. Continued pre-training needs large domain corpora. Pre-training from scratch needs massive data and compute.

Q3. A financial services company uses a foundation model to answer questions about account policies. Answers should be consistent and factual, with minimal creative variation. Which inference parameter change helps most?

A. Increase temperature
B. Decrease temperature
C. Increase maximum tokens
D. Remove stop sequences

Show answer

Answer: B. Lower temperature makes token selection more deterministic.

A increases randomness.
C only allows longer outputs.
D affects where generation stops, not consistency.

Q4 (matching). Match each metric to the task it's best suited to evaluate.

Metric
1. ROUGE
2. BLEU
3. BERTScore
4. LLM-as-a-judge

Tasks:

A. Machine translation against reference translations
B. Open-ended responses scored against a quality rubric at scale
C. Summaries compared to reference summaries
D. Generated text where paraphrased meaning should count as correct

Show answer

1 → C, 2 → A, 3 → D, 4 → B.

Q5. Several teams edit the prompt used by a production GenAI application. After a recent change, answer quality dropped, and no one can identify what changed or restore the previous prompt. What should the company adopt?

A. Amazon Bedrock Prompt Management with prompt versions
B. Fine-tuning to remove the dependency on prompts
C. A larger foundation model
D. Amazon Bedrock Guardrails

Show answer

Answer: A. Versioned, managed prompts let teams compare changes, test variants, and roll back to a known-good version.

B and C don't address the governance problem.
D filters content but doesn't track prompt changes.

Q6. A healthcare company's model struggles with clinical terminology. The company has millions of unlabeled clinical notes and few labeled examples. Which approach fits?

A. Few-shot prompting
B. Instruction fine-tuning
C. Continued pre-training
D. Model distillation

Show answer

Answer: C. Continued pre-training uses large amounts of unlabeled domain text to teach vocabulary and domain knowledge.

A can't close a vocabulary gap this large through a few examples.
B needs labeled examples.
D transfers an existing skill to a smaller model; it doesn't add domain knowledge.

Q7 (multiple response). A team is choosing a vector database for a Bedrock Knowledge Base. Which TWO AWS services from the exam guide's list can store and search embeddings?

A. Amazon OpenSearch Service
B. Amazon Redshift
C. Amazon Aurora PostgreSQL-Compatible Edition
D. AWS Lake Formation
E. Amazon CloudFront

Show answer

Answer: A and C. OpenSearch provides vector search. Aurora PostgreSQL supports vectors with pgvector.

B is a data warehouse.
D governs data lakes.
E is a CDN.

Q8. A RAG application's evaluation shows high context relevance but low faithfulness: the retrieved passages are on-topic, but answers include claims not found in them. Where is the problem, and what's a good fix?

A. Retrieval; change the chunking strategy
B. Generation; instruct the model to answer only from the provided context and add a contextual grounding check
C. Retrieval; switch vector databases
D. Neither; fine-tune the model on the documents

Show answer

Answer: B. Good context relevance means retrieval worked. Low faithfulness means the model added unsupported claims during generation. Tighten the prompt and add grounding checks, such as the Bedrock Guardrails contextual grounding check.

A and C fix a retrieval problem that doesn't exist.
D doesn't reliably stop the model from inventing claims.

Q9. A company runs a single high-volume classification task on a large foundation model. Quality is excellent, but inference cost is too high. They want to keep quality close to current levels. Which approach fits best?

A. Continued pre-training
B. Model distillation
C. RAG
D. Increase temperature

Show answer

Answer: B. Distillation trains a smaller student model on the large model's outputs for this specific task, keeping near-teacher quality at much lower cost per call.

A adds domain knowledge; it doesn't reduce cost.
C adds retrieval cost.
D changes randomness, not cost.

Key takeaways#

Pick the smallest model that meets the requirement, and check context window, modality, latency, and prompt caching support.
Inference parameters shape content, not speed. Lower temperature means more consistency. Maximum tokens caps length and cost.
RAG is the answer for current, proprietary, or citable facts. Bedrock Knowledge Bases is the managed way to do it. Kendra is legacy.
Vector stores: OpenSearch by default, Aurora or RDS for PostgreSQL with pgvector, Neptune for graphs.
Climb the customization ladder only as far as needed: prompting → RAG → fine-tuning → continued pre-training → pre-training. Distillation cuts inference cost.
Prompt structure: instruction, context, input data, output indicator, negative prompts. Hijacking takes over the task; jailbreaking breaks the safety rules.
Prompt Management versions prompts; Flows sequences them deterministically.
Evaluate at every level:
- Models: ROUGE, BLEU, BERTScore, LLM-as-a-judge.
- RAG: relevance for retrieval, faithfulness for generation.
- Business: task completion rate, user satisfaction, cost per interaction.

Next: Agentic AI on AWS.

AIF-C01 Domain 3: Applications of Foundation Models

Domain at a glance#

Objective checklist#

Design considerations#

Choosing a foundation model#

Inference parameters#

Retrieval Augmented Generation#

Vector stores on AWS#

The customization ladder#

The role of AI agents#

Prompt engineering#

Anatomy of a prompt#

Prompting techniques#

Best practices#

Prompt risks#

Bedrock Prompt Management#

Training and fine-tuning#

How foundation models are trained#

Fine-tuning methods#

Preparing data for fine-tuning#

Evaluating foundation model performance#

Evaluation approaches#

Evaluation metrics#

Does it meet the business objective?#

Evaluating whole applications#

Business alignment metrics#

Service cheat sheet#

Commonly confused#

Practice questions#

Key takeaways#

Sources#

Related Articles

AIF-C01 Domain 2: Fundamentals of Generative AI

AIF-C01 Domain 1: Fundamentals of AI and ML

AWS Certified AI Practitioner (AIF-C01) in 2026