ISTQB® CT-GenAI chapter 4: LLM-powered test infrastructure, RAG, agents and LLMOps

In short

Chapter 4 of the ISTQB® CT-GenAI v1.1 syllabus looks under the hood of test tools that embed an LLM. How are they built? How do you give them access to your project's documents? What changes when an agent can act instead of just answer? How do you adapt a model to your domain, then run it over time? The syllabus allocates 110 minutes of training to it.

You do not need to be a developer: the exam asks you to understand the principles and know which one fits which need, not to implement them. It is the lightest chapter on the exam, yet its concepts are exactly what you need to assess a vendor's tool.

What the exam expects

Chapter 4 accounts for 5 of the 40 questions, worth 5 points: K2 (understand) questions only, one per objective.

Objective	Level	Questions	What you need to be able to do
GenAI-4.1.1	K2	1	Explain the components of an LLM-powered test infrastructure
GenAI-4.1.2	K2	1	Summarize how Retrieval-Augmented Generation (RAG) works
GenAI-4.1.3	K2	1	Explain the role of LLM-powered agents in automating test processes
GenAI-4.2.1	K2	1	Explain fine-tuning a language model for a test task
GenAI-4.2.2	K2	1	Explain LLMOps and its role in deploying and managing LLMs

Expect questions framed as scenarios, for example: "the team wants the tool to take the latest version of the specifications into account; what should they put in place?"

Key concepts

The architecture. An LLM-powered test infrastructure has a front end (the interface where the tester enters a request), a back end (authentication, data retrieval, prompt preparation, calling the LLM, post-processing the response), data sources (relational databases for structured data such as test cases, vector databases for semantic search) and the LLM itself, either hosted by a third party and called through an API or deployed in-house. What sets it apart from a classic client-server setup: the LLM interprets testware and builds answers from context instead of following a script.

RAG. Retrieval-Augmented Generation gives the model access to your own documents without retraining it. Up front, documents are split into chunks, each chunk is converted into an embedding and stored in a vector database. At query time, the question is encoded in turn, the chunks closest in meaning are retrieved, and they are passed to the LLM, which generates an answer grounded in them. Two steps to remember: retrieval, then generation.

LLM-powered agents. An agent does not just answer: it acts by calling predefined functions (its "tools"), for example reading a ticket, triggering a test suite or filing a report. It can be autonomous or semi-autonomous, and several specialized agents can work together (orchestration). Agents inherit the LLM's hallucinations, reasoning errors and biases: for critical tasks, the syllabus recommends semi-autonomous agents and automated verification of their results.

Fine-tuning. Further training a pre-trained model (LLM or SLM) on a targeted dataset so it picks up your organization's vocabulary, reasoning or output format. A fine-tuned SLM can be enough for a narrow task, at lower cost. Four challenges: training data quality, overfitting, model opacity, and computing resources.

LLMOps. The practices, tools and processes for developing, deploying and maintaining LLMs in production. The decisions depend on how generative AI is used: an AI chatbot, a test tool with generative AI capabilities, or an in-house tool. These modes can coexist within one organization.

Common pitfalls

• Confusing RAG with fine-tuning. RAG supplies context with each request and leaves the model unchanged. Fine-tuning changes the model through additional training. Need up-to-date documents: RAG. Need an in-house format or vocabulary: fine-tuning.
• Picturing a keyword search. RAG retrieval relies on semantic similarity between embeddings, not on exact word matches.
• Trusting an agent because it is automated. An agent that gets something wrong acts on its mistake. The more it does on its own, the more verification matters.
• Assuming you must pick a single usage mode. A team can use a chatbot for some tasks and an in-house tool for others, adding RAG or fine-tuning on top.

How to study it

Two to three hours is usually enough. The best exercise: draw, from memory, the diagram of an LLM-powered test infrastructure, then the path of a RAG request step by step. Next, for a test tool you know (or a vendor demo), try to work out whether it uses RAG, an agent or a fine-tuned model, and what that means for checking its output.

Assess your level first with the free level check. The certification guide places this chapter in the overall path, the exam guide recaps the format, and the full course compares RAG, fine-tuning and agents on concrete test cases. To be clear on the scope (you use these technologies to test; you do not test them), read CT-GenAI vs CT-AI. To go further: RAG for software testing and AI agents in software testing.

Frequently asked questions

Do I need to know how to build a RAG system for the exam?

No. Objective GenAI-4.1.2 asks you to summarize the principle: chunking the documents, embeddings, a vector database, retrieval by semantic similarity, then generation. You need to know what each step is for and when RAG helps, but the objective does not cover any specific tool or piece of code.

What is the difference between an agent and an AI chatbot?

An AI chatbot answers questions in a conversation. An LLM-powered agent can act: it calls tools to interact with other systems and chain tasks together, autonomously or semi-autonomously. That is what lets it automate part of the test process, and also what makes its mistakes more costly.

Why is chapter 4 worth only 5 points?

Because the syllabus targets testers who use these tools, not specialists who build them: every objective is at K2 level. Five points still count when the pass mark is 30 out of 46, and these concepts come back in chapter 5 when choosing a model or an approach for a test organization.