SAP TechEd opens this week at Messe Berlin and online, from 27 to 29 October, and agent-building will be all over the stage. Plenty of SAP teams will fly home with a plan to build their first Joule agent in Joule Studio. Far fewer will come home with a plan to test it.
Here’s why that gap matters. Picture an accounts payable team that builds a Joule agent to answer supplier payment queries. It’s grounded on the company’s payment policy, and it works well in the pilot. Two months later, someone uploads a revised policy to the agent’s document grounding, the old version stays in the same folder, and the agent starts quoting both. No code changed. No transport moved. Nothing in the SAP change process noticed.
That’s a composite, but it shows what makes Joule Studio testing different from testing the agents SAP ships. SAP builds, tests and updates its own Joule agents. The agents you build are yours: your team owns them, they’re grounded on your data, and they change whenever someone edits them. This piece covers what’s inside a custom Joule agent, the changes that don’t look like releases, and six things to test before (and after) one goes live.
If you’re looking for SAP’s delivered agents instead, our piece on SAP agentic automation covers those.
What’s Inside a Custom Joule Agent
Joule Studio is a capability in SAP Build. Its agent builder lets you define an agent in natural language, give it tools and ground it in your business context. According to SAP’s Joule Studio page, that means instructions that set the agent’s goals, a choice of AI models, tools and MCP servers, existing skills and automations, and grounding in “data, documents, and the SAP Knowledge Graph”.
| Part | What it is | What a change does |
| Definition | Name, description, expertise and natural-language instructions | Changes how the agent plans, what it refuses and how it phrases answers |
| Model | The LLM behind planning and reasoning, chosen from the models SAP offers | Changes behaviour across every scenario at once, with nothing else edited |
| Tools | Joule skills, MCP servers connected through destinations, and other custom agents used as subagents | Changes what the agent can do, and which tool it picks for a request |
| Grounding | Document grounding resource groups, business data and the SAP Knowledge Graph | Changes what the agent knows, often silently, when a document is added or replaced |
| Identity | Agents run in the user’s security context, through SAP Cloud Identity Services and principal propagation | Changes what the agent may do for each person who asks |
Joule Studio is also moving fast. Agent builder reached general availability early this year, after a preview at TechEd 2025. At Sapphire in May, SAP announced Joule Studio 2.0, adding pro-code agents built in Python and connected to GitHub, support for frameworks such as LangGraph and AutoGen, and AI Agent Hub for managing agents in one place. Expect more news from Berlin this week. The testing questions below apply whichever way an agent is built.
Changes That Don’t Look Like Releases
SAP teams have decades of habit around change control. A change is a transport: it’s requested, built, imported into test, tested and promoted. Custom Joule agents live in SAP Build on BTP, and the changes that matter most don’t arrive that way.
| Change | Who usually makes it | What it can break |
| Edit the instructions | Agent owner, often a business user | Scope, refusals, tone, which tool gets picked |
| Add, remove or rename a tool | Agent owner or BTP developer | Tool selection for requests that used to work |
| Switch or update the model | Agent owner, or SAP’s platform | Everything, a little |
| Add or replace a grounding document | Whoever maintains the document folder | Answers that quote stale, duplicate or wrong content |
| An MCP server changes its tools | A third party, outside your landscape | Arguments, results, and tool descriptions the agent reads |
| A subagent changes | Another team | Every agent that calls it |
A custom Joule agent can change behaviour without a single transport. Your SAP change control won’t see it, so your tests have to.
The practical consequence: version every agent, and treat each of these events as a trigger for a regression run, the same way an import into the quality system triggers one today. That includes the change you don’t make yourself. When a model is updated underneath an agent, nothing in your configuration changed, yet behaviour can. Our piece on regression testing after an LLM upgrade covers how to build that baseline.
Who Owns a Custom Agent
For a delivered Joule agent, SAP owns the build and the updates, and your job is to validate what it does in your landscape. For a custom agent, every part of that chain is yours.
| Agents SAP ships | Agents you build in Joule Studio | |
| Who builds and updates it | SAP | Your team: often a CoE, sometimes a business unit |
| When it changes | On SAP’s release cadence | Whenever someone edits it, or a model, document or tool underneath it changes |
| Who tests a change | SAP first, then you in your landscape | Only you |
| What it knows | SAP’s content plus your configuration | Whatever you grounded it on |
That last row is the easy one to miss. A custom agent is only as good as the documents and data you gave it, and nobody outside your company will notice when those drift. It’s the same problem Microsoft customers face with agents built in Copilot Studio, which our piece on testing agents you build in Microsoft Copilot Studio covers from the Microsoft side.
Six Things to Test in a Custom Joule Agent
One principle runs through all six: judge the agent by what happened in SAP, not by how good the chat reply sounded.
1. Tool selection and arguments
For each scenario, assert which skill or tool the agent called and with which parameters: the right company code, the right supplier, the right date range. Joule Studio lets you review an agent’s reasoning steps, tool calls and data flow before deployment, which makes this assertable. Include near-miss requests that should go to a different tool, or to no tool at all.
2. Grounding
Ask questions whose answers sit in one specific document, and assert the answer matches that document. Then test the drift cases: a revised document added next to the old one, a document removed, and a question the grounding doesn’t cover at all, where the right behaviour is to say so rather than improvise.
3. Authorization
Agents run in the user’s security context, so the same request should succeed for one role and fail for another. Test with users who hold different authorizations, and assert the agent never becomes a route around them. Our guide to SAP SoD testing covers what an agent’s access should never combine.
4. The outcome in SAP
When an agent creates or changes something, check the record itself: the document posted, the fields set, the status it ended in. A confident chat reply saying “done” isn’t evidence. For tools that reach SAP through APIs or destinations, our SAP integration testing guide covers the interface layer underneath.
5. Human approval where it belongs
SAP says human-in-the-loop controls are on by default for critical decisions. Test that they still fire after every change to instructions or tools, and that the agent can’t reword a request so it no longer looks critical. Our piece on where humans belong in agentic testing covers how to decide which decisions those are.
6. MCP servers and outside content
An MCP server outside your landscape can change its tools, or its descriptions of them, without telling you, and the agent reads those descriptions to decide what to call. Pin the tool list you tested against and alert on differences. Treat anything the agent reads from outside, from tool results to supplier documents, as untrusted input: our piece on indirect prompt injection in ERP shows how to test that.
Building a Regression Set for Joule Studio Agents
A useful regression set for a custom agent is small, specific and run often:
- Twenty to fifty scenarios per agent, each with the request, the user’s role and the expected outcome in SAP. Cover the main jobs, the near misses and the requests the agent must refuse.
- Assert decisions, not wording. The same correct answer can be phrased many ways. Assert the tool called, the record created and the document cited; allow the sentence around them to vary.
- Run it on every trigger in the change table above, and on a schedule, because some triggers (a model update, an MCP change) won’t tell you they happened.
- Keep the results. Each run is evidence of what the agent did on a given version, which is what an auditor will ask for once agents touch financial records.
Pro-code agents built with Joule Studio 2.0 add unit tests for their code, which is welcome. They don’t replace scenario tests, because the model still decides what the code gets asked to do.
Where Sofy Fits
Sofy’s AI agents for SAP testing validate SAP business processes end to end, and that’s where a custom agent’s work lands. According to the product page, one test flow covers SAP GUI, Fiori, web and API layers simultaneously, and the agents read metadata, custom fields, Fiori structure, GUI logic, workflows and integrations. Fiori apps are tested “via business intent and OData validation, not fragile UI locators”.
Two more claims on the page matter here. The agents detect and adapt to SAP updates, transports and configuration changes autonomously, and Sofy supports pipeline triggers, so a regression run can start when an agent version changes. And business analysts, QA leads and process owners can build tests without ABAP or scripting, which suits the people who often own a custom agent.
TESTING A JOULE AGENT YOU BUILT?
See Sofy’s agents validate the SAP outcomes behind your custom agents, across GUI, Fiori, web and API layers.
What Still Needs a Human
Tests can tell you whether an agent did what you specified. They can’t tell you whether you specified the right thing: which requests the agent should handle, which it should hand to a person, and which documents count as the source of truth. Those are decisions for the agent’s owner and the process owner, and they should be written down before the first scenario is.
And someone has to read the failures. When a grounding test fails because two policy versions disagree, the fix is a decision about the policy, not about the agent.
Frequently Asked Questions
What is Joule Studio?
Joule Studio is a capability in SAP Build for creating custom Joule skills and Joule agents. Its agent builder lets you define an agent’s goals in natural language, choose a model, give it tools such as Joule skills and MCP servers, and ground it in your documents, business data and the SAP Knowledge Graph.
How is testing custom SAP Joule agents different from testing SAP’s own agents?
SAP builds, tests and updates the agents it ships. A custom agent is yours: your team owns it, it’s grounded on your content, and it changes whenever someone edits its instructions, tools, model or documents. Only you will test those changes.
Do changes to a Joule Studio agent need a transport?
The changes that most often alter behaviour, such as edited instructions, a new tool, a different model or a replaced grounding document, happen in SAP Build rather than through ABAP transports. Your existing SAP change control won’t see them, so version each agent and trigger regression tests on those changes.
How do you test a Joule agent’s grounding?
Ask questions whose answers live in one specific document and assert the answer matches it. Then test drift: a revised document beside the old one, a removed document, and a question the grounding doesn’t cover, where the agent should say it doesn’t know.
Can a Joule agent do more than the user is allowed to?
It shouldn’t. Joule agents run in the user’s security context through SAP Cloud Identity Services and principal propagation. Test that by sending the same request as users with different roles and asserting it succeeds only for those authorized.
What should a Joule Studio regression set include?
Twenty to fifty scenarios per agent, each with a request, a user role and the expected outcome in SAP, covering main tasks, near misses and refusals. Assert tool calls, records and cited documents rather than wording, and run it on every agent change and on a schedule.
The Bottom Line
TechEd will make building a Joule agent look easy, and in Joule Studio it increasingly is. The harder part is everything after: an agent owned by your team, grounded on your content, and changed by edits that never pass through a transport.
Treat every one of those edits as a release. Version the agent, keep a small regression set of real scenarios, and judge each run by what happened in SAP. That’s what turns a promising pilot into an agent your auditors, and your process owners, can trust.