The moment I thought, “As long as the model sticks to JSON format, we should be fine,” was when the real problem-solving began.
We worked on adding an AI agent to our internal platform so users could operate services using natural language. The system was divided into two services. The MCP server handled the tool list, policies, approvals, and auditing, while the agent host handled conversation processing, LLM calls, and the tool execution loop. There was one guiding principle: delegate reasoning to a general-purpose LLM, but keep execution control in the hands of the platform.
The initial implementation worked. But “works” and “can be trusted” were not the same thing. The loop’s decisions depended on a tool list embedded in the prompt and on directly parsing JSON written by the model as text. If the format was off even once, the loop retried and then terminated without calling any tools.
This article describes how we replaced a tool loop based on prompt JSON parsing with provider-native tool calling, and replaced our hand-built LLM calls and MCP client with Spring AI 2.0. There were multiple symptoms, but they had a single root cause. And while we replaced the entire execution layer, we did not change a single line of the control layer. Those were the key points of this work.
1. Background to the technology choice — Why prompt JSON parsing failed
Before deciding to switch, we conducted a thorough review of the loop that was already working. The review uncovered the following problems.
-
Decisions were coupled to the format. The tool list was appended to the prompt as a string, and JSON written by the model as text, such as {"action":"call_tool", ...}, was parsed directly. If the format was invalid, the loop retried twice and then terminated without calling a tool.
-
Observations accumulated without limit. At every step, the full text of all previous observations was appended again to assemble a new user message. As the number of steps grew, the prompt grew linearly.
-
There was no safeguard against answering immediately without calling a tool. The system prompt merely said, “Do not fabricate business data,” but there was no code to enforce it.
-
There were more than 40 tools, and all of them were sent with every request. The tool list alone consumed about 4,000 tokens per step.
-
We implemented the MCP client ourselves using raw HttpClient and JSON-RPC, and there was a risk of violating the protocol because we did not send notifications/initialized after initialize.
At first, we listed these items as separate improvement tasks. But when we traced them back to their causes, the first three turned out to share a single root: AgentAskRequest, the LLM call port, consisted of just two strings, prompt and systemPrompt.
[The root of the problem — an engine port with just two strings]
// Before: the engine port had no notion of a messages array.
public class AgentAskRequest {
private String prompt; // question + tool list + observations, all in here
private String systemPrompt; // rules go here
}
Because the structure had no concept of a message array, there was nowhere to put conversation history or include a tool list. As a result, the only option was to push both the tool list and observations into the prompt as strings. In other words, the structure wasn’t simply not using native tool calling—it made native tool calling impossible.
The conclusion was clear. Instead of fixing symptoms one by one, we decided to upgrade the engine port to a messages + tools model and switch the execution path to provider-native tool calling. At the same time, we decided to replace our hand-built provider REST calls and MCP client with Spring AI 2.0. Although spring-ai-bom:2.0.0 was already declared in the root build.gradle, the client-side module did not import a single Spring AI class and was reimplementing the same functionality by hand.
2. Implementation process
The work proceeded in four steps: defining ownership boundaries, switching the engine port, controlling tool names and tool lists, and verifying that the control layer remained unchanged. We ordered the steps so each could be rolled back independently.
2.1 Defining ownership boundaries — What to delegate to the library and what to retain
Before moving any code, we defined the boundaries. The task was to distinguish what could be replaced by a library from what we had to own ourselves.
|
Category |
Handled by Spring AI |
Handled directly by the agent |
|---|---|---|
|
Scope |
LLM calls, message and tool_calls conversion, MCP protocol/client, tool schema, metrics and tracing |
Loop progression policy, tool authorization policy, risk and approval, call-count and time limits, audit, SSE events |
|
Guideline |
Protocols and transport — areas with standards where we have no reason to do a better job ourselves |
Control and responsibility — areas governed by our platform’s policies |
This table became the basis for every decision that followed. Whenever the question “Where should this go?” came up, the answer became clear by asking whether it was a matter of standards or policy.
2.2 Switching the engine port — Call ChatModel directly, but don’t hand over execution
The aspect we paid the most attention to when introducing Spring AI was automatic tool execution. Spring AI has a path where the framework automatically executes tools requested by the model. It’s convenient, but taking that path would bypass our policies, approvals, and auditing.
We confirmed that Spring AI 2.0’s ChatModel layer called only ToolCallingManager.resolveToolDefinitions and did not call executeToolCalls. In other words, automatic tool execution does not happen as long as ChatModel is called directly. We used this behavior to add an adapter that passes tool definitions as schemas only and returns execution intents without executing them.
[An engine port that receives execution intents only]
// The engine returns only the model's tool-call intent. It never executes.
public interface NativeToolCallingEngine extends AgentEngine {
AgentNativeToolResponse askWithTools(AgentNativeToolRequest request);
}
// Calling ChatModel directly: ToolCallback only supplies the schema, it is not run.
ChatResponse chatResponse = chatModel.call(new Prompt(nativeMessages(request), options));
AssistantMessage assistant = chatResponse.getResult().getOutput();
List<AgentNativeToolCall> toolCalls = toolCalls(assistant); // structured tool_calls
Instead of appending the previous step’s tool calls and results again as strings, we accumulated them in a message array as AssistantMessage and ToolResponseMessage. The observation accumulation problem had moved from rebuilding the prompt to its proper place: the conversation array.
We accepted multiple tool calls and executed them sequentially, ensuring that every tool_call_id received a response. Disallowed tools and duplicate calls did not throw exceptions that killed the loop; instead, they were returned as observations beginning with ERROR:, allowing the model to make a new decision.
2.3 Tool names and lists — Handle constraints only at the boundary
We encountered one practical constraint during the transition. Tool names used the format navi.<service>.<verb>, but the OpenAI-style function name specification did not allow periods (.).
Changing the naming scheme was an option, but it would also have changed the names used in MCP, logs, policies, and audits. Since the constraint applied in only one place—the field passed to the LLM—we decided to convert the names using a lookup table at that boundary alone.
[Handle the naming constraint at a single boundary]
// Convert the name only at the LLM boundary.
// MCP, policy and audit keep the original name.
private String nativeName(String qualifiedName) {
String sanitized = qualifiedName.replaceAll("[^A-Za-z0-9_-]", "_");
if (sanitized.length() <= MAX_NATIVE_NAME_LENGTH) return sanitized;
String hash = String.format("%08x", qualifiedName.hashCode());
return sanitized.substring(0, MAX_NATIVE_NAME_LENGTH - hash.length() - 1) + "_" + hash;
}
// If two names collapse into one, fail fast instead of silently overwriting.
String previous = nativeNames.putIfAbsent(nativeName, qualifiedName);
if (previous != null && !previous.equals(qualifiedName)) {
throw new AgentNativeToolException("Native tool name collision: " + previous);
}
Because names sometimes had to be truncated to meet the maximum length, we made the system throw an exception immediately if different tools were shortened to the same name. If this kind of collision is silently ignored, it may not be discovered until much later, when it appears as “the model called the wrong tool.”
We also reduced the tool list itself. We added a selector that narrows the list based on domain vocabulary in the prompt, but keeps the full list if there are no matches. We also respond to requests that don’t need tools, such as greetings or help requests, before opening an MCP session, and use a short-TTL cache for the tool catalog.
2.4 Leave the control layer untouched
The most important metric for this work was not performance, but “what did not change?” Since we were replacing the entire execution layer, any change to ToolExecutionGateway, which handles policies, approvals, and auditing, would have meant that we had drawn the boundaries incorrectly.
As it turned out, ToolExecutionGateway did not change by a single line. The execution method had changed from prompt JSON parsing to native tool calling, but from the gateway’s perspective, all that still came in was “server id + tool name + arguments.”
We kept the fallback, however. Native tool calling guarantees only the format; empty responses and model silence are still possible. We did, however, expose the loop mode and finishReason together in the SSE completion event and audit log, so the path taken to completion is always recorded.
3. Lessons learned through trial and error
First, it’s faster to find the root cause than to count symptoms. Initially, we listed observation bloat, format errors, and argument validation as separate issues. All three stemmed from the same constraint—the engine interface had just two strings—and fixing the root cause resolved all three at once. If we had addressed each one separately, we would have fixed three issues only to encounter a fourth.
Second, the LLM integration port should treat the conversation array as a first-class concept from the start. Even if you begin with single-turn question answering, using messages as the default data structure means you won’t have to overhaul the interface later to add tools or multi-turn conversations. In fact, half of this transition was spent fixing the interface rather than adding functionality.
Third, a dependency is not fully adopted just because it has been declared; it is adopted when it is actually used. The BOM had already been declared, but the client module was still implementing the same functionality by hand, and that state persisted for some time. Once the hand-built JSON-RPC implementation was removed, the bug caused by omitting notifications/initialized disappeared at its source as well.
Fourth, before using a convenient automation path, check whether it can bypass your control boundaries. The framework’s automatic tool execution is appealing, but using that path would have allowed executions to occur without approval or auditing. We confirmed in the code that the ChatModel layer did not execute tools before deciding to adopt it.
Fifth, fallback paths quietly grow stale when they aren’t exercised. In a later cleanup, we removed both the shortcut that hard-wired sentences to tools using regular expressions and the JSON decision loop. Both paths had been created “just in case,” but in practice, their configuration was empty, so they were never used and only doubled the number of flows we had to maintain.
4. Results
The results of the migration can be summarized as follows.
|
Item |
Before |
After |
|---|---|---|
|
Decision path |
Parse prompt JSON + retry twice |
Provider-native tool calling (structured tool_calls) |
|
Conversation state |
Reassembled the observation transcript as a string at every step |
Accumulated assistant and tool messages in the messages array |
|
Tool list |
Sent all 40+ tools, about 4,000 tokens per step |
Selected by domain with a TTL cache; skipped MCP when unnecessary |
|
MCP client |
Raw HttpClient + custom JSON-RPC implementation |
Spring AI 2.0 (protocol omission bug fixed) |
|
Control layer |
Policy, approval, and audit gateway |
No changes (0 lines) |
Both the native and fallback paths now use Spring AI for transport, so the provider code wasn’t split into two separate implementations. The build and tests passed, and we added seven types of unit tests.
There was a side benefit, too. The messages array required by native tool calling is exactly the same data structure needed for conversation sessions, so this migration also laid the groundwork for conversation memory. We didn’t need to redesign the server contract when adding multi-session conversation history later.
5. Limitations and future plans
First, we still have a dependency on the model. We needed to measure whether the model used internally returns one tool call at a time or multiple calls, and the parallel-execution flag included in the policy remains unused by the loop. We believe it’s safest to enable parallel execution gradually, limiting it to read-only tools.
Second, narrowing the tool list using domain vocabulary has its limits as the number of domains with overlapping vocabularies grows. In subsequent work, we expanded this approach toward replacing full tool exposure with tool indexing and name-directed loading. We plan to cover this separately.
Third, as long as we let the model put tool results into its own words, the presentation will vary each time. This led us toward an approach where the server provides a display contract and the UI renders it as is.
This work reaffirmed that building an agent ourselves doesn’t mean building inference ourselves. The LLM always makes the decisions; what we build is the loop, the tool connections, and the policies and auditing. By making that boundary explicit in the code structure, we were able to replace the entire execution layer without changing a single line of the control layer.
References
-
Anthropic, “Model Context Protocol Specification”, https://modelcontextprotocol.io/specification
-
Spring AI, “Tool Calling”, https://docs.spring.io/spring-ai/reference/api/tools.html
-
Spring AI, “Model Context Protocol (MCP)”, https://docs.spring.io/spring-ai/reference/api/mcp/mcp-overview.html
-
OpenAI, “Function calling”, https://platform.openai.com/docs/guides/function-calling
-
Ollama, “Tool support”, https://ollama.com/blog/tool-support
Junny