Harness Disappeared Due to Model Change

Introduction

Over the past few months, I applied LLM agents to projects I worked on. When a user made a request in natural language, the agent selected the necessary query tools, checked the results, and then either generated an answer or asked another question when further judgment was needed. This article summarizes what I learned while building an agent harness that wrapped the model to control tool calls and conversation flow, and while trying out several different models.

At first, I thought of the harness as a general-purpose control layer that would make any connected model operate reliably. So whenever a local model violated the schema or made an incorrect tool call, I added parsers, retries, and reasoning rules. At the time, this seemed like a reasonable response because I was blocking failures one by one.

However, when I repeatedly ran the same harness while changing only the model, the results differed. Local open models repeatedly produced format errors, incorrect arguments, empty responses, and excessive delays before completing the overall task. In contrast, gpt-5.6 completed the task from beginning to end most reliably with the same tools and the same request. The difference was not marginal; it was overwhelming enough to determine whether the system was actually usable.

The compensatory harness was not a general-purpose safety net, but code tailored to the failure distribution of the models I had observed. As the model improves, that code becomes unnecessary. Worse, compensations tailored to an earlier model can interfere with the newer model's straightforward execution path.

That does not mean the entire harness disappears. We need to distinguish between a compensatory harness that makes up for model shortcomings and a harness responsible for system-level concerns such as approvals, permissions, budgets, and auditing. In this article, I revisit the compensatory code I actually built and explain why choosing a good model is an architectural decision, not merely a cost issue outside the code.

1. Why I Chose Local Models and How the Harness Began

There were clear reasons for prioritizing local models at the beginning of the project. We needed a configuration in which business data did not leave the organization, we already had an internal inference server available, and we could reduce invocation costs. I judged that if the model and provider could be switched through configuration, the application code could remain unchanged.

Simplified to exclude project identifiers that cannot be exposed externally, the configuration looked like this:

Configuration for switching the provider and model

agent:
  provider: ${LLM_PROVIDER:local}
  endpoint: ${LLM_ENDPOINT}
  model: ${LLM_MODEL}
  capabilities:
    tool-calling: true
    structured-output: false

Looking only at the configuration, switching models appears to be as simple as changing one string. In practice, it was not. The first local model changed the names of fields declared in the schema to different words, while the second model generated an answer before completing the necessary queries or took too long to respond. Another model used incorrect tool arguments or returned an empty turn with neither content nor tool calls.

Whenever a failure occurred, I added code so the model could succeed on another attempt. I accepted synonyms for field names, inferred incorrect types from the number of options, retried empty responses, and increased the number of steps to match one model's calling habits. Individual errors decreased, but the harness increasingly accumulated model-specific knowledge.

There was an important trap in this process. Each change passed the tests and resolved a real error. Therefore, when looking only at the code, all of it appeared to be necessary defensive logic. However, its necessity came not from the product contract but from the behavior of the model being used at the time. It was code whose justification could disappear along with a model change.

2. The Compensatory Harness I Actually Added

Although the compensation logic I added took different forms, it can all be described with the same sentence: “Because the model failed like X, the code fixes it as Y.” I have organized representative cases into generalized code that can be disclosed externally.

2.1 I Absorbed Schema Field Names as Synonyms

The tool for sending additional questions to users required fields such as questionId, question, and inputType. However, some models sent id or key instead of questionId, and type or kind instead of inputType. If parsed as-is, the entire question would be omitted, so I created a function that read several names in sequence.

A parser that accepts field-name variations

private String firstText(Map<String, Object> values, String... names) {
    for (String name : names) {
        String value = text(values.get(name));
        if (!value.isBlank()) {
            return value;
        }
    }
    return "";
}
 
String questionId = firstText(values, "questionId", "id", "key");

The problem was that the synonym list did not come from the specification. It was the result of adding words one by one after seeing them in execution logs. Even the same model generated different words from one run to another, and changing the model produced yet more variations. The parser became broader, but the contract became more ambiguous.

2.2 The Code Inferred Incorrect Input Types

Even after finding the field names, the values were still a problem. I allowed only closed values such as Text, Radio, and Select, but the models generated expressions such as multiple_choice, dropdown, and checkbox. To avoid discarding the question, I normalized the strings and, if the type was still unknown, determined the UI type from the number of options.

Logic for normalizing and inferring model output values

String normalized = raw.toLowerCase(Locale.ROOT).replaceAll("[^a-z]", "");
QuestionType declared = switch (normalized) {
    case "text", "freetext", "string" -> QuestionType.Text;
    case "radio", "singlechoice", "choice" -> QuestionType.Radio;
    case "select", "dropdown", "list" -> QuestionType.Select;
    case "multiselect", "multiplechoice", "checkbox" -> QuestionType.MultiSelect;
    default -> null;
};
 
return declared != null ? declared : inferFrom(options);

This logic made it possible to render the UI, but it introduced the risk that the application would arbitrarily interpret ambiguity created by the model. An incorrect inference could be stored as normal input, and even when a new model sent the correct value, the old fallback would remain. It was useful for short-term recovery, but not suitable as a long-term contract.

2.3 I Repeated the Contract in the Prompt Instead of Using Structured Output

In some local execution environments, enabling structured output caused generation to appear to terminate normally while the body came back empty. In the end, I disabled native format enforcement, wrote the JSON schema and the instruction “output JSON only” back into the prompt, and made the parser validate the result.

An example of bypassing format enforcement with a prompt and parser

request.disableNativeSchema();
request.addInstruction("Respond with JSON only.");
request.addInstruction(renderSchema(responseSchema));
 
Response parsed = parser.parse(modelResponse);
validator.validate(parsed);

This approach worked immediately, but it duplicated the contract in three places. The tool schema, the prompt text, and the parser's defensive rules all represented the same content in different ways. Changing any one of them required changing the others as well, and even when the model improved, this duplication did not disappear automatically.

2.4 I Covered Empty Turns and Repeated Errors with Retries

After successfully performing several tool queries, the system sometimes returned a response in which both content and tool calls were empty. To avoid losing all the observations already obtained because of a single final turn, I added a path that retried empty responses and finalized the answer using the remaining information.

Empty-response recovery and execution budget

private static final int MAX_EMPTY_RESPONSE_RETRIES = 2;
 
if (response.hasNoContent() && response.hasNoToolCalls()) {
    retryOrFinishWithCollectedEvidence();
}
 
RunBudget budget = RunBudget.of(maxSteps, maxDuration, maxTokens);

Retries themselves can be a necessary safeguard. However, I determined how many times to retry and how many steps to allow in one execution based on the failure frequency and calling habits of the model at the time. The numbers were insufficient for one model and excessive for another, and the same constant represented entirely different costs depending on the model.

2.5 The Prompt Also Became a Record of a Specific Model's Failures

Not only the code but also the system prompt grew to include instructions such as “copy identifiers exactly,” “do not create placeholders,” and “do not reach a conclusion before checking the tool results.” Most of these instructions originated from a real failure that had occurred once.

As the prompt grew longer, it became difficult to distinguish which instructions were product policies and which were compensations for a specific model. By repeatedly instructing even behaviors that a new model already handled well, we obscured the priority of important policies. This, too, was evidence that the harness had been fitted to the model.

3. What Happened When I Changed Only the Model in the Same Harness

3.1 Comparison Method

To verify the hypothesis, I changed only the model while keeping the system prompt, tool schema, query data, and execution budget identical. I repeatedly ran complex requests like those used in actual operations. The task involved checking the required information through multiple tools and asking questions about items that required a decision from the user.

I checked whether the overall task completed from beginning to end without interruption.

I checked whether the tool names and arguments matched the schema.

I checked whether response format errors, empty turns, and repeated retries occurred.

I checked whether the results varied significantly when the same request was run again.

I checked whether normal results were maintained even without intervention from the harness.

In the first draft I wrote, I summarized some subordinate metrics numerically. On review, however, I found that those values did not properly represent the actual experience or overall success. For example, even if one question JSON object matched the format, it would be difficult to call the task successful if an earlier tool call had failed and the overall task had not been completed. In this document, rather than creating new figures from memory, I summarized only the results actually reproduced through repeated runs as statuses.

3.2 Actual Observations

Table 1. Observed results from repeated runs with only the model changed in the same harness

Observation item	gpt-oss:120b	gemma4:31b	qwen3.6:35b	gpt-5.6
Task completed in full	Repeated errors during execution made stable completion difficult.	Tool call errors and empty responses occurred repeatedly.	Failures occurred repeatedly due to response delays and format errors.	It completed the task from start to finish most consistently across repeated runs.
Tool calls	It continued making calls that matched the schema.	It often omitted necessary calls.	There were incomplete terminations.	It continued making calls that matched the schema.
Response format	Format validation was occasionally necessary.	Recovery from empty responses was necessary.	Retries and format validation were necessary.	Responses were stable with one or two retries.
Execution variance	There was relatively little variance in the results and execution flow.	There was variance in the response format.	There was variance in the response format.	The results and execution flow showed the least variance.
Overall assessment	It was difficult to deploy in production under the current conditions.	It was difficult to deploy in production under the current conditions.	It was difficult to deploy in production under the current conditions.	It was the only one to consistently meet the criteria for actual deployment.

Because quantitative logs for each run were not preserved in the same format for all models at the time, I did not create new success rates or average times that might be inaccurate. The table records only repeatedly observed failure patterns and actual deployment judgments.

The results overwhelmingly favored gpt-5.6. When another model was fixed in one area, an error occurred again elsewhere, and the failure pattern changed on every run. In contrast, gpt-5.6 completed the flow from tool calls through the final question most reliably on the same harness. The model's own tool-use and contract-compliance capabilities had a greater impact on the results than the amount of reward code I wrote.

The particularly important difference was not whether it could produce a format-compliant answer once or twice, but whether it could repeatedly complete the entire task. The local models also achieved partial success. However, a feature that can be provided to users requires the entire flow to be completed. Viewed by this criterion, the differences between the models became much clearer.

The harness brought weaker models up to a certain level, but it did not eliminate the failure distribution. Adding synonyms produced new synonyms; increasing retries increased time and cost; and raising the step budget caused incorrect calls to be repeated for longer. The reward logic was effective, but its limitations were also clear.

3.3 What it means for a harness to fit a model

The inputs to a compensatory harness are not product requirements but the model's observed failures. These observations come from a specific model, specific version, specific prompt, and specific inference server. Therefore, the reward logic that translates those observations into code is also tied to the same conditions.

The model failed like X. → The harness compensates with Y. → The harness fits a model that exhibited failure X.

Wrapping nondeterministic outputs in deterministic code can make the system appear stable. However, when the model changes, the output distribution changes as well. Errors that were common in the old model may not occur in the new model, and the new model's calling patterns may differ from the sequence assumed by the old reward logic. In the end, code thought to be a general-purpose layer becomes an empirical model approximating the behavior of a specific model.

From this perspective, replacing a model is not simply a provider change. It is an effort to reassess the coupling between the harness and the model. If the new model is connected while the existing reward code is left unchanged, unnecessary fallbacks may alter correct values, or unnecessary retries may introduce delays. After a model upgrade, deletion should be considered before adding more code.

4. What disappeared after switching to gpt-5.6

The biggest change after switching to gpt-5.6 was not merely that it passed the existing harness more effectively. The situations requiring compensation themselves decreased substantially. As the rate increased at which the model preserved exact field names and argument formats, selected the necessary tools, and completed the entire task, the rationale for the recovery code in the shared loop weakened.

Table 2. Changes in the compensatory harness before and after the model replacement

Compensation item	Models with recurring errors	After applying gpt-5.6
Field-name synonyms	It explored multiple aliases in sequence.	It used the defined schema names as-is, making this unnecessary in most cases.
Input-type inference	It inferred unknown values from string rules and the number of options.	It used the permitted types, making it possible to remove the inference fallback.
Repeating the schema in the prompt	It re-explained the contract in long sentences.	The contract could be focused directly on the tool and response schemas.
Empty response recovery	The retry and observation-preservation paths intervened frequently.	The normal path stabilized, allowing us to reduce them to exceptional failure handling.
Model-specific step adjustments	We increased the constants to match the calling patterns.	Unnecessary calls decreased, allowing us to apply a simpler budget policy.

Here, saying that something is “no longer necessary” does not mean that all of the code was immediately deleted. Before actually deleting it, we need to run regression tests with the new model and verify that the results remain stable with the compensation logic disabled one piece at a time. The important change is that the focus of verification shifted—not toward adding more compensation, but toward verifying that existing compensation can be removed.

A good model does not merely increase the accuracy of the answers. It also reduces parser branches, retry counts, model-specific settings, failure-log analysis, and combinations of regression tests. A local model may be cheaper if you look only at the per-call cost, but the conclusion changes when you include the development cost of investigating failures and maintaining compensation code. In the actual project, choosing gpt-5.6 lowered the overall cost and risk.

Another lesson was that model evaluation should not end with the quality of a single answer. For agents, tool selection, argument accuracy, state preservation across multiple turns, recovery after failure, and the final response together constitute performance. gpt-5.6’s advantage was not merely that it wrote better sentences, but that it completed the entire execution graph.

5. The harnesses that should nevertheless remain

Even if a good model reduces compensating harnesses, it does not take over the system’s responsibilities. No matter how accurate the model is, it must not read data without authorization, execute difficult-to-reverse changes without approval, or prevent the evidence used and the execution results from being traced.

5.1 Questions for distinguishing compensation from responsibility

When classifying code, I used the following question: “Would this code still be necessary if the model followed the contract perfectly?” If the answer was “no,” it was a compensating harness; if the answer was “yes,” it was closer to a system responsibility.

Table 3. The boundary between compensating harnesses and responsibility-bearing harnesses

Category	Compensating harness	Responsibility-bearing harness
Basis for occurrence	An error observed in a specific model.	Product policy, security, and operational responsibility.
Model dependency	High. It varies by model and version.	Low. It remains in place even when the provider changes.
Representative examples	Synonym parsing, value inference, empty-turn retries, and model-specific constants.	Authorization checks, approval gates, budget limits, and audit records.
After applying a good model	Delete it after regression verification or reduce it to an adapter.	Keep it as is and strengthen the tests.
Failure handling	If possible, do not fix it silently; make it explicit.	When a policy is violated, stop execution and record the reason.

5.2 User approval and change boundaries should remain

We did not treat read-only queries and actual changes as equivalent tool calls. We automated the process through the stage where the agent checks information and explains its plan, but required the user’s explicit approval at the point where data changes or results are published externally. This boundary is unrelated to model performance.

An approval flow generalized by removing internal names

사용자 요청
  -> 읽기 전용 조회
  -> 변경 계획과 영향 제시
  -> 사용자 승인
  -> 실제 변경 실행
  -> 결과와 근거 기록

We also kept the practice of having the system deliver questions or approval requests to the user in a prescribed structure, rather than having the model rewrite them. This is not merely compensation to work around the possibility that the model might get field names wrong; it is a product principle that separates user decisions from generated text.

5.3 Budgets, permissions, and audit records should remain

A structure that places limits on the steps, time, and tokens that a single execution can use is necessary. However, the numbers themselves may vary depending on the model’s speed and calling method, so they should be separated into configuration. The policy of having a limit is a responsibility-bearing harness, while the value tailored to a specific model is a compensating setting.

We do not leave tool allowlists or authorization checks to the model either. A situation in which the user lacks query permission and a situation in which the query result is empty have completely different meanings to the user. Even if the model explains the difference well, the system must be the entity that determines whether a call is allowed.

Finally, we must record which tools were called for which requests and which results served as the basis for the response. If a problem occurs in production and only the model’s final sentence remains, we cannot reconstruct the cause. Audit records do not become less important as model quality improves; they become more important as the scope of actual use expands.

6. How to design model selection and harnesses together

After this experience, we changed the process of choosing a weak model first and compensating with a harness. First, we evaluate candidate models with a thin executor that has only the minimum contract and safety mechanisms, then select a model capable of carrying representative scenarios through to completion. After that, we compensate only for errors that continue to recur, within an isolated adapter.

6.1 Measure overall task success first

If we look only at the success rate of parsing JSON once or successfully making a single tool call, it is easy to overestimate actual usability. We need to consider together whether the user’s desired result was reached, whether the result remains consistent when the same request is repeated, and whether the cause is clear when a failure occurs.

Execute representative work scenarios from start to finish, rather than in short units.

Use the same prompt, tools, data, and budget for each model.

Record tool-argument errors, empty turns, retries, and elapsed time along with the final success status.

Compare side by side the result with the harness enabled and the result with the compensation logic disabled.

After changing models, verify whether the existing compensation can be deleted before adding new compensation.

6.2 Confine compensation code within the adapter

When model-specific compensation logic is placed in the shared execution loop, it becomes difficult to tell which model the code exists for. It is better to keep compensation inside the provider or model adapter and have the shared loop assume that a clear contract is being followed. When the contract cannot be met, failing with a diagnosable error is safer in the long run than silently inferring a solution.

When adding compensation is unavoidable, the model name, version, reproduction conditions, and removal conditions should be recorded in comments and tests. Then, when switching to the next model, you can immediately identify candidates for removal. Compensation code needs not only a description of its function but also grounds for determining how long it remains valid.

6.3 Include Maintenance Costs in Model Costs

A model selection table tends to include only token prices and inference server costs. In real projects, however, time spent analyzing failures, implementing and testing compensation code, delays caused by retries, and the possibility of operational incidents are also costs. Even if a local model has a low invocation cost, the total cost can be higher if you must continually clean up its errors.

In my case, gpt-5.6 had a greater impact by simplifying the development workflow than by reducing model costs. The work of adding a new branch every time a failure was reproduced decreased, allowing me to focus on core policies and tool contracts. Choosing a high-performing model became a choice that reduced code and operational complexity, rather than merely an improvement in quality.

7. Practical Checklist Compiled After Application

Currently, when introducing or replacing a model, I check the following in order.

Confirm that the candidate model can reliably perform representative scenarios from beginning to end.

Record failures by categorizing them into output format, tool selection, argument accuracy, state maintenance, and latency.

Do not compensate for problems with the model itself in shared business logic.

Isolate compensation logic in model-specific adapters and configuration, and record the removal conditions alongside it.

Keep approval, authorization, execution budgets, and audit records as policies independent of the model.

When upgrading a model, first consider deleting unnecessary harness code before adding functionality.

When quantitative logs are unavailable, do not invent numbers based on memory; share only reproducible phenomena.

The purpose of this checklist is not to eliminate the harness. It is to distinguish what the model is responsible for from what the system is responsible for. If the code is allowed to absorb the model's defects indefinitely, the harness will continue to grow thicker and become an intermediate layer that is optimal for no model.

Conclusion

At first, I thought a good harness could erase the differences between models. In reality, a considerable part of the harness was code that transcribed the defects of a single model. That code solved problems at the time, but it was not a general-purpose rule that would continue to be necessary with other models.

When I connected the same harness to multiple models, gpt-5.6's performance was overwhelming, while errors continued to occur with the other models. This experience made me realize that a model's performance is not merely a component specification; it determines the size and shape of the harness. The more code that accumulates to compensate for a weak model, the more strongly the system becomes fit to that model.

A good model does not eliminate the entire harness. It reduces compensatory harness code that was filling in for model defects and leaves only the harness the system must remain responsible for through to the end, such as approval, authorization, budgeting, and auditing.

Therefore, when choosing a model, you should not compare invocation costs alone. You should also consider the overall task success rate, the repeatability of errors, the amount of compensation code, operational risks, and the time developers spend cleaning things up. The reason for choosing gpt-5.6 for the project can be explained using the same criteria. Using a high-performing model not only improved the results but also made the system simpler and more explainable.

Even if the model changes again in the future, I plan to ask the same question first: Is the code I am about to add the system's responsibility, or is it temporary compensation for shortcomings in the current model? Keeping this distinction documented has become the criterion that most quickly tells me what to delete and what to preserve when moving to the next model.

IAN