Before Trusting LLMs, I Started by Questioning Them

Introduction

While carrying out a project, I needed to organize work documents written by people into structured data that could be used in subsequent processes. The documents described business objectives, people in charge, processing procedures, rules, and exceptions in natural language. People can read and understand the context, but for programs to use the documents for subsequent tasks such as search, classification, and review, they must be converted into consistent fields and formats.

At first, I thought it would be sufficient to provide the document and conversion instructions to an LLM and ask it to organize the content as JSON. In practice, the LLM produced neat-looking results in a short time. However, procedures that were not present in the original text were sometimes added naturally, or an important sentence would quietly disappear from the result. The actions performed by the person in charge and the processing results were also sometimes mixed together.

For example, the original text may only state, “The reviewer may approve the request or request additional information,” but the LLM could add the rule, “The team leader gives final approval,” based on general business practices. The sentence sounds natural, but it is not in the original text. If this result is used simply because the JSON format is valid, the model’s guess becomes a business fact.

Therefore, I redefined the goal from “having the LLM create complete data” to having the LLM create structured candidates, the system check the structure and the evidence in the original text, and a person decide whether to use the result finally.

This article introduces Structured Output, which receives responses in a specified JSON format; JSON Schema, which defines the output contract; Evidence, which provides the basis in the original text; and a limited Repair process that uses validation errors. The key issue is how to treat probabilistically generated results as verifiable data.

The Final Structure to Examine First

Figure 1. A flow that provides an unstructured work document together with a Prompt and JSON Schema, then validates the result deterministically

First, let us examine the final structure we put together. The original text, Prompt, and JSON Schema are provided to the LLM together to create structured candidates, and the system checks the structure, input coverage, evidence in the original text, and missing information. Here, deterministic validation means a check that always produces the same result when the rules are applied to the same input. If validation fails, the system performs limited correction, including the errors. If it fails again, the result is not applied automatically and is instead sent for manual review.

The following sections explain, in order, what problems occurred with the initial approach and how we added each validation step to address them.

The Initially Conceived Structure: Would a Prompt and JSON Schema Be Enough?

The initial input was a work document written by a person. To explain the conversion process more easily, this article uses a simple example called the “Task Request Review Procedure.” The data fields below were also designed to show the relationship between the document and the structured result.

Sample work document to be converted

작업 요청 검토 절차
 
배경
현재 작업 요청을 메일과 메신저로 전달하고 있어 요청 상태와 처리 이력을
일관되게 확인하기 어렵습니다.
 
목표
요청자는 작업 내용을 등록하고 검토 결과를 확인할 수 있어야 합니다.
검토자는 검토가 요청된 작업을 승인하거나 보완을 요청할 수 있어야 합니다.
 
담당자
- 요청자는 작업 내용을 작성하고 검토를 요청합니다.
- 검토자는 검토가 요청된 작업 내용을 확인합니다.
 
처리 절차
1. 요청자가 작업 내용을 작성합니다.
2. 요청자가 검토를 요청합니다.
3. 검토자는 요청을 승인하거나 보완을 요청합니다.
4. 검토자는 처리 결과와 검토 의견을 기록합니다.
 
업무 규칙
- 검토를 요청한 작업만 검토할 수 있습니다.
- 검토 결과에는 처리자와 처리 시각이 포함되어야 합니다.

Result sought from this document

The result I wanted was not a short summary of the document. I wanted to connect the people in charge and the processing procedures described in the document, and create structured candidates that a program could search and review.

Content of the sample document	Desired structured result
Background and objective	The purpose the document aims to achieve (documentPurpose)
Requester and reviewer	A list of participants with identifiers (participants)
Writing, requesting review, approving, requesting additional information, and recording the result	Processing steps with responsible parties and an order (processSteps)
Review conditions and record items	Business rules to be managed separately (rules)
Criteria for assigning people in charge that are not in the document	Not filled in arbitrarily as nulland confirmation questions (missingFields)
Source of the result	A quotation from the original text linked to the relevant value (evidence)

In other words, the desired output consists of values structured from the document, questions that cannot be determined from the document alone, The original-source basis for each valuehad to be included as well. Only when the target output is defined can we consistently determine what was taken from the source and what is missing. We expressed this criterion as a JSON Schema.

A simple JSON Schema used as the output contract

This is a simplified Schema example that retains only the items needed to understand the overall flow.

{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "documentPurpose",
    "participants",
    "processSteps",
    "rules",
    "assignmentRule",
    "missingFields",
    "evidence"
  ],
  "properties": {
    "documentPurpose": { "type": "string" },
    "participants": {
      "type": "array",
      "items": { "type": "object", "required": ["id", "name"] }
    },
    "processSteps": {
      "type": "array",
      "items": { "type": "object", "required": ["order", "actorId", "action"] }
    },
    "rules": { "type": "array" },
    "assignmentRule": {
      "type": ["string", "null"],
      "description": "원문에 배정 기준이 없으면 null"
    },
    "missingFields": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["fieldPath", "question"]
      }
    },
    "evidence": { "type": "array" }
  }
}

In this process, we divided the responsibilities of the Schema, Prompt, and Validator as follows.

Schema    : 필수 필드와 데이터 형식을 제한합니다.
Prompt    : 원문에 없는 값은 추측하지 않고 null과 확인 질문으로 반환하게 합니다.
Validator : 참조 관계, 원문 근거, null과 확인 질문의 대응 여부를 검사합니다.

For example, the Schema requires assignmentRule as a mandatory field but also allows null. The sample document only explains that a reviewer may approve or request supplementation; it does not provide a criterion for determining who is responsible among multiple reviewers. Therefore, the Prompt is designed not to guess this value, but to return null along with a clarification question, while the Validator checks whether the two values actually exist together.

Structured candidates created from the document

When the document above is converted into structured candidates that a program can read, it takes the following form. This data structure is an example created to explain the article; it is not the final answer, but an intermediate result that must undergo validation and human review.

{
  "documentPurpose": "작업 요청의 진행 상황과 처리 이력을 일관되게 확인한다.",
  "participants": [
    { "id": "requester", "name": "요청자" },
    { "id": "reviewer", "name": "검토자" }
  ],
  "processSteps": [
    {
      "order": 1,
      "actorId": "requester",
      "action": "작업 내용을 작성한다."
    },
    {
      "order": 2,
      "actorId": "requester",
      "action": "검토를 요청한다."
    },
    {
      "order": 3,
      "actorId": "reviewer",
      "action": "승인하거나 보완을 요청한다."
    },
    {
      "order": 4,
      "actorId": "reviewer",
      "action": "처리 결과와 검토 의견을 기록한다."
    }
  ],
  "rules": [
    "검토를 요청한 작업만 검토할 수 있다.",
    "검토 결과에는 처리자와 처리 시각을 포함한다."
  ],
  "assignmentRule": null,
  "missingFields": [
    {
      "fieldPath": "assignmentRule",
      "question": "여러 검토자 중 담당자는 어떤 기준으로 정합니까?"
    }
  ],
  "evidence": [
    {
      "fieldPath": "processSteps[1]",
      "sourceExcerpt": "요청자가 검토를 요청합니다."
    }
  ]
}

Using this result in subsequent processes such as search and review requires more than valid JSON syntax. We must additionally verify that no responsible person or procedure from the source has been omitted, that Evidence actually exists in the source text, and that the model has not arbitrarily invented an undetermined policy.

At first, we expected to obtain stable results by providing a sufficiently detailed Prompt and JSON Schema. We restricted required fields, data types, and object structures with the Schema, and received responses as JSON using the Structured Output feature.

This approach solved many basic problems. It significantly reduced issues such as explanations being added before or after the JSON, Markdown code blocks being mixed in, and field names varying between runs. Schema delivery, response parsing, and error-handling paths could also be repeatedly verified through automated tests.

However, once the structure became stable, an even more important problem began to emerge clearly. The fact that JSON is valid and the fact that the business content is accurate are entirely different matters.

First problem: the content could be wrong even when the format was correct

A problem frequently identified during evaluation was confusion among the responsible person, the action, and the processing result. Even when the source stated, “The reviewer approves the request or requests supplementation,” some responses created a new participant called the approver or added a step called “review completed,” which did not exist in the source.

This result can also pass the JSON Schema. That is because participants and processSteps are arrays and the types of their values are correct. However, the business content has changed because a person or procedure not present in the source has been added.

Another problem was inconsistency in reference relationships. The actorId of a processing step might not exist in participants, or the same participant might be represented by different IDs. Each object looks correct on its own, but the relationships between objects are broken. Simply normalizing identifier formats does not correct a misinterpreted meaning.

From this point on, we separated validation into two layers.

구조 검증
  - JSON 문법
  - 필수 필드
  - 타입
  - enum
  - key 형식
 
의미 보존 검증
  - 원문 항목 누락 여부
  - 참여자와 처리 단계의 참조 일관성
  - 처리 순서의 중복과 역전 여부
  - 원문 근거 존재 여부
  - 원문에 없는 값의 임의 생성 여부

The Schema remained important, but we treated it only as a necessary condition. On top of it, we added a rule-based Validator that always makes the same judgment for the same input, and retained differences between the source and the result as separate evaluation data.

Second problem: how can we find sentences the model did not read?

When some results were missing from an LLM response, it was difficult to distinguish whether the information truly did not exist in the source or whether the model had overlooked it. This difference cannot be determined by checking only the output.

To solve this, we divided the input document into non-empty sentence units and assigned identifiers to them. Each sentence had to be linked to one or more result fields or have a reason why it was not transformed.

MAPPED    : 하나 이상의 결과 필드와 연결됨
UNMAPPED  : 현재 출력 구조에 대응하는 필드가 없음
EXCLUDED  : 변환 대상이 아닌 문장으로 명시적으로 제외됨

The completeness of required fields was managed separately from the status of the input sentences.

PRESENT      : 원문 근거를 가진 값이 존재함
NEEDS_INPUT  : 원문만으로 확정할 수 없어 사용자 확인이 필요함

The reason for separating them this way is that information entirely absent from the source cannot be represented as the status of a specific input sentence. Input coverage tracks “which sentences were processed,” while field completeness determines “whether the required values are sufficient.”

After applying this to the evaluation document, we could immediately find unprocessed sentences and distinguish omissions by the model from limitations in the output structure. We could prevent sentences from silently disappearing without forcing every piece of content into a field.

Third problem: even a sentence that looks like evidence may have been created by the model

To increase confidence in the transformation results, we attached original-source Evidence to each result item. At first, we stored the sourceExcerpt returned by the model as-is. However, evaluation showed that Evidence sometimes contained sentences summarized or rewritten by the model rather than sentences from the source.

Suppose, for example, that the source text is as follows.

검토자는 요청을 승인하거나 보완을 요청할 수 있습니다.

If the model returns the sentence below as Evidence, its meaning may appear similar.

검토자는 모든 요청의 최종 처리 결과를 결정합니다.

However, the second sentence does not exist in the source. It implies stronger authority and includes the model’s interpretation. If we accept it as evidence, a circular situation arises in which the model uses a claim it created as evidence for its own result.

Therefore, we normalized only whitespace and required sourceExcerpt to actually exist in the input document before allowing it to pass. This prevented sentences summarized or rewritten by the model from being approved as Evidence.

However, Exact Match guarantees only that the sentence exists in the source. Whether that sentence actually supports the field to which it is linked is a separate issue, so we distinguished Evidence validation as follows.

1단계: 원문성 검증
  sourceExcerpt가 입력 문서에 실제로 존재하는가?
 
2단계: 연결 검증
  Evidence가 가리키는 fieldPath가 실제 결과 항목과 일치하는가?
 
3단계: 의미 정합성 검증
  이 문장이 해당 결과를 충분히 뒷받침하는가?

Whether the text is from the source could be checked by rules, but semantic consistency required human review. We clearly limited the scope guaranteed by the validator to whether the text exists in the source.

Do not fill in unknown values; make the model ask questions

When LLMs encounter a blank, they tend to fill it in naturally. This is an advantage in general writing, but it is risky when structuring business documents. When the source lacks a business purpose or boundaries of responsibility, if the model writes common industry practices, the result may sound natural but still be untrue.

To prevent this, we changed the contract so that when a required value cannot be confirmed in the source, the model returns a question instead of generating the value.

{
  "missingFields": [
    {
      "fieldPath": "documentPurpose",
      "question": "이 업무가 최종적으로 달성해야 하는 결과는 무엇입니까?"
    }
  ]
}

In this design, a question is not a failure message. It is a normal result of the transformation process. It distinguishes situations where the source information is insufficient from situations where the model has overlooked content, and allows the user to supplement it in the next step.

In actual evaluations, user clarification questions were sometimes generated more frequently than expected. At first, we thought we should reduce the number of questions, but when we examined their content, we found that many represented uncertainty that the model had previously been silently inferring. Rather than simply reducing the number of questions, it was more appropriate to combine duplicate questions and prioritize those necessary for actual decisions.

The Repair Loop was correction using validation results, not a retry

If the same request is sent again unchanged after validation fails, the result may be different, but the model will not know what needs to be fixed. Therefore, we allowed at most one Repair request and included the following information in the second request.

- 최초 사용자 요청
- 최초 모델 응답
- Schema 및 결정적 Validator 오류
- 누락되거나 원문과 일치하지 않은 항목

The processing flow is as follows.

Result first = generate(request);
Validation check = validate(first);
if (check.isValid()) return first;
 
Result repaired = repair(request, first, check.errors());
return validateOrSendToManualReview(repaired);

We did not allow unlimited retries. A result that happens to pass after repeated calls can conceal quality problems. If the second result also failed, we stopped automated processing and recorded the original response and the error.

In one evaluation, most input sentences were processed correctly, but some Evidence items slightly rewrote the source. Even though the overall result looked quite good, the system did not treat it as an approvable result and sent it through the Repair path. This experience made us reconsider the standard for whether “mostly correct” should count as system success.

What changed after implementation

The changes after applying the validation structure can be summarized as follows.

Before	After implementation
We compared the source and the output manually to determine whether anything was missing.	We immediately identified unprocessed input sentences.
We used the evidence returned by the model as-is.	Blocked Evidence that does not exist in the source text.
Values that were not in the document were also supplemented naturally.	Instead of finalizing the value, I switched to asking a clarification question.
When validation failed, I repeated the same request.	I conveyed the specific error and made a limited correction.
Discarded the failed response.	Preserved it as regression evaluation data along with the validation error.

Separated execution records from verified data

When I applied the same document and instructions to multiple models, all of them returned JSON, but their points of failure differed. Some models omitted rules, others added explanations that were not in the source text, and still others made errors in reference relationships. Therefore, it was difficult to trace the causes of problems by storing only the final results.

The execution records retained the model used, the Prompt version, the input document, the raw response, the parsing results, the validation status, and the reasons for failure. These records were not finalized business data, but materials for debugging and regression evaluation. Only results that passed validation and were reviewed by a person were converted into usable data.

Summary of Key Points

• LLMs should be limited to the role of generating structured candidates, rather than being the source of finalized data.

• Structured Output reduces parsing errors, but it does not prevent omissions or misinterpretations.

• To find omissions, you must track the processing status of each input sentence rather than the number of results.

• Whether Evidence exists in the source text and whether it supports the relevant field must be assessed separately.

• Failed responses and validation errors must be preserved together to enable regression evaluation after changing the Prompt, Schema, or Validator.

Conclusion

When I started this work, I thought it was about converting natural-language documents into JSON. As implementation progressed, I realized that the real problem was not JSON generation, but how far to allow uncertain interpretations, what to mechanically validate, and when to ask a person again.

The final structure was designed to constrain the format with JSON Schema, check input coverage and Evidence, and leave unknown information as questions. When applying generative AI to business operations, what matters is not obtaining the most natural answer, but preventing incorrect results from proceeding to the next step.

My conclusion from this work is that what makes LLMs trustworthy is not a stronger prompt, but a system that questions and validates their output.

Brown