blog

Generating Large-Scale Technical Documents with LLM

NEXTREE.

• 11 min read
Generating Large-Scale Technical Documents with LLM

"I thought that if I just threw the data to the LLM, a document would be quickly produced," and from that moment, the actual problem-solving began.

In a recent architecture consulting project, I had to write a large service definition document that organized dozens of business boundaries (Bounded Context) and hundreds of services. The input materials consisted of a service catalog Excel with multiple sheets and a 24-page Word document that organized the existing classification system. The goal was to create a new definition document based on this material, altering the classification axis, formatted consistently in Word.

At first, I approached it simply. I thought I could just pass the data to the LLM and request, "Please create a document in this structure." However, the actual output was completely different from what I expected. The first output was as much as 187 pages long, and both the formatting and structure were in a state that could not be used as is.

This article summarizes the process of tackling the issues of volume explosion and format collapse encountered while generating large technical documents with LLM, through four stages: input partitioning, structural redesign, tool switching, and consistency verification. Ultimately, it discusses the trial and error involved in transforming a 187-page failure version into a usable 43-page final version, along with how the LLM output was connected to and verified against the code.

1. Background of technology selection — Why did ‘leaving it all to the LLM’ fail?

The first attempt was the simplest approach. It involved passing the entire input data to the LLM and describing the desired document structure in natural language before receiving the output. However, this method revealed limitations in three aspects.

The first was volume explosion. The first output (v1) was 187 pages long. Upon analyzing the cause, I found that the LLM had expanded each of the hundreds of individual services into separate sections (headings). Since each service had a title and a description paragraph attached, the volume increased exponentially. What was expected by a person was a "compressed form of services as rows in a table," but the LLM tended to elaborate on every item as much as possible unless explicitly controlled.

The second was inconsistency in formatting. The style of the tables varied from section to section, heading levels were erratic, and the cover page and table of contents were either not generated at all or not formatted correctly. There were no consistent formatting rules throughout the document.

The third was the absence of reproducibility. Even when the same data was requested again, the section composition and volume changed each time. In a large document, this was fatal. Even after one review was completed, generating it again would shake the structure, rendering the review itself meaningless.

Ultimately, I realized that the LLM was a "technology for generating content," not a "technology for structurally assembling large documents." As long as the responsibility for volume and formatting was entirely left to the LLM's natural language output, stable deliverables could not be achieved.

I then completely changed the direction of my approach. I decided to separately control the next two aspects.

  • Structure and volume: explicitly define the skeleton of the document (what to compress into a table and what to keep as sections) by a human.

  • Formatting and rendering: consistently allocate visual elements like table styles, colors, headings, cover page, and table of contents to code.

In other words, I transitioned to a structure where the LLM was responsible only for "what to write (content)," while "how to organize and render (structure and formatting)" was controlled by humans and code.

2. Implementation process and application

The entire process was conducted in four main stages: input segmentation, structural redesign, generation tool transition, and consistency validation. Each stage was separated to enable independent validation so that if a problem occurred, it could be quickly traced back to the originating stage.

2.0 Input Segmentation — BC unit calls considering context window limitations

The first thing that needed to be decided prior to implementation was the method of input delivery. Since the input data contained dozens of business boundaries and hundreds of services, there were two concerns if everything was pushed into a single prompt. One was the token limit (Context Limit), and the other was the phenomenon of lost information in long inputs (Lost in the Middle).

Therefore, instead of throwing the input data all at once, we chose to split it into BC units and loop through to call the LLM individually. Each call only received the service list of that BC and the common classification system (like layer definitions) as context, and was designed to generate only the document segments corresponding to that BC. The generated segments were then merged into a single document at the end.

The advantages of this approach were clear. The input for each call was short, resulting in almost no information loss, and if issues arose in a specific BC, only that BC needed to be regenerated, eliminating the need to recreate the entire document. We confirmed that for large inputs, it is more stable to cycle through in 'meaning units' rather than 'all at once'.

2.1 Structural Redesign — Changing 'Service = Section' to 'Service = Row in Table'

The direct cause of the surge in volume was the structure. Therefore, we explicitly redefined the skeleton of the document first. The core principle was to 'not create separate sections for individual services'. Instead, we grouped services belonging to the same business boundary into a single table, compressing each service into a row of the table.

This was explicitly constrained in the LLM prompt. It was structured not simply as 'write concisely', but as 'do not create each service as an H4 section. Compress into rows of the domain table. Do not exceed an average of 1.5 pages per business boundary', controlling the structure numerically.

[구조 제약 — 프롬프트에 명시]
1. 개별 서비스를 H4 섹션으로 만들지 말 것 (그러면 180쪽 초과).
   → 서비스는 도메인 표의 '행'으로 압축.
2. 비즈니스 경계(BC)당 평균 1.5쪽, 총 40~45쪽 목표.
3. 문서 골격(헤딩 구조)은 아래 고정 템플릿을 그대로 따를 것.
   - H1 = Part, H2 = 비즈니스 경계, H3 = 하위 섹션

With this one change, the volume dramatically decreased. The document reduced from 187 pages to 31 pages (v2). We confirmed that 'asking the LLM to reduce volume' and 'fixing the structure to a form that cannot increase in volume' produce completely different results.

2.2 Generation Tool Transition — The Limitations of Default pandoc Conversion

The v2 (31 pages) that addressed the volume issue had an appropriate structure, but another problem arose. The markdown generated by the LLM was converted to Word using pandoc, but pandoc's default style did not ensure the visual completeness of elements like cover pages, tables of contents, colors, and table designs. Although the content was correct, it was difficult to present it as a document suitable for clients.

Thus, we switched the rendering method. Instead of converting markdown as it is, we decided to use a docx generation library (docx-js) that directly assembles the document structure into code. This allowed us to precisely control formatting elements like cover pages, tables of contents, heading styles, and table designs at the code level.

A crucial point to highlight at this stage was the 'data bridge' connecting the LLM output to the code. Since the LLM is essentially a model that outputs natural language, if it received free-form text as is, the code could not reliably parse it. Therefore, we enforced a strict JSON schema for the final output format of the LLM. Instead of allowing the LLM to write natural language documents directly, it was instructed to fill out predefined schema fields.

// LLM 출력은 자연어가 아니라 이 스키마를 따르는 JSON 으로 강제
{
  "bc": "환자·방문",
  "overview": "환자/encounter/동의 등 ...",
  "coreServices": [
    { "id": "S-001", "name": "환자", "layer": "Data", "owner": "원무" }
  ],
  "events": ["PatientCreated", "EncounterStarted"],
  "kpis": ["환자 등록 정합성", "encounter 완료율"]
}

The code simply received this JSON and mechanically rendered it as a docx-js object. In other words, the LLM filled out what to write in JSON fields, and the code only read those fields to draw rows and paragraphs of the table. By binding the output format to the schema, cases where the LLM arbitrarily changed the format disappeared, and incorrectly formatted responses could be immediately filtered out at the parsing stage.

// 서식의 단일 책임: 스타일 규칙을 코드 한 곳에서만 정의
const doc = new Document({
  styles: {
    default: { document: { run: { font: "Malgun Gothic", size: 20 } } },
    paragraphStyles: [
      { id: "Heading1", run: { size: 32, bold: true }, ... },
      { id: "Heading2", run: { size: 28, bold: true }, ... },
    ],
  },
  sections: [{ children: [/* 표지 → 목차 → 본문 */] }],
});

The key to the tool transition was 'the separation of content and formatting'. The content created by the LLM (row data of the table) was left as is, while the method of rendering it (fonts, colors, borders) was exclusively controlled by the code. Thanks to this, even if the LLM output changed slightly, the overall appearance of the final document remained consistent.

2.3 Formatting Styles — Color Coding and Table Design

Finally, visual rules for readability were incorporated into the code. Since the document contained various areas of different natures, different colors were assigned for each area to allow for easy differentiation at a glance. Additionally, background colors were added to the table headers, and zebra striping was applied to alternating rows to enhance the readability of the table.

[영역별 색상 코딩]
Core 영역    : 딥블루 (#1F4E79)
AI 지원 영역  : 퍼플   (#7030A0)
Analytics    : 그린   (#00875A)
공통(Shared) : 앰버   (#BF8F00)

By consolidating all these formatting rules in one place in the code, they were consistently applied throughout the document, and any future modifications could be made in just one place, reflecting across the entire document. The completed version 3 final draft was 43 pages long. This volume fell within the target range and became a document that was ready for actual delivery, complete with a cover, table of contents, colors, and table design.

2.4 Consistency Verification — Preventing Omissions and Hallucinations at the Source

The aspect I was most mindful of in large-scale generation was the consistency of the data. Even one omission or distortion (hallucination) in a technical document with hundreds of services can be fatal. Since LLMs can generate plausible items that were not in the input or silently omit some items, relying on a person to manually cross-check 43 pages was neither trustworthy nor sustainable.

Therefore, I placed a consistency verification script at the last stage of the pipeline. The core concept was simple. It involved mechanically cross-referencing the service ID list from the original Excel file with the service ID list from the JSON data generated by the LLM. By calculating the differences between the two sets, if there was even one missing ID or an ID not present in the original (hallucination), the verification would be marked as failed and would not proceed to the next stage.

source_ids = set(load_ids_from_excel())   # 원본 카탈로그
output_ids = set(s["id"] for bc in result for s in bc["coreServices"])

missing = source_ids - output_ids   # 누락된 서비스
halluc  = output_ids - source_ids   # 원본에 없는 서비스(환각)

assert not missing, f"누락: {missing}"
assert not halluc,  f"환각: {halluc}"

Thanks to this verification, it became possible to automatically catch instances where some services were missing from a specific BC call or where services not present in the input were generated before it could be seen by a human. Since it was only necessary to regenerate that specific BC if a problem was detected, costs were minimized. This step, guaranteeing that the output was generated to 'exactly match the input' rather than just stating 'generated', proved to be essential in large-scale document automation.

3. Lessons Learned Through Trial and Error

This work was not completed in one go; it was improved through three versions. The lessons learned from each version can be summarized as follows.

버전   분량     상태       핵심 문제 / 개선
----   ----     ------     -------------------------------
v1     187쪽    실패       서비스마다 별도 섹션 → 분량 폭증
v2      31쪽    분량 OK    구조는 잡힘, 그러나 서식 미흡(pandoc 기본)
v3      43쪽    최종       docx-js로 표지·목차·색상·표 완성

First, volume control should be achieved through structure, not requests. Simply asking the LLM to 'write briefly' did not reduce the 187 pages. Only when structural constraints such as 'compress services into rows of the table' were specified did the volume come under control.

Second, 'content accuracy' and 'transferability' are different. Version 2 had appropriate content and volume, but it was difficult to transfer as is due to a lack of formatting. Only after changing the tool from pandoc to docx-js to control the formatting through code did the quality improve.

Third, formatting should be controlled from one place. By consolidating rules such as colors, tables, and headings in one place in the rendering code rather than scattering them throughout the content, consistency was maintained and modifications became easier.

Fourth, LLM outputs are not to be trusted but verified. If the output hadn’t been enforced as a JSON schema, and ID consistency hadn’t been automatically cross-referenced against the input, omissions and hallucinations would have remained in the document until a human discovered them. The fact that verification was built into the pipeline code influenced the level of trust.

4. Results

As a result, it was possible to transform an unusable draft of 187 pages into a deliverable document of 43 pages, complete with a cover, table of contents, colors, and table design. Even looking solely at volume, there was about a 77% reduction, and above all, the most significant achievement was the stabilization of the structure and formatting that had previously been inconsistent.

There have been changes in terms of the working method. Initially, the output from the LLM had to be manually refined by a person, but after fixing the structure and controlling the format through code, human intervention has been significantly reduced when repeatedly generating the same type of document. Since the input data can be regenerated through the same pipeline when updated, maintenance costs have also decreased.

5. Limitations and Future Plans

Through this work, the basic framework for LLM-based large-scale document generation has been sufficiently verified. However, there are clear areas that need improvement.

First, currently the structural constraints and formatting rules are defined by a person. In the future, I believe it would be more efficient to have a stage that automatically suggests appropriate quantity and compression levels based on the scale of the input data.

Second, while it worked well when the input data was in table (Excel) format, it is necessary to perform preprocessing to extract structure when receiving unstructured documents as input. I expect that combining this aspect with the experience of previously organizing document layout analysis will allow me to expand to the entire flow of 'reading unstructured documents to understand the structure and reconstituting with a new definition.'

Through this experience, I was able to directly perceive that simply 'having AI write well' does not lead to large-scale outcomes that can be used in practice. Ultimately, what is important is not 'generating content' but rather 'having the structure controlled by a person, the formatting handled by code, and validating the results,' which was once again confirmed through this experience.

References

Junny

Site footer