1. Overview
The first concern when developing Lakey, an AI system based on RAG (Retrieval-Augmented Generation), was how much we can trust the answers provided by LLM.
In the initial development stage, we simply set up the system to search documents and generate answers using LLM.
However, during the actual testing, some issues began to arise.
The most representative problem was the phenomenon of hallucination. LLMs sometimes generate information that does not exist in actual documents or exaggerate some content.
Especially in real work environments, incorrect answers do not simply result in user experience issues.
There was a possibility of influencing the decision-making process and leading to incorrect handling of tasks.
For example, when generating responses based on internal policy documents, there could be issues such as providing answers with non-existent policy content or describing past policy information as if it were current.
For this reason, the project has decided to add a Verifier layer that not only performs simple 'response generation' but also validates the generated responses separately.
In other words, it is a structure designed to verify the results generated by the LLM once again and filter out responses that have low confidence or are incorrect.
We implemented the structure using the AI Agent and workflow automation tool n8n in the project.
In this article, I would like to focus on why we chose n8n rather than simply discussing 'how we implemented it', and what the advantages and disadvantages were that we experienced in the actual operating environment.
2. Overall Architecture Composition
The Verifier system is largely structured in the following flow.
- Lakey AI system
- Kafka-based event delivery
- n8n Workflow
- Validation Sub Workflow
- Error Workflow
- Result Return Structure
If validation of the response is required in the Lakey system, a validation request event is published through Kafka.
This event contains the following information.
- User Question
- Retrieved Context Document
- LLM generated response
- Request identifier
- Verification Metadata
In n8n, the Kafka Consumer Workflow is configured to receive the event and call the Sub Workflow that performs the actual verification logic.
The validation logic has been separated into an independent Sub Workflow.
The reason for choosing this structure is that the validation criteria are likely to change continuously.
For example, there could be a validation logic like the following.
- Verify if the response is context-based.
- Check for specific keyword inclusion
- Detect prohibited phrases
- Verification of policy violations
- Calculation of trust score
- JSON format validation
By separating these validation flows into distinct nodes and Sub Workflows, maintainability has been improved.
I also configured the Error Workflow separately.
This is because various exceptions such as LLM call failures, Kafka connection errors, timeouts, and JSON parsing errors could occur.
n8n allows for error handling configuration on a workflow basis, so in case of a failure, it is configured to move to a separate Error Workflow, log it, and send notifications.
Finally, once the validation is completed, the results are published again in the form of Kafka events so that the Lakey system can consume them.
In other words, the overall structure is designed based on Event-Driven Architecture.
3. Why did we choose n8n?
In the initial phase of the project, we also considered implementing the AI workflow directly based on Python.
For example, it was a method of creating an API server based on FastAPI and handling the verification flow internally with LangChain or custom Python logic.
However, in reality, the prompt and validation criteria were changed very frequently.
In particular, the following requests occurred repeatedly during the operational process.
- Modify Prompt
- Add Validation Step
- Special condition exception handling
- Check logs
- Change response format
- Change workflow order
If we managed all of this through code, we would have to repeat the following process each time.
- Code modification
- Build
- Deploy
- Server restart
- Test
On the other hand, n8n has a workflow-based structure, allowing nodes to be modified in the UI and immediately reflected.
This aspect felt like a very big advantage.
Especially, AI Agent-based systems often have rapidly changing requirements. Therefore, fast iterative development and experimentation are very important.
n8n was a tool that fit these requirements very well. Additionally, n8n is very easy to connect with various external systems.
With numerous nodes such as Kafka, HTTP API, Slack, Redis, Database, and Webhook provided by default, there was almost no need to write separate Connector code.
In actual projects, I was able to configure Kafka event reception and publication very quickly.
One of the advantages was the ability to visually represent the workflow.
Since complex AI processing flows can be expressed in terms of nodes, operators and other developers can easily understand the entire flow.
Especially, the AI Workflow had a significant visualization effect because understanding the flow itself is more important than just simple code.
As a result, the project chose n8n due to its 'quick iterative development' and 'operational convenience'.
4. Advantages Felt in Actual Operations
The greatest advantages felt during the actual operation process were the debugging and log tracing features.
In the existing code-based Workflow, it was often difficult to determine what data was generated at specific stages.
On the other hand, in n8n, it was possible to view both the input and output data for each node.
In other words, I was able to track the entire process of executing the Workflow step by step.
It was very useful in situations like the following.
- Checking specific prompt results
- Validating LLM response data
- Analysis of JSON Parsing Failure Causes
- Check Kafka Messages
- Check the results of conditional branching
- Analyze specific node Timeout
Especially in AI systems, it was very important to have the ability to check intermediate data because the results are not always the same. Even in actual operational environments, it was often possible to quickly identify the cause of problems with just the n8n execution logs.
Additionally, the code minimization structure was also an advantage.
If it were the traditional way, we would have to write all the Kafka Consumer, HTTP Client, Retry Logic, Error Handling, etc. in code.
However, in n8n, most of it could be implemented just with node settings. Thanks to that, the development speed has significantly increased, and we could focus more on the business logic itself.
Additionally, the reusability of the Workflow was good.
For example, if a specific validation logic is separated into a Sub Workflow, it can be reused in multiple Workflows.
This was quite useful in a prompt-based AI workflow environment.
There were also significant advantages from an operational perspective, as changes to the workflow could be reflected immediately, allowing for a very quick response time in emergencies.
It was very convenient that modifications, especially to the prompt, could be made immediately without separate distribution.
5. Disadvantages felt during the development process
Of course, n8n was not a perfect tool in every situation. While working on actual projects, I was able to experience several drawbacks as well.
The first drawback I felt was the difficulty of version control.
n8n can save change history, but it was difficult to intuitively compare changes based on Git Diff like Java code.
In particular, as the Workflow structure became more complex, it was necessary to manually check which node settings had changed.
The second issue was the readability problems arising from complex workflows.
At first, the node-based structure felt very intuitive, but as the workflow size increased, the screen started to become more complicated. In particular, when there were many conditional branches and loop logic, the connections themselves became tangled and complex.
The third was the lack of test automation.
In a typical code-based environment, automation tests can be set up using test frameworks like JUnit. However, since n8n is fundamentally a Workflow-centric tool, code-level test automation was not easy.
As a result, we had to rely mostly on manual testing. This could become burdensome as the operational scale increased.
Another issue was the difficulty of large-scale refactoring.
For example, when changing a specific variable name or node name, the automatic global change was not performed like the IDE Refactoring function.
In other words, I had to manually go and modify the connected nodes. As the scale of the Workflow increased, such tasks felt quite cumbersome.
Ultimately, I found that while n8n is very powerful for rapid development, it can also introduce a certain level of operational complexity in very large systems.
6. Conclusion
Through this project, we confirmed that n8n has the potential to go beyond a simple automation tool to become an AI Workflow platform.
In particular, AI systems have requirements that change very quickly.
Changing just one prompt can significantly alter the quality of the results, and the validation criteria are often subject to ongoing modifications.
In such an environment, rapid corrections and immediate implementation are crucial.
n8n provides a structure that fits these requirements very well.
I felt it has a significant advantage, especially in the following situations.
- Repetitive prompt modification
- Rapid experimentation environment setup
- Various system integrations
- Event-driven workflow configuration
- AI Agent flow visualization
- operation log tracking
Of course, if the complexity becomes excessively high, it may be more challenging to maintain than a code-based structure. However, if fast iterative development and operational automation are critical for the project, I believe n8n can be a very powerful choice.
Through this experience, I was able to once again realize that in AI systems, not only model performance is important, but also the operational structure and validation system are crucial.
I believe that the Verifier layer will play an increasingly important role in AI systems in the future.
It was a highly satisfying experience, to the extent that if there are similar AI Workflow projects in the future, there is a high possibility of reusing n8n.
sby